ppt-c4-pre0

SmolLM-135M trained on C4 with no pre-pretraining (the PT baseline). Part of the EPFL CS-552 (Spring 2026) Power Rangers project: Mechanisms of Pre-Pretraining Transfer in Language Models.

• Architecture: LlamaForCausalLM (30 layers × 9 query heads × 576 d_model, GQA with 3 KV heads, head_dim 64)
• Vocab: 49 152 (SmolLM BPE)
• Training data: allenai/c4 (en)
• Phase 1: none. Phase 2: 10 000 steps on C4 from random init.

Reproducibility

Both checkpoints in this pair were trained on the EPFL Alps cluster by team collaborators (M. Rofin and G. Yüce). The original training artifacts live in the tml-ppt/ppt-project wandb project; this HF copy is for downstream mechanistic-interpretability analysis.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("VityaVitalich/ppt-c4-pre0")
tokenizer = AutoTokenizer.from_pretrained("VityaVitalich/ppt-c4-pre0")