T2MLR_Baseline_136M_50B_FineWebEdu

A dense Transformer baseline for T2MLR (Transformer with Temporal Middle-Layer Recurrence): a 136M-parameter Llama-style model pretrained from scratch on FineWeb-Edu for ~51.5B tokens, with no recurrence.

It is parameter-matched to its T2MLR counterpart JupiterZhu/T2MLR_135M_lstart13_lend18_50B_FineWebEdu: same SmolLM2-style architecture and training recipe, with the hidden size widened so the total parameter count matches the T2MLR model's (which spends extra parameters on its recurrent mixing module).

Configuration

Parameters	136.4M
Layers	30
Hidden size	584
Intermediate (MLP) size	1536
Attention heads / KV heads	9 / 3
Vocabulary	49,152 (SmolLM2 tokenizer), tied embeddings
Precision	bfloat16
Training tokens	~51.5B (100,000 steps)
Final training loss	2.7776

Usage

This is a standard LlamaForCausalLM — no custom code required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "JupiterZhu/T2MLR_Baseline_136M_50B_FineWebEdu"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval()
tok = AutoTokenizer.from_pretrained(repo)

inputs = tok("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Provenance

Trained with the T2MLR codebase's wrapper with recurrence disabled, then exported to a plain LlamaForCausalLM (dropping the wrapper's unused mixing-module weights). The export was validated against the original training checkpoint: identical weights, bitwise-identical logits in fp32 and bf16, and identical greedy generations.

Notes

• This is a base model with no instruction tuning or alignment. Greedy decoding from short prompts is often repetitive.
• For exact reproduction, load with from_pretrained(..., dtype=torch.bfloat16) rather than loading in fp32 and calling .to(torch.bfloat16): the latter also casts the RoPE frequency buffer, which Hugging Face otherwise keeps in fp32.
• Outputs may be inaccurate, biased, or offensive, reflecting the pretraining data.

Citation

Official implementation: https://github.com/princeton-pli/T2MLR