1pp-0.58b-v2-raw-base

One Persona Pretraining (1PP) scaling-ladder model: 0.58B parameters, pretraining condition original documents, seed 28.

Part of ladder v2: four sizes (0.58B, 0.98B, 1.66B, 2.81B) × three conditions × three seeds (28, 42, 1234), each size trained from scratch for 20.3 assistant tokens per parameter on the same step-aligned data stream. Conditions differ only in the document text and the loss mask; seeds differ only in the parameter initialization (the batch order is the same). The 1.66B rung is the 1.7B model set of the main study (1pp-1.7b-*). Models are grouped in the 1pp collection.

Architecture

Llama-style decoder, 24 layers, hidden 1,152, FFN 4,608 (SwiGLU), attention heads / KV heads 9 / 3 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus <|pad|>; <|endoftext|> is the end-of-document token.

Pretraining

Data: the original DCLM-edu documents; loss on all document tokens and on <|endoftext|>. The first 11,106 steps of the 1PP step-aligned stream (23.29B token positions), which are the first steps of the main study's 31,777-step pass in the same order; the raw stream is the rebuild of the original documents, which over the full 31,777 steps holds one document fewer than the original packing on 60 steps. Global batch 512 × 4,096 tokens, cross-document attention masking. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for the other parameters (LR 3e-3 for embeddings and the output layer, 3e-3 for norms); warmup 699 steps, constant, linear decay over the last 1,110 steps to 1/100; weight decay 0.1; init std 0.0119; bf16.

Validation loss (per token, 2,433 held-out documents, final checkpoint):

assistant text	user text	document text	this condition's training tokens
2.898	2.967	2.617	2.618 (raw_doc_eod(8+5))

The last column weights adversarial conversations at their training share (11.18%) for the conversation conditions.

Chat format

ChatML without a system turn (the models never saw one):

<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n

The bundled chat_template renders exactly this. Generation stops at <|im_end|> (id 2) or <|endoftext|> (id 0); both are listed in eos_token_id. This is a base model; the conversation conditions produce chat-formatted text, the raw baseline plain text.

Verification

The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:

set	HF loss	Megatron reference	abs. diff
val50m segments [3]	2.8962	2.8979	0.0018
raw_val50m segments [8]	2.6185	2.6166	0.0019

Links

• Training logs: wandb projects 1pp-training and 1pp-sft
• Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.