theOG-50M

A 50.7M-parameter code generation model trained from scratch, aligned with DPO on execution-verified preference pairs — every "chosen" completion in the preference set actually passed the task's unit tests.

Small enough to run on a laptop CPU. Deep rather than wide (10 layers × 512), a choice made from measurement: at this scale depth beat width on every benchmark we ran.

Results

HumanEval, all 164 tasks, execution-graded with the canonical checker. Greedy is true argmax; pass@32 samples at temp 0.8 / top-p 0.95.

	greedy pass@1	pass@32
base (pre-DPO)	15/164 · 9.1%	32/164 · 19.5%
theOG-50M (this model)	21/164 · 12.8%	36/164 · 22.0%

The difference between these rows is not statistically significant — see below.

Is that difference real? Measured, not asserted.

No — not at this sample size. Both models were re-run recording how many of the 32 samples pass per task, and compared paired:

test	result
McNemar (binary pass/fail)	+9 / -3 tasks, p = 0.146
Wilcoxon (paired pass counts)	31 tasks changed — 18 up, 13 down; +28 passes of 5,248 draws; p = 0.431

The +6 is 9 tasks gained and 3 lost, and the counted test — which uses all 32 draws instead of collapsing each task to one bit — shows 13 tasks got worse. Neither test clears p<0.05. At n=164 with ~20 passes, roughly 10 of 12 discordant pairs would have to fall one way to reach significance.

So the honest claim is: DPO on 43 pairs did not measurably change HumanEval performance. What it did demonstrably do is not damage the model — held-out cross-entropy moved 0.4146 → 0.4170 across 200 steps, a 0.6% drift, with the trajectory stored in the checkpoint (val_ce_hist) so anyone can check it.

That matters because DPO's characteristic failure is winning the preference objective while the language model degrades. That did not happen here. The method is sound; 43 preference pairs is simply too small to move a 50M model measurably.

MBPP-500, never used in preference-pair generation, agrees: greedy 18 → 19, pass@32 70 → 74, +3/-2 tasks, p = 1.0.

Architecture

parameters	50.7M
layers × hidden	10 × 512
attention heads	4
FFN	2048, SwiGLU
vocabulary	16,000 (custom BPE)
context	1024
positional	learned embeddings + RoPE, QK-norm
embeddings	tied
norm	RMSNorm, eps 1e-5

Training

1. Pretrain / SFT — DrunkkToys recipe (~225M tokens), curriculum-balanced across difficulty tiers.
2. Name-coupling fine-tune — the single largest lever measured on this track, worth more than any architecture change we tested.
3. DPO on execution-verified pairs: 43 preference pairs built by sampling k=16 completions per task and running each against the real unit tests. Passing completions become chosen, failing ones rejected — no human labels, no reward model. β=0.2, lr=5e-7, 200 steps.

Usage

import torch
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
ckpt = torch.load("model_state.pt", map_location="cpu", weights_only=False)
weights = ckpt["model_state"]          # weights live under this key

prompt = "<|User|>\nWrite a function that reverses a string.\n<|Assistant|>\n```python\n"
ids = tok.encode(prompt).ids

The checkpoint is a dict carrying the weights and its own training record: val_ce_hist, dpo_loss_hist, base_ckpt_sha256, and the DPO hyperparameters.

Provenance

sha256	a0a5e83cdf931db5 …
trained from	2a28067c975515dc …

Every published artifact in this family records the checkpoint it came from. Verify before trusting any label:

from huggingface_hub import HfApi
print([f.lfs.sha256 for f in
       HfApi().model_info("VibeTheOG/theOG-50M", files_metadata=True).siblings
       if f.rfilename == "model_state.pt"])

Limitations

• Scale. 50.7M parameters. It writes short, self-contained functions. It is not a coding assistant and will not hold a long context or a multi-file task.
• Benchmark contamination. 10 of 164 HumanEval tasks have both question and answer present in the training corpus. The model scores 0/10 on exactly those tasks, and removing them slightly raises the score (9.1% → 9.7% greedy), so the headline numbers are not inflated by leakage — but the leakage is real and is being fixed in the corpus.
• pass@32 assumes an oracle. It counts a task solved if any of 32 samples passes the tests, which requires having the tests. Real use does not.
• Val CE split. The held-out slice used for the DPO monitoring above is the tail of the training corpus file, not the original trainer split (which was not recorded). It is comparable within that run, not against other figures.
• MBPP numbers are not published yet. They are being measured; nothing is claimed until they exist.