theOG-50M

A 50.7M-parameter code generation model trained from scratch, aligned with DPO on execution-verified preference pairs — every "chosen" completion in the preference set actually passed the task's unit tests.

Small enough to run on a laptop CPU. Deep rather than wide (10 layers × 512), a choice made from measurement: at this scale depth beat width on every benchmark we ran.

This version — DPO v4 (2026-08-16)

model_state.pt is theOG_50M_dpo_v4, sha256 c2479a076101fd79…, DPO from the deep fine-tuned base 2a28067c9755… on 43 execution-verified pairs (beta 0.2, lr 5e-6, 300 steps, accum 8, reference = frozen base). Held-out val CE moved 0.4311 → 0.4504 (+0.019), 20x under the degradation flag — the preference objective moved the logits without damaging the LM.

Results

Greedy (true argmax, identical MLX harness, measured for this version)

benchmark	base 2a28067c…	this model c2479a07…	delta
graded L1-L10 v2	18/30	21/30	+3
HumanEval-164	15/164	19/164	+4
MBPP-500	18/500	20/500	+2

First DPO checkpoint with a verified, stable, across-the-board greedy gain — graded +3, HE +4, MBPP +2, no benchmark lost, val CE stable.

External calibration — vs Salesforce CodeGen-350M (357M, 7x larger)

model	params	graded v2 strict (greedy)	HE-164 pass@32
theOG-50M (this model)	50.7M	21/30	not run yet
theOG-50M base	50.7M	19/30	32/164
theOG-50M DPO v2	50.7M	20/30	36/164
CodeGen-350M	357M	20/30	51/164

On 18/30 vs 19/30 for "base". Both rows are the same weights (2a28067c…; theOG_50M_deep and theOG_50M_deep_dpo are byte-identical checkpoints under two directory names). The 18/30 in the greedy table is the MLX bench_graded harness; the 19/30 here is the torch autograde-strict harness. The +3 delta above is harness-consistent — base and DPO v4 were both measured on MLX.

Graded numbers from the benchmark record (BENCHMARK_PILOT_50M_2026-08-11.md, same autograde-strict harness, greedy). HE-164 pass@32 for the 50M family from the same record; CodeGen-350M measured 2026-08-15 (k=32, temp 0.8, results/he164_k32_codegen350_final.json). DPO v4's own pass@32 has not been run yet — the cell above says "not run yet" and will be filled in once measured.

Test-time — K8 self-consistency + MBR-exec (no retraining)

On MBPP-500 the k=32 candidate headroom converts to pass@1 via execution-clustered selection (MBR-exec, scripts/rerank_candidates.py): greedy 18 → MBR 53/500 (p = 3e-8), clean-solvable conversion 30.4% → 76.1%. On graded v2, K8 sampling raises strict from 19 → 25/30.

Use cases

• Local code autocomplete — a lightweight completion for your editor that runs entirely on your laptop, offline, no cloud.
• On-device assistant — a tiny model you can embed in an app; your code and prompts never leave the device.
• Learning — run and inspect a real language model on your own machine, no GPU or API required.

Architecture

parameters	50.7M
layers × hidden	10 × 512
attention heads	4
FFN	2048, SwiGLU
vocabulary	16,000 (custom BPE)
context	1024
position encoding	RoPE + learned

Files

• model_state.pt — theOG_50M_dpo_v4 weights (sha256 c2479a076101fd79…)
• tokenizer.json — custom 16k BPE tokenizer
• config.json — model configuration
• mlx/ — theOG_50M_dpo_v4 in MLX format (custom model.py with QK-norm + learned pos-embed, logits-parity verified vs the torch reference)
• gguf/ — theOG_50M_deep_dpo (the base 2a28067c…, not DPO v4) as f16 GGUF, 114 tensors, round-trip max abs err 4.88e-04. Written under the custom theog50m arch key: a stock llama.cpp cannot load it without an arch patch. See gguf/README.md.

Usage (PyTorch)

AutoModelForCausalLM does not work with this repo. The architecture is custom (model_type: elastic, ElasticGPT) and is not registered in Transformers, and this repo ships no modeling_*.py, so trust_remote_code does not help either. Loading it raises:

ValueError: The checkpoint you are trying to load has model type `elastic`
but Transformers does not recognize this architecture.

AutoTokenizer does work. For the weights, use model_state.pt with the model definition from the source repo, or use the MLX folder below.

from transformers import AutoTokenizer   # tokenizer only
tok = AutoTokenizer.from_pretrained("DrunkkToys/theOG-50M")

Usage (MLX)

The mlx/ folder is a ready-to-load MLX model. Note that mlx_lm.load() takes a repo id or a local path — "DrunkkToys/theOG-50M/mlx" is not a valid repo id (three segments) and raises HFValidationError. Download the subfolder first, then load the local path:

from huggingface_hub import snapshot_download
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

path = snapshot_download("DrunkkToys/theOG-50M", allow_patterns="mlx/*")
model, tokenizer = load(f"{path}/mlx")

sampler = make_sampler(0.4, top_p=0.85, top_k=10, min_p=0.2)
out = generate(model, tokenizer, prompt="write a function that returns the sum of two numbers",
               max_tokens=256, sampler=sampler)
print(out)

This is a 50M model: greedy sampling degenerates, and even loose sampling drifts into token-repetition loops. The tuned serving settings are temp 0.4 / top_p 0.85 / top_k 10 / min_p 0.2, with a repetition_penalty of 1.2 and a stop at the model's natural closing code fence.

Serve it with mlx_lm.server — again pointing at the downloaded local folder:

python -c "from huggingface_hub import snapshot_download as d; print(d('DrunkkToys/theOG-50M', allow_patterns='mlx/*'))"
mlx_lm.server --model <printed-path>/mlx \
  --temp 0.4 --top-p 0.85 --top-k 10 --min-p 0.2 --max-tokens 1024

mlx_lm.server only applies repetition_penalty and stop per request, so send them in the request body:

curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"sum two numbers"}],
       "temperature":0.4,"top_p":0.85,"top_k":10,"min_p":0.2,
       "repetition_penalty":1.2,"stop":["\n```\n"]}'

Provenance

Every model this family publishes carries its sha256 and its base checkpoint sha256, because this project previously shipped nine repos containing one identical file. Verify before trusting the label:

from huggingface_hub import HfApi
[f.lfs.sha256 for f in HfApi().model_info("DrunkkToys/theOG-50M", files_metadata=True).siblings
 if f.rfilename == "model_state.pt"]