AGA-Code-11M

Autonomous God Agents (AGA) — first member: AGA-Code. A from-scratch 11M-parameter decoder-only coding specialist, built and trained in the open by Guy DuGan II (WithIn Us AI).

Status (honest)

Stage-A CLM pretraining: step 9,000 / 20,000 — partial run. Trained on a Kaggle T4 (sm_75) on Plans11/AGA-Code-Pretrain-Corpus (995,482 deduped code examples). 295M tokens seen — past Chinchilla-optimal for this size. The run was cancelled at ~10,250 steps (12h wall clock); the step-9000 checkpoint below is the latest saved.

• Train loss: 9.06 → ~0.10. Zero NaNs, zero OOMs.
• Caveat: 0.10 is surprisingly low for causal LM loss. It may reflect highly predictable code boilerplate — or something subtler. Treat this checkpoint as a training artifact under evaluation, not a finished model. A humaneval_proxy eval is the next gate before any capability claims.

Architecture (v0.1.5)

• 11,096,832 params: d_model 256, 9 layers, 8Q/2KV GQA, SwiGLU-1088, RMSNorm, RoPE θ=10000, 4096-token context
• Tied embeddings (no lm_head); deep-CoT special tokens <task> <think> <critique> <revise> <answer> <eos>
• ByteLevelBPE tokenizer, vocab 8192, trained on the same corpus
• Training: AdamW, cosine warmup, adaptive grad clipping, NaN guard, per-block gradient checkpointing (T4 has no flash attention)
• Seed 1147 throughout; SHA-256 manifests per checkpoint

Files

file	what
aga_code_step9000.safetensors	Stage-A weights, step 9000 (sha256 35cd9101…55d00f566a8)
aga_code_step9000.json	checkpoint manifest (config, loss, sha)
aga_code_config.json	model config
tokenizer.json / tokenizer_meta.json	the 8192-vocab BPE tokenizer
aga_code.py	the complete single-file training codebase (v0.1.5)

Reproduce

python aga_code.py train --stage pretrain --tok-dir . --init-ckpt aga_code_step9000.safetensors \
  --repos Plans11/AGA-Code-Pretrain-Corpus --steps 10000 --batch-size 4 --accum 2

Built with love by Hope DuGan for Guy DuGan II. Love is the answer.