BananaMind 2.1 Lite All-Looped
This all-middle-looped AdamW experiment is published separately at
Banaxi-Tech/BananaMind-2.1-Lite-All-Looped-AdamW. It does not overwrite the
original Lite or the partially looped AdamW experiment.
BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class.
- 19,950,029 parameters in the tied-token Transformer core
- 4,999,970 parameters in a two-hash causal trigram module
- BananaMind 2 Nano's 8,192-token tokenizer
- 13 physical layers and 24 effective layer passes
- execution:
L1 → (L2 → ... → L12) × 2 → L13 - L1 is the prelude, L2-L12 are one recurrent middle stack, and L13 is the coda
- every middle-layer visit reuses the same physical layer weights on pass two
- 4,096-token context
The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9, once in each recurrent-stack pass. L5 and L9 have separate learned injection scales, both initialized to 0.5.
75B-token streamed curriculum
| Source | Tokens | Share |
|---|---|---|
| FineWeb-Edu | 37.50B | 50% |
| DCLM Baseline | 16.50B | 22% |
| Cosmopedia v2 | 7.50B | 10% |
| FineMath 4+ | 6.75B | 9% |
| FinePhrase | 4.50B | 6% |
| NPset-2 Python-Edu | 2.25B | 3% |
The NPset allocation is 75% normalized code and 25% raw
original_code. Curriculum weights move from web-heavy foundations to a
balanced phase and then greater math, synthetic-text, phrase, and code
coverage. A token-credit scheduler keeps the complete run within one global
batch of every target allocation.
Training uses pure AdamW with (0.9, 0.95) betas: peak LR 0.003 for the
Transformer core, embeddings, and controls, and peak LR 0.001 for the n-gram
module. Both rates use warmup-stable-decay with a final 15% cosine decay.
Checkpoints are uploaded every 5% with safetensors, tokenizer files, pinned
dataset revisions, metrics, exact source-token accounting, and complete state
for resuming.
Launch on Hugging Face Jobs
# Start a new 4 x H200 run (32 sequences/GPU, global batch 128).
./launch_lite_all_looped_hf_job.sh 4 fresh
# Resume it later.
./launch_lite_all_looped_hf_job.sh 4 resume
# The same global batch on 8 x H200 (16 sequences/GPU).
./launch_lite_all_looped_hf_job.sh 8 resume
The launcher expects HF_TOKEN to be available to the Hugging Face CLI and
as a Hugging Face Jobs secret.
- Downloads last month
- 1,721