Cascade 1B BPE Logic v2
Experimental 976M-parameter causal language model with about 385M active parameters per token.
Architecture
- Mistral 32K BPE tokenizer
- four 384-wide local transformer layers
- surprise-driven causal patching with a 256-patch budget
- eight shared 1024-wide attention layers
- three top-1 routed FFN experts per core layer
- Muon for matrix parameters and AdamW for embeddings, norms, heads, and routers
v2 fixes a future-information leak in overflow patch compaction. Patch IDs now depend only on the current prefix. It also replaces three independently trained full routes with shared attention and learned sparse FFN routing, so shared reasoning layers receive every update while active compute stays near the earlier model.
Weights are initialized from Asilarkness/cascade-1b-bpe-logic checkpoint step 35000. Compatible shared weights are copied, route attention is merged, and route FFNs become sparse experts.
Training mix
- FineWeb-Edu sample-10BT: 58%
- FineMath 4+: 22%
- Cosmopedia v2: 10%
- OpenR1 Math 220k: 7%
- Codeforces CoTs: 3%
This is an experimental base model, not an instruction-tuned assistant. Early checkpoints can produce incoherent text and should not be used for factual or safety-critical tasks.
- Downloads last month
- 516