BananaMind 2.1 Unified - 100%
The 100% training preview of BananaMind 2.1 Unified, a 34,999,041-parameter three-tower base causal language model. Not instruction tuned.
Architecture
Three transformer stacks share one d=384 embedding. A and C are the outer towers and each owns an output head; the next token is a probability-space mixture of the two. B is the relay: no head, no solo loss term, and the only path between A and C.
| Tower A | Tower B (relay) | Tower C | |
|---|---|---|---|
| Layers | 14 | 5 | 6 |
| Hidden | 256 | 320 | 384 |
| MLP (SwiGLU) | 704 | 960 | 1024 |
| Attention | 4Q / 1KV | 5Q / 1KV | 6Q / 2KV |
| Layer params | 9.86M | 5.84M | 9.44M |
| Output head | 2.10M | none | 3.15M |
Head dim 64, QK norm, RoPE theta 100k, RMSNorm eps 1e-6, vocab 8,192, 4,096 context. Shared embedding 3.15M, input projections 384->256 for A and 384->320 for B; C reads the embedding natively. Three bridge points per pair in both directions, 1.23M total.
Exchange schedule, 1-indexed. Bridge output is added before the receiving block, so the relay always has processing between taking a signal in and handing one back out:
| round | A read | C read | -> B | B runs | B read | -> A | -> C |
|---|---|---|---|---|---|---|---|
| 1 | 5 | 2 | pre-1 | 1-2 | 2 | 7 | 3 |
| 2 | 9 | 4 | pre-3 | 3-4 | 4 | 11 | 5 |
| 3 | 12 | 5 | pre-5 | 5 | 5 | 14 | 6 |
Gates are per-channel and initialise to 0.01 rather than 0: B carries no solo loss, so a zero init risks the middle never waking up.
Training
| Field | Value |
|---|---|
| Progress | 100% |
| Tokens seen | 37,999,869,952 |
| Target tokens | 38,000,000,000 |
| Hardware | 8 x NVIDIA RTX PRO 6000 Blackwell Server Edition |
| Optimizer | AdamW, single group |
| Peak LR | 0.0023 |
| Schedule | WSD, cosine to 0 over final 15% |
| Loss | L_mix + 0.3 * (L_A + L_C) |
| Precision | bfloat16 autocast |
| Source | Share |
|---|---|
| fineweb_hq | 50.957% |
| dclm | 20.766% |
| cosmopedia_v2 | 20.043% |
| finemath | 8.234% |
Usage
.logits holds a normalised log-probability vector, not unnormalised
logits, because the two heads are mixed in probability space. log_softmax is
the identity on it, so scoring, generate() and sampling all behave normally.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Banaxi-Tech/unified-2.1-test"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
Set cut_bridges=True to sever every bridge, which leaves A and C as two
ordinary standalone transformers:
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, cut_bridges=True)
- Downloads last month
- -