BananaMind 2 Micro Preview - 100%
This is the 100% training preview of BananaMind 2 Micro, a
2,933,193-parameter base causal language model. It is not an
instruction-tuned chat model.
Architecture
| Field |
Value |
| Parameters |
2,933,193 |
| Layers |
9 |
| Hidden size |
128 |
| Intermediate size |
512 |
| Query/KV heads |
4 / 2 |
| Head dimension |
32 |
| Context |
4,096 |
| Vocabulary |
2,048, tied |
| Attention |
GQA, pre-RoPE QK norm |
| MLP |
SwiGLU |
| Refresh |
detached attention signal, kernel-9 causal depthwise gate, embedding reinjection |
The learned refresh scalar is initialized to zero. During cached generation,
the causal convolution history is carried alongside each layer's K/V cache.
Training
| Field |
Value |
| Progress |
100% |
| Tokens seen |
74,998,349,824 |
| Target tokens |
75,000,000,000 |
| Hardware |
8 x NVIDIA RTX PRO 6000 Blackwell Server Edition |
| Matrix optimizer |
Stock torch.optim.Muon |
| Muon LR |
0.05 |
| Embedding/1D optimizer |
AdamW |
| AdamW LR |
0.004 |
| Precision |
bfloat16 autocast |
| Token range |
FineWeb-Edu |
DCLM |
Cosmopedia v2 |
FinePhrase |
FineMath |
NPSet2 |
| 0.00B-15.00B |
55% |
30% |
6% |
4% |
4% |
1% |
| 15.00B-30.00B |
48% |
26% |
9% |
8% |
7% |
2% |
| 30.00B-48.75B |
42% |
20% |
12% |
12% |
11% |
3% |
| 48.75B-63.75B |
36% |
16% |
14% |
15% |
15% |
4% |
| 63.75B-75.00B |
30% |
12% |
15% |
18% |
21% |
4% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind-Model-Previewers/BananaMind-2-Micro-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
