BananaMind-2-Mini

BananaMind-2-Mini is a small decoder-only causal language model trained from scratch by BananaMind on a 30B-token curriculum. It is our first model in the BananaMind 2 Series!

The model has 25,178,752 parameters, a 4,096 token context window, and a custom 8k-token digit-aware byte-level BPE tokenizer.

Model Details

Field	Value
Parameters	25,178,752
Architecture	BananaMind2Mini decoder-only Transformer
Layers	14
Hidden size	384
Intermediate size	1,024
Attention heads	6
KV heads	2
Head dim	64
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocab size	8,192
Context length	4,096
Embeddings	Tied input/output embeddings
Weight format	safetensors
HF architecture	BananaMind2MiniForCausalLM
HF model type	bananamind2_mini
Final checkpoint	runs/bananamind2-mini/final.pt
Final training step	55,485
Tokens seen	29,999,726,592

Tokenizer

BananaMind-2-Mini uses a custom 8k byte-level BPE tokenizer trained from FineWeb-Edu text with digit-aware pre-tokenization.

Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.

Digit IDs:

Token	ID
0	19
1	20
2	21
3	22
4	23
5	24
6	25
7	26
8	27
9	28

Examples:

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Token	ID
<pad>	0
<bos>	1
<eos>	2
<unk>	3

Training Data

BananaMind-2-Mini was trained on a 30B-token mix of web, educational, synthetic textbook, and math data.

Dataset	Target Tokens	Share
FineWeb-Edu	16.5B	55%
DCLM	9.0B	30%
Cosmopedia-v2	3.0B	10%
FineMath-4+	1.5B	5%
Total	30.0B	100%

The run used a progressive curriculum rather than sampling the final aggregate mix from the first token.

Phase	Token Range	Mix
Web-heavy start	0B to 7.2B	60% FineWeb-Edu, 38% DCLM, 1% Cosmopedia-v2, 1% FineMath-4+
Curriculum ramp	7.2B to 8.0B	Ramps toward more synthetic and math data
Main mix	8.0B to 18.0B	55% FineWeb-Edu, 32% DCLM, 8% Cosmopedia-v2, 5% FineMath-4+
Aggregate taper	18.0B to 23.2B	Tapers toward the final aggregate target
Final mix	23.2B to 30.0B	50.957% FineWeb-Edu, 20.766% DCLM, 20.043% Cosmopedia-v2, 8.234% FineMath-4+

Training Setup

Field	Value
Sequence length	4,096
Micro batch	12
Gradient accumulation	11
Effective batch	132 sequences
Tokens per optimizer step	540,672
Planned optimizer steps	55,486
Actual final step	55,485
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	2.3e-3
Warmup steps	1,750
LR schedule	Warmup-stable-decay with cosine decay
Decay ratio	0.15
Weight decay	0.1, then 0.01 after 12B tokens
Gradient clipping	1.0
Z-loss coefficient	1e-4 until 12B tokens, then off
Compile	PyTorch compile enabled
Seed	1337

Evaluation

Self-reported benchmark scores using lm_eval. Scores may vary a bit depending on harness version, runtime settings, dtype, and evaluation environment.

All task scores below use acc_norm,none. Average is the mean over ARC Easy, PIQA, ARC Challenge, and HellaSwag.

Model	Average	ARC Easy	PIQA	ARC Challenge	HellaSwag
BananaMind-2-Mini	38.72	39.86	59.63	25.68	29.72
Zupra-1.6-50M-Instruct-Ultra-exp	39.47	43.14	59.47	25.60	29.68
BananaMind-1.5-Base	39.46	42.47	60.61	23.98	30.77
MiniBananaMind-v4-9M	35.07	34.97	55.39	23.04	26.87
Pythia-31M	34.79	34.01	56.47	21.42	27.28

Higher is better on the y-axis, and more parameters are farther right on the x-axis. The top-left region represents the most score-efficient models.

Repository Files

File	Description
config.json	Transformers config for bananamind2_mini
model.safetensors	Final exported model weights
tokenizer.json	Custom 8k digit-aware tokenizer
tokenizer_config.json	Tokenizer metadata
generation_config.json	Default generation config
configuration_bananamind2mini.py	Custom Transformers config class
modeling_bananamind2mini.py	Custom Transformers model class
checkpoint_metadata.json	Source checkpoint, step, and token metadata

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "BananaMind/BananaMind-2-Mini"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Suggested Generation Settings

For stable continuations:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=64 to 160

For more varied text:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• top_k=50
• repetition_penalty=1.1
• max_new_tokens=64 to 192

Intended Use

BananaMind-2-Mini is intended for lightweight language-model research, local experimentation, text continuation, tokenizer experiments, and small-model training comparisons.

Because this is a base model, prompts should be written as continuation prompts rather than chat messages.

License

Apache 2.0

🍌