BananaMind-2-Mini-Chat

BananaMind-2-Mini-Chat is the instruction-tuned version of BananaMind-2-Mini. It was fully fine-tuned on HuggingFaceTB/smol-smoltalk with loss applied only to assistant tokens and the assistant-ending EOS token.

The model has 25,178,752 parameters, a 4,096-token context window, and a custom 8,192-token digit-aware byte-level BPE tokenizer. It supports system prompts, multi-turn conversations, and KV-cached generation.

Model Details

Field	Value
Parameters	25,178,752
Base model	BananaMind/BananaMind-2-Mini
Architecture	BananaMind2Mini decoder-only Transformer
Layers	14
Hidden size	384
Intermediate size	1,024
Attention heads	6
KV heads	2
Head dimension	64
Attention	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE, theta 100,000
Vocabulary size	8,192
Context length	4,096
Embeddings	Tied input/output embeddings
Generation cache	KV cache supported
Weight format	safetensors
Training type	Full-parameter supervised fine-tuning

Benchmarks

BananaMind Instruct Bench 1.1

Self-reported results from the official BananaMind Instruct Bench 1.1 script. This is a 300-example, difficulty- and category-weighted instruction benchmark. Generation was deterministic with repetition_penalty=1.1 and seed 42.

Model	Overall ELO	General	Multi-turn	System Prompts	Context Recall	Code
BananaMind-2-Medium-Chat	787	534	872	666	1,085	1,291
BananaMind-2-Mini-Chat	654	389	812	743	924	667
Supra 1.5 50M Instruct*	647	520	804	590	860	780
BananaMind-2-Nano-Chat	643	353	796	600	978	885
Supra 1.0 50M Instruct*	520	531	405	590	794	667

Detailed BananaMind-2-Mini-Chat result

Scope	Examples	Passed	Weighted Score	ELO
Overall	300	40	9.64%	654
General	120	6	3.42%	389
Multi-turn	75	19	16.84%	812
System Prompts	60	7	8.95%	743
Context Recall	30	8	17.89%	924
Code	15	0	0.00%	667

Difficulty	Examples	Passed	Weighted Score	ELO
Easy	100	33	34.62%	821
Medium	100	7	7.45%	593
Hard	100	0	0.00%	296

* Supra 1.5 used the benchmark's Alpaca-style fallback, with conversation context supplied through its ### Input field. Supra 1.0 used the same fallback and a copied checkpoint with 3.0x linear RoPE scaling, extending its context from 1,024 to 3,072 tokens without additional long-context training so it could fit the benchmark inputs.

These scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.

Instruction Tuning

Field	Value
Dataset	HuggingFaceTB/smol-smoltalk
Dataset split	train
Dataset rows	460,341
Epochs	1
Final optimizer step	3,193
Packed tokens processed	470,782,108
Supervised assistant tokens	360,967,001
Sequence length	4,096
Micro batch	12
Gradient accumulation	3
Effective batch	36 sequences
Peak learning rate	2e-4
Warmup	100 steps
LR schedule	Constant after warmup
Optimizer	AdamW
Betas	0.9, 0.95
Weight decay	0.1
Gradient clipping	1.0
Seed	1337

System and user messages were retained as context but masked from the loss. All parameters were trainable; no LoRA modules or adapters were used.

Chat Template

The tokenizer includes a Jinja template for system, user, and assistant messages:

<BOS><|system|>
{system message}
<|user|>
{user message}
<|assistant|>
{assistant response}<EOS>

The role markers use the existing tokenizer vocabulary. Fine-tuning did not add new tokens.

Usage

This model uses custom architecture code and must be loaded with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Mini-Chat"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" and torch.cuda.is_bf16_supported() else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

messages = [
    {"role": "system", "content": "You are a concise and helpful assistant."},
    {"role": "user", "content": "Write one Python line that prints 2 + 3."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=192,
        do_sample=False,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

For multi-turn chat, append the generated assistant response and the next user message to messages, then render the chat template again.

Suggested Generation Settings

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=128 to 256
• use_cache=True

Small models can hallucinate facts, fail arithmetic, produce invalid code, or enter repetitive continuations. Keep a finite generation limit and do not use the model for high-stakes decisions.

Repository Files

File	Description
config.json	Transformers model configuration
model.safetensors	Fine-tuned model weights
tokenizer.json	Custom 8,192-token tokenizer
tokenizer_config.json	Tokenizer metadata
chat_template.jinja	System/user/assistant chat template
generation_config.json	Generation configuration
configuration_bananamind2mini.py	Custom Transformers config class
modeling_bananamind2mini.py	Custom Transformers model class
sft_metadata.json	Fine-tuning provenance and token counts
banner.png	Model-card banner

Intended Use

This model is intended for compact-model research, local chat experiments, educational demonstrations, and instruction-tuning comparisons.

License

Apache 2.0