BananaMind-2.1-Pico-Preview

BananaMind-2.1-Pico-Preview is a 1,480,516-parameter decoder-only base language model trained from scratch by BananaMind on 2B tokens of FineWeb-Edu. It is the first public checkpoint of the BananaMind 2.1 architecture line and is released as a preview: the architecture, not the checkpoint, is the point of the release.

The model has three physical Transformer blocks but executes four blocks per forward pass — block 2 runs twice (L1 → L2 → L2 → L3). Alongside the loop it carries three architectural additions over BananaMind 2.0: value-subtraction cross-head attention (XSA), a detached embedding-refresh gate, and a hashed causal trigram embedding injected mid-stack.

This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with trust_remote_code=True.

Release Status

Field	Value
Release type	Preview checkpoint
Loop configuration	partial, 2 passes
Tokens seen	2,000,000,000
Training status	Complete
Evaluation status	Complete

The execution schedule is set by loop_mode and loop_passes in config.json, not hardcoded in the modeling file. The released checkpoint was trained and evaluated under partial with 2 passes.

Model Details

Field	Value
Total parameters	1,480,516
Core parameters	980,547
N-gram table parameters	499,969
Architecture	BananaMind21Test decoder-only Transformer
Physical layers	3
Effective layer executions	4 (L1 → L2 → L2 → L3)
Hidden size	128
Intermediate size	384
Attention heads	4
KV heads	1
Head dimension	32
Attention style	Grouped-query attention with QK norm and value-subtraction XSA
Refresh gate	Detached-attention gate with causal depthwise convolution (kernel 9)
N-gram embedding	Hashed causal trigram table, 3,906 buckets, injected before block 2
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-5
Vocabulary size	2,048
Context length	3,072
Embeddings	Tied input/output embeddings
Generation cache	Not supported (see Limitations)
Weight format	safetensors
HF architecture	BananaMind21TestForCausalLM
HF model type	bananamind21_test

Architecture Overview

Tokens pass through a tied input embedding scaled by sqrt(hidden_size), then through the looped block stack, a final RMSNorm, and the tied language-model head. Each block is pre-normalized and contains causal grouped-query self-attention, the embedding-refresh gate, and a SwiGLU feed-forward network, with residual connections around each sublayer.

Partial looping. Only block 2 is reused. The first and last blocks run once each, so the model pays for three blocks of parameters and four blocks of compute. This sits between a fully looped stack (every block twice) and a plain three-block stack; the ablation below shows partial looping beating both.

Value-subtraction XSA. After standard grouped-query attention, each head's output has the component parallel to that head's current-token value vector projected out. Because four query heads share one KV head, the subtraction is applied per KV group across its four query heads, removing the self-value direction that otherwise dominates short-context attention output.

Embedding-refresh gate. Each block re-injects the original token embedding through a gated path. The gating signal is the block's own attention output, detached from the gradient graph, combined with a strictly causal depthwise convolution over that signal (left-padded, kernel size 9). The gate multiplies a projection of the original embedding, and the result is scaled by a zero-initialized scalar alpha, so the path starts as a no-op and the model learns how much lexical identity to restore at each depth.

Hashed causal trigram embedding. A 3,906-bucket table is indexed by an integer hash of the current token and the two preceding tokens, using only current and past positions. It is added once, before block 2, in every loop pass. At this scale the table is a third of all parameters — it is reported separately in parameter_summary() for that reason.

Parameter Breakdown

Component	Parameters	Share
Token embedding (tied)	262,144	17.71%
Trigram table	499,969	33.77%
Blocks (3 × 239,425)	718,275	48.52%
Final norm	128	0.01%
Total	1,480,516	100.00%

Per block: attention 41,024, refresh gate 50,689, SwiGLU 147,456, norms 256.

Evaluation

ARC Easy, ARC Challenge, PIQA, and HellaSwag use zero-shot acc_norm,none. ArithMark 3 uses length-normalized continuation accuracy. Base Bench is the public 350-item BananaMindBench 1.1 suite.

Benchmark	BananaMind-2.1-Pico-Preview	BananaMind-2-Micro	GPT-S-1.4M
Trainable parameters	1,480,516	2,933,193	1,426,000
Training tokens	2B	75B	6B
ARC Easy	29.50%	33.12%	31.27%
ARC Challenge	22.53%	21.93%	21.93%
HellaSwag	27.04%	28.27%	26.89%
PIQA	54.30%	53.21%	55.17%
ArithMark 3	33.00%	34.00%	30.20%
INT Index	5.369	6.01	5.34
Base Bench Elo	885	874	N/A
Base Bench accuracy	34.57%	34.57%	N/A
Base Bench weighted accuracy	33.65%	32.39%	N/A

BananaMind-2-Micro is the closest architectural relative — same 128-wide hidden size, same 2,048-token vocabulary, same XSA refresh path, same Muon/AdamW split — at nine physical layers rather than three. Pico-Preview is roughly half the parameters trained on 37.5× fewer tokens, and still leads it on Base Bench Elo and weighted accuracy while trailing on the four-choice academic benchmarks and INT Index.

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark 3:

N(s, c) = 100 * (s - c) / (100 - c)
INT = [N(H,25) + N((ARC_E+ARC_C)/2,25) + N(P,50) + 0.65*N(A3,25)] / 3.65

Compute Efficiency

Training compute is estimated consistently as 6 × parameters × training tokens.

Model	Parameters	Tokens	Estimated compute	INT Index	INT per PFLOP
BananaMind-2.1-Pico-Preview	1,480,516	2B	17.77 PFLOPs	5.369	0.302
GPT-S-1.4M	1,426,000	6B	51.34 PFLOPs	5.31	0.103
BananaMind-2-Micro	2,933,193	75B	1,319.91 PFLOPs	6.01	0.005

Pico-Preview reaches within 0.65 INT of Micro for 1.3% of the training compute, which is the result the architecture is meant to demonstrate. Compute here uses the full 1,480,516-parameter count; against models with no n-gram table, the 980,547 core count gives 11.77 PFLOPs and 0.456 INT per PFLOP.

All values are self-reported evaluations and can vary with harness version, tokenizer handling, dtype, and scoring configuration.

Loop Mode Ablation

The same three physical blocks were trained under three execution schedules, holding parameters, data, and step count fixed. partial reuses block 2 once (4 executions), all runs the whole stack twice (6 executions), none runs each block once (3 executions).

Loop mode	Blocks executed	Base Bench Elo	Throughput
partial	4	885	492K tok/s
none	3	884	553K tok/s
all	6	875	344K tok/s

Full looping reached a perplexity roughly 2 lower than partial looping early in training but scored worse on every downstream benchmark, and partial looping led full looping by 1.41 INT Index at the end. Adding a third pass over block 2 (L1 → L2 → L2 → L2 → L3) scored below two passes, so the returns from looping the middle block turn negative quickly.

All three modes are reachable from the released config by setting loop_mode and loop_passes.

Training Data

The full 2B-token budget is FineWeb-Edu, streamed raw with no curriculum, no reweighting, and no phase schedule.

Dataset	Tokens	Share
FineWeb-Edu	2.000B	100.00%
Total	2.000B	100.00%

Training Setup

Field	Value
Sequence length	3,072
Training tokens	2,000,000,000
Optimizer (hidden matrices)	Muon
Peak learning rate (Muon)	3e-2
Optimizer (embeddings, norms, scalars)	AdamW
Peak learning rate (AdamW)	3e-3
Gradient checkpointing	Enabled
Loop noise std	0.0
Hardware	1 × NVIDIA RTX 5070 Ti
Wall-clock time	~1.2 hours
Average throughput	~463K tokens/s

The two-optimizer split follows standard Muon practice: Muon updates the 2D hidden-layer weight matrices, while the tied token embedding, the trigram table, RMSNorm weights, and the scalar parameters (alpha, the n-gram scale) are handled by AdamW.

Tokenizer

BananaMind-2.1-Pico-Preview uses a 2,048-token vocabulary sized for the 1M-parameter class — at this width a full 32K vocabulary would consume more parameters than the entire block stack. The same tokenizer as BananaMind-2-Micro.

Usage

Install the runtime dependencies:

pip install -U torch transformers safetensors

Load the custom architecture with remote code enabled:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2.1-Pico-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For deterministic continuation scoring, use do_sample=False. For free-form sampling, a temperature from 0.6 to 0.8, top_p=0.9, and repetition_penalty=1.1 are reasonable starting points.

Switching Loop Modes

The execution schedule is a config field, so any of the three ablated modes can be run against the released weights:

model.model.config.loop_mode = "none"     # "all", "partial", or "none"
model.model.config.loop_passes = 2

Weights are shared across loop passes, so none and all load without modification — but the checkpoint was trained under partial, and the other modes will score below the numbers above.

Intended Use

BananaMind-2.1-Pico-Preview is intended for architecture research at the sub-2M scale: studying weight-sharing and looped-depth trade-offs, XSA and refresh-gate ablations, n-gram-augmented embeddings, and small-vocabulary training dynamics.

It is a research artifact. At 1.48M parameters trained on 2B tokens it is not a useful text generator.

Limitations

• No KV cache. The model class forces use_cache=False, so generation recomputes the full prefix at every step. Generation cost is quadratic in sequence length and slow for long outputs.
• This is a base model and does not follow instructions.
• The 2,048-token vocabulary produces long token sequences and coarse subword segmentation compared with standard 32K vocabularies.
• The 3,072-token context window limits long-document use.
• At 1.48M parameters, factual recall and multi-step reasoning are minimal; HellaSwag and ARC Challenge sit near chance.
• A third of the parameters live in the trigram table, so parameter-matched comparisons against models without n-gram embeddings should use the 980,547 core count.
• The training data is English-only FineWeb-Edu, so no other language or domain is characterized.
• The model has received no safety alignment and can produce inaccurate, biased, repetitive, or undesirable text.
• Loading requires repository-provided custom Transformers code with trust_remote_code=True.

Do not rely on the model for medical, legal, financial, safety-critical, or other high-stakes decisions.

License

Released under the Apache License 2.0.