BananaMind/Picolo-BananaMind

BananaMind è Picolo 🍌

100 parameters. Not 100M. Not 100K. One hundred.

We gave a banana two hidden dimensions and sent it to school. It attended 10 million tokens of FineWeb-Edu, then went home and said ffffffff. Tuition was non-refundable.

This is Picolo-BananaMind, a real causal self-attention transformer with exactly 100 unique trainable parameters, saved as safetensors, loaded through Transformers trust_remote_code. No GRU in a trench coat. No gigantic frozen teacher hiding behind the curtains. No pretrained embeddings. Just a very small banana with very large paperwork.

The tiny-model spirit of MicroSupra-1k was an inspiration; this architecture, tokenizer and text are original. Picolo has approximately one tenth the parameter budget and none of the confidence to use it responsibly.

Count them. There aren't many.

Part	Shape/count
Token embedding, tied to output head	20 × 2 = 40
RMSNorm	2
Query/key/value projection weights	2 × 6 = 12
Query/key/value biases	6
Attention output projection	2 × 2 = 4
Feed-forward up projection	2 × 9 = 18
Feed-forward down projection	9 × 2 = 18
Total unique trainable parameters	100

One layer, one attention head, hidden width 2, SiLU feed-forward width 9, causal scaled dot-product attention, residual connections, fixed sinusoidal positions. The output head shares the embedding tensor and isn't counted twice. All model parameters are trained. The safetensors payload has 100 float32 values (400 bytes); file headers and Python are bigger than its brain.

The tokenizer is doing the tiny-model thing

Each UTF-8 byte is split into two 4-bit tokens: two model tokens per byte, plus an EOS at each document boundary. Sixteen possible 4-bit values + PAD/EOS/BOS/UNK = vocabulary 20. Ordinary text round-trips losslessly. These are not 10 million wordpieces: token counts are not directly comparable to ordinary BPE models. No trained tokenizer or external neural model adds hidden parameters.

Training streamed the sample-10BT configuration of HuggingFaceFW/fineweb-edu, in dataset order, stopping at exactly 10,000,000 loss-target tokens. BOS prefixes are extra conditioning tokens, not part of that budget. No benchmark examples were intentionally used for training; no overlap audit was performed. Sequences: 256 tokens; batch: 64 sequences (final batch shorter); 611 optimizer updates; seed 42; AdamW, initial LR 0.003 linearly decaying toward 0.0003, weight decay 0.01, gradient clipping 1.0. Full training metadata and stream hash: training_results.json. Training code: train.py.

The report card

Full evaluation sets, zero-shot, local installed lm_eval harness, float32. Scores below are percentages; both metrics are shown so the banana cannot select whichever column makes it look taller.

Benchmark	Questions	Raw accuracy	Normalized accuracy
PIQA	1,838	53.10	50.71
ARC Challenge	1,172	20.99	27.82
ARC Easy	2,376	25.42	25.63
HellaSwag	10,042	25.42	23.70
ArithMark 3.0	1,000	27.30	28.40

Raw lm_eval standard errors, in percentage points: PIQA 1.16, ARC Challenge 1.19, ARC Easy 0.89, HellaSwag 0.43. Normalized errors: 1.17, 1.31, 0.90, 0.42 respectively. This is mostly a lesson in chance baselines and answer-length effects, not evidence that 100 parameters learned school mathematics. Multiple-choice likelihood scoring is not the same as successfully generating answers. Do not outsource your taxes to Picolo. It has fewer parameters than your accountant has pens.

ArithMark used the upstream bencharithmark-3.py, unmodified, with max context 1024, batch 32. Its primary metric is acc_norm. The local lm_eval run used max length 8192 4-bit tokens, batch 8, because a first attempt at 1024 failed when continuations exceeded the cap. That failed run produced no reported scores. Fixed sinusoidal positions permit this extended window; training context was only 256, so this is extrapolation, not trained long-context ability. Long inputs may still be left-truncated by evaluation tools. Normalization conventions differ between tools; do not treat their normalized metrics as identical.

Results: lm_eval JSON, ArithMark JSON with predictions. Exact task configs, sample counts and metric details are in the JSON. No result was invented, hand-picked from a subset, or selected across multiple trained seeds.

Invite the banana into Python

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = 'BananaMind/Picolo-BananaMind'
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
assert sum(p.numel() for p in model.parameters()) == 100
inputs = tokenizer('Banana', return_tensors='pt')
output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Greedy output is usually repetitive nonsense. Sampled output may contain invalid UTF-8; decoding replaces invalid byte sequences. Custom code is small enough to actually read before trusting it. No KV cache: generation recomputes the prefix. CPU works; GPU is comically unnecessary.

Reproduce the bruising

Tested with Python 3.14, torch 2.11.0+cu130, Transformers 5.7.0, on an RTX 5070 Ti. Install torch transformers datasets safetensors lm_eval, then:

python train.py
python -m lm_eval run --model hf \
  --model_args pretrained=.,trust_remote_code=True,dtype=float32,max_length=8192 \
  --tasks piqa arc_challenge arc_easy hellaswag --num_fewshot 0 \
  --batch_size 8 --device cuda --output_path eval-output --log_samples
# Download arithmark-3.jsonl from AxiomicLabs/Arithmark-3.0 first.
python evaluation/bencharithmark-3.py --model . \
  --data-path arithmark-3.jsonl --device cuda --dtype float32 \
  --batch-size 32 --max-context 1024 --results-dir arithmark-output --save-predictions

train.py writes a fresh checkpoint into model/ alongside this repository checkout; evaluate that directory when reproducing training. Benchmarks were run on the trained checkpoint before publishing. Causal masking and the 100-parameter count were checked locally. MIT applies to original code and weights; upstream benchmark code and datasets retain their own terms.

Named BananaMind è Picolo. Small brain. Big peel.