Boris-1.7-D60M-n30M

A 60M-parameter dense decoder-only transformer with a 30M-parameter n-gram embedding table spliced into the residual stream after block 2. One experiment, one variable: does 30M of lookup buy what 30M of dense weight would, at a fixed 2.5B-token budget on a single RTX 3060 12GB?

The trunk is deliberately conventional — the table is the only thing under test.

Model

Trunk	10 layers, d=768, 12 heads / 4 KV heads (GQA), SwiGLU 1920, RoPE, RMSNorm, tied embeddings
Context / vocab	1024 / 32,768 (own byte-level BPE, trained on this corpus)
Dense (non-embedding) params	59,981,568
Embedding params	25,165,824 (tied)
N-gram table	468,751 rows × rank 64 + 64→768 projection = 30,049,216

The table is keyed on the literal last 2 and 3 token ids — an explicit top-K vocabulary (281,250 bigram + 187,500 trigram rows, one shared OOV row), built by ngram.py over the corpus. No hashing, exact searchsorted lookup, one gather per position and no matmul, so compute per token barely moves.

Training

• 2.5B tokens: Ultra-FineWeb bulk, then cosmopedia-v2 + fineweb-edu anneal over the final 15%
• WSD schedule (2% warmup, 85% stable, 1-sqrt decay), Muon on 2-D linear maps, AdamW on everything else, bf16 + torch.compile
• 10,172 steps, 245,760 tokens/step, 22h37m wall clock
• Final val: loss 3.235, ppl 25.4, 1.0012 bits/char; n-gram hit rates ~65% bigram / ~21% trigram

Resumable to the byte (optimiser states, data cursor, all RNGs round-trip through the checkpoint).

Results (zero-shot)

Scored with the model's own tokenizer; harness validated against GPT-Neo-125M's published numbers first. --no-ngram ablates the table at inference, which measures how much the finished model leans on it — not the dense control (that needs its own training run).

Task	metric	intact	table ablated	Δ
HellaSwag	acc_norm	31.22	30.54	−0.68
ARC-Easy	acc_norm	45.88	43.77	−2.11
ARC-Challenge	acc_norm	26.02	22.95	−3.07
PIQA	acc_norm	62.57	60.55	−2.02
WinoGrande	acc	51.07	50.12	−0.95
LAMBADA	acc	26.66	25.33	−1.33
val loss	bits/char	1.0012	1.0699	+0.0687

Ablating the table costs 0.069 bits/char and 0.7–3.1 points across the six tasks. The two ARC splits move most; HellaSwag and WinoGrande barely notice.

Additional benchmarks, table intact only:

Benchmark	metric	score
ArithMark-3	acc_norm	36.1 ±1.5 (random 25)
BananaMind Base Bench 1.1	Elo	1022 (54.3% acc)

Usage

This is not a transformers checkpoint — it's a custom architecture (60M dense trunk + 30M n-gram lookup table) with a custom byte-level BPE tokenizer. Load it with the loader in this repo:

from loader import load_model, load_tokenizer

model = load_model(".").eval()     # reads config.json + model.safetensors
tok = load_tokenizer(".")          # reads tokenizer.json (custom 32,768 BPE)

The n-gram table and its top-K lookup arrays are inside model.safetensors, so that one file — plus config.json — is enough to run the full model. lm_head.weight is tied to the input embedding and stored once.

Tokenize with the tokenizer's GPT-2-style byte-level split (digit runs are capped at three characters):

ids = tok.encode("The capital of France is")

Generation loop (greedy, single token step):

import torch

def generate(prompt, n_tokens=32):
    ids = tok.encode(prompt)
    for _ in range(n_tokens):
        with torch.no_grad():
            logits = model(torch.tensor([ids]))[0]
        nxt = int(logits[0, -1].argmax())
        ids.append(nxt)
        if nxt == 0:            # <|endoftext|>
            break
    return tok.decode(ids)

print(generate("The capital of France is"))

AutoTokenizer also works (self-contained tokenizer.json), but the model itself has no transformers class:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.7-D60M-n30M")

Run python loader.py "The capital of France is" for a next-token smoke test.