Quazim0t0/Escarda-86M-Base-JL

Escarda-86M-Base

🔭 Jet-Long context extension (native 4K → 10K)

This is the Jet-Long edition of Escarda-86M-Base. It extends the usable context from the native 4,096-token training window to 10,240 tokens with no fine-tuning and no change to short-context behaviour, by adding dynamic bifocal RoPE from Jet-Long (arXiv:2607.07740, NVIDIA).

What was applied

Jet-Long pairs a local window (w0 = 2048, classic RoPE) with a remote window whose position map aliases far-apart tokens back onto the pretrained rotation grid:

f(x) = floor(x / G),   G = max(1, ceil(L / 4096))

G adapts to the current sequence length L, so:

• L ≤ 4096 → G = 1 → f is the identity → the model is bit-for-bit the base model. (Verified: max |Δlogit| between Jet-Long on/off within the window is 0.000e+00.)
• L > 4096 → the remote window keeps every rotation in-distribution, so the model extrapolates instead of collapsing.

Implementation notes specific to this SpikeWhaleLM build:

• Only the decoupled RoPE partition (16 of 64 head dims) is aliased; the NoPE partition is untouched. Softmax attention (use_derf=False) — the standard Jet-Long merge applies.
• The remote view is realized by an on-the-fly correction rotation on the already-RoPE'd KV cache (RoPE composes additively), so the cache is never rewritten and decode is cheap.
• Enabled via config: use_jetlong=true, jetlong_w0=2048, jetlong_w_pretrained=4096, max_position_embeddings=10240. Set use_jetlong=false to recover the exact base model.
• The inclusion–exclusion / CuTe throughput kernel from the paper is not included (it targets 100K+ contexts on H100); at 86M params the bifocal attention is computed directly.

Measured (PG-19-style perplexity on held-out text, lower is better)

Context length	Base model	This Jet-Long model
≤ 4,096 (in-window)	(identical — Jet-Long is a no-op)	(identical)
10,240	59.68	16.04

Beyond the training window the base model's perplexity blows up while Jet-Long stays flat — and long-context generation stays grammatical where the base model degrades into word-salad.

Usage

Jet-Long is on by default in this repo. Pass explicit position_ids so RoPE gets true absolute positions during cached decode:

import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("Quazim0t0/Escarda-86M-Base-JL", trust_remote_code=True)
ids = ...              # up to ~10,240 tokens
pos = torch.arange(ids.shape[1]).unsqueeze(0)
out = m(input_ids=ids, position_ids=pos, use_cache=True)   # prefill, then decode step-by-step

Method: Tang, Wang, Gu, Han, Cai — “Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE”, arXiv:2607.07740. Applied here zero-shot to SpikeWhaleLM; no weights were retrained.

Escarda-86M-Base is a ~86M-parameter, from-scratch decoder-only language model — the base sibling of Quazim0t0/Escarda-86M (the chat-tuned model). It shares the same SpikeWhaleLM architecture (Multi-head Latent Attention, an n-gram "engram" memory, hash-lookup layers, hyper-connections, an HRM refinement step, and JEPA / multi-token-prediction training objectives) and the same custom ChatML-aware tokenizer.

This checkpoint is a JEPA-distilled base. It is best used as a starting point for continued pretraining / fine-tuning rather than as a chat assistant.

Related models: SFT / chat model → Quazim0t0/Escarda-86M · live demo → Escarda-86M-Chat Space

Trained using Modal's credits during the Small Models, Big Adventures Hackathon.

---

Model summary

Parameters	~85.7M (tie_word_embeddings=True)
Type	Decoder-only LM (SpikeWhaleLM, model_type: spike_whale)
Hidden size / layers	640 / 16
Attention	10 heads (head_dim=64), 1 KV head (MQA), MLA low-rank Q/O, decoupled RoPE(16)+NoPE(48), QK-norm
Context length	4096 tokens
Vocab	16,512 (custom length-max tokenizer)
License	Apache-2.0

For the full architecture description see the chat model's card.

---

Architecture

These models are built on SpikeWhaleLM, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components:

• Multi-head Latent Attention (MLA + XSA) — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a single KV head (multi-query attention), with QK-norm for stable logits.
• Engram n-gram memory — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream.
• Hash-lookup layers (×2) — multi-head content-addressable features alongside the token embeddings.
• Hyper-Connections — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add.
• HRM refinement — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head.
• Multi-Token Prediction (MTP) — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost).
• Feed-forward is dense (the block is MoE-capable, but MoE is disabled in this release).

JEPA vs HRM. The Escarda models are trained with both HRM refinement and a JEPA (Joint-Embedding Predictive Architecture) auxiliary objective (use_hrm_refine=True, use_jepa=True) — the JEPA term predicts future latent states during training to shape the model's representations. The sibling Byrne models drop JEPA and use HRM refinement only.

Tokenizer

These models use SpikeTokenizer, a custom byte-level "length-max" (greedy longest-match) tokenizer with a 16,512-token vocabulary — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is ChatML-aware, with atomic special tokens for framing and reasoning/tool markers (<|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers) plus <bos>/<eos>/<pad>/<unk>. It ships as a PreTrainedTokenizer subclass (spike_tokenizer.py) and loads via AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

splits. byte_ppl is exp(sum_NLL_nats / total_UTF8_bytes) on WikiText-2 test (tokenizer- independent). BLiMP is fraction of minimal pairs with logprob(good) > logprob(bad) (12 paradigms × 150). Stderr is binomial sqrt(p(1-p)/n).

Language modeling

Metric	Value
WikiText-2 byte_ppl ↓	2.2228
BLiMP acc ↑	0.7144

Multiple-choice suite

Task	acc	±	acc_norm	±
arc_easy	0.3801	0.0100	0.3615	0.0099
arc_challenge	0.1886	0.0114	0.2235	0.0122
hellaswag	0.2759	0.0045	0.2832	0.0045
winogrande	0.5162	0.0140	—	—
piqa	0.5843	0.0115	0.5631	0.0116
openbookqa	0.1300	0.0150	0.2500	0.0194
boolq	0.5138	0.0087	—	—

ArithMark-2.0 (AxiomicLabs)

Metric	Value
acc	0.2536 ± 0.0087
acc_norm	0.2348 ± 0.0085

n = 2,500 · chance = 0.25.

Note: as a distilled base, this checkpoint has the lowest byte-perplexity of the Escarda family but trades off downstream task accuracy — a good reminder that perplexity alone is not a reliable capability ranking. For the strongest chat behaviour use Escarda-86M; use this model when you want a low-loss base to continue pretraining or fine-tune.

---

Training & token budget

• Tokens: ~20B (from-scratch pretraining of the SpikeWhale base, ~28k steps); this checkpoint is a JEPA-distilled snapshot of that base.
• Token/param ratio: ~233 tokens/param (20B / 85.7M) — roughly 11–12× the Chinchilla ~20-tokens/param compute-optimal heuristic, i.e. a deliberately over-trained small model (the inference-efficient trade-off).

Fitting the Chinchilla data term to this model's own pretraining loss curve gives:

L(D) ≈ 2.611 + 77,715 · D^(−0.537) (nats/token, R² = 0.92)

From that fit:

• Compute-optimal tokens for this 86M size ≈ 4.3B → the 20B run is ~4.6× past compute-optimal.
• Diminishing-returns knee ≈ 22.5B tokens (where +1B tokens buys < 0.005 nats) — the 20B stopping point lands right at the knee, a well-judged budget.
• The model is parameter-bound, not data-bound at 20B: the capacity term (0.82 nats) exceeds the data term (0.54), so extra tokens help little. Doubling to 40B is projected to lower loss only 0.07 nats (7% perplexity) with negligible downstream gain — the lever for better quality is more parameters, not more tokens. (This is also why, as a distilled base, it reaches the lowest perplexity of the family without the best downstream scores — it is already at its data-term floor.)

Caveats: single-size fit (folds irreducible loss + capacity floor into one constant); the cosine-LR decay inflates the fitted exponent, so treat β as an upper bound; token counts are anchored to the ~20B figure and scale linearly if that differs.

---

Usage

Custom architecture — load with trust_remote_code=True (the modeling code ships in this repo via auto_map):

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Quazim0t0/Escarda-86M-Base", trust_remote_code=True)

The tokenizer is the custom SpikeTokenizer (tokenizer.json, algorithm: length-max); load it with the spike_tokenizer.py helper from the project rather than AutoTokenizer.

Acknowledgements

Built with Modal credits during the Small Models, Big Adventures Hackathon, and released to the community as a base to build on.

Citation

If you use this model, please cite:

@misc{escarda86mbase,
  title        = {Escarda-86M-Base: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Escarda-86M-Base}},
  note         = {Quazim0t0/Escarda-86M-Base}
}

Escarda vs Byrne — vision family comparison

The Byrne family uses HRM refinement. Escarda = Byrne + JEPA (Joint-Embedding Predictive head) added alongside HRM in both the vision encoder and the LM trunk — auxiliary only, zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

	Byrne-VE	Escarda-VE
Params	39.34M	39.60M (+JEPA head)
CLS cosine	0.776	0.771
PATCH cosine	0.600	0.584
JEPA self-consistency	—	0.040

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags; Byrne-Docling is marginally more complete on the hardest samples (closes </formula>, includes the <code> wrapper), consistent with its slightly higher teacher-alignment. Escarda-Docling is structurally on par and adds the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1–3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M

Update: engram repair (behavior-preserving)

The n-gram Engram memory in the original weights was degenerate: with the frozen LSH compressor at init scale, every token hashed to bucket 0, so only one table row ever received gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.

Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram's hash now spreads across the full table and every bucket is independently trainable — so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.