Quazim0t0/Byrne-VLM-131M

Byrne-VLM (131M)

Model will be ungated for open download once I am done with the base..

Byrne-VLM is a compact (~130.9M-param) vision-language model built entirely from the SpikeWhale / Byrne family: a from-scratch 39M vision encoder, a lightweight connector, and the ~90M Byrne language model — wired LLaVA-style and trained by distillation. It is a research / architecture artifact, not a SOTA captioner (see Honest evaluation below).

image → Byrne-VE (39.3M, frozen) → 196 patch tokens (512-d)
      → Connector MLP (512→640, 1.18M)
      → spliced at <image> placeholders into Byrne-LM (90M + Family-LoRA)
      → caption

Components

part	params	what it is
Byrne-VE (vision)	39.34M	ViT-style encoder — RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM refinement. 224px / patch16 / 196 tokens / dim512 / depth12. Distilled from DINOv2-base, then improved teacher-free with DINO-style self-distillation.
Connector	1.18M	2-layer MLP (512→1024→640) projecting patch tokens into the LM space.
Byrne-LM	~90M (+Family-LoRA)	SpikeWhale LM — MLA attention, DERF, XSA, Engram n-gram, hyper-connections, MoE, MTP, HRM refine, QK-Norm, partial RoPE. Custom SpikeTokenizer (vocab 16512), 4096 context.
Family-LoRA	5.04M	Custom adapter whose bottleneck is a full family block: HRM iterative gated refinement + MoE-SwiGLU (shared+routed experts, sqrtsoftplus routing). r=16, zero-init → exact no-op at start, preserving the LM.
Total	130.9M

How it was trained

1 · Vision encoder (Byrne-VE). Distilled from a frozen facebook/dinov2-base teacher (cosine on CLS + patch grid), then improved teacher-free with DINO-style EMA self-distillation. Reaches ~88% of DINOv2's k-NN accuracy at ~45% of its params; the HRM gate is alive (tanh ≈ 0.35).

2 · Connector grounding (10 streamed rounds). The connector was trained to map vision features into the LM, streaming image–caption pairs live from a broad mix: CC3M, CC12M, Conceptual-Captions-12M, LLaVA-ReCap-558K/118K/CC3M, TextCaps, Flickr8k, LLaVA-NeXT-Data. (Vision encoder + LM frozen throughout.)

3 · Style fine-tune. Connector fine-tuned on ~25k images captioned by HuggingFaceTB/SmolVLM-256M-Instruct — sequence-level distillation (the teacher and the SpikeTokenizer have different vocabularies, so we distill the generated caption text, not logits).

4 · Stage-2 Family-LoRA. A Family-LoRA (HRM + MoE-SwiGLU adapter) is added to the LM decoder and trained with the connector. Base LM weights stay frozen; the zero-init adapter wakes during training (gates climb off zero, MoE routing learns, no collapse).

Honest evaluation

Byrne-VLM is a demonstration of the SpikeWhale family as a VLM, not a competitive captioner. On 500 COCO-val images it scores CIDEr ≈ 0.06 / BLEU-4 ≈ 4.8 — far below usable captioners (its own teacher, SmolVLM-256M, is much stronger). It reliably gets coarse scene gist and is correct on clear subjects (e.g. lion, tower) but cannot produce specific, reference-grade captions. This is a capacity-and-scale limit (90M from-scratch LM, 39M encoder, ~25k caption pairs), not a wiring bug. Example outputs:

image	Byrne-VLM
lion	"a lion on the ground. In the background there are trees."
tower	"a tower. In the background there is a sky."
cheetah	"a tiger in the water." (right family)

Usage

import torch
from generate import load_vlm, caption
from spike_tokenizer import SpikeTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
tok = SpikeTokenizer(vocab_file="tokenizer.json")
vlm = load_vlm("weights/byrne_vlm.pt", "lm", "weights/byrne_ve.pt", device)
print(caption(vlm, tok, "photo.jpg", device))

AnyRes tiling (higher effective resolution for dense images / documents) is supported via load_vlm(..., anyres_grid=(2,2)) — fits the 4096 context (980 image tokens) and needs a matching fine-tune.

Files

weights/byrne_vlm.pt (connector + Family-LoRA) · weights/byrne_ve.pt (vision encoder) · lm/ (Byrne base LM) · model code (vlm_model.py, modeling_byrne_embed.py, family_lora.py, anyres.py, …).

Citation

@misc{byrne_vlm_2026,
  title  = {Byrne-VLM: A Tiny From-Scratch SpikeWhale Vision-Language Model},
  author = {Quazim0t0},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-VLM-131M}}
}

License

Apache-2.0.

Escarda vs Byrne — vision family comparison

The Byrne family uses HRM refinement. Escarda = Byrne + JEPA (Joint-Embedding Predictive head) added alongside HRM in both the vision encoder and the LM trunk — auxiliary only, zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

	Byrne-VE	Escarda-VE
Params	39.34M	39.60M (+JEPA head)
CLS cosine	0.776	0.771
PATCH cosine	0.600	0.584
JEPA self-consistency	—	0.040

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags; Byrne-Docling is marginally more complete on the hardest samples (closes </formula>, includes the <code> wrapper), consistent with its slightly higher teacher-alignment. Escarda-Docling is structurally on par and adds the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1–3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M