spotlight-memory-synth-1b

Independent implementation of Spotlight Memory (Percepta Research, 2026-10-02): a decoder-only LM whose sequence mixer is 2D-lattice attention with per-cell DeltaNet states — constant per-token cost at any context length, addressing learned, not retrieved. Written and trained from scratch by Vibelet; not a Percepta release.

Model

• 48,964,160 params (~50M). tie_embeddings=True shares the LM head with the embedding matrix, so safetensors carries emb.weight only — with a freshly constructed SpotlightLM, load_state_dict(..., strict=False) restores the tie.
• n_embd 512, n_layer 8, n_head 8, d_k = d_v 16, lattice 32×32, chunk 128, ffn_hidden 1536, bump_radius 1.5, kernel_floor 0.7, SwiGLU head.
• Trained on the SYNTH corpus (~1.58B tokens in total: 51,250 steps @ ctx 8192, then 70,820 steps @ ctx 16,384; step 122070 is schedule end). Base val loss 2.7516.
• Annealed release (this checkpoint): +10,000 steps @ ctx 8192, cosine decayed to exactly-zero LR (min_lr_ratio=0), step 132,070, 81.9M tokens revisited. Same-window val loss 2.73971 vs the base 2.75385 (8 identical val windows, −0.0141). Every weight in this repo is now the annealed checkpoint.
• Numerics: fp32 master weights; the fused scan path (details in the code) tracks per-chunk states in fp32 registers.

Usage

import torch
from spotlight.config import SpotlightConfig
from spotlight.model import SpotlightLM
from safetensors.torch import load_file

cfg = SpotlightConfig(vocab_size=50257, n_embd=512, n_layer=8, n_head=8,
                      d_k=16, d_v=16, lattice=32, chunk=128, ffn_hidden=1536,
                      bump_radius=1.5, kernel_floor=0.7, tie_embeddings=True)
model = SpotlightLM(cfg)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()
_, logits = model(torch.tensor([[464, 2415]]), None)   # (" Hello", ...)

CPU out of the box; macOS via the Metal scan shader in spotlight.metal_spotlight; Intel XPU via spotlight.triton_spotlight (SYCL/graph kernels built separately).

Interactive demo

All-in-browser demo (WebGPU + WGSL; ~98 MB fp16 weights download to the client, no server model call):

vibelet-spotlight-demo.hf.space · sources: Vibelet/spotlight-demo

License

MIT.