LCGA v3 — learnable block-sparse attention on a frozen Qwen2.5-0.5B

This is a research checkpoint, not a chat model. It contains the LCGA router + LoRA weights for the frozen Qwen/Qwen2.5-0.5B base; the base weights are not included. The research code, full reports and raw evidence (results/results.jsonl) live in the archive byjameson/lcga-local; the streaming trainer (32B on a 6 GB GPU) is byjameson/stream-finetune.

It is not a standalone transformers checkpoint: a generic HF benchmark loader will not assemble the frozen base + router + LoRA. Use the LCGA code/report to run it.

Demos from the same project: 130 KB AI — 500 labelled examples, Complement NB fitted in the browser, 78.3%; 1.7 MB accuracy mode — a stored linear model on 75k human labels, 88.5%.

What it is

LCGA replaces dense O(N²) attention with a learned block-sparse path:

a lightweight latent concept graph router (lcsa/concepts.py) scores the sequence with K=64 learnable concept embeddings;
concept probabilities are aggregated into per-block scores, and the top-M blocks per layer are selected (M = round(rho * n_blocks), default rho=0.25);
exact softmax attention is then computed over only the selected blocks' K/V (no approximation inside the attention itself);
slot-level causality (kpos <= qpos) is the reference semantics in every path.

The base Qwen weights stay frozen. v1 trains only the router; v2/v3 add LoRA and train with LM + KL + MSE + recall-gain losses. v3 differs from v2 only by the LoRA learning rate (1e-3 -> 1e-4), which largely fixed out-of-domain forgetting.

Training

Item	Value
Base	`Qwen/Qwen2.5-0.5B` (frozen, fp16)
Data	wikitext-2, T=1024
v3 stage	router + LoRA, `lr_lora=1e-4`, 1250 steps
Checkpoint	`v3_best.pt` (19.1 MB): per-layer router + LoRA state
Losses	LM + KL + MSE + recall-gain

Measured results

From results/results.jsonl and RAPOR.md (single 6 GB laptop GPU, batch=1):

Axis	Result
`wt2_test`, T=1024, rho=0.25, 128 chunks	dense 12.46 / sparse 16.56 PPL — best of 4 trained models
`TinyStories` valid (OOD)	dense 6.09 / sparse 9.16 PPL
v1 / v2 / v2ctl (`wt2_test`)	14.30/21.84, 12.88/16.73, 12.80/17.26
Prefill speed	crossover ~5k tokens; 1.25× sparse-vs-dense @16k (A2 CUDA-graph router)
Decode (whole-step CUDA graph)	dense bitwise parity; graph-sparse 1.12–1.27× faster than graph-dense @4k–8k
Attribution (v2)	~83% generic LoRA adaptation, ~17% sparsity-specific (dense_control ablation)

Honest limits

Scale: 0.5B, batch=1, one 6 GB laptop GPU. No SOTA or "dense-quality at a fraction of the cost" claim; "quality preserved" means the measured wikitext-2 gap plus one OOD probe, not lossless attention.
OOD: a single TinyStories probe. v3 largely recovers v1-level OOD, but "never forgets" is not claimed.
This is a language-modeling checkpoint, not an instruction/chat model.

How to use

The checkpoint requires the custom lcsa package from the GitHub repository. experiments/exp3_eval.py contains the exact load_ckpt helper; the sketch is:

import torch
from lcsa.harness import load_model
from lcsa.model import patch_model, set_sparse_enabled
from lcsa.lora import apply_lora

model = load_model("models/Qwen2.5-0.5B", device="cuda")
ckpt = torch.load("v3_best.pt", map_location="cpu", weights_only=False)
margs = ckpt["meta"]["args"]
wrappers = patch_model(
    model,
    budget=margs.get("budget", 0.25),
    block=64,
    n_concepts=margs.get("n_concepts", 64),
    graph_rank=margs.get("graph_rank", 16),
    rel_bias=not margs.get("no_rel_bias", False),
    use_coarse=False,
    null_init=-1e9,
    recall_gain=True,
)
apply_lora(model, r_attn=margs.get("lora_attn", 16), r_mlp=margs.get("lora_mlp", 0))
# then copy ckpt["state"][i] into each attention layer, exactly as in
# experiments/exp3_eval.py:load_ckpt()
set_sparse_enabled(True)

The base model Qwen/Qwen2.5-0.5B is Apache-2.0. This derivative checkpoint is released under Apache-2.0; see the base model card for its terms.

Latest findings (25 Oct 2026) — corrected, extended, and a working recipe

Correction first. An earlier note said the router selects 0.0% content blocks and is identical to a budget-matched static window. Both numbers came from evaluation scripts that loaded the checkpoint before patching the model, so an uninitialised router was being measured. The scripts are fixed and every number below is re-measured. (The older exp3 PPL table — 14.30 dense / 21.84 learned — was never affected.)

1. On plain language modelling the negative result stands. At a 25% block budget the learned router is a position prior: content blocks are 1.7 / 6.0 / 7.0 / 6.7% of its selections at 1k / 4k / 8k / 16k, and perplexity is 25.166 / 13.788 / 12.176 / 12.001 versus a budget-matched static window's 25.051 / 13.783 / 12.160 / 11.868 — the learning adds nothing (it is 0.005–0.133 PPL worse). What helps is the budget: the gap to dense falls from +45% (1k) to +11% (16k) at a fixed 25% of blocks. On WikiText the dense teacher itself puts almost no mass on distant blocks, so there is nothing to distil — the degeneration is correct behaviour, not a training failure.

2. Retrieval is the regime where content matters. Needle at 8k, 25% budget: dense answers exactly (exact 1.00, CE 0.14–0.39), the budget-matched window cannot (exact 0.00, CE ≈ 3.0). Perplexity can never show this.

3. The router can learn it — and the three ingredients are separable:

ingredient	good	bad	needle exact
where the loss is applied	answer positions	all query rows	1.00 vs 0.75 (8k), 1.00 vs 0.00 (1k)
what the target is	teacher's content argmax	full attention distribution	works vs 0.11 p(fact), fails
router head	+ bilinear content path	concept graph only	works vs p=0.066, fails

The full distribution fails because ~50% of the teacher's mass at the answer rows sits on the sink block and ~20% on the query block; the content signal is a few percent of the gradient. Restricting the KL to content blocks helps at 8k but not at 1k (the teacher's content mass is still diffuse). Focusing on the teacher's content argmax works at both, and needs no needle labels.

4. Label-free result, 3 seeds × 6 prompts (trained at T=1024, evaluated at 1k/8k):

cell	dense	label-free router	budget-matched window
1k, depth 0.5 (out of window)	1.00	1.00 (1.00–1.00)	0.00
8k, depth 0.1 (out of window)	1.00	1.00 (1.00–1.00)	0.00
8k, depth 0.5 (out of window)	1.00	1.00 (1.00–1.00)	0.00
1k, depth 0.9 (in window)	1.00	0.39 (0.00–1.00)	1.00
8k, depth 0.9 (in window)	1.00	1.00	1.00

Selection: 24/24 layers pick the fact block at the query row (17/24 on a second probe); content blocks rise to 19.5 / 61.7 / 71.6% (1k/4k/8k); greedy generation with full re-prefill produces the same answer as dense (7311). At 8k the budget can be cut to 6.25% of blocks (8 of 128) and retrieval is still exact.

5. Controls and mechanism. Shuffled-label training learns nothing (p(fact) 0.067 = chance). The teacher's own per-instance content argmax points at the fact in only 15.3% of (layer, sample) pairs, yet the student selects it in 71% of layers: a shared router averages out per-instance teacher noise and learns the consistent part (the fact sentence). The softmax temperature drifts under a hard argmax target, so p(fact) flattens after training while the ranking (what top-k uses) stays correct — selection rate and exact match are the metrics to trust.

6. One router for both behaviours (the practical result). Mixing one needle step per seven LM steps (mix7, 900 steps) keeps retrieval where the window fails and keeps plain-LM quality:

schedule	1k d0.5	1k d0.9	8k d0.1	8k d0.5	PPL 1k	PPL 4k
needle only	1.00	0.17	1.00	1.00	27.70	18.22
1 needle : 2 LM	0.83	0.17	1.00	1.00	25.87	14.79
1 needle : 7 LM	1.00	1.00	1.00	1.00	25.66	14.15
static window (control)	0.00	1.00	0.00	0.00	25.05	13.78
dense	1.00	1.00	1.00	1.00	17.32	10.95

Seeds 0/1/2 give 15/15 cells at 1.00 (PPL 25.65–25.68 across seeds; 12.46 at 8k vs the window's 12.16). +0.60 PPL at 1k buys exact retrieval on every out-of-window cell at a 25% (or 6.25%) compute budget. Warm-starting the LM router on needles instead does not work: p(fact) 0.28, PPL 27.0.

Adding multi-fact + paraphrase training on top (mix7_nf2) keeps 5/5 cells at 1.00 and PPL 25.648, and raises the held-out-phrasing and two-fact cells at 2k depth 0.1 to 1.00 (three facts 0.70). That is the better general-purpose configuration.

7. Limits, stated plainly.

Transfer stops at 8k: 2k fails (exact 0.00 even when trained at 2k; the sparse path itself is bitwise-faithful at 2k, so this is a router effect we did not explain), 16k fails, and at 4k the 0.5B dense model itself fails.
A re-phrased fact scores exact 0.00 with the single-phrase model. Multi-fact + paraphrase training (mix7_nf2) reaches 1.00 on the held-out phrasing and two facts at 2k depth 0.1 (three facts 0.70), and keeps 5/5 cells + PPL 25.65.
The router selects fact-like content, not the block that answers the question: with two facts both are selected regardless of the query; making the budget bind (three facts), using labels, and switching to last-token query pooling all leave it non-selective. The cause is upstream of the router's objective.
Scale is partly demonstrated (new): the label-free recipe replicates on Qwen2.5-3B — dense-level exact match at 1k with a quarter of the training tokens the 0.5B needed. The 1.5B host is unusable (its dense attention does not find the needle at 1k at all). The full ablation grid remains 0.5B-only.
No KV-memory saving — compute/bandwidth only, so the context that fits on a 6 GB card does not grow.
One base model (0.5B), one task family (4-digit needle); all needle numbers are teacher-forced scoring of the answer span plus one greedy-generation check.

8. Speed is unchanged. Prefill 1.71× at 32k; graph decode 1.27× at 8k; eager decode loses; no KV-memory saving.

Trained routers in this repo (router-only, 22 MB each, fp32): router_needle_mix7.pt (single-task, +0.60 PPL) and router_needle_mix7_nf2.pt (paraphrase + multi-fact). Load them on the frozen base with patch_model(..., budget=0.25, n_concepts=64, graph_rank=0, content_rank=64, rel_bias=True, capture_dense=False, use_coarse=False, null_init=-1e9) and copy each ckpt["state"][i] key into model.model.layers[i].self_attn (see experiments/exp61_eval_check.py). The frozen base weights are not included.

Also in the same repository (a different line of work): long-context training on the same 6 GB card — a 32B nf4 base streamed from disk trains at 1024 tokens/step, 36.7 tok/s, 4.61 GB peak (3.1× the seq-128 throughput, 8× the context). 2k does not fit: 7.93 GB with the recompute cache on the GPU, 6.60 GB with it offloaded to host RAM (exactly the 1.33 GB cache) — the remaining ~0.6 GB is the bf16 lm_head, and attention is not the cause (a 40-head bf16 SDPA at 2048 allocates 42 MB). At a matched token budget, training at 2k is even slightly worse on 2k text than training at 512 (1.3033 vs 1.2833 CE), so the lever is data volume, not window length. See LONGCTX_FINDINGS.md and the practical write-up with the traps and one-command presets: GUIDE_6GB_32B.md.

Full tables, ablations, controls, limits and reproduction commands: LCGA_FINDINGS.md. The licence, weights and donation section above are unchanged.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for byjameson/lcga-v3-0.5b

Base model

Qwen/Qwen2.5-0.5B

Finetuned

(769)

this model

byjameson
/

lcga-v3-0.5b