LFM2.5-350M-HSSA

An experimental variant of LiquidAI/LFM2.5-350M-Base in which the six full-attention (GQA) layers are replaced with Hierarchical Searchable Sparse Attention (HSSA), then continued-pretrained so the HSSA routers learn content-dependent block selection on real data.

The convolutional (LIV) blocks of LFM2 are left untouched. Only the attention layers [2, 5, 8, 10, 12, 14] are converted. The donor Q/K/V/O projections and per-head Q/K norms are preserved; HSSA adds two small router projections (router_q_proj, router_k_proj, dim 128) per converted layer.

• Base model: LiquidAI/LFM2.5-350M-Base
• Parameters: ~356M (+ ~1.6M router parameters)
• Attention: HSSA on the 6 attention layers — O(n log n) routed sparse attention: local window (256) + top-8 routed blocks of 64 tokens, selected by beam search over a balanced router tree
• Precision: bf16
• Continued pretraining: 2048-token windows, LR 2e-5, router supervision loss weight 0.1
• Intended use: research on sparse attention / edge text generation experiments

GGUF builds (llama.cpp / Ollama / LM Studio)

Quantized GGUF builds are provided. Important: GGUF builds run with dense attention — the HSSA router tensors are stripped during conversion, because llama.cpp has no HSSA kernel. All fine-tuned backbone weights are identical to this checkpoint, so outputs are close to (but not exactly) the sparse-attention model.

file	quant	size
model-hssa-Q8_0.gguf	Q8_0	379 MB
model-hssa-Q6_K.gguf	Q6_K	293 MB
model-hssa-Q4_K_M.gguf	Q4_K_M	229 MB

llama-cli -m model-hssa-Q8_0.gguf -p "What is C. elegans?" -n 128

Q8_0 is recommended; at this model size Q4_K_M shows visible quality loss.

Using with transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DarkKnighToS223/lfm2.5-350m-hssa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")

Note: stock transformers loads the fine-tuned backbone with standard dense attention; the HSSA router tensors in the checkpoint are ignored (same behaviour as the GGUF builds).

The HSSA configuration is stored in hssa_conversion.json:

knob	value
block_size	64
query_group_size	16
local_window	256
top_blocks	8
router_dim	128
router_beam_width	16

Evaluation

Synthetic long-context retrieval suite (RULER-style; needle-in-a-haystack, multi-needle, associative recall, copy, variable tracing), 75 fixed prompts, answer_recall scoring. Contexts of 512/1024 tokens are within the CPT training window; 2048-token prompts fall outside it.

Continued pretraining lifts overall recall from 0.03 (freshly converted, random routers) to 0.27.

answer_recall	LFM2.5-350M-HSSA
overall (512+1024)	0.27
niah	0.70
multi_needle	0.63
associative_recall	0.00
copy	0.00
variable_trace	0.00

Out-of-window contexts (2048-token targets, ~3.4k actual tokens): recall 0.08 — degraded but non-zero, suggesting longer-window CPT would help. A dense baseline of the same family scores higher on this suite; that gap is the current cost of O(n log n) sparse attention after a short CPT run.

Training context

Continued pretraining was short (a few thousand steps on 2048-token windows) — enough for the HSSA routers to learn sensible block selection after conversion. This is a research checkpoint, not a production-tuned model. Expect quality below the original LFM2.5-350M on knowledge-heavy tasks; this variant exists to study HSSA on a real, pretrained backbone.

Limitations

• GGUF builds and stock transformers run with dense attention; the HSSA routers are exercised only by the research training/eval framework.
• Trained on 2048-token windows; retrieval degrades beyond that range.
• Base-model knowledge cutoff and biases are inherited; factual accuracy is limited at this scale.

Acknowledgements

Built on LiquidAI's LFM2.5-350M-Base. HSSA conversion, continued pretraining