Wanyun/hola-recency-340m

HOLA+recency 340M

This is the matched recency-control checkpoint used in the paper. For generation-style evaluations, set FLA_FORCE_NO_CACHE=1 to force full recomputation.

Model details

• Architecture: Gated DeltaNet with HOLA bounded exact cache
• Scale: 340M parameters
• Training data: SlimPajama-15B
• Tokenizer: Mistral-7B-v0.1 tokenizer, vocab size 32000
• Training context: 2048 tokens
• Cache: w=64, chunk=256, RMSNorm-gamma cache read
• Eviction: recency (Matched HOLA cache with recency eviction)

Usage

Install the HOLA code package first, then load with Hugging Face Auto classes:

import torch
import fla  # registers gated_deltanet
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "YOUR_ORG/hola-recency-340m"
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    path, torch_dtype=torch.bfloat16, device_map={"": "cuda"}
).eval()

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

For the recency-control model, generation-style evaluations should set:

export FLA_FORCE_NO_CACHE=1

Paper evaluation snapshot

Metric	Value
Wikitext-103 PPL	25.04
LAMBADA PPL	32.33
Six-task commonsense average	43.17
FDA retrieval	16.9
SWDE retrieval	29.9
RULER S-NIAH-1 at 32k	0.24

Citation

@article{hola2026,
  title = {HOLA: Hybrid Optimization with a Learned Attention Cache},
  author = {TODO},
  journal = {arXiv preprint},
  year = {2026}
}