FlyinGodzilla/AMOR-GatedDeltaNet-1.5B

Paper · Code · Files · All models

Block 0 receives the backbone's final normalized output. Blocks 1 and 2 apply their own pre-normalization. A shared, embedding-tied LM head supplies all three entropy evaluations and the final output prediction.

A complete recurrent backbone, refined by three entropy-gated attention blocks.

For a smaller download, see the 180M and 440M models below.

The idea

AMOR (Adaptive Metacognitive Output Router) uses the model's predictive uncertainty to decide where attention should refine its recurrent representation. Each appended block reads normalized output entropy, compares it with a frozen threshold at inference, and adds an attention update only where its gate fires.

The recurrent backbone and attention blocks are pretrained jointly from scratch. “Post-hoc” describes where the attention blocks sit, after the complete backbone. Keys and values come from residual-stream projections and are retained at every position for future retrieval.

This model uses the native entropy gate. Optional distilled deployment routers are not included in this release. Training uses dense attention with an output mask, so gating does not reduce training attention FLOPs.

This model

Property	Value
Backbone	Gated DeltaNet
Parameters	1,524,025,472
Recurrent mixer–MLP layers	24
Model width	2048
Appended attention blocks	3
Training corpus	FineWeb-Edu
Training tokens	30,748,520,448
Training context	3,072 tokens
Final training step	62,558
Seed	42
Tokenizer vocabulary	128,256
Model type	Pretrained base model; not instruction-tuned
Weight format	Safetensors; original weight precision

Detailed settings are in training_config.json; training_summary.json records the completed run.

Results

This model's canonical eight-task common-sense mean is 45.1%. The score uses raw accuracy on LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA and TruthfulQA-mc2.

Task	Accuracy (%)
LAMBADA	39.1
HellaSwag	39.2
PIQA	70.6
ARC-Easy	66.4
ARC-Challenge	30.5
WinoGrande	53.7
OpenBookQA	24.8
TruthfulQA-mc2	36.9
Eight-task mean	45.1

Per-task entries are point estimates; the figure reports evaluation-repeat spread.

Research comparison across all three scales; use the 1.5B panel for this model. Whiskers show the sample SD of five reevaluations of each fixed model, centered on its canonical score. They measure evaluation repeatability, not independent-training variation. Axes are zoomed per panel.

Retrieval and long context

The 1.5B AMOR variants improve average retrieval over their recurrent backbones; serial hybrids retain stronger six-task retrieval averages. LongBench results remain close to the recurrent backbones, with lower scores for the serial hybrids in this experimental setting.

Left: six-task retrieval mean. Right: averages of three single-needle variants at each context length; lengths are not pooled.

Unweighted mean of 14 task-native scores. Whiskers are benchmark standard errors propagated under task independence, not training-seed variability.

Per-task retrieval and LongBench tables

Task	Answer-containing outputs (%)
SWDE	47.4
SQuAD-completion	36.8
FDA	24.9
TriviaQA	43.1
Natural Questions	10.5
DROP	17.4
Six-task mean	30.0

Retrieval outputs do not store standard errors.

Task	Score ± benchmark SE
NarrativeQA	1.7 ± 0.17
Qasper	4.0 ± 0.35
MultiFieldQA-en	11.4 ± 0.87
HotpotQA	3.6 ± 0.49
2WikiMultihopQA	7.0 ± 0.74
MuSiQue	1.9 ± 0.32
GovReport	8.1 ± 0.30
QMSum	15.1 ± 0.50
MultiNews	10.8 ± 0.43
TREC	11.0 ± 2.22
TriviaQA	11.8 ± 1.22
SAMSum	11.4 ± 0.68
LCC	10.1 ± 0.38
RepoBench-P	10.3 ± 0.41
14-task mean	8.4 ± 0.22

Scores are on a 0–100 scale. The mean SE is propagated under task independence.

Per-task values are also available in benchmark_results.json.

Download and run

These are custom PyTorch models. Use the supplied loader and generation script; this repository is not a Transformers AutoModel.from_pretrained integration, a hosted chat endpoint, or a GGUF/PocketPal package.

A Linux machine with a compatible NVIDIA GPU is the intended accelerated setup. Install a CUDA-enabled PyTorch build appropriate for your driver first (PyTorch installation). Then, in a fresh Python environment:

python -m pip install huggingface_hub
hf download FlyinGodzilla/AMOR-GatedDeltaNet-1.5B --local-dir AMOR-GatedDeltaNet-1.5B
cd AMOR-GatedDeltaNet-1.5B
python -m pip install "transformers==5.0.0" safetensors einops
python -m pip install flash-linear-attention
python generate.py --prompt "The Eiffel Tower is" --max-new-tokens 32

For the tested CPU reference path, use PyTorch 2.6.0 and Transformers 4.51.3. Mamba2 has a slow CPU fallback (--device cpu); Gated DeltaNet requires its compatible accelerated kernels. GPU kernel versions must match the PyTorch/CUDA environment. CUDA smoke tests used PyTorch 2.6.0+cu124, Transformers 5.0.0, Triton 3.2.0 and safetensors 0.7.0. Gated DeltaNet kernels: the imported fla module reported version 0.4.2. These are actual imported versions from the tested environment; the installation commands above have not been checked in a fresh environment.

generate.py performs cached greedy text completion. It is intentionally a plain-prompt example: these models were not trained as chat assistants. The tokenizer is referenced from teknium/Llama-3.1-AlternateTokenizer and pinned in config.json to the release-packaging revision. The original training-time tokenizer revision was not recorded.

Verification

CUDA strict loading, finite forward/prefill, cached decoding and the published generation command passed on an NVIDIA H100 NVL. The short cached/full-forward check had a maximum absolute logit difference of 0.0625 (rtol 0.02, atol 0.1). These are smoke tests, not a benchmark rerun.

Every exported tensor was compared exactly with the original trained weights, including the entropy gate's frozen buffers. The tied embedding/head matrix is stored once and restored by the loader. Optimizer state is excluded. Export metadata records provenance and actual validation; manifest.json lists file hashes.

Model family

Backbone	180M	440M	1.5B
Mamba2	180M	440M	1.5B
Gated DeltaNet	180M	440M	1.5B

All six models use the three-block canonical AMOR layout. Choose a backbone and scale to match your experiment; larger models require more memory. The cards use a snapshot of the research figures and do not automatically track later manuscript revisions.

Intended use and limitations

Research on recurrent–attention hybrids, uncertainty-based routing, and pretrained language-model behavior. Outputs can be incorrect and may inherit training-data biases. Entropy measures predictive uncertainty; it cannot detect every confidently wrong prediction. Inference fire rates depend on the workload, and attention KV storage still grows with context. Training context is 3,072 tokens; long-context experiments do not establish reliable performance on every domain.

License

The code and model weights are released under the MIT license; see LICENSE.

Citation

@article{zheng2026amor,
  title={When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models},
  author={Zheng, Haoran and Shani, Chen},
  journal={arXiv preprint arXiv:2602.13215},
  year={2026}
}