Axiom-1.1-Chat (65M)
An experimental, attention-free 65M-parameter language model. Research artifact, not a product.
Axiom-1.1 is the second language model built on AXIOM, an experimental architecture by Lee Madover. It has the same architecture and size as Axiom-1-Chat, retrained from scratch with twice the pretraining data and chat data that teaches it to say when it doesn't know something. It was trained on a single consumer GPU (RTX 5070).
Compared with Axiom-1, it declines most questions about things that don't exist instead of inventing an answer, gets more easy facts right, and decodes faster. It is still wrong on most factual questions. Measured results are below; read the limitations before using it.
Code, design notes and every experiment: GitHub: leemadov/axiom
Quick start
git clone https://github.com/leemadov/axiom && cd axiom
pip install -r requirements.txt
import torch
from axiom.hub import load
from axiom.decoding import RECOMMENDED, default_exempt, generate
model, tok, meta = load("Atlas-AI-research/Axiom-1.1-Chat", device="cuda" if torch.cuda.is_available() else "cpu")
s = {k: tok.token_to_id(k) for k in ("<|endoftext|>", "<|user|>", "<|assistant|>", "<|end|>")}
nl = tok.encode("\n").ids
prompt = [s["<|endoftext|>"], s["<|user|>"]] + nl + tok.encode("What is the capital of France?").ids + [s["<|end|>"]] + nl + [s["<|assistant|>"]] + nl
reply = [t for t in generate(model, prompt, RECOMMENDED, stop=(s["<|end|>"],), exempt=default_exempt(tok)) if t != s["<|end|>"]]
print(tok.decode(reply)) # The capital of France is Paris.
There is also an interactive chat: python chat.py --model <local export folder>. The model needs this repository's
code; it is not a transformers architecture and has no trust_remote_code loader.
Prompt format. Every conversation starts with <|endoftext|>, then one block per turn:
<|user|>\nβ¦<|end|>\n, then <|assistant|>\n. An optional system turn uses <|system|>.
Model details
| Parameters | 64.7M, all used for every token |
| Layers | 12 Γ (working memory + Law-Cell bank), width 640, 10 memory heads |
| Working memory | gated linear recurrence with one decay per head. O(1) state per layer while generating; no attention, no KV cache |
| Law Cells | 2 per layer (640 β 1280 β 640), top-2 routing, so every token uses both cells. Fission was off for this run |
| Tokenizer | byte-level BPE, 16,384 tokens (the same tokenizer as Axiom-1) |
| Context | trained on 1,024-token windows |
| Precision | weights in fp32 (bf16 autocast during training) |
The config carries "variant": {"dense_cells": true}. That only selects a faster training kernel that computes the same
function; the stock axiom.lm.LM loads and runs these weights as they are.
What changed from Axiom-1
- Twice the pretraining data: 6.3B tokens from all 10 shards of FineWeb-Edu
sample-10BT, instead of 3.15B from 4. - No fission and no reserved expert slots. All 9 fission trials were rejected in Axiom-1, so the mechanism was switched off.
- A softmax-attention hybrid (attention in 3 of the 12 layers) was tested first and did not win by the required margin, so 1.1 stays attention-free.
- Reshaped chat data, described below.
Training
| Stage | Data | Tokens | Steps |
|---|---|---|---|
| Pretraining | FineWeb-Edu sample-10BT, all 10 shards |
6.3B | 6,000 Γ 1M-token batches |
| Chat tuning | smol-smoltalk + everyday-conversations + identity conversations, reshaped (below), + 12.5% FineWeb replay | 262M | 2,000 |
- Hardware: one NVIDIA RTX 5070 (12 GB), about 78k tokens/s with
torch.compile, about 25 GPU-hours in total. - Pretraining: AdamW (Ξ² 0.9/0.95), peak LR 1e-3, cosine to 1e-4, weight decay 0.1.
- Chat tuning: LR 3e-4 β 3e-5. Loss only on assistant tokens. Windows aligned to conversation starts.
- Chat data changes from Axiom-1:
- more weight on everyday conversations and on short answers;
- conversations with code dropped;
- a quarter of the short user turns retyped casually (lower case, no apostrophes, sometimes a typo);
- a small share of synthetic one-turn chats added. Half ask about invented people, places and things, answered by saying it doesn't know them. The other half ask real questions of the same kinds, answered in one correct sentence, so it doesn't learn to refuse every question with a name in it. None of these overlap the evaluation questions.
Evaluation
All numbers are measured, not estimated, with the same prompts, seeds and decoding for both models. Scripts and raw outputs are in the GitHub repository.
| Measure | Better is | Axiom-1-Chat | Axiom-1.1-Chat |
|---|---|---|---|
| Held-out FineWeb-Edu loss, base model (64 Γ 1,024-token windows) | β | 3.277 | 3.130 |
| Chat validation loss (assistant tokens) | β | 1.627 | 1.685 |
| Easy facts answered correctly, hand-audited (of 50) | β | 8 | 15 |
| Questions about invented things declined (of 50) | β | 0 | 29 |
| Easy facts with one typo, automatic grading (of 50) | β | 10 | 9 |
| Coherent and on topic, judged by a local Qwen3.5-0.8B (of 60) | β | 32 | 35 |
| Knows its identity (4 phrasings Γ 2) | β | 75% | 75% |
| Generation speed, RTX 5070, batch 1, fp32 | β | 54.1 tok/s | 66.7 tok/s |
| Peak GPU memory in that speed test | β | 758 MiB | 295 MiB |
How to read this:
- The clearest gain is declining to answer about things that don't exist: 29 of 50 instead of 0.
- Easy-fact accuracy rose from 16% to 30%. A paired bootstrap over these 50 questions puts the gain at 14 points with a 95% interval of 0 to 28 points, so it is real but imprecise.
- The judge difference (32 vs 35 of 60) is small and not a robust quality improvement.
- Chat validation loss got worse and typo robustness did not improve.
- Several things changed at once (data size, fission, chat data), so these results don't show which change caused which gain.
- This is a small development suite that was used during the work, not a new blind test.
Limitations and risks
- It still makes things up. It gets 70% of easy facts wrong; it fails "how many days are in a week" and "what is 2 plus 2". Never use its output as fact.
- Abstention is partial and imperfect. It still invents answers about about 4 in 10 made-up subjects, and a misspelled real question can look made-up to it, so it sometimes refuses questions it could answer.
- Says it is "Axiom-1". Its identity answer names the model Axiom-1 and usually does not mention who trained it.
- No code, maths or reasoning ability worth the name.
- English only. Trained on web text, so it can reflect the biases and errors of that text. No safety tuning beyond what is in the chat datasets.
Intended use: research into the AXIOM architecture and small-scale training: architecture comparisons, ablations, abstention training, interpretability, teaching. Not intended for factual question answering, advice, or any user-facing product.
Data and attribution
- FineWeb-Edu by Hugging Face, ODC-By 1.0
- smol-smoltalk by Hugging Face, Apache-2.0
- everyday-conversations-llama3.1-2k by Hugging Face, Apache-2.0
- The identity, abstention and short-fact conversations were written for this model.
Citation
@misc{madover2026axiom11,
title = {Axiom-1.1: an experimental attention-free language model with Law Cells},
author = {Madover, Lee},
year = {2026},
url = {https://github.com/leemadov/axiom}
}
- Downloads last month
- -