Ivonar/ivonar-nano

Ivonar Nano

Ivonar Nano is a 349M-parameter language model with natively ternary weights: every large matrix holds only −1, 0 and +1, stored as 2-bit codes with one 16-bit scale per 128 weights. It was trained that way from random initialization on 100 billion tokens, so the ternary weights are what the optimizer produced, not a compressed copy of a larger model. No existing model was used as a starting point, and nothing was quantized after training.

The packed model is 94.6 MB and decodes at 1,078 tokens per second on an RTX 4060 Ti.

ivonar.com · runtime: github.com/LuisOezdem/ivonar-inference

Benchmarks

Zero-shot multiple choice, full validation splits, one H100. Every option is scored by its summed log probability, the rule the published tables below use, so each column means the same thing. Best per row in bold.

	Ivonar Nano	TriLM 390M	FloatLM 390M	QuantLM 4-bit	BitNet b1.58	LLaMA 700M
Parameters	349M	390M	390M	390M	700M	700M
Training tokens	100B	300B	300B	300B	100B	100B
Weights	ternary	ternary	FP16	4-bit	ternary	FP16
ARC-Easy	56.0	48.6	51.0	49.6	51.8	54.7
ARC-Challenge	24.8	21.2	21.3	21.3	21.4	23.0
HellaSwag	32.8	32.0	35.7	35.1	35.1	37.0
PIQA	64.1	65.0	68.4	68.3	68.1	68.9
Winogrande	51.9	52.2	51.8	53.7	55.2	54.8
BoolQ	60.8	55.1	54.7	50.8	58.2	60.0
Average (6)	48.4	45.7	47.2	46.5	48.3	49.7
Format size	95 MB	98 MB	785 MB	220 MB	175 MB	1,400 MB

TriLM, FloatLM and QuantLM are the Spectra suite: one lab, one recipe, one 300B-token run, differing only in how the weights are stored. Nano leads TriLM 390M, the only other natively ternary model of its size, by 2.7 points at a third of its training tokens, and it leads QuantLM 390M 4-bit by 1.9, so training in the format beats quantising into it afterwards. It matches BitNet b1.58 700M with half the parameters and stands level with FloatLM 390M. Only LLaMA 700M is ahead, by 1.3 points at twice the parameters.

Format size is parameters × bit width. TriLM and QuantLM ship unpacked in FP16 at 785 MB, BitNet 700M and LLaMA 700M were never released, and Nano's 95 MB is the measured file. Sources: Spectra (arXiv 2407.12327) tables 6 and 7, BitNet b1.58 (arXiv 2402.17764) table 2, raw-accuracy columns.

The supervised pass keeps what pretraining built: under the project's own scoring rule the eight-task mean is 43.8 against 44.0 for the pretrained checkpoint, a 285-question knowledge probe reads 66.0 against 66.3, and validation loss is 1.148. Standard errors run ±1.5 to ±2.6 per task, so differences under 3 points are noise. Both scoring rules sit in nano_sft_base.csv.

Speed and size

Weights on disk	94,645,329 bytes, 2.17 bits per parameter: 92.4% packed 2-bit codes, 5.8% FP16 scales, 1.8% FP32 norms
RTX 4060 Ti, packed 2-bit kernels	1,078 tokens per second, single stream, median of six runs

The kernels read the packed codes directly and accumulate INT8 activations with integer dot products, one scale multiply per 128 weights. Decode is bound by the 95 MB the weights occupy, not by arithmetic.

Quick start

git clone https://github.com/LuisOezdem/ivonar-inference
cd ivonar-inference
pip install .

Put packed_inference_checkpoint.pt, tokenizer.json and attribution_bundle.md into models/ivonar-nano/, then run ivonar chat for the terminal or ivonar serve for a chat page and an OpenAI-compatible API on http://127.0.0.1:8000/. ivonar verify compares the decoder against the single-precision reference. Python 3.12 or newer; a CUDA build of PyTorch for GPU decoding.

Without a system message the model answers "who are you" with a persona from its training data, since it carries no identity of its own. The runtime ships one by default.

Files

File	Size	Purpose
packed_inference_checkpoint.pt	94,645,329 bytes	the model
tokenizer.json	5.7 MB	65,536-token BPE tokenizer, trained for this model
attribution_bundle.md, dataset_origin_approvals.json		every training source with revision, license and attribution text
nano_sft_base.csv, nano_raw_base.csv, nano_dpo_base.csv		the benchmark rows behind the tables above
SHA256SUMS.txt, LICENSE		checksums; Apache 2.0, copyright 2026 Luis Oezdem

Only the first two are needed to run the model. Keep attribution_bundle.md beside them when you redistribute the weights.

The full training checkpoint, 4.19 GB, sits on the step2100 branch of this repository, for anyone continuing the training:

from huggingface_hub import hf_hub_download
hf_hub_download("Ivonar/ivonar-nano", "checkpoint_step_00002100.pt", revision="step2100")

Architecture

Parameters	349M, all active (dense)
Blocks	24: 18 state-space, 6 attention, one attention block after every three
Width	768 hidden, 12 heads; SwiGLU feed-forward 4,480 wide
State space	Mamba-3 MIMO: causal depthwise convolution of width 4, then an exponential-trapezoidal oscillatory recurrence with a 16-dimensional complex state per head channel
Attention	exact causal multi-head latent attention, 192-dimensional latent, decoupled rotary encoding
Vocabulary	65,536 BPE tokens; input embedding and output head share one ternary matrix
Context	4,096 tokens
Weights	ternary {−1, 0, +1}, 2-bit codes, one FP16 scale per 128 inputs; norms, biases and recurrence stability terms in FP32
Activations	INT8 per token at every ternary matrix

Training

Stage	Tokens	Steps	Learning rate	Result
Pretraining	100.0B	678,168	3e-4 peak, 2,000 warmup, decay to 10%	the pretrained checkpoint
Supervised fine-tuning	309.7M	2,100	2e-5	the shipped model
Preference tuning (DPO)	6,120 pairs	170	1e-6, β 0.1	trained, evaluated, not shipped

One H100 80GB, 147,456 tokens per optimizer step, FP32 shadow weights and a straight-through estimator. DPO kept the model's knowledge but did not improve its answers at this size, so the supervised checkpoint ships and the preference data never entered the released weights. The supervised checkpoint that produced them, optimizer state included, is on the step2100 branch.

Pretraining mixture by configured weight: FineWeb-Edu 46%, FineWeb-Edu score-2 30%, FineMath 3+ 10%, Cosmopedia 5%, permissively licensed source code 5%, Project Gutenberg 4%. Fine-tuning by realized token share: Magpie-Ultra 62%, OpenCodeInstruct 12%, science answers 7%, summaries 6%, rewrites 4%, persona instructions 3%, OpenThoughts3 3%, system chats 2%, rewriting drills 0.6%, small talk 0.1%. All SmolTalk2 no_think splits apart from OpenCodeInstruct, because a model this size imitates a reasoning trace without being able to carry it.

Training data

Every source carries a permissive license and a pinned revision; exact revisions and required wording are in attribution_bundle.md.

Source	Used in	License
HuggingFaceFW/fineweb-edu, fineweb-edu-score-2	pretraining	ODC-BY, from Common Crawl
HuggingFaceTB/finemath (finemath-3plus)	pretraining	ODC-BY, from Common Crawl
HuggingFaceTB/cosmopedia (web_samples_v2)	pretraining	Apache-2.0, generated with Mixtral-8x7B-Instruct
common-pile/project_gutenberg	pretraining	public domain; the Project Gutenberg trademark implies no endorsement
common-pile/stackv2_edu_filtered	pretraining	Blue Oak Council permissive licenses, per-file notices retained upstream
HuggingFaceTB/smoltalk2, nine no_think splits	fine-tuning	Apache-2.0; Magpie-Ultra generated with Llama-3.1-405B-Instruct (built with Llama), summaries and rewrites with Qwen2.5-72B-Instruct, science answers distilled from DeepSeek-R1, persona instructions from Tulu 3
nvidia/OpenCodeInstruct	fine-tuning	CC-BY-4.0
HuggingFaceH4/ultrafeedback_binarized	preference stage, not shipped	MIT

Benchmark datasets were used for evaluation only.

Intended use and limitations

A reference for native ternary training and a small, fast local chat model: 2-bit inference experiments, on-device demos, short answers, rewriting, summarizing a paragraph, simple code.

• Facts are right about two thirds of the time and arithmetic about a third, on the runtime's own test suite, and it invents details fluently, so read what it writes before acting on it.
• English only, 4,096-token context, no tool use, no retrieval.
• It sometimes declines a question it could answer, and repeats itself at low temperature; the runtime's repetition penalty and system message reduce both.

License

Copyright 2026 Luis Oezdem. The weights, tokenizer and this card are released under the Apache License 2.0 (LICENSE). The training data licenses require attribution when the weights are redistributed: keep attribution_bundle.md with the model. One fine-tuning source, 0.19% of all training tokens, carries conversations generated with Llama 3.1: built with Llama.

Citation

@misc{ivonar-nano-2026,
  title  = {Ivonar Nano: a 349M natively ternary language model},
  author = {Oezdem, Luis},
  year   = {2026},
  note   = {Apache-2.0. Trained from scratch on 100B tokens, 2.17 bits per parameter, 94.6 MB packed.}
}