ThoxMicro-1bit-9M

A BitNet b1.58 ternary language model trained from scratch. Weights in the transformer blocks are {-1, 0, +1} with a per-tensor absmean scale; activations are int8 per-token (W1.58A8).

Licence pending. Trained from scratch on roneneldan/TinyStories, which is CDLA-Sharing-1.0. Whether that licence's share-alike terms extend to trained weights is not settled. Treat as unlicensed for redistribution until THOX confirms. This is deliberately unresolved rather than guessed at.

What this is — and is not

It writes simple children's stories. That is the whole of it.

• ❌ Not an assistant. Not instruction-tuned, no chat template, no system prompt.
• ❌ Cannot answer questions or follow instructions. It has never seen an instruction.
• ❌ Not a general-purpose model. Its world is the TinyStories vocabulary and domain.
• ✅ A completion model, and an edge/on-device research artifact demonstrating that ternary quantization-aware training works end to end at this scale.

Prompt it with the start of a story and it continues the story.

Architecture

Plain Llama decoder — RMSNorm, RoPE, SwiGLU. Nothing custom, so it converts to GGUF with stock tooling.

Parameters	8,917,248 (6,815,744 ternary = 76.4%)
vocab_size	8,192 (own byte-level BPE)
d_model	256
n_layers	8
n_heads	8 (head_dim 32)
ffn_hidden	768
seq_len	512
Embeddings	tied

Every dimension is a multiple of 256, so the block quantizers can pack every tensor rather than silently falling back to F16.

Following BitNet, the embedding and output head are not ternarized — only the linear layers inside transformer blocks.

Training

Trained from scratch — this is not a fine-tune or a post-hoc quantization of an existing model. The latent fp32 weights spend the whole run adapting to the ternary grid via a straight-through estimator.

Data	TinyStories, 393,216,000 tokens
Steps	24,000
Hardware	one RTX 4060 Ti, 2.38 hours
Cloud cost	$0

Results

Measured against an fp16 baseline sharing config, seed, data order and token budget — so the delta isolates ternary and nothing else.

	fp16 baseline	ternary (this model)
Validation loss	1.5476	1.7385
Perplexity	4.7004	5.6887
Artifact size	6,484,316 B	4,420,028 B

Ternary costs +0.1909 nats / +21.0% perplexity, for a 31.8% smaller artifact.

Both arms are 8 layers, so this is a like-for-like comparison: the only variable is quantization. (The 16M card documents a case where that was not true and the published number had to be corrected.)

The finding worth reading

The ternary penalty widened with more data. At 131M tokens the gap was +0.1591 nats; at 393M tokens it was +0.1909. Tripling the data did not close it — both arms improved and the fp16 arm improved more.

Do not budget more tokens expecting to recover the ternary loss. At this scale that needs more parameters, not more data.

That prediction was tested. A 16-layer sibling was trained with an fp16 control at the same depth, giving the full grid:

	8 layers (this model)	16 layers
fp16	1.5476	1.4844
ternary	1.7385	1.6271

+0.1909 nats at 8 layers, +0.1427 at 16 — parameters do recover part of the penalty, about a quarter of it, where three times the data recovered none. The figure on this card is the 8-layer corner of that grid, not a universal constant. See ThoxMicro-1bit-16M.

Artifact size is identical at both token budgets, because size is set by architecture and packing rather than training length. So on a fixed device budget, extra training is free.

Sample output

Greedy, temperature 0:

Tom and Sara went to the park and saw a big slide. They wanted to go on the slide, but they were scared.

"Let's go on the slide, Sara!" Tom said. "It looks fun!"

4-gram repetition is 0.0 across the evaluation prompts.

Files

file	bytes	sha256	runs on
thoxmicro-1bit-9m.TQ2_0.gguf	6,252,640	bbc3ef3e1a9125ff6a9f1ad7e57181d2311c4fc48f9a1cb9054f4dbd5fc2ffc0	llama.cpp
thoxmicro-1bit-9m.Q8_0.gguf	9,771,104	d33cea1fc8234de024934f88cf6a6298d2dfc61d0150dc3bdbd2e77e0be1c26a	llama.cpp and Ollama

Why two GGUFs. TQ2_0 is the ternary-native quant and the smaller file, but Ollama cannot load it — its vendored ggml predates the type and fails with tensor "blk.0.ffn_down.weight" size overflow. The Python gguf package rejects it too (35 is not a valid GGMLQuantizationType).

The Q8_0 build is not a quality downgrade: the weights are ternary, so every value is one of three levels and Q8_0's 256 levels represent them exactly. It is simply a less dense container — 9.77 MB instead of 6.25 MB for identical numbers. Use TQ2_0 with current llama.cpp; use Q8_0 everywhere else.

Verified in llama.cpp (llama-completion): loads in ~207 ms, 300–499 tok/s eval on 4 CPU threads.

A second artifact exists for microcontroller deployment — TERN1, a packed ternary format (4,420,028 B) for ESP32-S3 class targets. It is not distributed here; it is not a GGUF and no general-purpose runtime reads it.

Usage

llama-completion -m thoxmicro-1bit-9m.TQ2_0.gguf \
  -p "Once upon a time, there was a little girl named Lily. She" \
  -n 60 --temp 0

This model is a completion model. llama-cli's conversation mode will not behave sensibly with it — there is no chat template to apply.

Limitations

• TinyStories domain only. Simple vocabulary, simple sentences, children's-story structure. It will not discuss anything else coherently.
• No instruction following, no factual grounding, no safety tuning.
• 9M parameters. It is smaller than most tokenizers.
• Ternary costs real quality — see the table above. That number is published because it is the point of the artifact, not despite it.
• Losses on this card are best_val_loss, matching the 16M card. An earlier version quoted final_val_loss (1.7412), 0.0027 nats higher; two cards using different conventions for the same quantity invites a wrong comparison.

Provenance

Trained in thoxllm-factory (thox_ternary/). The pipeline, the TERN1 format specification, a reference C reader, and the paired-run methodology are all in that repository.