cox-1-3b-int8

The INT8 build of cox-1-3b, Vitund's decision model: given a text and typed questions (yes/no, choice, score), it returns a probability for every option of every question, without generating text. Everything about what the model does, how to use it, its evaluation, limitations and training data is on the cox-1-3b model card; this page covers only what differs.

• Build: cox-1.2 (2026-10-30), the same build as cox-1-3b's main.
• GPU memory: on a CUDA GPU the weights stay in INT8, so the loaded model takes 3.0 GB instead of 5.9 GB. On an 8 GB RTX 3070, a single 4k-token request peaked at 3.7 GB against 6.5 GB in bf16, and an 8k-token request at 4.3 GB against 7.1 GB. A batch of eight 1k-token texts ran in INT8 but did not fit in bf16.
• Size: 2.9 GB to download, against 5.7 GB for the bf16 build.
• Accuracy: on the 15 public datasets of the cox-1-3b card (about 18,500 decisions), skill is +0.653 for both builds (paired difference -0.0004, 95% interval -0.003 to +0.002), accuracy 80.7% for both, and calibration error 0.067 against 0.069 in bf16. The top answer matches the bf16 build on 99.3% of decisions.
• Speed: not faster than bf16 yet. Each matrix product widens the INT8 weights to bf16 first, which costs time: on the RTX 3070, one request at a time (a yes/no and a four-option question) took 163 ms for a short text against 113 ms in bf16, 296 ms against 240 ms at 1k tokens and 787 ms against 710 ms at 4k. Use this build when GPU memory is the limit; use the bf16 build when latency is.
• How it is made: every weight matrix in the transformer blocks, and the token embedding table, is stored as INT8 with one scale per output channel (per row of the embedding table): symmetric round-to-nearest, no calibration data. Norms, the small linear-attention parameters and the decision head stay in their original precision. cox_config.json lists which tensors are which.

How to use it

import coxlm
model = coxlm.load("Vitund/cox-1-3b-int8")

This needs coxlm 0.7 or later; older versions stop with "upgrade coxlm". coxlm recognizes the INT8 format from the model's own configuration. On a CUDA GPU the weights stay INT8 (as torchao tensors if torchao is installed, pip install "coxlm[int8]"; otherwise coxlm's own INT8 layer computes the same products). On the CPU, Apple silicon (MPS or MLX) the weights are converted to bf16 as the model loads, so it answers like the bf16 build there and saves only download size. Fine-tuning and adapters need the bf16 build.

Builds

build	published	notes
cox-1.2 (current, main)	2026-10-30	First INT8 build.

Licence and contact

Apache-2.0, with the training-data attributions on the cox-1-3b model card. Questions, problems and licensing: opensource@vitund.ai. More about Vitund: vitund.ai.