altertable/vela3-fast-multilingual-onnx-int8

vela3-fast multilingual ONNX int8 (EN/FR/DE/ES/IT decisions on CPU)

Torch-free CPU packaging of vela3-fast-ml, a French/German/Spanish/Italian fine-tune of the decision backbone of vllm-sr/vela3-fast (0.38B UltraBERT-380M encoder, Apache-2.0, KR Labs × vLLM Semantic Router). Altertable continued training it by knowledge distillation on English and machine-translated FR/DE/ES/IT decision data (MLFT round 2, checkpoint c1-s762). The encoder is exported to ONNX and weight-only int8 quantised (MatMulNBits, 8 bit, block 128, dense attention). See NOTICE and DATA_CARD.md.

English-only sibling (vela3-fast weights unchanged, int8 ONNX): altertable/vela3-fast-onnx-int8.

Answers	choice (probability per option) and noul (yes/no, p(yes)) questions over a text or JSON state
Languages	English, French, German, Spanish, Italian (state and/or instructions; best with English instructions/criteria)
Files	model/model_int8.onnx (0.6 MB graph) + model/model_int8.onnx.data (539 MB int8 weights; keep both files in the same folder), model/head.npz (12 MB fp32 readout head), tokenizer.json, vela_onnx.py
Runtime	onnxruntime, tokenizers, numpy (no torch); up to 32,768 tokens per question (longer → Unsupported)
Licence	Apache-2.0 (model and code); see LICENSE, NOTICE, LICENSES.md

Quality

Multilingual probe (XNLI, PAWS-X and Belebele test items, never trained on; English instructions, localised content; accuracy, macro over the three tasks), PyTorch checkpoint c1-s762:

	EN	FR	DE	ES	IT	ML4 (FR/DE/ES/IT)
vela3-fast (base)	80.9	64.8	58.6	62.7	47.6	58.4
vela3-fast-ml (this)	80.2	73.0	64.8	67.5	64.1	67.3 (gate ≥ 66.4)

• English Decision Index preserved: paired 11,313-row public sample (sample8) 52.2 vs 52.0 for vela3-fast (gate ≥ 51.5). vela3-fast's own public Decision Index (0.3, authors' run) is 50.36; we did not run the full Index for this model.
• int8 ONNX vs PyTorch fp32 (996 rows: sample8 rows + probe rows, same rows for both): answer agreement 99.7%, mean |Δp| 0.0009 (p95 0.0045); sample8 accuracy Δ 0.00, probe accuracy Δ −0.39 pts (gate: both within ±0.5). A faster blocked-attention int8 (acc4) variant was rejected (probe Δ −0.79).
• Provenance caveats: vela3-fast was trained on train splits of public datasets that are also Decision Index sources; our English data includes MNLI/SNLI/WANLI train splits (task-related to XNLI). Translations are machine-made. See DATA_CARD.md.

Quick start

hf download altertable/vela3-fast-multilingual-onnx-int8 --local-dir vela3-fast-multilingual-onnx-int8
cd vela3-fast-multilingual-onnx-int8
pip install -r requirements.txt
python example.py        # -> billing {'billing': 0.994, 'tech': 0.001, 'sales': 0.005}; urgent p(yes) = 0.852

from vela_onnx import DecisionModel
m = DecisionModel("model", threads=8)   # run from the downloaded folder; "model" holds model_int8.onnx(.data) and head.npz
r = m.decide({"ticket": "Bonjour, ma carte a été débitée deux fois pour le même café ce matin."}, {
    "route": {"type": "choice", "instructions": "Which team should handle this support ticket?",
              "criteria": {"billing": "Payments, refunds, double charges and invoices",
                           "tech": "Bugs, login problems and app crashes",
                           "sales": "New purchases, pricing questions and upgrades"}}})
print(r["answers"]["route"]["choice"])   # -> billing (p 0.992)

Tip: give options a short description of what belongs in them; bare option names are answered less reliably. English instructions/criteria over non-English content is the best-tested setting. One encoder pass per question (state repeated per question); questions are batched under a token budget.

Decision Index kit engine: PYTHONPATH=. python -m decision_index run --engine decision_index_engine:VelaOnnxEngine --option threads=8 ...

CPU latency

Short requests (1–2 questions, a few hundred tokens): ~0.3–0.6 s on a shared 8-vCPU VM (6 threads, onnxruntime 1.31). This dense int8 (acc0) build is about 1.9× slower per row than the rejected blocked-attention build on the parity set (5.8 vs 3.1 s mean per row, 8 workers × 4 threads on n2-highmem-32). Non-English text costs ~1.7–2.1× more tokens than English.

Citation (upstream)

@misc{vela3_fast, title = {vela3-fast (Vela 3.0, 0.4B)}, author = {{KR Labs} and {vLLM Semantic Router}}, year = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/vela3-fast}}}