attentionlab/bayon-it-onnx

Bayon Instruct — ONNX

ONNX export and int8 quantization of attentionlab/bayon-it (Bayon Instruct), an approximately 109.0M-parameter (109,019,648 measured) instruction-tuned Khmer decoder-only language model.

Files

File	Format	Size on disk (approx.)
model.onnx	ONNX opset 17, fp32	≈447 MB
model-int8.onnx	ONNX, int8 dynamic quantization	≈114 MB

All file sizes are bytes on disk, not parameter counts. The checkpoint contains 109.0M parameters (109,019,648, summed from the safetensors header). Weights are stored fp32 at ~4 bytes/parameter, which is what makes model.onnx ≈447 MB on disk; the int8 build stores quantized MatMul/Gemm weights at ~1 byte/parameter plus fp32 residuals and scales, hence ≈114 MB.

model-int8.onnx was quantized with ONNX Runtime (onnxruntime.quantization.quantize_dynamic, QInt8, per-channel), which quantizes MatMul/Gemm weights while keeping activations in fp32. It runs on CPU with the standard CPUExecutionProvider — no extra ops/threads flags needed.

Usage

import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession("model-int8.onnx", providers=["CPUExecutionProvider"])
ids = np.array([[tokenizer.encode("...")]], dtype=np.int64)  # Khmer BPE, 5,000-token vocab
mask = (ids > 0).astype(np.int64)
logits = sess.run(None, {"input_ids": ids, "attention_mask": mask})[0]
# greedy decode: sample argmax(logits[:, -1]) and loop

Inputs: input_ids, attention_mask (int64). Output: logits of shape [batch, seq, 5000].

Notes

• The export was produced with Optimum (optimum-cli export onnx --task text-generation) from the source weights.
• This graph has no KV cache — each forward pass recomputes the whole sequence, so it's best suited to validation, short generations, or batched scoring.
• GGUF builds of the same weights (q8_0, q4_k_m) are available in the source repo family.