ShanelBastian/multilingual-e5-base-q8-reduce-range

multilingual-e5-base, 8-bit quantized with reduce_range

A dynamically quantized ONNX build of intfloat/multilingual-e5-base, made for koralg. This is a derivative of that model under the same MIT license. It is not an official release and isn't endorsed by the original authors.

Why this build exists

The base repository's quantized file (model_qint8_avx512_vnni.onnx) gives different results depending on the CPU. In our retrieval test, its average MRR across 12 languages was 0.841 to 0.852 on CPUs with VNNI and 0.773 without. This build uses 7-bit weight range, which ONNX Runtime recommends for CPUs that can saturate.

Build	AVX-512 VNNI	AVX-VNNI	No VNNI
Base repository's quantized file	0.841	0.852	0.773
This build	0.843	0.845	0.845
Full fp32 model	0.867	0.867	0.867

The test uses 40 passages and 10 questions per language, so it's useful for comparing builds, not for predicting scores on a real knowledge base. Vectors still differ slightly between AVX-512 CPUs and others (average cosine 0.998, lowest 0.964), so re-index if you move an index between them.

How it was built

quantize.py in this repository, with reduce_range as its last argument and ONNX Runtime 1.31.1, from onnx/model.onnx at base revision d128750597153bb5987e10b1c3493a34e5a4502a (SHA-256 84a4d426f7e87a6bf5bf195f0bae2c4a7d15f675b23ca96f42fab8326d7a77aa). Rebuilding gave a byte-identical file on seven machines.

onnx/model_q8_reduce_range.onnx SHA-256: 8db030f6ad537c840cb59043264ed3ad1b7e8f8b9740ffa52f26b628638e6a1e

onnx/tokenizer.json is unchanged from the base revision.

Usage

Prefix text with query: or passage: , apply mean pooling over the attention mask, then L2-normalize. Output has 768 dimensions and input is limited to 512 tokens.