arnhemlabs/bge-m3-int8-weightonly

BGE-M3, weight-only int8 ONNX

This is a weight-only int8 quantization of BAAI/bge-m3 at revision 5617a9f61b028005a4858fdac845db406aefb181. It outputs dense 1024-dimension embeddings. It is the in-process embedder of the Remanence memory engine (ADR-016 and amendment 1), which loads it from a pinned revision and checks every file against the SHA-256 values below.

Source and licence

• Source: BAAI/bge-m3 @ 5617a9f61b028005a4858fdac845db406aefb181. The FP32 ONNX export it publishes is onnx/model.onnx + onnx/model.onnx_data, and tokenizer.json is used unchanged.
• Licence: MIT, as declared in the source's model card. The source repository has no LICENSE file, so LICENSE here holds the standard MIT text, with a line naming the source and revision. This derivative is distributed under the same licence.

How it was made

• Script: scripts/export-models.sh in the Remanence repository, at commit 767aa23. It drives scripts/export-models.py.

• Command:

  scripts/export-models.sh ./models-export weight-only bge-m3
  

  This writes ./models-export/bge-m3-int8/.

• Environment, all pinned:

  • the image python@sha256:e697c81897302a4c4333bade000d98038fbb21eb064f8ee1fd42d02ebcd9d337;
  • onnx==1.23.2, onnxruntime==1.31.1, numpy==2.5.4, onnx-ir==1.1.0 and tokenizers==0.23.3.

  Every upstream input is fetched at the pinned revision and checked against its SHA-256 before use:

  upstream file	SHA-256
  onnx/model.onnx	f84251230831afb359ab26d9fd37d5936d4d9bb5d1d5410e66442f630f24435b
  onnx/model.onnx_data	1eebfb28493f67bba03ce0ef64bfdc7fc5a3bd9d7493f818bb1d78cd798416b4
  tokenizer.json	21106b6d7dab2952c1d496fb21d5dc9db75c28ed361a05f5020bbba27810dd08

• Quantization: onnxruntime's MatMulNBitsQuantizer with bits=8, block_size=128 and is_symmetric=True.

  • The MatMul weights become MatMulNBits with 8-bit weights.
  • Activations and compute stay FP32.
  • Weights are saved as external data (model.onnx.data).
  • The output for a text doesn't depend on what else is in the batch. Dynamic quantization was rejected because it does.

• Parity: cosine 0.9998 against the FP32 model on the Remanence parity set (ADR-016 amendment 1).

Files

file	bytes	SHA-256
model.onnx	690697	c383c2962c6408ff3d009845698ef2b73f1ba3e90233a3a066340352394858c9
model.onnx.data	1370353680	7039a3b86a769763e1bf516fc008a7fd566495262b2243c5a45ebb899d23f9f5
tokenizer.json	17098108	21106b6d7dab2952c1d496fb21d5dc9db75c28ed361a05f5020bbba27810dd08

model.onnx refers to its weights as model.onnx.data, so keep both in the same directory. tokenizer.json is the source's file, byte for byte.

Use

• Inputs: input_ids and attention_mask (int64, [batch, sequence]).
• Output: Remanence takes the first output, CLS-pools it (if it is [batch, sequence, 1024]) and L2-normalizes it. That matches how TEI serves BGE-M3.
• Sequence length: Remanence bounds input at 512 tokens by default.