colbert-muvera-small-onnx

ONNX export of NeuML/colbert-muvera-small (Apache-2.0) for fastlate and any ONNX Runtime client. The weights are unchanged; this repo adds the graph, an INT8 dynamic quantization of it, and the ColBERT settings the model was trained with.

Files

model.onnx: transformer, the PyLate Dense projection layer(s), and per-token L2 normalization, in one graph. Inputs input_ids, attention_mask (int64, dynamic batch and length); output embeddings of shape (batch, seq, 128). Opset 17.
model_int8.onnx: onnxruntime.quantization.quantize_dynamic of the above, QInt8 weights.
onnx_config.json: prefix ids, lengths, skiplist, expansion and padding settings copied from the PyLate model, plus fde_center.
tokenizer.json: the source tokenizer, with [Q] /[D] as added tokens.

Settings

Query prefix [Q] (id 30522), document prefix [D] (id 30523), inserted right after the sequence-start token. Query length 32, document length 300. do_query_expansion=True: queries are padded to query_length with [MASK] (id 103), attention off on the padding, and every query vector is kept. Punctuation tokens (skiplist_words) are dropped from document embeddings after encoding, as in PyLate.

fde_center=False: whether to subtract the corpus mean token vector before building MUVERA fixed-dimensional encodings. Measured on a 3,000-chunk code/notebook corpus with FDEs of 4,096 dims: centring lowered shortlist recall for this model, so leave vectors as they are.