Kainlan/Qwen3-Reranker-0.6B-int8-ov-lasttok

Qwen3-Reranker-0.6B — int8 OpenVINO IR (last-token export)

A graph-edited variant of OpenVINO/Qwen3-Reranker-0.6B-int8-ov optimized for reranking inference. The int8 weights are untouched — only the compute graph is edited.

What changed vs. the source IR

Qwen3-Reranker scores a (query, document) pair from the logits at the last sequence position only. The stock IR computes lm_head over every position (MatMul producing (batch, seq, vocab)), then discards all but the last row — wasting a full-sequence projection against the ~151k-wide vocab on every call.

This export inserts a single Slice(start=-1, axis=1) immediately before the lm_head MatMul, so the projection runs only on the final position. The output shape becomes (batch, 1, vocab) instead of (batch, seq, vocab).

This is mathematically exact: a matmul output row depends only on its own input row, so slicing before vs. after the projection is the same computation minus the discarded rows. Measured score parity with the source IR is within the int8 quantization noise band, and it is ~8.6–8.8× faster on the reranker hot path.

Files

Standard OpenVINO IR + tokenizer: openvino_model.xml / openvino_model.bin (int8), tokenizer.json, config.json, plus the source repo's tokenizer sidecars.

Provenance & license

Derived from OpenVINO/Qwen3-Reranker-0.6B-int8-ov (itself an OpenVINO int8 export of Qwen/Qwen3-Reranker-0.6B), apache-2.0. The graph edit was performed with openvino.opset13 (Core.read_model → insert Slice → save_model); no weights were retrained, requantized, or otherwise modified.

Produced for the codescout code-search engine's cross-encoder reranker stage.