DimQ1/sortformer-4spk-v2-onnx-int8-cpu

Sortformer 4spk-v2 — ONNX INT8 (CPU, MatMulNBits)

INT8 quantized ONNX model of NVIDIA's Sortformer 4spk-v2 speaker diarization. Uses MatMulNBits (ORT ≥ 1.18) for real 8-bit weight compression.

Model Details

• Base model: nvidia/diar_streaming_sortformer_4spk-v2
• Format: ONNX opset 21, INT8 MatMulNBits (block_size=64, symmetric)
• Parameters: 117.7 M (weights: 8-bit)
• File size: 129 MB total (21 MB ONNX + 109 MB external data)
• Quantizable MatMul nodes: 265 of 352 (87 attention-score MatMuls kept FP32)
• Input: processed_signal — log-mel spectrogram [B, 128, mel_frames]
• Output: speaker_logits — per-frame sigmoid probabilities [B, diar_frames, 4]
• Diar frame rate: 80 ms (10 ms mel stride / 8× subsampling)

Architecture

Audio (16 kHz mono) → STFT → 128 mel bins
  → ConformerEncoder (17 layers, d=512)
    → SortformerModules (4 speakers)
      → TransformerEncoder (18 layers, d=192)
        → Sigmoid → [B, diar_frames, 4] speaker logits

All 265 constant-weight MatMul nodes replaced with MatMulNBits (INT8, block_size=64). 87 dynamic MatMul nodes (attention scores) remain in FP32.

Performance (CPU, 12 threads, ORT 1.27.1)

Metric	Value
RTF	0.034 (29.4× real-time)
DER	21.64% (LibriCSS test set)
Size	129 MB (3.6× smaller than FP32)
cos_sim vs FP32	0.989–0.999

Benchmarked on 100-file LibriCSS subset (1954.8 s audio). ORT 1.27.1 with ORT_PARALLEL + memory optimizations.

🚀 INT8 is the fastest variant — 29.4× real-time on 12-core CPU (66.6 s for 1954.8 s audio).

Other Variants

Variant	Size	DER	RTF	Link
FP32	469 MB	17.82%	0.037	sortformer-4spk-v2-onnx-fp32-cpu
INT4	135 MB	19.11%	0.043	sortformer-4spk-v2-onnx-int4-cpu

Requirements

• ONNX Runtime ≥ 1.18 (MatMulNBits support)
• CPU Execution Provider
• .data file must be in the same directory as sortformer.onnx

Usage

Python

import onnxruntime as ort

session = ort.InferenceSession(
    "sortformer.onnx",
    providers=["CPUExecutionProvider"],
)
speaker_logits = session.run(None, {"processed_signal": mel})[0]

C#

using var session = new InferenceSession("sortformer.onnx");
var inputs = new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("processed_signal", melTensor),
};
using var results = session.Run(inputs);

Conversion

Quantized from FP32 ONNX using MatMulNBitsQuantizer (ORT ≥ 1.18, opset 21):

python scripts/quantize_nbits.py --int8

See nemotron-speech-csharp/DiarizationConverter

References

• NVIDIA Sortformer Paper
• ONNX Runtime MatMulNBits