DimQ1/sortformer-4spk-v2-onnx-int4-cpu

Sortformer 4spk-v2 — ONNX INT4 (CPU, MatMulNBits)

INT4 quantized ONNX model of NVIDIA's Sortformer 4spk-v2 speaker diarization. Uses MatMulNBits (ORT ≥ 1.18) for 4-bit weight compression with the best accuracy-to-size ratio.

Model Details

• Base model: nvidia/diar_streaming_sortformer_4spk-v2
• Format: ONNX opset 21, INT4 MatMulNBits (block_size=32, symmetric)
• Parameters: 117.7 M (weights: 4-bit)
• File size: 135 MB total (21 MB ONNX + 115 MB external data)
• Quantizable MatMul nodes: 265 of 352 (87 attention-score MatMuls kept FP32)
• Input: processed_signal — log-mel spectrogram [B, 128, mel_frames]
• Output: speaker_logits — per-frame sigmoid probabilities [B, diar_frames, 4]
• Diar frame rate: 80 ms (10 ms mel stride / 8× subsampling)

Architecture

Audio (16 kHz mono) → STFT → 128 mel bins
  → ConformerEncoder (17 layers, d=512)
    → SortformerModules (4 speakers)
      → TransformerEncoder (18 layers, d=192)
        → Sigmoid → [B, diar_frames, 4] speaker logits

All 265 constant-weight MatMul nodes replaced with MatMulNBits (INT4, block_size=32). The finer block_size=32 gives better accuracy than INT8 despite 4-bit weights. 87 dynamic MatMul nodes (attention scores) remain in FP32.

Performance (CPU, 12 threads, ORT 1.27.1)

Metric	Value
RTF	0.043 (23× real-time)
DER	19.11% (LibriCSS test set)
Size	135 MB (3.5× smaller than FP32)
cos_sim vs FP32	0.996–0.999

Benchmarked on 100-file LibriCSS subset (1954.8 s audio). ORT 1.27.1 with ORT_PARALLEL + memory optimizations.

⚡ INT4 is more accurate than INT8 (DER 19.11% vs 21.64%) due to finer block_size=32 quantization granularity. Best accuracy-to-size ratio for production.

Other Variants

Variant	Size	DER	RTF	Link
FP32	469 MB	17.82%	0.037	sortformer-4spk-v2-onnx-fp32-cpu
INT8	129 MB	21.64%	0.034	sortformer-4spk-v2-onnx-int8-cpu

Requirements

• ONNX Runtime ≥ 1.18 (MatMulNBits support)
• CPU Execution Provider
• .data file must be in the same directory as sortformer.onnx

Usage

Python

import onnxruntime as ort

session = ort.InferenceSession(
    "sortformer.onnx",
    providers=["CPUExecutionProvider"],
)
speaker_logits = session.run(None, {"processed_signal": mel})[0]

C#

using var session = new InferenceSession("sortformer.onnx");
var inputs = new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("processed_signal", melTensor),
};
using var results = session.Run(inputs);

Conversion

Quantized from FP32 ONNX using MatMulNBitsQuantizer (ORT ≥ 1.18, opset 21):

python scripts/quantize_nbits.py --int4

See nemotron-speech-csharp/DiarizationConverter

References

• NVIDIA Sortformer Paper
• ONNX Runtime MatMulNBits