NealCaren/Nemotron-3-Diarization-ONNX

Nemotron 3 Diarization (ONNX)

An ONNX export of nvidia/Nemotron-3-Diarization, a 100M-parameter streaming Sortformer speaker-diarization model (up to 8 speakers, 10 ms resolution). These files are meant for running the model in a web browser with ONNX Runtime Web, or anywhere else ONNX Runtime runs.

This is an unofficial conversion. All credit for the model goes to NVIDIA. It is redistributed under the same OpenMDW-1.1 license.

Files

File	Size	What it is
embed.onnx	2 MB	Feature stacking + projection: log-mel features [1, N, 128] → embeddings [1, ceil(N/8), 512]
step_int8.onnx	103 MB	Encoder + head, dynamic int8 weights: embeddings [1, T, 512] → speaker logits [1, 8T, 8]
step.onnx	396 MB	The same in fp32
mel_filters.bin	128 KB	float32 [128, 257] Slaney mel filterbank (librosa, n_fft 512, 16 kHz)
silence_embeds.bin	2 KB	float32 [512] learned silence embedding used by the speaker cache
export_onnx.py, quantize.py		Scripts that produced these files from the transformers implementation

How to use it

The graphs cover only the neural network. The front end and the streaming speaker cache must be implemented around them, following Nemotron3DiarizationForAudioFrameClassification in transformers:

1. Features. 16 kHz mono audio, pre-emphasis 0.97, STFT (n_fft 512, symmetric Hann window of 400 centered in 512, hop 160, center=True with zero padding), power spectrum, mel filterbank, log(x + 2^-24), no normalization. Keep floor(len / 160) frames.
2. Embed all features with embed.onnx.
3. Chunked encoding. Offline mode uses chunks of 340 embedding frames with 40 frames of right context. Each step feeds [speaker cache, FIFO, chunk, right context] to step_int8.onnx. Positions restart at 0 every step.
4. Speaker cache update. Arrival-order speaker cache of 264 frames, FIFO of 40 frames with an update period of 300 (offline values), and score-based compression. See Nemotron3DiarizationSpeakerCache in transformers.
5. sigmoid(logits) gives per-speaker activity every 10 ms, with speakers numbered in order of first appearance.

A complete JavaScript implementation of all of this (about 250 lines, no dependencies apart from onnxruntime-web) lives in the web app that uses these files: nealcaren/local-interview-transcriber (see diar.js).

Accuracy of the conversion

On an 87-second, 4-voice test clip, compared against the transformers reference:

• step.onnx with the JavaScript pipeline: identical speech segments, and 100% agreement on the thresholded (p > 0.5) speaker decisions per frame.
• step_int8.onnx: 99.994% agreement per frame, with segment boundaries within 10 ms.

Speed in Chrome on an Apple M3 Max: step.onnx on WebGPU ran at about 160× real time, and step_int8.onnx on multithreaded WASM at about 25× real time.

License

OpenMDW License Agreement v1.1, as for the original model. Model by NVIDIA; see the original model card for training data, intended use, limitations, bias, and safety information.