FluidInference/nemotron-3.5-transcribe-lite-coreml

Nemotron 3.5 Transcribe Lite: Core ML + Core AI

Unofficial community port, not affiliated with or endorsed by NVIDIA or Apple. "Lite" is our name for this setup, not an NVIDIA model or release. It is the audio side of NVIDIA Nemotron 3.5 Transcribe, without its 30B language model, converted for Apple silicon:

• the ~600M-parameter ASR encoder and the CTC timestamp head (words with 40 ms start times), as Core ML and Core AI models
• run next to Nemotron-3-Diarization (who spoke when). The diarizer is not in this repo: FluidAudio downloads its Core ML port from FluidInference/nemotron-3-diarization-coreml

On an M5 Pro MacBook Pro, a 49.5-minute, 3-person meeting becomes a speaker-labeled transcript with word start times in about 6 s with the models preloaded and warm (500x real time): the encoder and CTC head run on the GPU while the diarizer runs on the Neural Engine at the same time. Model loading, warm-up and the diarizer's one-time Neural Engine compile (80 s) are not included (see Meeting demo).

This is not the full model. Without the LLM the transcript is much rougher (see Speed and accuracy). The encoder weights here are NVIDIA's, unchanged apart from the format conversion; the CTC head was trained by NVIDIA as a timestamp aligner for the LLM's output, and here it is used directly as a greedy CTC recognizer.

Files

Path	What	Size
coreml/lite_encoder_w48.mlpackage	Core ML encoder, one 48 s window (used by the demo)	1.26 GB
coreml/lite_ctc_head_w48.mlpackage	Core ML CTC head for the same window (used by the demo)	101 MB
coreml/lite_encoder_w48.mlmodelc	the w48 encoder above, compiled, for FluidAudio (Swift)	1.26 GB
coreml/lite_ctc_head_w48.mlmodelc	the w48 CTC head above, compiled, for FluidAudio (Swift)	101 MB
coreml/lite_ctc_fp16.mlpackage	Core ML encoder + CTC head in one model, 30 s window	1.36 GB
coreml/lite_ctc_w8.mlpackage	same, int8 weights (linear symmetric, per channel)	683 MB
coreml/lite_ctc_p4.mlpackage	same, 4-bit palettized weights (k-means, groups of 16), see below	389 MB
coreai/lite_ctc_gpu.aimodel	Core AI (beta, macOS 26+), same I/O as lite_ctc_fp16, via coreai-torch	1.36 GB
assets/frontend.npz	NeMo mel window (400) + filterbank (128 x 257) from the checkpoint	0.1 MB
assets/mel_window.bin, assets/mel_filterbank.bin	the same window and filterbank as raw float32 LE
assets/ctc_tokenizer.model	the CTC head's SentencePiece model (23,862 pieces; blank = 23862)	0.6 MB
assets/vocab.json	the same 23,862 pieces as a JSON list (index = token id), for FluidAudio
demo/	meeting demo (NumPy host code, no PyTorch)
scripts/, coreai/*.py	extraction, conversion and LibriSpeech eval scripts (no AMI meeting scorer)

Inputs and outputs (enc_len is (1,) int32; token_ids and token_logprob give one token per 40 ms):

• lite_encoder_w48: mel (1, 128, 4800) fp32 + enc_len → states (1, 1280, 600) fp16
• lite_ctc_head_w48: states (1, 1280, 600) + enc_len → token_ids, token_logprob (1, 1200)
• lite_ctc_fp16, _w8, _p4 and the .aimodel: mel (1, 128, 3000) + enc_len → token_ids, token_logprob (1, 750)

All Core ML models are fp16 ML Programs (macOS 15+) unless noted. w48 = 4800 mel frames: NeMo's long-form layout of 40 s cores with 4 s of context on each side; the demo keeps only each window's core frames. In lite_ctc_p4 the CTC output projection (23,863 x 1280) stays fp16, because 23,863 rows do not split into groups of 16.

The .mlmodelc folders, assets/*.bin and assets/vocab.json are for FluidAudio (Swift); FluidAudio support: FluidAudio #1069 (nemotron35-transcribe).

Meeting demo

Needs Python 3.10 to 3.13 (coremltools 9.0 has no Python 3.14 wheel). The download fetches only what the demo uses, about 1.4 GB of this repo (FluidInference/nemotron-3.5-transcribe-lite-coreml); leave out the --include flags to get everything.

python3.13 -m venv .venv && . .venv/bin/activate      # any Python 3.10-3.13
pip install -U huggingface_hub                        # provides the `hf` CLI
hf download FluidInference/nemotron-3.5-transcribe-lite-coreml --local-dir nemotron-lite \
  --include "coreml/lite_*_w48.mlpackage/*" --include "assets/*" --include "demo/*" --include requirements.txt

git clone https://github.com/FluidInference/FluidAudio && (cd FluidAudio && swift build -c release)
export FLUIDAUDIO_CLI=$PWD/FluidAudio/.build/release/fluidaudiocli  # diarizer (~200 MB) is downloaded on first use

cd nemotron-lite && pip install -r requirements.txt
python demo/lite_meeting.py meeting.wav --verbose          # 16 kHz audio; ffmpeg -i in -ac 1 -ar 16000 meeting.wav
python demo/lite_meeting.py meeting.wav --out t.txt --json t.json   # transcript with <spk:N> tags; word start times

--verbose logs every window; --wait loads and warms all three models, then starts the clock when you press Enter. bash demo/demo_tmux.sh meeting.wav opens a side-by-side view (needs brew install tmux macmon): CPU, GPU and Neural Engine power on the left (demo/sidemon.py), the run on the right. Run the shell scripts with bash: Hugging Face downloads do not keep the executable bit.

Pipeline: mel + per-feature normalization (NumPy) → encoder windows (GPU) → CTC head windows (GPU) → greedy CTC words, each stamped with its first piece's 40 ms frame (start time only). The diarizer starts at t=0 in its own process on the Neural Engine; each word takes the speaker active at its time (nearest active 80 ms frame within 1 s). Diarizer slots with under 2 s of speech in the whole recording are not counted as speakers.

The first Neural Engine run compiles the diarizer (~80 s on an M5 Pro). macOS keeps the compiled copy only for a while, so the demo warms it on a 10 s clip before the clock starts.

Speed and accuracy

M5 Pro (24 GB), macOS 27, models loaded and warmed before timing.

Meeting: AMI Meeting Corpus EN2002c (49.5 min, mixed headset audio, 3 speakers), scored against the AMI manual transcript with the Whisper English normalizer.

total	5.7 to 6.5 s (450x to 520x real time)
mel + encoder (CPU + GPU)	4.6 to 5.5 s
CTC head (GPU)	1.0 to 1.1 s
diarizer (Neural Engine, in parallel)	5.0 to 5.3 s
peak power during the run	GPU ~32 W, Neural Engine ~4.6 W
WER	26.3% (sub 1048, del 1729, ins 135 of 11,049 words)
cpWER (speaker-attributed)	28.7%

For comparison, NVIDIA reports 10.29% cpWER (individual headsets) and 17.34% (single distant mic) on AMI for the full model with its LLM. Those figures are for the AMI test set (full recordings), scored with NVIDIA's own pipeline. Ours is one meeting on the mixed-headset channel, scored with our own jiwer + Whisper-normalizer cpWER, so the comparison is only indicative.

LibriSpeech test-clean, first 400 utterances (7 of 40 speakers), greedy CTC, 30 s windows (shorter clips padded to 30 s), speed = audio duration / time inside the model call:

Model	GPU WER	GPU speed	Neural Engine WER	Neural Engine speed
PyTorch fp32 (reference)	4.64%
Core ML lite_ctc_fp16	4.64%	238x	4.67%	106x
Core ML lite_ctc_w8	4.91%	235x	4.95%	126x
Core ML lite_ctc_p4	5.09%	213x	5.04%	133x
Core AI lite_ctc_gpu	4.62%	234x	4.65%	87x

The 400-utterance subset is easier than the full set: on all 2,620 test-clean utterances the PyTorch encoder + CTC head scores 5.50%. NVIDIA reports 1.09% for the full model with its LLM. Core ML fp16 and Core AI stay within 0.03 points of PyTorch's WER (at most 2 word errors apart out of 9,004 words; Core ML fp16 on the GPU has the same error count). Neural Engine speeds here use the standard tensor layout, not Apple's ANE-optimized layout.

Only English was evaluated.

Core AI

coreai/lite_ctc_gpu.aimodel needs Apple's Core AI runtime (coreai-core==1.0.0b3, beta, Python 3.12) on macOS 26+ (its only Mac wheels are macosx_26_0_arm64). Loading and calling are async in Python:

import asyncio, numpy as np, coreai.runtime as rt

async def main():
    kind = rt.ComputeUnitKind.gpu  # or rt.ComputeUnitKind.neural_engine
    opts = rt.SpecializationOptions.from_preferred_compute_unit_kind(kind())
    m = await rt.AIModel.load("coreai/lite_ctc_gpu.aimodel", opts)
    fn = m.load_function(list(m.function_names)[0])
    mel = np.zeros((1, 128, 3000), np.float32)  # normalized mel, see demo/frontend.py
    r = fn({"mel": rt.NDArray(mel), "enc_len": rt.NDArray(np.array([375], np.int32))})
    r = await r if hasattr(r, "__await__") else r
    print(r["token_ids"].numpy().shape)  # (1, 750): one token per 40 ms

asyncio.run(main())

coreai/eval_aimodel.py is the LibriSpeech evaluation above.

How it was made

1. scripts/extract_lite.py: reads the index of NVIDIA's 67.5 GB model.safetensors and downloads only the 893 perception.* and ctc_timestamp.* tensors (1.67 GB) with HTTP Range requests → ckpt/lite.safetensors at the repo root, where the other scripts read it.
2. scripts/export_assets.py: mel window, filterbank and CTC tokenizer → assets/.
3. scripts/lite_model.py + scripts/attn_patches.py: the encoder + CTC head in PyTorch via NeMo, with FlexAttention replaced by an exact masked-matmul attention so it traces.
4. scripts/convert_coreml.py (lite_ctc_*), scripts/convert_w48.py (the demo's encoder + head), coreai/export_aimodel.py (Core AI).
5. scripts/eval_librispeech.py, scripts/eval_coreml.py, coreai/eval_aimodel.py: the LibriSpeech numbers above. The AMI meeting WER/cpWER scorer is not included.

Run the scripts from the repo root. The demo needs none of the following.

• NeMo Speech must come from git: PyPI nemo-toolkit 3.0.0 lacks the RoPE encoder and the CTC timestamp head. pip install -r scripts/requirements-convert.txt installs the toolchain the Core ML models were built with (Python 3.12; NeMo Speech commit 16b0ff2584, torch 2.7.0, coremltools 9.0, numpy 1.26.4: coremltools 9.0 fails to convert under numpy 2.5).
• coreai/export_aimodel.py used a separate env: Python 3.12, torch==2.9.0, coreai-core==1.0.0b3, coreai-torch==0.4.3, coreai-opt==0.3.0 and the same NeMo commit. Its parity gate needs PyTorch reference outputs (coreai/oracle.npz) that are not shipped; without them the gate is skipped.
• The parity check in scripts/convert_coreml.py and the eval scripts read LibriSpeech test-clean (CC BY 4.0) from data/LibriSpeech/test-clean: mkdir -p data && curl -L https://www.openslr.org/resources/12/test-clean.tar.gz | tar xz -C data

Changelog

Date	Files	Change	Issue	mobius PR	FluidAudio PR	HF commit
2026-10-28	all	first release: Core ML + Core AI ports, demo, scripts				c6d4024e
2026-10-28	coreml/*_w48.mlmodelc, assets/vocab.json, assets/mel_*.bin	compiled models + Swift-readable tables (new files)			#1069	7d740811

License

The model materials are NVIDIA's, released under the OpenMDW License Agreement 1.1 (copy in LICENSE). This repository redistributes them converted to Core ML and Core AI formats, including 8-bit and 4-bit compressed variants. Unofficial port, not affiliated with or endorsed by NVIDIA or Apple.

scripts/attn_patches.py is adapted from NVIDIA NeMo Speech (Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES) and is licensed under the Apache License 2.0 (copy in licenses/NeMo-LICENSE.txt).

Meeting numbers use the AMI Meeting Corpus (CC BY 4.0); no AMI audio is included here.