agelwarg/supertonic-3-coreml

Supertonic 3 for Core ML

Supertone/supertonic-3 (31 languages, 10 voices, 44.1 kHz) converted to Core ML, to run on the Apple Neural Engine of an iPhone or Mac. Made for Pocket Chatter, an on-device voice chat app.

What's here

Path
coreml/DurationPredictor.mlpackage	seconds of speech for the text
coreml/TextEncoder.mlpackage	text embedding, conditioned on the voice
coreml/VectorEstimator.mlpackage	one flow-matching step (with Supertonic 3's classifier-free guidance)
coreml/Vocoder.mlpackage	latents to 44.1 kHz audio
unicode_indexer.json, tts.json, voice_styles/	unchanged from Supertone's repo
conversion/	the scripts that made the packages

Each package is an ML program (iOS 18 / macOS 15) with one function per fixed length, sharing one copy of the weights: n64, n128, n192, n256, n384 and n512. The Neural Engine only runs fixed shapes, so inputs are padded up to the next length, with masks; text and latent frames use the same length. A latent frame is 3,072 samples, so n512 holds about 35 seconds of speech. Inputs and outputs are float32; the networks compute in FP16.

Load a length with MLModelConfiguration.functionName = "n128". Run the duration predictor and text encoder once per sentence, the vector estimator once per step (current_step 0 to total_step - 1, Supertone uses 5 to 10), then the vocoder, and keep the first duration × 44100 samples.

Compute units: run the vector estimator and vocoder (nearly all the work) on the Neural Engine (.cpuAndNeuralEngine), and the duration predictor and text encoder on the GPU (.cpuAndGPU). On the Neural Engine the text encoder's output is wrong at n64 (correlation about 0.4 with the CPU's, on an M5 Mac and an iPhone 17 Pro), which makes the speech noise; every other model and length matches the CPU there. Inputs and outputs have the names and layouts of Supertone's ONNX graphs.

How it was made (changes from the original)

Supertone publishes ONNX graphs. The network was rebuilt as PyTorch modules (conversion/st3_torch.py, after typomonster's MLX port of Supertonic 2), loaded with the weights from Supertone's ONNX files (conversion/onnx_weights.py), checked against ONNX Runtime (float32 rounding on every part) and converted with coremltools 9 (conversion/convert_all.py). Changes:

• FP16 compute. Against ONNX Runtime with the same noise, the audio correlates 0.97.
• Padding. Before each depthwise convolution, padded frames are filled with the last real frame, the edge padding an unpadded input gets, so a padded input gives the same result as an unpadded one.
• Masks. The text attention's masked logits are -1e4 instead of -inf (FP16).

License

The model is Supertone's, under the BigScience Open RAIL-M license (LICENSE), including its use restrictions (Attachment A), which apply to these converted models too. The conversion scripts are MIT.