arraypress/stable-audio-3-small-sfx-coreai

Stable Audio 3 Small SFX — Apple Core AI (.aimodel)

Powered by Stability AI

A community conversion of stabilityai/stable-audio-3-small-sfx to Apple's Core AI format, for on-device generation on iOS 27 / macOS 27. Sound effects, textures and ambiences.

Not an official Stability AI release. The weights are unchanged — this is a format conversion, not a retrain or a fine-tune.

What's here

A single .aimodel holding four named inference functions, plus the tokenizer:

Function	Signature
conditioner	(input_ids, attention_mask, seconds) → cross_attn_cond, cross_attn_mask, global_cond
dit	(x, t, cross_attn_cond, cross_attn_mask, global_cond, local_add_cond) → v — any length
decoder_N	(latent) → audio — several fixed lengths, see below
encoder_N	(audio) → latent — for audio-in, continuation and inpainting

let model = try await AIModel.load(url, options)
let dit = model.loadFunction("dit")

small-sfx.aimodel is 2.9 GB and the tokenizer folder is 34 MB. All weights are float32 — precision was chosen by measurement, not preference; see Fidelity.

The conditioner's output matches the original library's conditioning assembly exactly (0.0 difference), so nothing about the prompt or duration handling is approximated.

Why several decoder lengths. The DiT takes any length, but the encoder and decoder are compiled at fixed sizes — and a fixed-length decoder makes you pay its maximum on every call, since short latents have to be padded up to it. Measured on the 0.6 B model, decoding the same 24 s of content costs 0.30 s / 0.5 GB through a 47.6 s graph but 2.38 s / 2.1 GB through a 380.4 s one. So the asset ships a ladder — pick the smallest decoder_N that fits your clip. Weights are shared between the functions, so the extra rungs cost about 0.4 MB each rather than a full copy.

Capabilities

Feature parity with the original, with one structural difference:

	Original	This conversion
Text → audio	✅	✅
Audio length	up to 380.4 s	up to 380.4 s
Audio in / continuation / inpainting	✅	✅ (via the encode_N functions)
Negative prompts / CFG	✅	✅ vanilla CFG, verified (see note)
Prompt length	256 tokens	256 tokens
Steps, seeds, determinism	✅	✅
44.1 kHz stereo	✅	✅

The structural difference: the DiT takes any length (its shape is [1, 256, -1]), but the encoder and decoder are compiled at a fixed maximum of 380.4 s. For shorter audio, zero-pad the latent to that length, decode, and trim. The padding influences only the last 7–26 ms, which falls inside the margin you should be trimming anyway.

Classifier-free guidance

Run the DiT twice per step and blend on the host:

v = v_cond + (cfg_scale - 1) * (v_cond - v_uncond)

This reproduces the reference implementation's vanilla CFG exactly (cosine 1.0000000, SNR 81.8 dB, measured). Note the reference defaults to adaptive projected guidance (apg_scale=1.0), which projects out the parallel component; that variant is not implemented here. example.py --negative "..." --cfg-scale 3 uses the vanilla form.

Getting good output — read this

Three settings are not optional. Getting them wrong produces audio that is clearly broken, not subtly worse:

1. Sampler must be pingpong. This is an rf_denoiser model. Euler drives the output past full scale (+7.8 dBFS measured) and crushes dynamic range by ~10 dB.
2. Generate at 256 latent frames (23.8 s) or longer. The model's distribution_shift_options.min_length is 256. Below that, output picks up gross high-frequency content — we measured 16–27 % of energy above 10 kHz against ~1 % for correct output.
3. 8 steps, CFG scale 1.0. These are the model's own defaults for this family.

Latent frames ↔ seconds: frames = seconds × 44100 / 4096.

Usage

example.py in this repo is self-contained — four graphs, the tokenizer, numpy and coreai.runtime. No stable-audio-tools, no PyTorch, no virtualenv beyond the Core AI runtime itself.

pip install coreai-torch tokenizers numpy      # brings the Core AI runtime

# text to audio
python3 example.py "loud crackling campfire with crickets" out.wav --seconds 24

# continue or inpaint from existing audio (44.1 kHz WAV)
python3 example.py "a dog barking" out.wav --seconds 24 --init-audio in.wav --keep 12

--keep sets how many seconds of --init-audio are held as context; everything after it is regenerated from the prompt. Omit --init-audio for plain text-to-audio.

The sampler, schedule and conditioning assembly are all in that one file, ~120 lines, if you are porting to Swift. The ping-pong loop is four lines:

denoised = x - t[i] * dit(x, t[i], **cond)
x = (1 - t[i+1]) * denoised + t[i+1] * randn_like(x)

Fidelity

Measured against the original PyTorch model on identical inputs:

Graph	Cosine	SNR
DiT	1.000000000	112.6 dB
Decoder	1.000000000	45.3 dB
Encoder	1.000000000	~100 dB
End-to-end audio	0.999999981	74.1 dB

Performance

On an M-series Mac (36 GB), 23.8 s of 44.1 kHz stereo:

• 2.21 s — about 10.8× realtime
• Roughly 2× faster than the same model running under MLX with 8-bit weights

Memory scales with duration and it is the real limit, not disk. A rough guide from the same hardware: ~7.7 GB fixed plus ~0.32 GB per second of audio for the 2 B model. Check before you ask for long clips.

First load compiles the model

Core AI specialises an .aimodel for your device the first time you load it, then caches the result. That first load is slow and the cost grows sharply with decoder length — measured on the 2 B decoder, per rung: 23.8 s of audio compiles in 12 s, 47.6 s in 44 s, 95.1 s in 8.5 min, and 190.2 s in over an hour. Subsequent loads are effectively instant.

To skip it, precompile ahead of time:

xcrun coreai-build compile small-sfx.aimodel --platform macOS --preferred-compute gpu

That writes one .aimodelc per architecture, which loads without compiling (2.2 s versus 12 s on an M3 Max). .aimodelc runs only on the architecture it was built for, so the portable .aimodel is what ships here. This is the workflow Apple recommends for apps (WWDC26, "Integrate on-device AI models into your app using Core AI").

Known issue

fp16 crashes the Neural Engine, whatever the shape (ANE inference operation failed, Code=-19) — retested on macOS 27.0 build 26A5416b. Fixing the length at export does not avoid it, and nor does asking for the GPU: allowed_compute_unit_kinds always includes the Neural Engine and has no setter. These graphs therefore ship at float32, which sidesteps the bug entirely since the Neural Engine cannot execute float32 at all. Reproduced on both model sizes.

Licence

Stability AI Community License — see LICENSE.md. Free for research, non-commercial and limited commercial use; register with Stability AI for commercial use, and the licence terminates above USD $1M annual revenue. The T5Gemma text encoder is additionally subject to the Gemma Terms of Use — see LICENSE_GEMMA.md. See NOTICE for attribution.