Audio2Face-3D v2.3 Mark — MLX

NVIDIA Audio2Face-3D v2.3 (Mark) exported for the MLX Swift runtime in speech-swift. Speech audio in, timestamped facial-animation coefficients out — fully on-device on Apple Silicon.

The bundle contains the learned tensors from the official ONNX checkpoint in safetensors form plus the NVIDIA runtime metadata. The hand-written MLX graph in speech-swift runs the real model forward pass; outputs are parity-checked against the official ONNX model.

Output

Each frame carries 301 coefficients: 272 skin blendshapes, 10 tongue blendshapes, 15 jaw controls, and 4 eye controls, with a 10-value explicit + 16-value implicit emotion input (16 kHz mono audio, 8320-sample windows).

Files

File	Purpose
`audio2face3d.safetensors`	Learned tensors for `MLX.loadArrays`
`network_info.json`	NVIDIA runtime geometry/audio metadata
`model_config.json`	Runtime configuration (input strength)
`default_emotion.f32`	Default emotion vector
`graph_metadata.json`	Source ONNX graph provenance

Usage (Swift)

import Audio2Face3D

let model = try await Audio2Face3DModel.fromPretrained(
    modelId: "aufklarer/Audio2Face-3D-v2.3-Mark-MLX")
let frames = try model.frames(for: samples, sampleRate: 16_000)

Or from the speech CLI:

speech avatar-motion input.wav --model aufklarer/Audio2Face-3D-v2.3-Mark-MLX --output frames.jsonl

Identities

Audio2Face-3D-v2.3-Mark-MLX (this repo) — 301 coefficients (272 skin)
Audio2Face-3D-v2.3.1-Claire-MLX — 169 coefficients (140 skin)
Audio2Face-3D-v2.3.1-James-MLX — 169 coefficients (140 skin)

Coefficients are identity-specific: renderers need the matching rig or a retarget projection for the chosen identity.

License and attribution

The weights are NVIDIA's Audio2Face-3D v2.3 Mark model, redistributed in MLX-ready form under the NVIDIA Open Model License. All model credit goes to NVIDIA.