Sentis Audio2Lipsync (com.sky.sentis.audio2lipsync)

A Unity package for on-device, audio-driven lipsync built on Unity Sentis (com.unity.ai.inference 2.6.1). It is a port of aaryansachdeva/unreal-audio2lipsync (weights): HuBERT-Large followed by a transformer decoder that predicts 52 ARKit blendshapes at 60 fps from 16 kHz audio. You don't need the Python server: the model runs inside Unity, and the server's chunking and post-processing are reimplemented in C#.

Install

The package keeps its model files in Git LFS, so install git-lfs before Unity resolves the package.

Packages/manifest.json:

"com.sky.sentis.audio2lipsync": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2lipsync.git"

You can also use Package Manager → + → Install package from git URL… with the same URL.

Requirements: Unity 6 (tested on 6000.3.23f1) and com.unity.ai.inference 2.6.1. The demo sample also needs URP, the Input System and uGUI.

Quick start

There is no model setup step. In the editor the model loads straight from the package's Model~ folder. When you build a player, a build hook copies it into StreamingAssets/Audio2Lipsync/ and removes it again after the build. To use your own model, put an audio2lipsync.sentis in StreamingAssets/Audio2Lipsync/; it takes precedence over the packaged one.

1. Add a BlendshapePlayer (it requires an AudioSource) to your character and assign its SkinnedMeshRenderers. Edit meshNames if your rig doesn't use the sample head's blendshape names.
2. Generate the frames and play them:

using Audio2Lipsync;
using Unity.InferenceEngine;

var runner = await Audio2LipsyncRunner.CreateAsync(BackendType.GPUCompute);  // loads the packaged model
runner.Warmup();                                        // one silent pass: shader compile + CFG baseline

float[] audio16k = AudioUtils.ToMono16k(clip);          // any AudioClip -> mono 16 kHz
float[] frames = await runner.GenerateAsync(audio16k, LipsyncSettings.Default); // [n * 52] @ 60 fps
player.Play(frames, clip);                              // locked to the AudioSource clock
// ...
runner.Dispose();

Demo sample

Import it from Package Manager ▸ Sentis Audio2Lipsync ▸ Samples ▸ Demo, open Scenes/Audio2LipsyncDemo.unity and press Play. Tools ▸ Audio2Lipsync ▸ Build Demo Scene rebuilds the scene from code. It also sets a default Microphone Usage Description, which iOS and macOS player builds require.

• Generate & Play (or Space) runs the selected clip (EN / DE / JA / KO).
• Record (Mic) → Stop & Lipsync runs microphone audio.
• The sliders set gain, CFG scale, smoothing window and silence gate.

Package layout

Path	Contents
Runtime/Audio2LipsyncRunner.cs	Owns the worker. Splits audio into 3 s chunks with a 30-frame crossfade and overlap-adds them. Provides warmup, Generate and GenerateAsync, which uses async readback.
Runtime/LipsyncPostProcess.cs	Port of server.py run_inference: CFG → clip → mouth-only mask → centered gain → smoothing → RMS silence gate → fades.
Runtime/BlendshapePlayer.cs	Maps ARKit names to mesh blendshapes and plays them at 60 fps. Also drives tongue_ and teeth_ variants.
Runtime/AudioUtils.cs	Mono downmix and a windowed-sinc resampler to 16 kHz.
Runtime/Audio2LipsyncModel.cs	Finds and loads the model: the package in the editor, StreamingAssets in players (UnityWebRequest on Android and WebGL).
Editor/Audio2LipsyncBuildHook.cs	Copies the model into StreamingAssets for each player build and cleans it up afterwards, including after a failed build.
Editor/Audio2LipsyncSerializer.cs	Serializes an imported ONNX to .sentis (fp32, or optional Float16/Uint8).
Model~/audio2lipsync.sentis	The model, fp32 weights, 1.3 GB.
Samples~/Demo	Sample head, dialogue clips, demo manager and a code-built UGUI scene.
Tools~/export_onnx.py, Tools~/compare_clip.py	Export to ONNX from best.pt, and comparison against the original server.

Model I/O: input is audio [1, 48000] (3 s, 16 kHz mono, raw). Output is blendshape [1, 180, 52] (ARKit order, de-normalized, not clipped).

Accuracy and speed

Measured on Apple Silicon with Unity 6000.3.23f1 and Sentis 2.6.1:

Check	Result
torch vs ONNX Runtime (3 s speech chunk)	max abs diff 6.1e-6
torch vs Sentis, packaged .sentis, GPUCompute / CPU	4.7e-6 / 4.5e-6
Original server pipeline vs Unity, full 3.87 s clip	max abs diff < 1e-4 (JawOpen corr 1.0000)
Warm time per 3 s chunk, GPUCompute / CPU	~45 ms / ~320 ms

Differences from the reference server

• Chunk normalization: HuBERT-Large normalizes the waveform with F.layer_norm(x, x.shape). The server batches its chunks, so that normalization spans every chunk in the batch and the chunks affect each other. Here each chunk is normalized on its own, which matches the server with MAX_CHUNK_BATCH=1.
• Exported layer norm: the export writes that normalization out as mean/variance ops, because the Sentis importer rejects a rank-2 LayerNormalization scale.
• Short clips: clips of 3 s or less are zero-padded to 3 s, the same way the server pads its last chunk.
• Resampling: done in C# with a windowed-sinc filter. The server uses torchaudio.functional.resample.

License

MIT — see LICENSE.md. This package includes:

• Weights converted from fotonlabs/unreal-audio2lipsync (MIT).
• The HuBERT-Large encoder from fairseq via torchaudio (MIT).
• The sample head and dialogue audio from skykim/NeuroSync_Unity (MIT).