OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
Authors: Ziqiao Peng
Jiwen Liu
Haoxian Zhang
Xiaoqiang Liu
Songlin Tang
Pengfei Wan
Di ZHANG
Hongyan Liu
Jun He
Venue: NeurIPS 2025 spotlight ()
Keywords: Lip Synchronization
Diffusion Transformers
Talking Head Generation
Primary area: applications
TL;DR: OmniSync enables universal lip synchronization for diverse visual content using mask-free diffusion with dynamic guidance.

Abstract: Lip synchronization is the task of aligning a speaker’s lip movements in video with corresponding speech audio, and it is essential for creating realistic, expressive video content. 
However, existing methods often rely on reference frames and masked-frame inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality.
In this paper, we present OmniSync, a universal lip synchronization framework for diverse visual scenarios. Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited-duration inference while maintaining natural facial dynamics and preserving character identity.
During inference, we propose a flow-matching-based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To address the weak conditioning signal of audio, we develop a Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space.
We also establish the AIGC-LipSync Benchmark, the first evaluation suite for lip synchronization in diverse AI-generated videos. Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip sync accuracy, achieving superior results in both real-world and AI-generated videos.

OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
Ziqiao Peng'*
Jiwen Liu
Haoxian Zhang²
Xiaogiang Liu² Songlin Tang"
Pengfei Wan
Di Zhang"
Hongyan Liu Jun He
Renmin University of China
Kling Team, Kuaishou Technology Tsinghua University
Project Leader
*Corresponding Author
https://ziqqaapeng github..o//mmiin..... io/OmmiSync/
LONE
High Identity Consistency
Strong Occlusion Robustness
Rich Stylistic Diversity
Figure I: OmniSyne demonstrates universal lip synchronization capabilities, effectively handling facial occlusion, while maintaining visual consistency and generating accurate lip movements.
Abstract
Lip synchronization is the task of aligning speaker's lip movements in video with corresponding speech audio, and is essential for creating realistic, expressive video content. However, existing methods often rely reference frames and masked-framt framee--feeeae inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality. In this paper, present OmniSyne, universal lip synchronization framework for diverse visual scenarios. Our approach intro- duces mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited -duitted-durtiono inference while maintaining natural facial dynamics and preserving character identity. During infer- ence, we propose a now-matching-bee based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To ad- dress the weak conditioning signal of audio, we develop Dynamic Spatiotemporal Classifier Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space. We also establish the AIGC-L Lic--L Benchmark, the first evaluation suite for lip synchronization in diverse Al Al-generated videos.
WWok done during internship Kling Team, Kuaishou Technology.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

inference Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip syne aceuracy, achieving superior results in both real-world and Al-generated videos.
Introduction
Lip synchronization, matching mouth movements with speech audio, is essential for creating com- pelling visual content across film dubbing (5T), digital avatars [32 31 55 [ and teleconferenc- ing [24] 28 53]. With the rise of Al-generated content, this technology has evolved from a specialized technique to fundamental aspect of the video generation landscape [47 B4 20). Despite significant advances in text-to- text-to-video (T2V) models [4] 2 48 16 39 creating increasingly realistic footage,
achieving precise and natural lip synchronization remains an unsolved challenge.
Traditional lip synchronization approaches rely heavily on reference frames combined with masked- frame inpainting [33]55]]]] ( [O]. This methodology extracts appearance information from reference frames to inpaint masked regions in target frames-a process that introduces several critical limitations. These methods struggle with head pose variations, identity preservation, and artifact elimination, especially when target poses differ significantly from references 30
Furthermore, the dependence on explicit masks cannot fully prevent unwanted lip shape leakage, compromising synchronization quality and restricting upplicability across diverse visual representa- tions []] The challenges intensify in the context of audio-driven generation. Unlike strong visual cues, audio signals provide relatively weak conditioning, making precise lip synchronization diffi- cult [40]. Additionally, existing methods rely on face detection and alignment K] techniques that break down when applied to stylized characters and non-human entities, precisely the diverse content that modern text-to-video models excel at generating.
This technical gap compounded by the absence of standardized evaluation frameworks for lip sync in stylized videos. Current benchmarks [22 focus almost exclusively on photorealistic human faces in controlled settings, failing to capture the visual diversity inherent in Al-generated videos.
To address these challenges. we introduce OmniSync, a universal lip synchronization framework designed for diverse videos. Our approach eliminates reliance on reference frames and explicit masks through a diffusiio--ased direct video editing paradigm. In addition, we establish AIGC-LipSyne Benchmark, the first comprehensive evaluation framework for lip synchronization across diverse AIGC contexts. OmniSync's ' technical approach is built upon three key innovations:
First, we implement mask-free training paradigm using Diffusion Transformers (DiT) 26 for direct cross-frame editing. Our model learns a mapping function (Ved, Aut) + Vab. where V represents video frames and A represents audio. The indices (a b, c d) represent different segments sampled from the same video. The model modifies only speech speech-relevane relevant regions according to target audio without requiring explicit masks references. This approach enables while maintaining natural facial dynamics and preserving character identity.
Second, we introduce a flow-matching-bae based progressive noise initialization strategy during inference. Rather than beginning with random noise [38], we inject controlled noise into original frames using Flow Matching 11), then execute only the final denoising steps. This approach maintains spatial consistency between source and generated frames while allowing sufficient flexibility for precise mouth region modifications, effectively mitigating pose inconsistency and identity drift.
Third, we develop a dynamic spatiotemporal Classifier-Fres Guidance (CFG) framework [33 that provides tine-graine control over the generation process. By adaptively adjusting guidance strength across both temporal and spatial dimensions: temporally reducing guidance strength as denoising progresses, and spatially applying gouusian-weeghhee control centered on mouth-relevant regions. This balanced approach ensures precise lip synchronization without disturbing unrelated areas.
Our contributions can be summarized as follows:
A universal lip synchronization framework that eliminates reliance on reference frames and explicit masks, enabling accurate speech synchronization across diverse visual representa- tions.
A now-matchingg-aase based progressive noise initialization strategy during inference, effectively stabilizing the early denoising process and mitigating pose inconsistency and identity drift.

---

A dynamic spatiotemporal CFG framework that provides fine- aheeggraieed control over audio influence, addressing the weak signal problem audio-driven generation.
A comprehensive AIGC-LipSyne Benchmark for evaluating lip synchronization in Al- generated content, including stylized characters and non-human entities.
2 Related Work
2.1 Audio-driven Lip Synchronization
41 7 23 12 have GAN-based Lip Synchronization. Traditional GAN-based 99] methods 83
tablished important foundations in lip synchronization. Wav2Lip [33] pioneered the use of pretrained SyncNet to supervise generator training. setting benchmark for subsequent research. DINet [5]] enhanced synchronization quality by performing spatial deformation on reference image feature maps, better preserving high-frequency details. IP-LAP [44] introduced two-stage approach that first infers landmarks from audio before rendering them into facial images. ReSyncer [10) incorporated
3D mesh priors for facial motion, effectively reducing artifacts.
have enabled significant progress in audio-driven lip synchronization. LatentSyne 17 represents
Diffusion-based Lip Synchronization. Recent advances in diffusion models 23
50
end-to-end framework based on audio-conditioned latent diffusion models without intermediate motion representation. SayAnything [22] employs denoising UNet architecture that processes video latents with audio conditioning, MuseTTal (50) proposes a novel sampling strategy that selects reference images with head poses closely matching the target.
However, these methods still rely on reference frames combined with massedd-frame inpainting, leading head pose limitations, identity preservation issues, and blurry edge generation. Our OmniSync framework addresses these limitations through a mask-free training paradigm that enables application across diverse visual representations.
2.2 Audio-driven Portrait Animation
37 49 differs fundamentally from lip syne. Portrait animation 45 8 follows an image-to-vided framework without constraints on head poses or facial expressions, eliminating the need to integrate generated content back into original video. This approach is unsuitable for gost-generation lip synchronization in video generation pipelines. In contrast, lip synchronization [33] 17 operates within video-to-video framework, modifying only lip movements while maintaining compatibility with existing footage. This represents more constrained task, requiring precise modification of lip regions while preserving surrounding facial features.
Audio-driven portrait animation
38 15
Recent models like OmniHuman-I 18 and Mocha use audio directly as a conditioning signal for image-to- tonge-oo-video text-to-video frameworks. However, due to limitations in talking head datasets, their generative capabilities don't match the versatility of advanced video generation models, This gap highlights why specialized lip synchronization for Al-generated videos remains critical.
Method
3.1 Overview
In this section, present OmniSync, universal lip synchronization framework designed for diverse visual content (Fig. 2 Our approach comprises three key components: 1) a mask-free training paradigm that eliminates dependency reference frames and explicit masks, 2) flow-mate fow-matchingg based progressive noise initialization strategy for enhanced inference stability, and 3) dynamic spatiotemporal Classifier-Fre Guidance (CFG) that optimizes lip syne while preserving facial details. The following subsections provide comprehensive explanations of each component.
3.2 Mask-Free Training Paradigm
Traditional lip synchronization methods 33 rely on masked-frame inpainting, isolating the mouth region before generating content based audio input. Despite their prevalence, these approaches

---

Mask-Frec Training Paradigm
Training Input & Target
Inpui
Text Encoder
Toeyy
Kiifle
Target Pas
Vidco Descipticn
Text Flceznce
Bllckk
Timesiop
Vileo
Frumes
Audio
[[0 | Mo
20 VAE
3
Doodeer
Input
Timestep Dependent Sampling Stralegy
Diftasion Timestp
Aadio Eaceder
Trainille
Aodle Fratues
Froezz
Vi
Mideo
Inference Stage
(Frm She)
Early Stage
Middle Stage
Late Stage
((F)
Text Frinures
Flow Meading
Spuidlly-Adapive Guidanoe
Tempenill-Adappin Chiibenee
Neiiz Generat
OnmisSyn
Focus os: Puse & Monity
Focus Lip Shape
Focas Meaiity & Deail
Lip-Syaced
Natodd Vidao
Progressive Noise Initialization
Assho Finitanee
Dynamic Spariotemporal Classiffer-Free Guidance
Proste-Paied
Arharayy Inaning Liranct
Figure 2: Overview of OmniSync. A mask-free training paradigm employs
sampling to predict the lip-synchronized targets Vab- During inference, progressive noise initialization and dynamic spatiotemporal CFG ensure consistent head pose and precise lip synchronization.
produce boundary artifacts and struggle with identity preservation. Crucially, they require explicit face detection and alignment- techhnguuse that fail with stylized characters and non-human entities.
An alternative approach is direct frame editing, which aims to transform frames according to target audio without relying on masks or references. However, this approach requires perfectly paired training data with identical head poses and identity differing only in lip movements. Such paired data is extremely rare and would severely restrict the model's generalizability to diverse visual results.
To address these limitations, we leverage the progressive denoising characteristic of diffusion models, introducing a novel training strategy that varies data sampling based on diffusion timesteps. This allows for stable learning without requiring perfectly paired examples. Our goal is learn conditional generation process mapping (Ved, Aab) + Vas through iterative denoising, where V represents video frames and represents audio.
We employ Flow Matching [19] as our training objective. Given an input video segment Vod from frames to d, and target audio segment Aub from frames a to b, our model generates the corresponding video segment Vas via the diffusion process:
Tt- = DiT( Vods Aab, t),
where v, represents the noised version of target video Vab at timestep 4, and DiT denotes our diffusion transformer, which predicts the denoised state at timestep t 1.
The CFM loss used for training is defined as:
LCF M (0) -E,,,, Vas I1lvo Vod, Aany 1) u. (
where (1 Ved, Aab, t) is the learned velocity field predicted by DiT with conditioning on input video Vod and target audio Aab. ue uz is the conditional velocity field typically defined as ue (2r//VV.)) (Vab (1 t) for the linear interpolation path x; (1 f).. + (Vat.
Timestep-Dependent Sampling Strategy. critical insight in our approach is recognizing that the diffusion process can be decomposed into distinct phases, each with different learning requirements. Specifically, early timesteps focus on generating fundamental facial structure, including pose and identity information; middle timesteps primarily generate lip movements driven by audio; while late timesteps refine identity details and textures. To capitalize on this natural progression, we utilize different datasets for distinct timesteps.

---

For early timesteps (approximately t = T), responsible for generating overall facial structure, we employ pseudo-paired data from controlled laboratory settings. These samples maintain nearly identical pose information, with variations only in lip movements, providing stable learning signals for structural features and ensuring pose alignment between input and output.
For middle and late timesteps, we transition to more diverse data, sampling from arbitrary videos. During middle timesteps, the model learns to generate lip shapes guided by audio input, whereas in late timesteps (approximately & RJ 0), it focuses on refining identity details and ensuring texture consistency. This menestep-dependeed training strategy can be formalized as:
Preeuto-painne (Vat, Vab) Purbitrary (Veat, Vab)
if > Ltreshold,
p(Ved; Voolt)
otherwise.
Here, Ppecude-paired indicates sampling from controlled datasets with minimal pose variations, while Parbitrary signifies sampling from our diverse collection of videos. The conditional generation process can be expressed mathematically as:
f
po (Vas Ved, Aub) = po Po (ooaallzo)po (FolVed, Aat)dro.
where pe (oVaslaar) represents the mapping from the fully denoised state the output video, and (xol Vod, Aas) captures the relationship between input conditions and the denoised state. Here, Zo refers to the completely denoised latent representation (at timestep t 0).
This progressive training approach aligns well with the natural learning progression of diffusion models. By providing appropriate training signals each stage, enable stable learning even without perfectly paired data, allowing our model to generalize effectively to diverse real-world scenarios while maintaining identity consistency.
3.3 Progressive Noise Initialization
Standard diffusion-based generation (38 typically begins from random noise (timestep 7) and progressively denoises toward the final output (timestep 0). However, this approach often results in subtle but noticeable pose misalignments between generated content and original video frames, creating undesirable boundary artifacts and compromising identity preservation.
The fundamental issue lies error accumulation during the early stages of diffusion. Even minor deviations in early timesteps-wher basic facial structure is being formed can lead to significant misalignments in the final output. This problem is relevant for lip synchronization, where the goal is to modify only speech-relevant regions while maintaining perfect spatial consistency elsewhere. To address this challenge, we introduce How-matching-baan -baedd progressive noise initialization strategy that transforms the traditional diffusion process.
Eloc-hat Noise Initialization. Rather than starting the diffusion process from random noise at timestep T, we initialize from original video frames with controlled level of noise. This simulates an intermediate state in the diffusion trajectory, corresponding to a normalized parameter т. The initialization is performed by adding this controlled noise to the original video frame:
Tinit FM (aaatuaVonn Voource T) = (1 T) Vseurce + TE,
(5)
where Timit is the initial noised state derived using the parameter т, Vsource is the source video frame, and ~ (0, 7) random noise. Let (start be the discrete timestep corresponding to this initialization point (7 is the total number of diffusion steps, and 7 E [0, 1)).
This initialization strategy provides two significant advantages. First, it bypasses the early stages of diffusion (from T' down to starr) where general facial structure is formed. This ensures that head pose and global structure are directly inherited from the source frame. Second, it reduces computational requirements by performing denoising only for the remaining steps, from sstart down to O.
The complete progressive denoising process can be expressed as:
Pinit
if tstarts
(6)
(IT(1
Vanunces Auages ++1 1) if fuurt > 2 0,
5

---

where Agrger is the target audio used to guide the denoising process, and t here represents discrete diffusion timesteps.
This approach effectively creates a two-stage process: (1) initialization using the flow- -atching inspired noise addition (Éq. 5 to reach a state equivalent to timestep fsart. and (2) guided denoising from Esturt that focuses on modifying mouth regions according to the target audio while preserving the overall facial structure, identity features, and head pose from the source frame. By skipping the early noisy stages where basic structures form, we maintain spatial consistency while allowing sufficient flexibility for precise mouth region modifications.
3.4 Dynamic Spatiotemporal Classifier- Guidance
Audio-driven lip synchronization faces fundamental challenge: audio signals provide relatively weak conditioning compared to visual cues [40]. Standard Classifier- Free Guidance (CFG) [3] can enhance audio conditioning, but applying uniform guidance across spatial and temporal dimensions creates an unavoidable dilemma: higher guidance scales produce more accurate lip syne but introduce texture artifacts, while lower scales preserve visual fidelity but yield less precise lip movements.
To resolve this tension, we introduce Dynamic Spatiotemporal Classifier FFeee Guidance (DS-CFG), novel approach that provides fine-grained control over the generation process across both spatial and temporal dimensions. Our method applies varying guidance strengths to different regions of the frame and different stages of the diffusion process, achieving optimal balance between lip synchronization accuracy and overall visual quality.
Opatially-Adaptiin Guidance. The key insight for spatial adaptation is that audio information primarily affects the mouth region, while other facial areas should remain largely unchanged. We implement this through Gaussian-weighted spatial guidance matrix that concentrates guidance strength around speech-relevant regions:
(x - Im ) + (y ymm
Gspatia (r. y) Whase + (wipesk - Whase) exp
2
where (Zm- Vm represents the mouth center, a controls the spread of the Gaussian distribution, Woms the baseline guidance strength applied non-mouth regions, and wpeak is the peak strength applied at the mouth center. This spatial adaptation ensures that audio conditions strongly influence lip and surrounding regions while minimally affecting other facial features.
cumporally-Adaptivy Guidance. We observe that audio conditioning plays different roles at different stages of the diffusion process. In early diffusion timesteps, strong guidance helps establish correct
lip shapes, while in later stages, excessive guidance can disrupt fine texture details. [2] 43 351 To address this, we implement& a temporally decreasing guidance schedule:
t T
((t = Wpeak
(8)
where f is the current diffusion timestep. T is the total number of timesteps, Wpeak is the maximum guidance scale, and 7 controls the decay rate, with a value of 1.5. This temporal adaptation ensures strong guidance during early and middle diffusion stages when coarse structures are formed, gradually reducing influence during later stages when fine details and textures refined.
Unified Dynamic Spatiotemporal CFG. Combining both spatial and temporal adaptations, our DS-CFG approach modifies the standard CFG formulation to:
( c, t) = co (r,, 0, t) + G Gspanial (t) (Co(Ie, C, t) - ca (I,, 0,t))
(9)
where c0(1e, c,t) t) and cæ 0, t) are the noise predictions with and without conditioning, respectively,
Through this DS-CFG, our method achieves precise control over the generation process, effectively addressing the weak audio signal problem in audio-driven generation.
4
Experiments
4.1
Experimental Settings
Datasets. We trained OmniSyne using the MEAD dataset 42 and 400-hour dataset collected from YouTube. MEAD's controlled laboratory recordings with diverse facial expressions but minimal

---

Table 1: Quantitative comparison with previous methods on HDTF Dataset.
HDTF Dataset
Full Reference Metries
No Reference Metries
Lip Syne
Method
FID
FVD 4
CSIM
NIQE + BRISQUE 4 HyperlQA t
LMD 1
LSE-C
Wav2Lip
33
14.912
543.340
0.852
6.495
53.372
45.822
10.007
7.630
VidcoRc Talking
11.868
379.518
0.786
6.333
50.722
48.476
8.848
7.180
TalkLip 41
16.680
691.518
0.843
6.377
52.109
44.393
15.954
5,880
IP-LAP [54]
9.512
325.691
0.809
6.533
54.402
50.086
7.695
7.260
Diff2Lip 25
12.079
461.341
0.869
6.261
49.361
48.869
18.986
7.140
MuseTTalk [50]
8.759
231,418
0.862
5.824
46.003
55.397
8.701
6.890
LatentSync 11
8,518
216,899
0.859
6.270
50,861
53.208
17.344
8.050
Ours
7.855
199.627
0.875
5.481
37.917
56,356
7.097
7.309
Table Quantitative comparison with previous methods on AIGC-LipSYne Benchmark.
AIGC-L.ipSync Benchmark
Full Reference Metries
No Reference Metries
Generation Success Rate
Method
Stylized
FID
FVD
CSIM t NIQE
BRISQUE HyperIQA f
All Videos
Characters
Wav2Lip
33
22.989
562,245
0.727
5.392
42.816
50,511
71.38%
26.67%
IideoReTalking
7
20.439
329.460
0.669
5.947
45.047
48.645
48.78%
7.78%
TalkL.p
41
31.180
619.179
0.754
5,239
41.692
50.608
52.36%
34.44%
IP-LAP
54
14.686
247.402
0.796
5.546
45.153
53.174
45.53%
6.67%
Dift2Lip
25
23.542
403.149
0.692
5.440
42.442
50.335
74.63%
36.67%
MuseTTalk
[50
17.668
297.621
0.667
4,935
36.017
58,334
92.20%
67.78%
LatentSyne
17
15.374
263.111
0.751
5.342
41.917
54.648
74.96%
35.56%
Ours
10.681
211.350
0.808
4.588
25.485
61.906
97.40 %
87.78%
head movement provided ideal data for training early denoising stages, while the YouTube dataset enhanced generalization across varied weal-world conditions for middle and late stages.
To address the limitations of existing benchmarks that focus solely on real-world vidcos with frontal views and stable lighting, created the AIGC-LipSyne Benchmark. This comprehensive text-to-
evaluation framework comprises 615 human-centric videos generated by video models such as Kling, Dreamina, Wan 39]. and Hunyuan [16]. The benchmark specifically captures challenging visual scenarios such as large facial movements, profile views, variable lighting. occlusions, and stylized characters-condiiieee conditions that traditional benchmarks fail address. Details about benchmark construction can be found in the supplementary materials.
Implementation Details. We implement our OmniSync model using the Diffusion Transformer ar- chitecture. The model is trained on a combined dataset for 80,000 steps using AdamW optimizer [21] with learning rate of 1e-5. Training is completed in 80 hours using 64 NVIDIA A100 GPUs with batch size of 64. Audio features are extracted via a pre-trained Whisper encoder, and text conditioning utilizes T5 encoder. Training employs the timestep-dependent sampling threshold fuheshold 850. The experimental results indicate that excessive thresholds induce significant misalignment while in- sufficient values will leak the original lip shape. During inference adopt Now-matching-bart based progressive noise initialization starting at 7 0.92, followed by 50 denoising steps.
4.2 Quantitative Evaluation
We evaluate OmniSync against state-of-the-at methods including Wav2Lip (33, VidcoReTalking 77]. TalkLip 4T. IP-LAP [54]. Dift2Lip 225, MuseTalk [50), and LatentSyne 17 using a comprehensive suite of metrics. For visual quality assessment, we employ FID (Fréchet Inception Distance) to measure frame-level fidelity, FVD (Fréchet Video Distance) for temporal consistency, and CSIM (Cosine Similarity) to quantify identity preservation. Perceptual quality is assessed using no-reference metries including NIQE (Natural Image Quality Evaluator), BRISQUE (Blind/Referrnceelss Image Spatial Quality Evaluator), and HyperiQA [36]. For audio-visual synchronization, we measure LMD (Landmark Distance) between predicted and ground truth facial landmarks in the mouth region, and LSE-C (Lip Syne Error Confidence) to evaluate lip synchronization quality.

---

Ours
IP-LAP
LatentSyne
MuscTalk
VidkoReTalking
Diff2Lip
Wav2Lip
Iaput Video
than!
Goyar Agan!
Error
Manager idy!
Error
Evvery
Chaneel PiP
Error
Error
Error
Figure 3: Qualitative comparison with previous methods across diverse subjects and phonemes. Our approach produces accurate lip synchronization and better identity preservation.
Table 3: User study results comparing various audio-driven lip-syne methods.
Video ReTalking
Latent Sync
Metric
Wav2Lip
TalkLip
IP-LAP
Diff2Lip
Muse
Ours
Lip Syne Accuracy
2.684
2.769
1.940
2.359
3.000
2.632
3.812
3,923
Identity Preservation
2.410
2.786
2.222
2.991
2.889
3.197
3.658
4.128
Timing Stability
2.556
2.376
2.162
3.034
2,812
3.145
3.581
4.043
Image Quality 2.017
2.607
1.889
3.171
2,402
3.094
3.632
4.051
Video Realism
2.120
2.419
1.838
2.316
2,479
2.761
3.453
3.872
For the AIGC-LipSyne benchmark, we report the Generation Success Rate across 615 videos and specifically for stylized characters. This metric shows the percentage of videos that are successfully synchronized and pass human verification. This evaluation is essential for universal lip synchro- nization in Al-generated content, where traditional metrics may not fully capture the challenges of stylized characters, extreme poses, and other atypical visual conditions.
The experimental results in Tab. and Tab. 2 demonstrate that our approach achieves superior performance on multiple metrics. the HDTF dataset, our method reduced FID by 7.8% and FVD by 8.0% compared to LatentSync, with remarkable 23.2% improvement in BRISQUE over Diff2Lip. For lip synchronization, we achieved the lowest LMD, outperforming IP-LAP by 7.8%, while LatentSync maintained slight edge in LSE-C due to its SyncNet-based loss constraint.
On the challenging AIGC-LipSync benchmark. OmniSyne demonstrated exceptional capabilities with 30.5% FID reduction and 19.7% FVD reduction compared to LatentSyne, alongside improved identity preservation. Most significantly, our method achieved 97.4 97.40% Generation Success Rate across all videos substantially higher than MuseTaak (92.20%) and other methods (below 75%). For stylized characters, our success rate of 87.78% outperformed MuseTaak (67.78%), demonstrating OmniSync's 'ss capability to handle diverse visual representations including stylized characters.
4.3 Qualitative Evaluation
present qualitative comparisons between OmniSync and existing methods in Fig. Our approach produces more natural facial expressions and superior lip synchronization. Due to lip shape leakage, IP-LAP 54 and LatentSync [17] frequently fail mouth shape modification, resulting in poor lip synchronization effects. MuseTalk [50] and VideoRe Takking [7] modify lip movements but

---

Table 4: Ablation study for our method.
Methods
FID 4
FVD 4
CSIM
NIQE
BRISQUE J
HyperIQA 1
LSE--
Ours
15.710
287.168
0.814
5.321
29.588
57.288
7.06
w/o Cemestep-Dependdee
21.552
549.768
0.727
5.462
30.346
56.204
7.00
Sampling Strategy
w/o Progressive Noise
16.731
361.282
0.805
5.349
29.789
56.511
7.03
Initialization
w/ Low Static CFG
5.359
29.724
56.568
4.16
w/ High Static CPG
22.725
348.335
0.782
5.473
29.678
56.289
7.10
wio Depeedep- Sampling Strategy
Static
High Static DS-CFG
Sampling Strategy
Figure 4: Ablation study for timestep-dependent sampling strategy and different CFG settings.
commonly exhibit lip sync errors, mouth artifacts, and identity drift, particularly in challenging or stylized cases. In contrast, OmniSyne consistently maintains identity details and generates realistic, expressive lip movements, demonstrating robust performance. Our approach effectively balances audio and visual cues, addressing the challenge of weak audio conditioning.
frequently lose identity and visual quality. Diff2Lip 25 and Wav2Lip
33
User Study. To assess perceptual quality, we conducted user study with 39 participants evaluating 32 video sets generated by OmniSync and seven competing methods, with standardized Cronbach's a coefficient of 0.98. Participants rated each video on a 5-point Likert scale across five criteria: Lip Sync Accuracy, Character Identity preservation, Timing Stability, Image Quality, and Video Realism. As shown in Tab. 3 OmniSync outperformed all competitors across all metrics, achieving superior scores in Lip Sync Accuracy (3.923 vs. 3.812 for LatentSync). Character Identity (4. (4.128 vs. 3.658), Timing Stability (4.043 vs. 3,581), Image Quality (4.051 vs. 3.632), and Video Realism (3.872 vs. 3.453). These results confirm OmniSync's superior ability generate high-quality talking videos.
4.4
Ablation Study
To clarify the contributions of each core component in framework, conduct ablation study targeting three key modules: the timestep-dependent data sampling strategy, progressive noise initialization, and the Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism. Quantitative results are presented in Tab. 4 and corresponding visual examples are shown in Fig.
Removing the timestep-dependee tion and pose consistency, with 10.7% decrease CSIM and substantial increases FID and FVD. As shown in Fig. 4 without this sampling strategy, the generated faces often exhibit clear mismatches with the original image, including noticeable facial misalignment issues. This validates our design choice of aligning pseudo-paired data with early diffusion steps, which proves critical for generating structurally stable outputs. Similarly, removing progressive noise initialization leads to evident temporal inconsistencies and increase in FVD, confirming the importance of our flow-matching initialization in preserving spatial anchoring and motion coherence.
sampling strategy results in significant drop in identity preserva-
We also compare our proposed DS-CFG with both low and high static CFG settings. As illustrated in Fig. 4 low CFG provides insufficient audio conditioning, resulting in articuaate lip movements (LSE-C: 4.16), whereas high CFG improves synchronization ILSE-C: 7.10) but introduces noticeable artifacts and distortions facial details. contrast, DS-CFG achieves an optimal balance by applying strong localized guidance in early diffusion stages and gradually reducing it in later steps. These results confirm that dynamic control across temporal and spatial dimensions is essential for producing expressive and visually coherent lip synchronization in generative video content.

---

5 Conclusion
In this paper, we introduce OmniSyne, a universal lip synchronization framework for diverse content that addresses critical limitations of traditional approaches. Our three key innovations a mask- free training paradigm eliminating mask dependencies, a now-matching--eee -based progressive noise initialization strategy ensuring identity preservation, and dynamic spatiotemporal Classifier-Free Guidance balancing synchronization with visual coalitt--oollctivvely enable precise lip movements across diverse visual representations. To support systematic evaluation in this field, we establish the AIGC-LipSyne Benchmark, the first comprehensive framework for assessing lip synchronization in varied AIGC contexts. Extensive experiments demonstrate OmniSync's superior performance across challenging scenarios, establishing a robust foundation for integrating precise lip synchronization into the broader AI video generation ecosystem.
Acknowledgments
This work was supported by National Natural Science Foundation of China (NSFC) under Grant Nos. 62436010, 72572090, 62172421 and 62572474. This work also supported by the Outstanding Innovative Talents Cultivation Funded Programs 2023 of Renmin University of China.
References
[1] Antoni Bigata, Rodrigo Mira, Stella Bounareli, Michal Stypułkowski, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Keysync: A robust approach for leakage-free lip synchro- nization in high resolution. arXiv preprint arXiv:2505.00 004497 2025.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 2023.
[3] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017.
[4] Haoxin Chen, Menghan Xia. Yingging He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang. et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.1955 2023.
[5] Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang, Yuan Zhang, Pengfei Wan, Di Zhang, and Shuai Li. Cafe-talk: Generating 3d talking face animation with multimodal coarse-and fine-grained control. arXiv preprini arXiv:2503. /4517, 2025.
[6] Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echominic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages
2025.
[7] Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[8] Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. arXiv preprint 4rXiv:2410.07711. 2024.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1:139-1444 2020.
[10] Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, et al. Resyncer: Rewiring style- generator for unified audio-visually synced facial performer. In European Conference on Computer Vision,
pages 348-367. Springer, 2024.
10

---

[11] Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang. Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding. Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 1505-1515, 2023.
[12] Anchit Gupta, Rudrabha Mukhopadhyay, Sindhu Balachandra, Faizan Farooq Khan, Vinay P Namboodiri, and CV Jawahar. Towards generating ultra-high resolution talking-face videos with lip synchronization. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 5209-5218, 2023.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207. /2598, 2022.
arXiv preprint
[14] Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation. arXiv preprint 2rXiv:2411.16331 2024.
[15] Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency, arXiv preprint arXiv:2409.0m 02634, 2024.
[16] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.
[17] Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsyne: Audio conditioned latent diffusion models for lip sync. arXiv preprint arXiv:2412.09262, 2024.
[18] Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang, Omnihuman-l: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025.
[19] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2210.02747, 2022.
[20] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139-2211 22149, 2024.
[21] Ulya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017.
[22] Junxian Ma, Shiwen Wang, Jian Yang, Junyi Hu, Jian Liang, Guosheng Lin, Kai Li, Yu Meng, al. Sayanything: Audio-driven lip synchronization with conditional video diffusion. arXiv preprint 25x. 2025.
[23] Yifeng Ma, Suzhen Wang. Zhipeng Hu, Changjie Fan, Tangjie Lv,, Yu Ding, Zhidong Deng, and Xin Yu. Styletalk: One-shot talking head generation with controllable speaking styles. In P Proceedings of the AAAI conference artificial intelligence, volume 37, pages 1896-1904, 2023.
[24] Ming Meng, Yufei Zhao, Bo Zhang, Yonggui Zhu, Weimin Shi, Maxwell Wen, and Zhaoxin Fan. A comprehensive taxonomy and analysis of talking head synthesis: Techniques for portrait generation, driving mechanisms, and editing. arXiv preprint arXiv:2406. 100553330..053, 2024.
[25] Soumik Mukhopadhyay, Saksham Suri, Ravi Teja Gadde, and Abhinav Shrivastava. Diff2lip: Audio conditioned diffusion models for lip-synchronvvation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 5292-5302, 2024.
(26) William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF international conference computer vision, pages 4195-4205, 2023.
11

---

[27] Ziqiao Peng, Yi Chen, Yifeng Ma, Guozhen Zhang, Zhiyao Sun, Zixiang Zhou, Youliang Zhang. Zhengguang Zhou, Zhaoxin Fan, Hongyan Liu, al. Actavatar: Temporally-awas precise action control for talking avatars. arXiv preprint arXiv:2512. 19546, 2025.
[28] Ziqiao Peng. Yanbo Fan, Haoyu Wu, Xuan Wang. Hongyan Liu, Jun He, and Zhaoxin Fan. Dualtalk: Dpal-speaker interaction for 3d talking head conversations. arXiv preprint arXiv:2505. 18096, 2025.
[29] Ziqiao Peng, Wentao Hu, Junyuan Ma, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Hui Tian, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk++: High-fidelity and efficient synchronized talking heads synthesis using gaussian splatting, arXiv preprint arXiv:2506. 14742, 2025.
[30] Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 666-676, 2024.
[31] Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Selftalk: A self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5292-5301, 2023.
[32] Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 20687-20697. 2023.
[33] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484-492, 2020.
[34] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv::209. 14792, 2022.
[35] Michat Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięha, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091-5100, 2024.
[36] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge. Jinqiu Sun, and Yanning Zhang Blindly assess image quality in the wild guided by self-adaptive hyper network. In Proceedings of the IEEE/CFF conference on computer vision and pattern recognition, pages 3667-3676, 2020.
[37] Zhiyao Sun, Ziqiao Peng. Yifeng Ma, Yi Chen, Zhengguang Zhou, Zixiang Zhou, Guozhen Zhang, Youliang Zhang, Yuan Zhou, Qinglin Lu, et al. Streamavatar: Streaming diffusion models for real-time interactive human avatars. arXiv preprint arXiv:2512.22065, 2025.
[38] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference Computer Vision, pages 244 260. Springer, 2024.
[39] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-We Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative 2025.
models. arXiv preprint ar arXiv:2503.20:
[40] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.0251/1 2024.
[41] Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pages 14653-14662 2023,
12

---

[42] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuogian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio- visual dataset for emotional talking-face generation. In European conference on computer vision, pages 700-717. Springer, 2020.
[43] Xi Wang, Nicolas Dufour, Nefeli Andreou, Marie-Paule Cani, Victoria Fernández Abrevaya, David Picard, and Vicky Kalogeiton. Analysis of classifier-free guidance weight schedulers. arXiv preprint arXiv:2404.13040, 2024.
[44] Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, et al. Mocha: Towards movie-grade talking character synthesis. arXiv preprini arXiv:2503. 23307, 2025.
[45] Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio synthesis of photore- alistic portrait animation. arXiv preprint arXiv:2403. 2024.
[46] Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang. Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406. 000000 2024.
[47] Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2 10157, 2021.
[48] Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 060722 2024.
[49] Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, and Limin Wang. Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions. arXiv preprint arXiv:25/1 1.03344, 2025.
[50] Yue Zhang, Minhao Liu, Zhaokang Chen, Bin Wu, Yubin Zeng, Chao Zhan, Yingjie He, Junxin Huang, and Wenjiang Zhou. Musetalk: Real-time high quality lip synchronization with latent space inpainting. arXiv preprint arXiv:2410. 10122, 2024.
[51] Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, and Yu Ding. Dinet: Deformation inpainting network for realistic face visually dubbing high resolution video. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 3543-3551, 2023.
[52] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow- Flow-guided one-shot talking face generation with righ-resonution audio-visual dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3661-3670 2021
[53] Zongzheng Zhang, Jiawen Yang, Ziqiao Peng, Meng Yang, Jianzhu Ma, Lin Cheng, Huazhe Xu, Hang Zhao, and Hao Zhao. Morpheus: A neural-driven animatronic face with hybrid actuation and diverse emotion control. arXiv preprint wxXiv22507.16645 2025.
[54] Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao Liang Lin, and Guanbin Li. preetering talking face generation with landmark and appearance priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2023.
[55] Xukun Zhou, Fengxin Li, Ziqiao Peng, Kejian Wu, Jun He, Biao Qin, Zhaoxin Fan, and Hongyan Liu. Meta-lcarning empowered meta-face: Personalized speaking style adaptation for audio-driven 3d talking face animation. arXiv preprint arXiv:2408. 09357, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction fully and accurately reflect the scope and contributions of the paper, including the development reference-free diffusion transformer framework (OmniSync), a Now-matching--oet dynamic spatiotemporal CFG method, and the introduction the AIGC-LipSyne Bench- mark. All claims are well-supported by theoretical exposition and extensive experimental results.
noise initialization strategy,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The supplementary materials discuss the limitations of our method. We highlight, for example, that while our framework shows strong generalization on diverse AIGC scenarios, performance on certain rare, highly stylized, or non- non-human characters may still pose challenges.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper is focused on algorithmic and applied contributions; no formal theorems or proofs are required.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All details necessary to reproduce the main experimental results are included in the Sec. 4.1, such model architecture, dataset details, evaluation metrics, training schedules, hardware, and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
15

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Due to ongoing anonymization and preparation, the benchmark will be made publicly available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
popllccgguide.. httes//oooSuumiiiiooiiiieee oonooicc)y for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips wttp::cccccpccggddceeeceeeeeepee htapl/ic/guides//ooeeeiee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all experimental settings in the Sec. 4.1 and 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
16

---

Justification:
This paper reports the statistical significance of user study.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The compute resources (GPU type and count, training time, batch size) required for the experiments are detailed in the Sec. 4.1.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurips.ceem
http://euriippc.c. htweerpppp..c/ppllcc/ccccccciiie
Answer: [Yes]
Justification:
The research complies with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [Yes]
Justification: The broader impacts of our work are discussed in the paper. OmniSyne enables more robust and flexible lip synchronization in Al-generated video, benefiting content creation, accessibility, and virtual communication. We acknowledge the potential risk of misuse (e.g., deepfakes) and recommend responsible deployment and further safeguards.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high-risk models or datasets are publicly released at submission time.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external datasets and code used in this work are properly cited and credited, and used in accordance with their respective licenses.
Guidelines:
18

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new AIGC-LipSyne Benchmark is thoroughly described in the paper. Upon publication, documentation and access instructions will be provided to the community,
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We described the details of the user study in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
19

---

Justification:
No IRB review was required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "eBzf5Ayonv", "reviewer": "Reviewer_h8zb", "summary": "This paper proposes OmniSync, a universal lip synchronization framework designed for various visual scenarios. The framework uses diffusion transformer models for mask-free training and direct frame editing, thus eliminating the need for explicit masks. Overall, the paper is well-written with a clear and logical explanation of the motivation behind the framework. However, its relevance to the question is unclear, and a few aspects are puzzling.", "questions": "a. Greater attention should be paid to the precision of language and image descriptions. The effectiveness of the proposed method should be presented objectively, without overstating its impact.\n\nb. The authors should make both the model and the dataset publicly available. Providing an anonymized repository during the review stage would not compromise the double-blind process and would greatly improve the manuscript’s transparency.\n\nc. It is recommended to include key results in the abstract and introduction to better highlight the strengths of the proposed method and help readers quickly grasp its advantages.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\na. The manuscript is clearly structured, and the methodology demonstrates some level of novelty. \n\nb. The benchmark dataset provided by the authors could serve as a useful resource for future research in this field.\n\nWeaknesses:\n\na. The abstract states that the method enables mask-free training and unlimited-duration inference while preserving facial dynamics and character identity. However, the description is somewhat vague, and it's not entirely clear how these characteristics relate to the task of role description. This lack of clarity also extends to the Introduction. While the unlimited inference capability is noted, a clearer explanation of how the approach maintains natural facial dynamics would be helpful.\n\nb. The methodology section is concise and appropriate for a conference setting, but it lacks important details. Since the authors state that the code and dataset will be open-sourced, with no privacy or confidentiality concerns, an anonymized repository should have been provided at the review stage to clarify the implementation of key modules.\n\nc. From the subsequent descriptions, it appears that the proposed approach primarily focuses on aligning lip movements with speech and maintaining consistency. However, in several examples shown in Figure 1, the model also seems to exhibit gesture or interaction-related actions. It is unclear whether this is due to selective case presentation or if the model is indeed capable of generating interactive gestures. This should be clarified.", "was_revised": true, "final_justification": "The manuscript received positive feedback from different reviewers. The authors also provided a detailed response, which addressed most of my concerns. After reading the other reviewers' comments and the authors' response, I believe the manuscript is suitable for publishing in NeurIPS, so I raise my rating from 4 to 5.", "rebuttal": "# Response to Reviewer h8zb\n\nWe sincerely thank the reviewer for the constructive feedback and thoughtful evaluation. We address each concern systematically below:\n\n---\n\n### **Q1: Clarity of facial dynamics preservation**\n\nThank you for your valuable feedback. We appreciate the opportunity to clarify these points. \n\n**Unlimited-duration inference:** Our mask-free framework achieves this capability because, unlike previous diffusion-based methods that rely on a single reference frame and mask-based inpainting, we use the current frame itself as reference at each timestep. This design fundamentally eliminates the accumulation of alignment or identity drift errors over time. Consequently, our model can perform frame-wise editing for videos of arbitrary duration without introducing compounding artifacts or identity inconsistencies. We have experimentally validated this on sequences up to 18,000 frames (10 minutes) without quality degradation.\n\n**Natural facial dynamics preservation:** Our method leverages a progressive denoising process guided by both temporal (audio-driven) and spatial (mouth-centric) cues, ensuring that only lip regions are modified in alignment with the audio, while preserving all other facial features and dynamics present in the source video. Our ablation studies demonstrate that this approach maintains both identity and natural motion, which can be objectively observed in our results.\n\nWe acknowledge the importance of precise language and objective presentation. We have carefully ensured throughout the paper that our method's strengths and limitations are presented accurately. We will refine the descriptions for enhanced clarity in the revision.\n\n---\n\n### **Q2: Implementation details and code availability**\n\nWe appreciate your emphasis on reproducibility. While we will make effort to release the source code of our project, rebuttal stage restrictions currently prevent us from providing additional links or an anonymized repository.\n\nTo ensure reproducibility, we provide comprehensive implementation details:\n\n**Model architecture**: Our system adopts a DiT-based latent video diffusion framework with 12 transformer blocks (hidden dimension: 1024, MLP dimension: 4096, 16 heads, RMSNorm, learned 3D positional embeddings), and uses a 3D VAE encoder/decoder (6 layers, 4 latent channels, 1/8 spatial reduction, KL=0.05) for latent representation. Audio is conditioned via a pretrained Whisper encoder (1024-dim features, segmented to frame-length, linearly projected for DiT cross-attention), and additional text conditioning is provided by T5-large (768-dim embeddings) for video descriptions.\n\n**Training configuration**: Training is conducted for 80,000 steps with batch size 64 (gradient accumulation=2, effective batch=128) over 64 NVIDIA A100 GPUs (~80 hours). We use AdamW optimizer (lr=1e-5, weight decay=0.01, betas=0.9/0.999). Data consists of both the MEAD dataset (for early-timestep pseudo-paired structural supervision) and a 400-hour collection of YouTube videos (for diverse mid/late-timestep audio-lip mappings). Training uses a timestep-dependent sampling strategy ($t_{threshold}$=850, $T$=1000): for $t$>850, pseudo-paired ($V_{cd}$, $V_{ab}$) are sampled from MEAD to ensure pose/identity stability, while for $t$≤850, arbitrary ($V_{cd}$, $V_{ab}$) pairs from YouTube broaden lip shape and context diversity.\n\n---\n\n### **Q3: Gesture generation capabilities**\n\nThank you for raising this point. Our method is specifically designed and optimized for audio-driven lip synchronization while preserving all original head pose and non-lip facial dynamics present in the input video. \n\nThe gesture and interaction-related actions observed in Figure 1 are **not generated by our model**; rather, they are present in the source videos and faithfully retained by our approach. We intentionally selected challenging cases featuring hand or object occlusions to demonstrate our method's robustness in complex real-world scenarios.\n\nThis design choice highlights a key advantage of our mask-free approach: maintaining natural scene dynamics even under partial occlusions, where traditional mask-based methods would fail.\n\n---\n\n### **Q4: Language precision and key results**\n\nWe appreciate your suggestion regarding result presentation. We will revise the abstract and introduction in the revision to more prominently feature key quantitative results."}, {"review_id": "DPZgEe0SRg", "reviewer": "Reviewer_zvtB", "summary": "This paper introduces OmniSync, a novel and comprehensive framework for universal lip synchronization designed to work across a diverse range of videos, including stylized and AI-generated content. The work makes several significant contributions: (1) A mask-free training paradigm based on Diffusion Transformers that eliminates the need for reference frames, enabling robust direct video editing. (2) A flow-matching-based progressive noise initialization strategy for inference, which improves pose and identity consistency. (3) A Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism for fine-grained control over audio conditioning. (4) The creation of the AIGC-LipSync Benchmark, a new and much-needed evaluation suite for lip-sync in challenging, modern AI-generated videos.\n\nThe paper is exceptionally strong, addressing critical limitations of existing lip-sync methods with a series of well-motivated and technically sophisticated innovations. The experimental results, validated on both standard and their new, challenging benchmark, are outstanding and clearly demonstrate state-of-the-art performance. This is a top-tier submission that pushes the boundaries of lip synchronization.", "questions": "1. Could you provide the inference speed of OmniSync (e.g., in FPS on an A100 GPU)? Are there clear avenues for accelerating inference, for instance, by using consistency models or reducing the number of denoising steps, and how does this trade-off with quality?\n\n2. The supplement mentions a \"center-biased crop\" as a fallback when facial landmarks cannot be detected. While this is a reasonable heuristic, it would be interesting to know how often this fallback is triggered on the AIGC-LipSync benchmark, especially for non-human characters, and how it impacts performance in those cases.\n\n3. Line 147 indicates that \"For middle and late timesteps, we transition to more diverse data, sampling from arbitrary videos\", is this means that Vab from video1 and Vcd can from totally different video2?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles the highly relevant problem of \"universal\" lip synchronization, extending beyond photorealistic human faces to the diverse and stylized content common in modern AI video generation. The core idea of a mask-free, reference-free direct editing framework is a major conceptual leap from traditional inpainting-based methods and addresses their fundamental limitations.\n2. The proposed method is novelty, which includs three main technical innovations: 1) timestep-dependent sampling strategy, 2) progressive noise initialization and 3) dynamic spatiotemproal cfg.\n3. The authors proposed a new challenging benchmark -- AIGC-LipSync, which provide a valuable resource for the community and enables a much more rigorous evaluation of modern lip-sync methods.\n4. The experimental results are impressive. The shown videos demonstrate strong qualitative excellence compared to other methods.\n5. The ablation study is comprehensive and demonstrate the effectiveness of each design proposed.\n6. The paper is very well-written, with clear explanations of complex concepts.\n\nWeaknesses:\n1. The paper mentions using 50 denoising steps for inference but does not report the inference speed. Given the use of a large Diffusion Transformer and an iterative process, it is likely not real-time. While this doesn't diminish the research contribution, a brief mention of the computational cost of inference would provide a more complete picture of the method's practical applicability.", "was_revised": true, "final_justification": "Based on its novelty and the interesting nature of the task, I recommend strong acceptance.", "rebuttal": "# Response to Reviewer zvtB\n\nWe sincerely thank the reviewer for the positive evaluation and insightful questions. We address each concern systematically below:\n\n---\n\n### **Q1: Inference speed and acceleration strategies**\n\nThank you for the question and suggestion. We acknowledge the importance of reporting inference speed for practical deployment considerations.\n\n**Current Performance:** On an NVIDIA A100 GPU, OmniSync generates a 5-second video in approximately **3.5 minutes** using the default 50 denoising steps. By incorporating lightweight acceleration techniques such as sequence parallelism and test-time caching (e.g., TeaCache), and leveraging our flow-matching-based progressive noise initialization to reduce the number of denoising steps, we can shorten inference time to approximately **1 minute** for a 5-second video without noticeable quality loss.\n\n\n**Future Acceleration Directions:** Recent advances such as consistency models and distillation-based methods present promising directions to further reduce inference latency. Integrating these techniques is feasible and forms part of our ongoing research; we expect significant acceleration with minimal quality degradation.\n\nThank you for highlighting this important practical aspect—we will include detailed performance benchmarks in the revised manuscript.\n\n---\n\n### **Q2: Center-biased crop fallback frequency and impact**\n\nThank you for this valuable question regarding our fallback mechanism's behavior and performance impact.\n\n**Frequency Analysis:** On the AIGC-LipSync benchmark, our center-biased crop fallback is triggered in approximately **7.29% of cases**, primarily occurring with highly stylized or non-human characters where facial landmark detectors fail to provide reliable keypoints.\n\n**Performance Impact:** Qualitative inspection demonstrates that this fallback mechanism does not introduce noticeable artifacts or degrade lip synchronization quality. This robustness stems from our mask-free and reference-free design philosophy, which reduces dependency on precise facial landmark localization compared to traditional mask-based approaches.\n\nWe will clarify both the trigger frequency and performance impact of this fallback mechanism in the revised paper. Thank you for this constructive suggestion.\n\n---\n\n### **Q3: Timestep-dependent sampling clarification**\n\nThank you for raising this clarification regarding our training data sampling strategy.\n\nIn our framework, both $V_{ab}$ and $V_{cd}$ are **always sampled from the same original video** to ensure consistent identity and detailed facial appearance preservation. The segments may differ in head pose or lip movements, but using entirely different videos for $V_{ab}$ and $V_{cd}$ would break identity consistency and hinder the learning of fine facial details. This same-video constraint is crucial for maintaining the temporal and identity coherence that enables our method's performance on diverse content types.\n\nWe appreciate you pointing out this potential confusion and will revise the manuscript text to make this requirement clearer and more explicit. Thank you for this valuable feedback."}, {"review_id": "xVzYiE8nbN", "reviewer": "Reviewer_X2Zu", "summary": "The paper introduces OmniSync, a groundbreaking universal lip synchronization framework that effectively addresses the challenges of aligning lip movements with audio in diverse visual scenarios, including AI-generated videos and stylized characters. Leveraging Diffusion Transformers, OmniSync eliminates the need for traditional masked-frame inpainting, utilizing a mask-free training paradigm that enhances robustness and adaptability. Key innovations include a timestep-dependent sampling strategy, flow-matching-based progressive noise initialization, and Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG). Additionally, the authors present the AIGC-LipSync Benchmark, a comprehensive evaluation framework for assessing lip synchronization in varied AI-generated content.", "questions": "Threshold Selection and Sensitivity: The choice of $t_threshold$ for timestep-dependent sampling appears critical. Can you provide more principled guidance on how to select this threshold? How sensitive is the method's performance to this parameter, and does the optimal threshold vary across different types of content (realistic vs. stylized)?\n\nComputational Efficiency Analysis: How does the computational cost of OmniSync compare to existing GAN-based methods like Wav2Lip or VideoReTalking? Given that diffusion models typically require multiple denoising steps, what is the inference time comparison, and are there opportunities for acceleration without sacrificing quality?\n\nTraining Data Requirements: The method relies on pseudo-paired data from MEAD for early timesteps. How much pseudo-paired data is actually needed for effective training? Could the method work with less controlled data, or could synthetic pseudo-pairs be generated to reduce dependence on laboratory-recorded datasets?", "limitations": "yes.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1.Innovative Approach: The use of a mask-free training paradigm is a significant advancement over traditional methods, allowing for more robust performance across diverse content types.\n\n2.Comprehensive Evaluation: The introduction of the AIGC-LipSync Benchmark fills a crucial gap, providing a standardized method for evaluating lip synchronization in AI-generated videos, which enhances the paper’s relevance and impact.\n\n3.High Video Quality: The generated outputs exhibit exceptional visual quality, producing realistic and expressive lip movements that enhance overall viewer experience.\n\n4.Strong Empirical Results: Extensive experiments demonstrate OmniSync's superior performance in both visual quality and lip synchronization accuracy, outperforming existing methods across multiple metrics.\n\n5.Practical Relevance: The framework effectively addresses real-world challenges in lip synchronization, making it applicable for modern AI-generated content, including stylized characters and non-human entities.\n\nWeaknesses:\n\n1.Limited Theoretical Insights: While empirical results are strong, further theoretical analysis could strengthen the understanding of certain design choices, such as the selection of parameters like $t_threshold$ .\n\n2.Computational Efficiency: A detailed comparison of computational costs with existing methods would provide a clearer picture of the trade-offs involved.\n\n3.Dataset Limitations: The training relies heavily on MEAD dataset for pseudo-paired data, which may limit diversity in the early-stage training.", "was_revised": true, "final_justification": "Thank you for your detailed response, which has resolved all my questions. As Reviewer zvtB commented, the paper is exceptionally strong. I will raise my rating", "rebuttal": "# Response to Reviewer X2Zu\n\nWe sincerely thank the reviewer for the evaluation and questions. We address each concern systematically below:\n\n---\n\n### **Q1: Threshold selection and theoretical understanding**\n\nThank you for your questions regarding the theoretical understanding and parameter sensitivity, especially concerning the selection of $t_{threshold}$ in our timestep-dependent sampling strategy.\n\nWe agree that the timestep threshold is crucial for balancing the use of pseudo-paired and diverse data. In practice, **$t_{threshold}$ represents the diffusion timestep where the transition occurs from structure/pseudo-paired to diverse/speech-driven sampling**. Early diffusion steps predominantly control global structure (pose, identity), while middle and late steps drive finer details (such as lip shape and texture). We provide detailed explanation of this mechanism in the supplementary materials (Sec. 2 Training Procedure Details), noting that excessively high thresholds cause misalignments (structure not adapted), while overly low values lead to lip shape leakage.\n\n**Parameter Sensitivity:** Our experiments indicate that performance is relatively robust within a reasonable interval around the chosen threshold (850 in our setting, out of 1000 steps), with **±5% variation not significantly affecting results**. However, extreme values did degrade performance substantially.\n\n**Generalization to Stylized Content:** Due to our method's training on real datasets and its strong generalization capability, it maintains excellent performance on stylized content. However, during training, we still set the optimal threshold based on realistic content characteristics to ensure robust learning foundations.\n\n---\n\n### **Q2: Computational efficiency analysis**\n\nWe appreciate your concern about the computational cost and efficiency of OmniSync compared to prior GAN-based methods, such as Wav2Lip and VideoReTalking. It is true that diffusion-based methods generally require multiple iterative denoising steps, which can impact inference speed.\n\n**Optimization Strategy:** To address this challenge, our approach incorporates a **progressive noise initialization mechanism**: rather than starting the diffusion process from pure noise, we initialize from the input frame at an intermediate noise level, enabling us to reduce the required denoising steps during inference to just 30-40 steps.\n\n**Performance Benchmarks:** We tested our method's inference speed on an A100 GPU, where generating a 5-second video requires **3.5 minutes**. Through implementing a series of acceleration techniques including TeaCache and sequence parallelism, and benefiting from our flow-based noise initialization that eliminates the need for complete 50-step denoising, we achieve excellent results with reduced computational overhead. **After these optimizations, generating a 5-second video requires only 1 minute**.\n\nThis represents a significant improvement in practical deployment efficiency while maintaining our quality and generalization capabilities.\n\n---\n\n### **Q3: Training data requirements and pseudo-paired data strategy**\n\nWe agree that a key aspect of our method is leveraging pseudo-paired data (i.e., with minimal pose variation) during early diffusion timesteps to stabilize learning of head pose and identity.\n\n**Data Requirements:** We utilized approximately **40 hours of MEAD data** for pseudo-paired training. We experimented with using in-the-wild data for training but found that the model struggled to converge effectively, confirming that controlled data with strong pose consistency is necessary for stable learning of structural features.\n\n**Generalization Capability:** Despite this controlled training requirement, our model demonstrates **strong generalization and robustness**, proving that our pseudo-paired data strategy is well-suited to the task requirements. The model successfully handles diverse real-world scenarios and stylized content that were not present in the training data.\n\n**Future Enhancement:** Your suggestion is excellent—we can leverage our current trained model to generate additional pseudo-paired data for incorporation into future training iterations. This self-supervised data augmentation approach could further enhance our model's performance and robustness across even more diverse scenarios.\n\nThank you for this feedback, which opens promising directions for future improvements."}, {"review_id": "6p9YGlQQa6", "reviewer": "Reviewer_tzZJ", "summary": "This paper proposes a new framework for lip synchronization, which is the task of going from video of person A saying X + audio of any person saying Y to video of person A saying Y. This is useful for dubbing or video avatar generation, for example. The authors propose the first reference-free and mask-free lipsync model, which removes a lot of the constraints posed by existing models. They use a DiT (diffusion transformer) as a backbone, and use Whisper and T5 to encode audio and text respectively. The text serves to customize/improve the resulting video. To achieve good results without masks or references, the authors adopt a two-stage training/inference logic. In the first stage, the model is trained with pseudo-paired data from MEAD where poses etc. match exactly between two different videos (input and target). Instead of starting from total noise, the model starts from a noised version of the input, which is important so that the pose etc. is maintained. During the second stage, arbitrary clips from a broader dataset extracted from Youtube are used for training, to generate finer details. During inference, CFG (Classifier-free guidance) is used on the audio. It varies spatially with a gaussian kernel to strengthen guidance around the lip region, and also over time, to strengthen guidance in earlier steps. Finally, the authors show state-of-the-art results, present an ablation on the main components, a new benchmark on AI-generated composed of human-centric stylized videos, on which they also achieve the best results. A user study and a variety of qualitative evaluations are shown.", "questions": "- Does C mean concatenation in Figure 2? I would think the noise should be added, not concatenated. Also, \"Flow matching noise generator\" is just adding noise from what I understand, which seems a bit misleading.\n - Do you plan on releasing the 400-hour dataset? Or at least releasing instructions on how to obtain or reproduce it?\n- Do you plan to release code?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nFor me the greatest strength is the fact that this method is mask-free and reference-free, which is a substantial achievement that in my view should be more highlighted in the abstract, intro, etc. The way in which this is achieved is interesting and requires some tweaks compared to normal diffusion models. The use of MEAD for pseudo-paired data is clever. The two-stage training is also interesting, and the initialization makes sense. The CFG methodology, while not groundbreaking, is adequate and clearly makes a difference according to the ablations. The new AI-generated benchmark is appreciated, and so is the user study.\n\nWeaknesses:\nI'd say the biggest weakness is the constant use of the term flow matching when, from my understanding, nothing in the model is at all related to flow matching. The model uses a diffusion loss, as is clearly explained, and the initialization is just a noised version of the input. In my view, none of this has anything to do with flow-matching, and the nomenclature and overall narrative around this should be adjusted accordingly. Some specific problematic segments are highlighted below. Apart from this, there are some clarity issues also highlighted below. Regarding lip shape leakage, I believe this should be more discussed in the main paper since it is discussed in the abstract (I know it's discussed in the supp material, but I think it's also important to have some text discussing it in the main text, even if it is brief). Furthermore, some claims are made around the fact that this problem is alleviated compared to other works, but this is only shown via a qualitative comparison in the supp mat. Ideally, it would be great to evaluate this more objectively via some kind of metric that could be computed for the whole evaluation dataset. Finally, I believe that despite being concurrent work, it would be interesting to compare results with KeySync and mention it in the related work, since it is also a prevalent recent lipsync work. However, given that it came out very recently and qualifies as concurrent work, my score will not be affected by this - I just think it would be interesting to include if possible.\n\nSpecific comments and nitpicks:\n - \"existing methods rely on face detection and alignment [19] techniques that break down when applied to stylized characters and non-human entities, precisely the diverse content that modern text-to-video models excel at generating\" not sure if this is true, at least KeySync has demos with statues, paintings, etc. and it appears to work.\n - \"We employ Flow Matching [24] as our training objective.\" this doesn't seem to be accurate, since straight after writing this you proceed to describe your objective as a \"standard diffusion loss\".\n - In lines 141-142, you should specify which datasets and why they were chosen, otherwise it's a bit confusing. In the paragraph below you should also specify more clearly what you mean by pseudo-paired data and how obtaining this is possible. I think I understand what was done but it took me a while.\n - Temporally-Adaptive Guidance section on line 216 requires some citations since other works have used this before (aka vary the CFG scale over time). Examples would be Video Stable Diffusion and also \"Analysis of Classifier-Free Guidance Weight Schedulers\" (2024), by Wang et al. Not sure about the spatial weights for CFG, but other works such as Diffused Heads (\"Diffusion Models Beat GANs on Talking-Face Generation\") and KeyFace (among others) have used very similar ideas for losses that are weighted differently around the mouth region, so they should probably be cited as well.\n - It'd be nice to report LSE-D as well as LSE-C, since this is the standard pair to use. Landmark distance is a reasonable metric, but likely a bit suboptimal to evaluate lipsync. I think it's okay to keep it though. Finally, it might be interesting to include LipScore from KeyFace/KeySync since it has been shown to correlate better with human perception than SyncNet-based metrics.\n - Some more qualitative analysis on the impact and limitations of text prompts would be quite interesting and could be included in the supp material. I feel that the existing exploration is slightly insufficient.", "was_revised": true, "final_justification": "Happy with the rebuttal as claimed below", "rebuttal": "# Response to Reviewer tzZJ\n\nWe sincerely thank the reviewer for the thorough technical review and valuable feedback. Your observations have significantly helped us improve both the technical accuracy and clarity of our manuscript. We address each concern systematically below:\n\n---\n\n### **Q1: \"Flow Matching\" terminology confusion**\n\nThank you for this crucial observation. We found that there was indeed a technical error in our manuscript's mathematical presentation.\n\nYou are right that our method description created confusion between Flow Matching and diffusion terminology. While we correctly stated using \"Flow Matching as our training objective\" (Line 127), Equation (2) incorrectly presented diffusion loss notation instead of proper CFM formulation. This created unnecessary confusion about our approach.\n\nOur method is indeed a standard Conditional Flow Matching (CFM) approach following Lipman et al.[1]. The DiT model predicts velocity field $v_θ(x_t, t)$ to regress conditional velocity (not noise $ε$), with training objective being standard CFM loss for velocity field learning, and generation process through integration of learned velocity field: $dx/dt = v_θ(x_t, t)$.\n\nThe proper CFM loss function for Equation (2) should be:\n\n$L_{CFM}(\\theta) = E_{t, x_t , V_{cd}, A_{ab}, V_{ab}}[\\|v_\\theta(x_t, V_{cd}, A_{ab}, t) - u_t(x_t|V_{ab})\\|_2^2]$\n\nwhere $v_θ(x_t, V_{cd}, A_{ab}, t)$ is the learned velocity field predicted by DiT with conditioning on input video $V_{cd}$ and target audio $A_{ab}$, $u_t(x_t|V_{ab})$ is the conditional velocity field typically defined as $u_t(x_t|V_{ab}) = (V_{ab} - x_t)/(1-t)$ for the linear interpolation path $x_t = (1-t)x_0 + tV_{ab}$, and this differs from diffusion models that predict noise $ε$.\n\nIn the revision, we will correct Equation (2) to proper CFM mathematical formulation, clarify throughout that DiT predicts velocity fields rather than noise, and ensure consistent Flow Matching terminology across the manuscript.\n\n[1] Lipman Y, Chen R T Q, Ben-Hamu H, et al. Flow matching for generative modeling[J]. arXiv preprint arXiv:2210.02747, 2022.\n\n---\n\n### **Q2: Limited discussion of lip shape leakage in main paper**\n\nThank you for your feedback. We appreciate the reviewer’s careful attention to the issue of lip shape leakage. As described in detail in Supplementary Section 5 (lines 148–170), we analyze the phenomenon, its typical causes in mask-based approaches, and our proposed solutions in OmniSync. We agree, however, that this important topic should be explicitly addressed in the main paper as well. In our revision, we will add a dedicated paragraph to the main text to clearly define lip shape leakage, explain its prevalence in mask-based methods, and describe how our mask-free training paradigm and dynamic spatiotemporal CFG jointly alleviate this problem.\n\nWe appreciate your suggestion to quantitatively assess lip shape leakage. Although no standard open-source implementation is available for KeySync's LipLeak metric, we have followed the methodology described in their paper and utilized the LipScore codebase to compute LipLeak scores on our entire evaluation set. The results are summarized below:\n\n| Method | Wav2Lip | VideoReTalking | TalkLip | IP-LAP | Diff2Lip | MuseTalk | LatentSync | KeySync | Ours |\n|--------|---------|----------------|---------|--------|----------|----------|------------|---------|------|\n| LipLeak↓ | 0.44 | 0.34 | 0.49 | 0.26 | 0.23 | 0.29 | 0.30 | **0.19** | $\\underline{\\text{0.21}}$ |\n\nCompared to most previous works, our method significantly reduces lip shape leakage. Even compared to KeySync, which is specifically designed to minimize lip leaks, our scores are comparable.\n\n---\n\n### **Q3: Face detection limitations for stylized content**\n\nThank you for your valuable comment. We agree that for stylized but anthropomorphic subjects—such as statues and paintings resembling human faces—existing face detection and alignment techniques can still function reliably, which is also reflected in KeySync's demos. However, the true limitation arises in scenarios involving non-human or highly non-anthropomorphic characters (e.g., animal faces, fantasy creatures), where face detectors often fail to localize landmarks accurately or at all.\n\nAs demonstrated in **Figure 3 (second row, \"wolf girl\")** of our paper and further visualized in our supplementary videos, existing methods relying on face alignment frequently struggle or fail in such challenging cases, resulting in poor lip sync and severe artifacts. In contrast, our mask-free approach does not depend on explicit facial keypoints, enabling robust lip synchronization even when standard detectors fail. Thus, while KeySync and similar approaches may work for \"easy\" stylized humans, they cannot generalize to the full diversity of AI-generated content—an important advantage highlighted by our method.\n\n---\n\n### **Q4: Dataset specification and pseudo-paired data extraction**\n\nThank you for your valuable feedback. We apologize for the confusion regarding the dataset selection and the construction of pseudo-paired data. To clarify:\n\nAs described in lines 234–236, for early timesteps, we use the MEAD dataset, which consists of controlled lab recordings with minimal pose variation—ideal for stabilizing structural learning. For the middle and late timesteps, we utilize a diverse 400-hour YouTube dataset to ensure generalization to varied real-world scenarios.\n\nBy \"pseudo-paired data,\" we refer to pairs of video segments from MEAD where the head pose and identity remain nearly identical and only the lip movement differs, enabling us to simulate paired conditions without requiring perfect alignment or explicit pair annotations.\n\nThis timestep-dependent sampling strategy is a key innovation of our method. For full details on the dataset choices and the practical process for constructing pseudo-paired data, please refer to Supplementary Section 54–110. We will revise the main text to explicitly highlight these points for greater clarity.\n\n---\n\n### **Q5: Missing citations for temporal and spatial CFG approaches**\n\nWe will add comprehensive citations acknowledging prior work, including temporal CFG approaches from Video Stable Diffusion[1] and \"Analysis of Classifier-Free Guidance Weight Schedulers\"[2], as well as spatial mouth attention mechanisms from Diffused Heads[3], KeyFace[4] and KeySync[5].\n\n[1] Blattmann A, Dockhorn T, Kulal S, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets[J]. arXiv preprint arXiv:2311.15127, 2023.\n\n[2] Wang X, Dufour N, Andreou N, et al. Analysis of classifier-free guidance weight schedulers[J]. arXiv preprint arXiv:2404.13040, 2024.\n\n[3] Stypułkowski M, Vougioukas K, He S, et al. Diffused heads: Diffusion models beat gans on talking-face generation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 5091-5100.\n\n[4] Bigata A, Stypułkowski M, Mira R, et al. Keyface: Expressive audio-driven facial animation for long sequences via keyframe interpolation[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5477-5488.\n\n[5] Bigata A, Mira R, Bounareli S, et al. Keysync: A robust approach for leakage-free lip synchronization in high resolution[J]. arXiv preprint arXiv:2505.00497, 2025.\n\n\n---\n\n### **Q6: Enhanced evaluation metrics and KeySync comparison**\n\nThank you for your valuable suggestions. Due to space limitations in the main paper, we were unable to report LSE-D previously. As requested, we now provide LSE-D results for all compared methods below. Additionally, following your advice on evaluation metrics, we have also computed LipScore using the official KeySync implementation. As shown in the table below, our method not only achieves competitive LSE-D (7.33), but also leads on the LipScore metric (0.5668), **outperforming KeySync with a relative improvement of 15.7%**. This further demonstrates the strong perceptual quality and accurate synchronization achieved by OmniSync.\n\n| Method | Wav2Lip | VideoReTalking | TalkLip | IP-LAP | Diff2Lip | MuseTalk | LatentSync | KeySync | Ours |\n|--------|---------|----------------|---------|--------|----------|----------|------------|---------|------|\n| LSE-D↓ | 7.528 | 7.453 | 9.716 | 7.358 | 7.421 | 10.647 | **7.240** | 14.652 | $\\underline{\\text{7.336}}$ |\n| LipScore↑ | **0.662** | 0.519 | 0.367 | 0.198 | 0.207 | 0.542 | 0.245 | 0.490 | $\\underline{\\text{0.567}}$ |\n\n---\n\n### **Q7: Text prompt analysis and exploration**\n\nThank you for your valuable feedback. We appreciate your interest in the impact and limitations of text prompts in our framework. In Section 6 of the supplementary material, we provide qualitative analysis and ablation studies demonstrating how video description prompts affect lip clarity and movement amplitude. Specifically, our results show that incorporating descriptive prompts such as \"clear facial and tooth movements\" leads to more expressive and visually distinct lip synchronization, with enhanced dental visibility and mouth articulation.\n\n---\n\n### **Q8: Figure 2 notation confusion**\n\nThank you for your insightful comments. You are correct: in Figure 2, the \"C\" should more appropriately be a \"+\" to indicate addition, not concatenation. We apologize for this oversight and will revise the figure to use the correct notation in the camera-ready version.\n\nRegarding the \"Flow Matching Noise Generator,\" it indeed adds noise to the original frames, but unlike standard random noise addition, the process is controlled based on the flow-matching principle.\n\n---\n\n### **Q9: Dataset and code release plans**\n\nWe are actively working to release our 400-hour dataset with extraction scripts, subject to institutional data policy approval. We commit to providing comprehensive technical specifications including DiT backbone architecture, training configuration, and evaluation mehtod to ensure reproducibility."}]

Author rebuttal:

Meta-review: In this paper the authors propose a lip synchronization framework that eliminates reliance on reference frames and explicit masks, This contribution is appreciated by all reviewers who have finally all provided accept or strong accept recommendations. 

There are a number of limitations pointed out, such as the need to use the right flow matching equations instead of the diffusion equations, the validations provided in the discussion phase, especially the new lipsync results that have been provided, the additional details regarding the lip-shape leakage. All of these additions need to be included in the revised final paper as promised and providing the final code and benchmarks as promised.

Given that the authors have promised to provide all these modifications in the revised version and provide the code and benchmarks, I believe this paper can be accepted.
Decision: Accept (spotlight)