Title: OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking
Venue: NeurIPS 2025
Accepted: yes
Keywords: talking head

audiovisual generation

text-driven talking head
Abstract: Although significant progress has been made in audio-driven talking head generation, text-driven methods remain underexplored. In this work, we present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text while emulating the target identity's speaking and facial movement styles, including speech characteristics, head motion, and facial dynamics. Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis.
At the shallow layers, cross-modal fusion modules are introduced to integrate information between the two modalities. In deeper layers, each modality is processed independently, with the generated audio decoded by a vocoder and the video rendered using a GAN-based high-quality visual renderer. Leveraging DiT’s in-context learning capability through a masked-infilling strategy, our model can simultaneously capture both audio and visual styles without requiring explicit style extraction modules.  Thanks to the efficiency of the DiT backbone and the optimized visual renderer, OmniTalker achieves real-time inference at 25 FPS.
To the best of our knowledge, OmniTalker is the first one-shot framework capable of jointly modeling speech and facial styles in real time. Extensive experiments demonstrate its superiority over existing methods in terms of generation quality, particularly in preserving style consistency and ensuring precise audio-video synchronization, all while maintaining efficient inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OmniTalker, the first unified framework for one-shot real-time talking head generation based on text. By developing a DiT-based backbone with audio-video fusion, OmniTalker generates high-quality synchronous video and audio that mimics the styles of the reference. The contributions include a dual-branch DiT-based backbone, a unified framework for training and one-shot generation, and a whole system that could generate at a high frequency of 25Hz. The framework is evaluated on publicly available datasets and custom datasets, compared with TTS and audio-driven methods, and shows improvements. This work provides a prototype for real-time text-driven talking head generation and provides further study directions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work is the first one-shot real-time text-driven talking head generation framework and fill the gap in this area.\n2. The entire pipeline of the framework is well demonstrated, providing a clear methodology for further research in this area.\n\nWeakness:\n1. Lack of comparison with existing text-driven THG baselines (e.g., TTSF [1] referred to in related works).\n2. The custom dataset is not available, and the results are hard to check. Although the authors claim the availability of the dataset, I did not find a link to the dataset. \n3. The ablation is not adequate, and the effectiveness of CFG and dual-branch DiT is not fully validated.\n\n[1] Jang, Youngjoon, et al. \"Faces that speak: Jointly synthesising talking face and speech from text.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n1.  Could the authors clarify whether they have a plan to release the custom dataset?  This will increase the soundness and the influence of this work.\n\n2. In Table 1, OmniTalker underperforms previous work on SEED dataset. Could the authors give a further discussion on this? Additionally on Custom dataset, OmniTalker gives the best result. It is explained in the paper that the video could also help improve audio performance. However, why  SIM-A with $\\alpha_{C_r}=2.5$ is worse on the Custom dataset?\n\n3. In Related Work section, line 62, it is said a detailed comparison with prior methods is included in the Appendix. However the paper do not include an appendix and I did not find it in the supplementary material as well.\n\n4. How is the hyperparameters, like block size, determined?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new setting: given a reference video and a driving text prompt, the model can generate synchronized audio-visual talking heads while preserving the speaking style (both vocal and facial) of the reference video. To achieve this, the authors first employ a shared encoder to encode the text prompt, reference video, and audio, followed by two separate heads dedicated to generating audio and video, respectively. Additionally, they adopt a masked infilling strategy during training, which helps maintain each individual’s unique speaking style during generation. Notably, the proposed approach is capable of running in real time on a single GPU, and demonstrates strong audio-visual synchronization performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. They are the first to introduce a novel setting: given a reference video and a driving text, the system is able to generate synchronized audio-visual talking heads.\n2. The generated video preserves the speaking style of the reference, including both vocal characteristics and facial expressions.\n3. The proposed method achieves real-time inference at 25 FPS on a single GPU.\n4. The generated results exhibit high-quality audio-visual synchronization.\n\n\nWeakness:\n1.  This paper uses a 3DMM model as the motion representation, which inherently limits the method to the scope of 3DMM. For example, it can only handle real human faces, and lacks the flexibility of former diffusion-based approaches （\\eg, ominihuman, loopy) that can generalize to a wider range of scenarios, such as AIGC-generated avatars, cartoons, or animals. Additionally, 3DMM reconstructions can become inaccurate under large head poses.\n2. The paper’s claim of being end-to-end seems somewhat misleading. In fact, the audio and video generators are first pre-trained separately, and the DiTs are only used later to generate intermediate representations. This setup is quite similar to previous methods such as Float and VASA-1.\n3. The paper lacks comparisons with other real-time and open-source methods, such as JoyVASA and DiTTO, which should be included for a more comprehensive evaluation.\n4. Some training details are missing. For instance, how is the renderer trained? What exactly do the “3D keypoints” represent? Since 3DMM only accounts for the facial region, how is the dynamic behavior of hair ensured during generation?\n\nQUESTIONS:\n1. How does this method perform in singing scenarios? \n2. Is it possible to generate results without a reference video? What happens if only audio or video is provided? Can the system still generate plausible outputs in those cases?\n3. How is the renderer trained? Are the 3D keypoints you use the commonly adopted 68 facial landmarks? If so, how do you ensure the motion consistency of features like hair and shoulders, which are not explicitly covered by those keypoints?\n4. Does the reported 24 FPS include the 3D face reconstruction step?\n5. Since your rendering module essentially operates as an image-based model, and the reference image is randomly selected, how do you ensure the consistency of features like teeth, which may not be clearly visible in all reference images?\n6. Given that many video generation methods based on Stable Diffusion tend to degrade when generating long videos due to overlapping inference artifacts, can this method generate long videos without such degradation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OmniTalker, a novel and unified framework for generating synchronized audio and video of a talking head from text input. The method is one-shot, meaning it can mimic a person's identity and speaking style from a short reference video. The core of the architecture is a dual-branch Diffusion Transformer (DiT) that jointly generates audio (mel-spectrogram) and visual motion features. The key innovation is the use of a masked infilling strategy, which allows the model to learn multimodal styles (vocal characteristics, head motion, facial expressions) in-context, without requiring an explicit style encoder.\nThe paper is exceptionally strong, presenting a well-motivated and elegant solution to a challenging problem. The experimental results are comprehensive and demonstrate state-of-the-art performance, particularly in style preservation, where it vastly outperforms existing methods. The work is well-written, the architecture is clearly explained, and the claims are robustly supported by quantitative metrics, qualitative examples, and a user study. This is a high-quality submission that represents a significant advance in the field of talking head generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors proposed an elegant method to solve text-driven talking head generation task in a unified, end-to-end framework that jointly synthesis both modalities.\n2. The proposed method is effective, which show really good visually results that achieve high lip-sync and high rendering quality simultaneously, and achieve comparable quality in text-to-speech results with the sota tts methods.\n3. The proposed method is one-shot, only a reference video and a reference audio needed to generate same style results.\n4. The paper is well-writen and easy to follow.\n\nWeaknesses:\n1. The generated video and audio are rely on off-the-shelf video render and audio vocoder, which limite the output quality to upper-bound of the decoder they adopt.\n\nQUESTIONS:\n1. Could you clarify the exact input representation for the text projector? Does the model use raw text, or an intermediate format like phonemes? Given the challenge of generating mel-spectrograms directly from text, more detail on this part of the pipeline would be very helpful.\n2. Did you consider an end-to-end approach that generates video pixels directly, instead of intermediate motion representations? My concern is that the GAN-based renderer might be tightly coupled with the learned motion codes, potentially limiting their applicability with different rendering architectures and may perform worse when applying a difference alignment method to input video.\n3. Did you apply another pre-trained weights when you perform large-scale pre-training, or just train from scatch?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes a unified framework for generating high-quality audio-visual talking heads from text input.

STRENGTHS AND WEAKNESSES:
Strengths:

The method enables real-time talking head generation on a single NVIDIA RTX 4090 GPU.

Weaknesses:

1、There are many writing issues. For example, the formatting of bolded subheadings in Section 3.2 is inconsistent. There is an extra comma at the start of line 200. The "CMF" in Eqn.3 should be "CFM". 

2、The architecture of OmniTalker heavily relies on a dual-branch DiT framework.

3、 The ablation study only evaluates different settings of Audio-Video Fusion.

4、The technical contributions are not clear. 

5、The symbols in Eqn. 3 and Eqn.4 lack explanations.

QUESTIONS:
1、 The authors should evaluate the core technical contributions through an ablation study

2、Please clarify the technical contributions.

3、Please clarify the symbols in Eqn. 3 and Eqn.4.

4、How does Figure 2 validate the proposed method's performance in reproducing complex facial motion styles through the visualization of jaw angle dynamics?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02