Title: OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human-computer Interaction

Human Behavior Understanding

Multi-model Generation
Abstract: In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Text Markup, which precisely timestamps generated text tokens, and TempoVoice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Comprehensive evaluations on ResponseNet demonstrate that OmniResponse outperforms baseline models in terms of semantic speech content, audio-visual synchronization, and generation quality. Our dataset, code, and models are publicly available at https://omniresponse.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Online Multimodal Conversational Response Generation (OMCRG) task, which focuses on generating synchronized listener feedback in real-time conversations. To address this, the authors propose OmniResponse, a Multimodal Large Language Model (MLLM) that uses text as an intermediate step to synchronize generated audio and facial reactions. The authors also created a new dataset, ResponseNet, for this task. Experiments show OmniResponse outperforms baselines, creating more natural and temporally aligned conversational responses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper defines a new task (OMCRG) that requires generating synchronized audio and facial reactions in real-time, which is more advanced than previous work.\n2.\tThe proposed OmniResponse uses text as a bridge to align audio and video, introducing new techniques (Chrono-Text Markup and TempoVoice) for precise temporal synchronization.\n3.\tThis paper introduces ResponseNet, a new, high-quality dataset of synchronized and richly annotated conversational videos.\n\nWeaknesses:\n\n1.\tThe LSE-D scores presented in Table 2 show only a marginal improvement over the baseline. Metrics from SyncNet [1], which is a more reliable indicator of lip-sync accuracy, may be a better choice.\n2.\tHuman evaluation is absent. Since automated scores often fail to capture the nuances of multimodal synchronization, a user study would be essential to validate the model's perceptual performance and strengthen the paper's claims.\n3.\tThe qualitative results in the supplementary materials are not convincing. First, there are noticeable discrepancies and a lack of synchronization between the generated lip movements and the audio. Second, the quality of the synthesized audio itself is poor, suffering from low fidelity and poor intelligibility.\n4.\tOmniResponse is designed for \"online generation\", yet the paper lacks a quantitative analysis of its inference speed or real-time viability.\n5.\tThe phrase \"accumfd udfsalated\" in line 144 is confusing and seems to be a typo.\n\n[1] Prajwal, K. R., et al. \"A lip sync expert is all you need for speech to lip generation in the wild.\" Proceedings of the 28th ACM international conference on multimedia. 2020.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nGiven the potential for malicious applications of talking-head techniques, the paper should include more discussion on methods for proactive and reactive protection of personal portraits to address potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a novel task, Online Multimodal Conversational Response Generation (OMCRG), which aims to generate synchronized audio and facial listener responses in real-time during dyadic interactions. To tackle this challenging task, authors propose d OmniResponse, a MLLM framework that uniquely uses text as an intermediate modality to bridge and synchronize the audio and visual outputs. The core are two novel components: Chrono-Text Markup, which temporally aligns text tokens with visual frames, and TempoVoice, a controllable online Text-to-Speech module that generates speech synchronized with the facial reactions. A dataset of OMCRG is collected for benchmarking the model performance.  Experimental results show OmniResponse outperforms baselines in generation quality and synchronization across modalities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. vovel and practical task formulation. The paper defines and formalizes the OMCRG task, a practical step beyond existing work that only focuses on single-modality response generation (e.g., text-only or face-only). This task addresses a more holistic and natural form of human-agent interaction.  besides, the dataset contribution is also worth noting. \n\n2. interesting methodological approach. The core idea of using text as an intermediate modality to synchronize audio and facial generation is novel. This approach cleverly circumvents the difficult problem of direct audio-visual synthesis by leveraging the semantic richness and discrete nature of text, making the problem more tractable. \n\n3. comprehensive evaluation. the authors conduct a thorough evaluation using a wide array of metrics including text, audio, and visual modalities. The inclusion of ablation studies clearly demonstrates the effectiveness and necessity of the proposed Chrono-Text and TempoVoice components, supporting the paper's main claims.\n\nWeakness:\n\n1. small scale of ResponseNet. While ResponseNet is of high quality, its total duration (14.2 hours) is relatively small, which may limit the model's ability to generalize to unseen conversational styles or demographics. A more detailed analysis of the dataset's demographic diversity would be beneficial to understand potential biases.\n\n2. potential oversimplification of temporal dynamicsl. the Chrono-Text Markup uses only two special tokens to align text and vision, which might be too coarse to model highly complex temporal dynamics. This approach may struggle with typical phenomenons like overlapping speech, or subtle non-verbal vocal cues that require more fine-grained temporal controls. \n\n3. lack of user study. The evaluation relies entirely on automated metrics, which may not fully capture the nuances of human perception. A human user study assessing the naturalness and audio-visual synchronization shall be included.\n\n4. there are many typos and grammar errors in current version of manuscript, e.g, line 74, challenge -> challenging, line 116 steaming -> streaming, line 144 accumfd, line 234 frequently -> frequent\n\nQUESTIONS:\nplease see the weakness part above.\n\nLIMITATIONS:\nyes, limitations are discussed in appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Online Multimodal Conversational Response Generation (OMCRG), a novel task to generate real-time synchronized verbal and non-verbal listener feedback based on the speaker’s multimodal inputs. The proposed OmniResponse framework, a Multimodal Large Language Model (MLLM), addresses the challenges of audio-visual synchronization by using text as an intermediate modality, enhanced with Chrono-Text for temporal anchoring and TempoVoice for synchronized text-to-speech. Additionally, the ResponseNet dataset is presented, comprising 696 high-quality dyadic interactions with synchronized multimodal data. Experiments show that OmniResponse outperforms baselines in semantic content, audio-visual synchronization, and generation quality, laying the foundation for future human-agent interaction research.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This work introduces a novel task, Online Multimodal Conversational Response Generation (OMCRG), which addresses a critical gap in modeling natural dyadic interactions and holds significant value for advancing human-computer interaction research.\n2. The proposed OmniResponse framework demonstrates robust effectiveness in generating synchronized audio-visual-textual responses, outperforming baselines in semantic alignment and multimodal coherence.\n3. The creation of ResponseNet provides an essential resource for the research community.\n\nWeakness:\n1. A more detailed analysis of temporal dynamics, such as response delay and real-time processing latency, is needed to fully characterize the framework’s performance in online interaction scenarios.\n2.  The paper lacks detailed elaboration on the real-time design mechanisms, particularly in explaining how the framework handles dynamic input streams, manages processing latency, and ensures responsive timing in natural conversational flows.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper introduces Online Multimodal Conversational Response Generation (OMCRG), a novel task focused on generating synchronized verbal and non-verbal listener feedback in real-time based on a speaker's multimodal input. To address the challenges of synchronization between generated audio and facial responses, the authors propose OmniResponse, a Multimodal Large Language Model (MLLM) that uses text as an intermediate modality. OmniResponse incorporates two new components: Chrono-Text, for temporally anchoring text tokens, and TempoVoice, a controllable online Text-to-Speech (TTS) module synchronized with facial reactions. Additionally, the paper presents ResponseNet, a new high-quality dataset of 696 dyadic interactions with synchronized split-screen videos, multichannel audio, transcripts, and facial behavior annotations to support OMCRG research. Evaluations on ResponseNet show that OmniResponse significantly outperforms baseline models in semantic speech content, audio-visual synchronization, and generation quality

STRENGTHS AND WEAKNESSES:
Advantages

1.This work introduces OmniResponse, the first online model to jointly process and generate synchronized streams of conversational human behavior.

2.This work further propose ResponseNet, an annotated dyadic conversation dataset and benchmark, enabling standardized evaluation of OMCRG models.

3.This work consider a real-world scenario: Online Multimodal Conversational Response Generation.

Disadvantages

1.The baselines in the main experiment could be more comprehensive (just see the GPT-series models)

2.The auxiliary experiments lack depth.

QUESTIONS:
See the disadvantages.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Online Multimodal Conversational Response Generation (OMCRG), a new task that advances multimodal conversational AI for real-time human-computer interactions. It presents a Multimodal Large Language Model (MLLM) that generates high-quality multimodal listener responses autoregressively. The paper also introduces ResponseNet, a dataset of 696 dyadic interactions with synchronized split-screen videos, multichannel audio, transcripts, and facial behavior annotations. The model generates synchronized verbal and non-verbal responses based on the speaker’s inputs. Key innovations include Chrono-Text, which aligns text tokens with facial expressions and speech, and TempoVoice, a controllable TTS system that synchronizes speech with facial reactions. Experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) Applicability: This paper proposes a novel task of real-time synchronization, designing a model that addresses the complex challenge of synchronizing multiple modalities—audio, facial expressions, and text—positioning it as a promising candidate for future human-computer interactions.\n(2) Effectiveness: The proposed dataset proves to be highly beneficial for this task, and the model achieves performance that surpasses existing alternatives.\n\nWeaknesses:\n(1) Dependency on Text Modality: Although text offers clarity, relying on it for synchronizing audio and facial reactions may constrain the richness of non-verbal communication. The paper lacks a thorough discussion on the quality of non-verbal generation and the evaluation of text as an intermediary modality.\n(2) Real-Time Applicability: While this paper introduces an online task, it does not address the potential time consumption required for real-time streaming audio-facial generation. Given that the training data includes sequences up to 863 seconds in duration, questions remain about how to efficiently handle visual and textual history when processing long sequences and how to minimize latency in such scenarios.\n(3) Lack of Error Analysis: The paper lacks a detailed error analysis or discussion on failure cases, which would be useful to understand the model's limitations in edge cases.\n\nQUESTIONS:\nAs listed in the Weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}