OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions
Authors: Cheng Luo
Jianghui Wang
Bing Li
Siyang Song
Bernard Ghanem
Venue: NeurIPS 2025 poster ()
Keywords: Human-computer Interaction
Human Behavior Understanding
Multi-model Generation
Primary area: applications

Abstract: In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Text Markup, which precisely timestamps generated text tokens, and TempoVoice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Comprehensive evaluations on ResponseNet demonstrate that OmniResponse outperforms baseline models in terms of semantic speech content, audio-visual synchronization, and generation quality. Our dataset, code, and models are publicly available at https://omniresponse.github.io/.

OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions
Cheng Luo', Jianghui Wang', Bing Li Siyang Song", Bernard Ghanem' *King Abdullah University of Science and Technology, Univvrrity of Exeter
Abstract
In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Ters -ext Markup, which precisely timestamps generated text tokens, and Tempo Voice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Compre- hensive evaluations ResponseNet demonstrate that OmniResponse outperforms baseline models in terms semantic speech content, audio visual synchronization, and generation quality. Our dataset, code, and models are publicly available at stttps://omit htttps://omirresposse wrrinns..ggttie github. io/
Introduction
Generating realistic human conversational responses has substantial potential across numerous appli- cations, spanning from human-computer interactions (39], immersive metaverse experiences [30], to mental health interventions [31]. However, human communication is inherently multimodal and com- plex. In face-to fo-cee interactions, speakers convey their messages not only through spoken language but also through non- verbal cues, such as lip movements and facial expressions. Correspondingly, listeners provide multimodal responses consisting of verbal (e.g., audible affirmations disapprovals) and non-verbal responses (e.g., subtle head nods). While considerable efforts (10 67 have been dedi- cated to modeling text dialogue, particularly in language-based interfaces (35] modeling multimodal
conversational interactions has been much underexplored.
In this paper, we explore new task: learning to simultaneously generate verbal and non-verbal listener responses in an online dyadic conversation setting, conditioned on the speaker's verbal and non-verbal inputs (see Figure D. We refer to this task Online Multimodal Conversational Response Generation. Although various audio-to-video generation methods (c.g. talking head generation (82 84 79]) have shown impressive performance, these methods focus synthesizing visual content aligned with input audio signals, which ignores explicitly modeling multimodal conversational interactions. Recent studies 41 47 61 propose to generate facial reactions for
"Corresponding author.
Prrevious studies g defined ppeaker-listener listener framework for dyadic interactions, in which the listener both attends the speaker's utterances and provides verbal and nonverbal feedback.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Time
Inputs:
Inputs:
Offiine Multimodal Conversational Response Generation
Online Multimodal
Conversationa Response Generation
Outputs:
(|ji= [is])--aa-s joi
Outputs:
11--I-DDa-l11--0044/0 joo
Time
(8) Offline Multimodal Conversational Response Generation
Online Multimodal Conversationa Response Generation
Figure I: Illustration of the new OMCRG task. (a) In offline tasks, the generation model generates the listener's full response only after receiving the entire input sequence from the speaker. (b) Differ- ently, OMCRG task requires sequentially processing the speaker's incoming input and generating multi-modal responses for the listener on the fly.
listener; however, these methods overlook verbal responses, which are essential to engage in dialogue fully.
The OMCRG task is complex and poses major challenges three aspects. First, non-trive non-trivial to directly achieve synchronization between the generated audio and facial reactions of the listener for OMCRG task. As revealed in existing talking-hed head works [82] [6], achieving precise alignment between facial motion and audio is already challenging, even when the entire audio signal is given. In contrast, OMCRG is to generate both audio and facial reactions simultaneously and incrementally. Such online and multimodal generation settings make face-audio synchronization much more difficult, due to the high variability and semantic ambiguity of audio modality. Second, due to the online setting, the model has to reason partial speaker input and generate audio-visual responses the fly, which requires both powerful audio-visual understanding and generation abilitics. While powerful pre-trate me-itrined models have been developed for language and vision, audio modeling remains comparatively underdeveloped, making more challenging to generate expressive and appropriate audio and facial reactions. Third, the lack of high-quality datasets for dyadic multimodal interaction significantly hinders the development of OMCRG.
We address the above challenges by proposing unified framework, OmniResponse, which autore- gressively generates high-quality multimodal listener responses. Rather than directly synchronizing generated audio and facial reactions, our key insight is to introduce text intermediate modality for the OMCRG task. Compared with audio, text offers clearer semantics and reduces uncertainty, making it more tractable for learning multimodal reaction generation. However, text is static modality without inherent temporal information, posing challenges for synchronizing spoken words with visual frames in an autoregressive generation setting. To overcome this, introduce Multi- modal Large Language Model (MLLM) augmented with two innovative modules: Chrono-Tec and Tempo The Chrono-Tex: module temporally anchors generated textual tokens by incorporating additional tokens (markers) that explicitly encode time, ensuring alignment between words and visual frames. Tempo
controllable, online text-to-speech module designed to produce synchro- nized audio from these temporally annotated textual embeddings, ensuring accurate synchronization between audio and facial reactions.
In addition, we construct a high-quality dataset named ResponseNet, comprising 696 dyadic conver- sation pairs. Each pair includes synchronized split-screen video streams of both speaker and listener, multichannel audio recordings, verbatim text transcriptions, and detailed facial- behivir annota- tions (i.e., facial expressions and head movements). Through extensive retrieval for scarce dyadic video data, rigorous content filtering, meticulous camera-shiff alignment, and manual annotation, ResponseNet delivers unique and valuable resource for benchmarking OMCRG.
Our contributions are summarized follows: (1) we present OmniResponse, the first online model to jointly process and generate synchronized streams of conversational human behavior, establishing

---

foundation for future work in human agent interaction; (2) we introduce ResponseNet, an annotated dyadic conversation dataset and benchmark, enabling standardized evaluation of OMCRG models.
2 Related Work
Facial Reaction Generation. Facial reaction generation (FRG)
63
83
60
is a particularly chal-
lenging new task as it requires to predict the non-deterministic human facial reactions under different contexts. Early FRG approaches [27] relied on Generative Adversarial Networks (GANs) [43] 21
typically conditioned the generation process on the speaker visual-speech behaviors. Since FRG is a non-dcterministic process (i.e., different facial reactions can be triggered by the same speaker behavior [3]), recent advances have shifted towards more sophisticated generative frameworks, For example, Ng et al. [7 introduces a non non-deterministie approach based on Variational Autoencoders (VAEs) [3), which enabled sampling diverse human facial motions. This work was complemented by novel dataset containing paired recordings of active speakers and silent listeners, providing essential training data for modeling natural reactions. Zhou el al. [83] developed a specialized speaker-listenee video dataset for head motion generation, which is somewhat limited by its relatively short clip durations (median length of 9.0 sec) and modest dataset scale (1.58 hours total), and thus constraining their model's 's ability learn long-term temporal dependencies. More recent works have attempted to address these limitations through innovative architectural choices or larger-scale datasets Luo et al. 15 and Zhu et al. [85] proposed transformer-based [70] VAE and diffusion
61 62]
models 64 24 respectively, training them on a hybrid collection of videos from three different
human-human dyadic interaction datasets 12 57 [50].
Spoken Dialogue Models. Spoken dialogue models generate natural speech responses in real-time, requiring systems to process both verbal content and paralinguistic elements of communication. Early approaches including AudioPALM [58], Spectron [46), and SpeechGPT (77) adopted pipelines combining automatic speech recognition (ASR), text generation, and text-to-speech (TTS) synthesis. However, their requirement to complete the entire response before the speech generation makes them unsuitable for live human-computer interactions. Recent developments [44 [8] 49 have shifted towards end-to-end approaches that directly model speech-to-speed examples include Moshi et al. [18] and dGSLM [49], which operate full-duplex speech dialogue systems capable of processing continuous speaker input while generating appropriate vocal responses. While these advances are significant, they focus exclusively on speech and text modalities, overlooking the crucial visual aspects of human communication. Even recent work by Park et al. [5I] visual-speech data is limited to intermittent speaker-listener interactions.
generation. Representative
that includes
Autoregressive Generative Model. Transformer-bant autoregressive models 70 have revolution- ized numerous domains in AI, demonstrating remarkable success in language modeling [10] 67],
72 665. Their success can be attributed to their inherent scalability and ability to unify multi-modal training under single autoregressive objective, enabling seamless integration different data modalities. The adaptation of transformers to visual tasks was pioneered by approaches such VQVAE [69] and VQGAN (19), which introduced effective methods for quantizing visual information into discrete tokens. They align visual generation with the successful paradigm of language modeling by employing decoder- only transformers to predict sequences of image tokens. Subsequent research [3] has focused on enhancing both the efficiency of tokenization processes [42 37 and sampling procedures (76.. while simultaneously scaling up model architectures to handle increasingly complex tasks.
multi-modal processing 40 3 34 45), and generative tasks [56] 81 74 73
3 Methodology
Problem Definition. Let Fÿ and A; be the speaker's facial and audio cues at time t. respectively. Given the speaker's streaming facial sequence Fi, and audio sequence Ai from time to t, the goal of OMCRG is online generate facial reactions F and audio feedback A time step t. Such multi-modal generation has been much less underexplored, different from recent works [83] mainly focusing on single-modal response generation. To provide natural responses, is crucial to ensure that the generated facial reactions and audio are temporally synchronized and react appropriately to the speaker. However, this significantly challenging due to the inherent difficulty of online audio-visue understanding and generation.
41 [8 77
3

---

Next Vision
Next Audio Responsa: ju
Response:
Speaker
am not
TempoVoice
Vision Decoder
Noxt Language Response:
Large Language Model (LLM)
Listaner
is I.....
Vision Projection
Yoyystmmb-ou are active am-dasting token?.. not-lasting tokens... <lasting token?.... spause_ tokeeet...
<peuse tokenn what-lasting token?.... clasting. token?...
listener face-to-face dyadic interaction.... <lendll clusedt- What's the hardlest thing about...clendl>
Chrono- Markup
Chrono- Text Markup
classistantl that's a really good question... <lendi- me very...
cuser- <landi>
am not sure
what
Facial Expression Head Posc
Language Instruction
Convorsation History
Speaker
Listener Text
Figure 2: Overview of the proposed OmniResponse. The model takes textual conversational history and newly coming multimodal information (e.g., facial cues) from the speaker and listener as input, and generates temporally synchronized facial and textual responses for the listener by leveraging pre- pre-trained LLM enhanced with our proposed Chrono-Tex: Markup. The generated text embeddings converted into audio synchronized with the facial response by the proposed TempoVoice
Voice module.
Instead of generating audio and visuals directly, we treat text as an intermediate modality and decompose OMCRG into two subproblems: (i) joint text-and -ad--nd--fa face response generation producing temporally aligned facial reactions F/ and textual responses W:; and (ii) synchronous text-to-speech W into audio waveform segments A that are aligned with the facial reactions. However, because text lacks explicit temporal information, achieving tight alignment with facial and audio streams is challenging for both subproblems. We address this issue with two novel modules.
Overview. We present OmniResponse, a novel framework for the OMCRG task (see Figure 2 where OmniResponse is new MLLM enhanced by two proposed key components: Chrono- Text Markup and Tempo Voice. In particular, our OmniResponse leverages the capability of a pretrained LLM to understand and interpret the speaker's multimodal inputs and autoregressively generate meaningful responses in terms of textual and facial responses. To address the lack of temporal information in text, the proposed Chrono-Tex Markup embeds explicit temporal marks between text tokens, endowing the input and output text with time-aware embeddings and ensuring precise alignment with the generated facial reactions. Furthermore, the proposed Tempo Voice generates audio responses temporally synchronized with both the generated textual response and the listener's facial movements.
3.1 OmniResponse
Model Architecture. As shown in Figure OmniResponse processes multiple modalities from the speaker and the listener, temporally aligns different modalities, and outputs synchronous multimodal responses to the speaker. In particular, at each time step i, OmniResponse consumes: (I) Static text inputs: a task-specific instruction prompt W Wisttruct and the conversation history prior to time t (T t), denoted W Whistory,<77 < and (2) Temporal inputs: the previously generated facial features of the listener F'. Fi the facial features of the speaker F - and the accumulated text sequences from both participants (W W W over the interval [T, - 1]. Using these inputs, OmniResponse predicts the next facial features Fl, the verbal response W!. and the corresponding speech segment A! in the current frame, ensuring precise temporal alignment all modalities. Formally, we defined this process as:
E A W/ = MMWinstruet. instrucct,, W Whistory, <r; F 1 F
r:t-1: W W! W

---

Vision Projection. We introduce the vision projection layer to enable the pretrained LLM (Phi-3.5 mini-instruce with 3.8B parameters (T) to process visual facial features. The layer implemented as a multilayer perceptron (MLP) that maps the the listener's and speaker's past facial features Fi,-11
and - into embedding features Vi.-- aligned with the LLM token space. During autoregressive generation, the MLLM employs causal self-attention [70] to model temporal dependencies between the next token and previous one, and outputs the next listener vision embedding V.
Vision Decoder. A learnable vision decoder, comprising transformer layers, converts V back into the original coefficient space to produce the predicted listener facial coefficients F.. Subsequently, a pre-trained visual renderer maps these visual coefficients to 2D frames, using given portrait image. Please refer to the appendix for additional details.
Murono-Text Markup. Visual frames inherently encode temporal information, whereas text tokens are static and lack any temporal dimension. Additionally, visual frames and textual tokens typically differ in length due to their fundamentally different modalities, making unified autoregressive prediction challenging. To resolve this mismatch, we propose Chrono-Texl Markup, a novel yet straightforward approach that explicitly embeds temporal information into textual data, aligning the textual sequence precisely with the visual frame sequence. Unlike prior approaches such as TimeMarker (14), which inserts timestamps only between visual frames the method by Ng et al. [48]. which integrates timestamp embeddings into textual tokens, our method employs only two special markers, ensuring that the textual and visual sequences have identical lengths. Specifically, insert two special tokens into the transcript: [PAUSE] to denote silent intervals between utterances, and [LASTING] indicate that the previous textual word continues speaking to the current time. Each text token is placed between pause and lasting tokens.
Multimodal Context Modeling. Our synchronous Multimodal LLM integrates both static and dynamic inputs: Static inpuls: the instruction prompt and the accumulated conversation history. Dynamic inputs: frame-aligned visual embeddings and timestamped textual tokens for both speaker and listener. All tokens are jointly processed by an omni-attention mechanism that enforces causal, cross-modal interactions. Under this operation, each visual token attends to preceding visual tokens and to text tokens marked by chrono-text markers at earlier timestamps; similarly, each dynamic text token attends to past visual and textual tokens. However, this omni-attention prevents dynamic tokens from looking at future tokens. This ensures the generation adheres to temporal dynamics and cross-modal interactions. Meanwhile, static tokens remain globally accessible, ensuring that every dynamic update remains guided by the overarching instructions.
Tempo Voice. Generating natural speech that is precisely synchronized with text and facial frames poses a significant challenge. To address this, we introduce a dedicated synthesis pipeline, Tempo Voice. Our framework begins by combining the listener's voiceprint, extracted via the Spark-TTS global tok- enizer (71 to capture speaker identity, with the hidden states of the generated text (see Figure B.. We then apply sinusoidal positional encodings to the merged em- beddings. Since audio-token sequences typically differ in length from visual frames and textual tokens, we prepend a series of zero-initialized placeholder tokens, each endowed with positional information. These place- holders serve as queries in a cross- -otssnattenton module within a Transformer decoder, attending over the fused text-voice representations. This mechanism enables fully synchronous, autoregressive generation of audio tokens in lockstep with visual frames and text tokens. Fi- nally, a linear projection layer maps the decoder outputs to logits over the discrete audio-codec vocabulary.
Voice.
Next Audio Response:
Audio Do Tokenizer
Linear Projection
Query
Trans Decador Layer
Key
Positional Encoding
Positional Encoding
Zero Initialized
Generated Text Hidden States
The decoder logits are then quantized into discrete audio semantic tokens A as defined by the Spark-TTS audio tokenizer 77.. Conditioned on these semantics and the global speaker-identity embeddings, the tokenizer reconstructs the continuous waveform segment.
Placeholders
Voiceprint
Figure 3: Architecture of TempoVoice. Tempo TempoVoice transforms textual hidden- state embeddings into audio segments.
Voice.
5

---

3.2 Training Objectives
To train OmniResponse, the training objective a weighted combination of text generation loss Liexts vision reconstruction Liisions and audio generation loss Landio:
C= Lext + Avision-visson Naudio Caudio
(I)
where Avision and Aaudio are the scaling factors balancing text, vision, and audio loss terms.
Text Generation Loss. The text loss encourages accurate next-token prediction conditioned on both speaker context and past listener states:
Liext
Σ
Po po (W?
W Winstructs W Whistory, F. F
FL 1 W 7:t-1
W W
(2)
Vision Reconstruction Loss. To align predicted and ground-truth facial dynamics, we apply an l reconstruction loss on the listener's feature embeddings:
Cvision DILF -FIE
Audio Generation Loss. The audio loss operates over discrete semantic tokens A indexed by 1r, which correspond to frame indices / puk (k is the downsampling factor). We maximize the likelihood of each token conditioned on previous audio semantics and the listener's hidden states:
Caudio - -E log po (A
A H(-8+1:),
where H denotes the model's hidden representations for the corresponding listener text tokens W! -+1-- This formulation ensures coherent alignment across modalities throughout generation.
4 Dataset Construction
Existing publicly available dyadic video datasets do not satisfy the requirements of the OMCRG task (See Figure TT. For example, mono-view talking-head datasets and offline dialogue corpora (e.g., MultiDialog [5]) do not offer split-screen recordings that capture speaker and listener simul- taneously. Others, such as IEMOCAP [II] feature predominantly side profile views recorded in noisy environments and provide only mixed audio channels, thus preventing separate analysis of
each participant's s speech. Furthermore, datasets such as ViCo [82], ICD 47), and REACT2024 [6]]
lack comprehensive textual annotations, suffer from low video resolution 82 M 61]. or exhibit
inconsistent spoken languages [61]. To fill the dataset gap. we introduce ResponseNet that comprises 696 temporally synchronized dyadic video pairs, totaling over 14 hours of natural conversational exchanges. Each pair provides high high-resolution (1024 x 1024) frontal-face streams for both speaker and listener, along with separated audio channels to support fine- mne-grained analysis of verbal and nonverbal behavior. T shows ResponseNet is the only dataset that satisfies the key requirements: (1) online video streaming, (2) separate audio channels, and (3) textual word-lewe annotations for both participants.
The construction of ResponseNet follows rigorous workflow that integrates automated tools with extensive human-in-the-loww curation. (1) Initially, split-screen videos featuring simultaneous appear- ances of speaker and listener are sourced from YouTube according predefined topic and quality criteria. These clips are then filtered to remove low-resolution, noisy, or frequent camera transitions. (2) Human annotators perform a thorough review correct camera-view mis-alignments and ensure precise temporal synchronization between streams. (3) Next,
audio tracks are auto- matically separated into discrete speaker and listener channels using speaker separation tools such as MossFormer2 [80] and subsequently verified and refined by experts. Finally, word-level transcripts are generated via automatic speech recognition (55) and meticulously proofread to guarantee accuracy. By combining automation with meticulous manual oversight across data sourcing, preprocessing alignment, audio separation, and annotation, this pipeline yields high-quality, richly annotated dyadic video corpus ideally suited for multimodal conversational response generation.

---

denote speaker and listener data re- spectively. ResponseNet provides complete multimodal data (speaker+listener) with their separated
Table I: Comparison of conversation datasets. and
audios.
Dataset
Video
Audio
Text
Online
Separated Audios
# Dialogues
Total Duration
MultiDialog 51
8,733
339.7h
ICD [47
182,132
72h
ViCo 83
X
X
483
1.6h
REACT2024 62
/
5,919
71.8h
IEMOCAP
/
151
11,5h
ResponseNet
696
14.2h
relationship
Number of Videos
150
Duration: 39
Toxal Vidkes
1000
power stare th
Mudiam Dutation: 55.65
Min Duration: 27.13
Intimate
f camily y
863.13 seconds
vol Barray
comptiy
ResponseNet
inteddddpinary
Exart Interdeww
50
(696)
Ashool
Interveew
News
WOI
130
Commentary
Duration (eeconds)
(a)
(b)
(c)
Figure 4: Statistics of ResponseNet. (a) Distribution of video clip durations. (b) Distribution of
dyadic conversation topics. (c) Word cloud of spoken words in dyadic conversations.
The statistics of ResponseNet are shown in Figure The durations of speaker-listenes video clips 13 seNet. Figure shows that the average clip duration in ResponseNet is 73.39 seconds, significantly longer than that of other dyadic datasets such REACT2024 (30 seconds), and ViCo (9 seconds). This extended duration ensures that each clip captures sufficient conversational exchanges. Fig- ure (b) illustrates that the conversations span diverse range of topics, including professional discussions (e.g., economic interviews, news commentaries), emotionally driven interactions (e.g.. intimate conversations), educational settings (e.g., teaching interviews), and interdisciplinary expert discussions. (igure(4(c) presents a word cloud highlighting the most frequent words in the conversa- tions. Such diversity shows that ResponseNet captures rich and varied human-human interactions rather than being restricted to narrow or monotonic conversation patterns.
range from 27.13 seconds (short conversations) to 863.13 (3 seconds (long conversations) in Respon-
5 Experiments
Implementation Details. Our framework was implemented using PyTorch [52) and trained on four NVIDIA Tesla A100 GPUs The model optimization was performed using the Adam W optimizer [3) with learning rate of 2 x 10-5, 31 0.9, 32 0.999, and a weight decay of 10-4, accompanied by a cosine learning rate scheduler. Training was executed with a batch size of one for 2,000 epochs. Additionally, we fine-tuned the LLM using the LoRA (26) technique with a LoRA rank of 64 and LoRA alpha value of 16. More implementation details are provided the Appendix.
Evaluation Metrics. Quantitatively evaluating the quality of multimodal response generation remains non-trivial. We thereby employ comprehensive metrics evaluate generation results across text, audio, and visual modalities. For text response, we use METEOR [9], BERTScoreFL [78). and ROUGE-L [38] to measure how appropriate and natural the generated responses are, based on reference responses from the ResponseNet test set. We also adopt Distinct-2 36 to evaluate
diversity through the ratio of unique bi-grams. For audio response, we adopt UTMOSv2 6 a neural MOS predictor that estimates the perceptual naturalness, and employ LSE-D [54 16 SpeeSh Error Distance) to evaluate synchronization between generated speech and lip movements. For facial response, we compute Fréchet Distance (FD) between real and generated facial-feature distributions, and Fréchet Video Distance (FVD) 68 to assess the apatial-temporal visual quality of generated video sequences.
7

---

Table 2: Quantitative Results on ResponseNet test set.
Text
Audio
Video
Model
METEOR
BERTScoreF)
ROUGE-L11
Distinct-2
LSE-D
UTMOSv2
FD
FVD
Ground-Truth
Offinee Text Dialogue Generation System
GPT-4o 2
0.167
0.805
0.079
0.928
GPT-4 2
0.163
0.822
0.082
0.960
GPT-ol 2
0,189
0.822
0.113
0.948
Qwea-7B Bane-B--Chi Bha
7
0.167
0.807
0.090
0,920
Clande Sonnet
S
0.183
0.807
0.101
0.966
Gemini-2.5 F.5
17
0.175
0.824
0.085
0,932
DeepSeck R1
22
0.173
0.815
0.078
0.981
Online Auditory Dialogue Generation System
Moshi
[S
0.120
0.818
0.078
0.499
2.21
Reaction
System
ReactFace 41
32.72
340.28
ViCo 83
Online Multimodal Comersational Response Generation Baseline
LSTM
0.042
0.716
0.000
9.72
320.92
Audio-visual LLM
0.030
0.662
0.020
0.155
10.03
1.32
580.86
6815555
OmniResponse (Ours)
0.141
0.806
0.081
15.46
5.1
Quantitative Results
To the best our knowledge, few works have explored the OMCRG task before. We build two baselines and compare them in Table 2 (1) LSTM-based method employing recurrent neural network [25] for temporal sequence modeling; (2) Audio-visual LLM taking gpeaker-listena audio and visual inputs and leveraging pre-trained LLM to generate audio-visual frames autoregressively.
Table 2 further reports the generation performance of representative mooglitmddality baselines,
including offline, text-only dialogue models (e.g.. GPT variants 22. Dwen-7B-Chee 77]. Claude- Sonnet-4 (5] (version 2025-05-14), Gemini-2.5-Flaan
17), and DeepSeek-RI [22] (version 2025- 05-28)), online audio-only generation models (e.g., Moshi (18),, and facial reaction generation approaches 41 83 Different from these methods focusing on generating single modality, our method enables online, synchronized generation across audio, visual, and textual modalities for modeling human conversation.
Table 2 shows that OmniResponse achieves the best performance in dialogue speech content (METEOR, BERTScore F1. ROUGE-L, Distinct-2), audio quality (UTMOSv2), audio-visual syn- chronization (LSE-D), well temporal consistency and visual quality (FVD). Although the LSTM baseline achieves a lower FD owing to its tendency to produce repetitive static visual output, it fails to generate rich, synchronized multimodal responses. Audio-Visual LLM does not incorporate the text modality. compared to our method. Consequently, Audio- Visual LLM achieves much lower speech content quality (METEOR and BertScore #1) and struggles with audio-visual synchronization (LSE- D) than our method. Although Audio Visual LLM leverages a powerful LLM, it is still challenging directly synchronize generated audio with facial reactions, especially in the absence of strong audio foundation model.
Our OmniResponse model significantly outperforms Audio- visual LLM across all evaluated metrics, including non-verbal ones. These results demonstrate that introducing text as an intermediate modality greatly enhances the naturalness and realism of non-verbal responses, reflected by the FD and FVD scores. Moreover, we introduce a novel framework that effectively adapts pre-trained LLMs for audio-visual generation with the proposed Chrono-Text Markup and Tempo Voice.
5.2 Qualitative Results
Figure 5 presents a qualitative result. The synthesized listener remains silent while the speaker is speaking, but then produces an immediate or delayed response at the end of each speaker turn. This behavior demonstrates that OmniResponse effectively captures the temporal dynamics of online dyadic conversation and generates responses at appropriate timestamps. For example, between 100.97 and 132.05 s, the listener interjects briefly between 120.13 and 121.57 in response to the speaker's ongoing content, reflecting natural human conversational interaction. In contrast,

---

Generated Listener Multi-model Response
Timeline (second)
This part the conversation and then want tur to your All project, but just the idea of the masses. People born into dircumatances help.
0.00
<PAUSE><PAUSEY.... <PAUSE=-PAUSEN... <PAUSE><PAUSEE..
Now taking that context, what is the appropriate way to help the poor, the weak in the sense of being born somewhere without advantage, without being able maybe wealth, without access some opportuntiiss other people have accees to?
1000
How do we help the weak, the poor, those that reaily could use
<PAUSE><PUUSE>.. <PAUSE><PAUSES. <PAUSE>>PAUSSE...
up ladder?
91.50
96.43
so excited that's going interesting question. think would just the time and what you do with your life but that's a thing be able about something like this.
really
Well, obviously, honest one..
98.51
100.37
Everyone has helped way. There have been people helped There people generous along the way.
100.97
-PAUSE><PAUSE
<PAUSE>
<AUSE>>PAUSES
We need to steal from someone confiscate from someone give and redistribute
120.13
We were both so you' free
somewhere also.
121,57
when you just give something don't know, handed to you too casily.
<PAUSE><PAUSE>...
and you don't appreciate it, that's not really helpful either.
<PAUSE>=PAUSE>.
132.05
131,70
I'm sure too. were going to be talking about this topic. So i'm actually curious if you know the people that were on your side, and I'm curlous about why
their justifiatioon
difficult you
Oh yeah, was just going say that you know we need them the world they do a of good stuff and then there's like this other thing.
Speaker
Listener
142.57
Figure 5: Qualitative Results. Given the speaker's audio and video streams and corresponding utterances (left), OmniResponse autoregressively generates synchronized visual, audio, and textual response streams (right). For clarity, [LASTING] tokens are removed from the generated dialogue.
conventional pipeline that integrates Automatic Speech Recognition (ASR), dialogue generation, TTS, and talking-head components waits for predefined silence threshold before producing offline multimodal response, thus diminishing conversational behaviors such as interruptions, backchannels, questions, and immediate feedback. In contrast, OmniResponse maintains the continuous flow of dyadic conversation by continuously modeling and generating synchronized time series streams of textual, visual, and audio outputs.
5.3 Ablation Studies
Effectiveness of Chrono-Tex: Markup. We construct baselines removing the proposed Chrono-Ter Markup from our OmniResponse, In the baselines, each predicted word is assigned a timestamp indicating when it emerges; if this timestamp falls within temporal window around the current time, the word is retained and appended the spoken output; otherwise, is discarded. As shown in the last rows of Table 3 incorporating Chrono- Text Markup significantly improves audio-visual synchronization, reducing the LSE-D score from 11.51 to 9.56. In addition, it enhances the semantic alignment of speech with conversational context, increasing METEOR from 0.122 to 0.141 and BERTScoreF from 0.766 to 0.806. Improvements in FD and UTMOSv2 further indicate that Chrono- Text Markup boosts the quality of the generated audio and facial responses. These results demonstrate the effectiveness of Chrono-Teas Markup in generating high-quality multimodal responses.
Effectiveness of TempoVoice. To study the effect of our Tempo we remove it from our framework and instead directly feed the hidden states, which are trimmed or padded to match the target audio length, into multi-layer perceptron to predict audio token logits. As shown in Table removing Tempo Voice degrades audio-visual synchronization and reduces the quality of generated audio responses, where 2 drops from 1.41 to 1.23, and LSE-D increases from 9.56 to 11.91.

---

Table 3: Ablation study on the effects of the proposed Chrono-Text Markup and Tempo Voice.
Chrono-Text Markup
Tempo Voice
METEOR
BERTScore
LSE-D
UTMOSv2
FD
X
0.090
0.755
13.64
1.21
596.27
X
0.128
0.778
11.91
1.23
19.58
/
0.122
0.766
11.51
1.39
23.42
/
0.141
0.806
9.56
1,41
15.46
Table 4: User study (A/B preference: higher is better). Each cell shows the percentage of participants preferring Ours.
Criteria
Ours vs. LSTM
Ours vs. Audio-Visual LLMM
Speech Content Appropriateness
75.5%
81.6%
Audio Speech Quality
77.6%
85.7%
Visual Quality
67.3%
93.4%
Audio-Visual Synchronization
91.8%
95.9%
These results highlight the importance of Tempo in temporally aligning audio with the other modalities and enhancing the quality of the generated audio.
5.4
User Study
We conducted user study with 49 participants (28 male, 21 female). Each subject viewed 16 randomly ordered clips and rated speech content appropriateness, audio speech quality, visual quality, and audiovisual synchronization. All participants were proficient in English (53.1% 1% reported advanced proficiency or daily-commn daily-communication ability). Educational attainment was high: 95.9% held at least an undergraduate degree, 44.9% held master's degree, and 18.4% held Ph.D. Ages were distributed as follows: 14.3% under 25, 34.7% aged 26-35, 24.5% aged 36-45, and 26.5% aged 46-55. In direct A/B preferences, "Ours" achieved minimum preference of 67.3% (speech content appropriateness vs. LSTM) and maximum of 95.9% (audiovisual synchronization vs. Audio- Visual LLM).
6 Conclusion and Discussion
We have presented OmniResponse, online multimodal generation model that produces verbal and nonverbal listener responses to a speaker's multimodal behaviors. OmniResponse integrates techniques for processing multimodal imputs, synchronizing across modalities, and aligning responses with the speaker's content. To enable evaluation of this task. Online Multimodal Conversational Response Generation in Dyadic Interactions, introduce ResponseNet. dataset containing parallel recordings of speaker and listener streams. Our model and dataset lay the foundation for future research this emerging field. Experimental results demonstrate that OmniResponse significantly increases speech semantic content, audio- visual synchronisation, audio and visual quality.
Linitations. While approach performed well the evaluated datasets, the remaining challenges include the proposed approach (e.g., its results) may largely depend on the quality and diversity of training data, replying accurate speaker-listener segmentation and can be negatively affected in noisy or overlapping conversations. Additionally, generating well-aligned multi-modal responses remains difficult in fast-changing emotionally rich interactions, while paper lacks fairness analysis. Future work will focus on improving these aspects.
Risks and Potential Misuse. This system is developed for multi-modaa conversational AI, but certain risks should be acknowledged. For instance, realistic synthetic contents could be misused [59) for impersonation or misleading information. During real-time human-used user interactions, users may also develop misunderstandings or excessive reliance on system without proper contents control. To avoid these risks, we recommend clear labeling of the generated contents, appropriate usage monitoring, and the inclusion of protective measures against potential misuse.
20 75
(e.g., Deepfake Detection [53 29
10

---

Acknowledgments. This work is supported by the KAUST Center of Excellence for Generative Al under award number 5940. The computational resources are provided by IBEX, which is managed by the KAUST Supercomputing Core Laboratory.
References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404.14219 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2033::2303.08774, 2023.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 0122371--23738 2022.
[4] Helmut Alt and Michael Godau. Computing the fréchet distance between two polygonal curves. International Journal of Computational Geometry & Applications, 5(01n02):75-91, 1995.
[5] Anthropic. Introducing claude 4 (opus 4 and sonnet 4). news/claude- 2025. Model announcement and overview.
com/
[6] Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. The t05 system for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality synthetic speech. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 818-824, IEEE, 2024.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint vrXiv:2309.166000 2023.
[8] MM Bakhtin. The problem of speech genres. 1987.
[9] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[10] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
[11] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim. Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
[12] Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth André, and Michel Valstar. The noxi database: multimodal recordings of mediated novice-expert interactions. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 350-359, 2017.
[13] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-1133 11355, 2022.
[14] Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-Ilm for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv:24/ 2. 2024.
[15) Luo Cheng, Song Siyang, Yan Siyuan, Yu Zhen, and Ge Zongyuan. Reactdiff: Fundamental multiple appropriate facial reaction diffusion model. arXiv preprint arXiv:2510.04711 2025.
11

---

[16] Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops. Taipei. Taiwan, November 20-24, 2016, Revised Selected Pupers, Part II 13, pages 251-263. Springer, 2017.
[17] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06611, 2025.
[18] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time sale dialogue. arXiv preprint arXiv:2410.000 10.000077. 2024.
[19] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEE/CVE conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[20] Yuan Gan, Jiaxu Miao, Yunze Wang, and Yi Yang. Silence is golden: Leveraging adversarial examples nullify audio control in Idm-based talking-head generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13434-13444 13444, 2025.
[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 ())11))399-14, 2020.
[22] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[23] Dirk KJ Heylen. Understanding ppeaker-listenee interaction. In 10th Annual Conference of the International Speech Communication Association, INTERSPEECH 2009, pages 2151-2154. International Speech Communication Association (ISCA), 2009.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[25] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1735:1735 1997.
[26] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. 1(2):3, 2022.
[27] Yuchi Huang and Saad Khan. Dyadgan: Generating facial expressions in dyadic interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 11-18, 2017.
[28] Yuchi Huang and Saad M Khan. Generating photorealistic facial expressions in dyadic interac- tions. In BMVC, page 201, 2018.
[29] Naseem Khan, Tuan Nguyen, Amine Bermak, and Issa Khalil. Unmasking synthetic realities in generative ai: A comprehensive review of adversarially robust deepfake detection systems. arXiv preprint arXiv:2507.21117. 2025.
[30] Do Yuon Kim, Ha Kyung Lee, and Kyunghwa Chung. Avatar-mediated experience in the metaverse: The impact of avatar realism on user-avatar relationship. Journal of Retailing and Consumer Services, 73:103382, 2023.
[31] Everlyne Kimani, Timothy Bickmore, Ha Trinh, and Paola Pedrelli. You'll be great: virtual agent- cognitive restructuring to reduce public speaking anxiety. In 2019 8th international conference on affective computing and intelligent interaction (ACII), pages 647- IEEE, 2019.
[32] Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 1.N3::3312.6144... 2013.
12

---

[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[34] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, 2024.
[35] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52008, 2023.
[36] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promooing objective function for neural conversation models. arXiv preprint arXiv:1510.03055, 2015.
[37] Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024.
[38] Cew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[39] Christine Lisetti, Reza Amini, Ugan Yasavur, and Naphtali Rishe. I can help you change! an empathic virtual agent delivers behavior change health interventions. ACM Transactions on Management Information Systems (TMIS), 4(4):1-28, 2013.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[41] Cheng Luo, Siyang Song, Weicheng Xie, Micol Spitale, Linlin Shen, and Hatice Gunes. Reactface: Multiple appropriate facial reaction generation in dyadic interactions. arXiv preprint arXiv:2305. 15748, 2023.
[42] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. /5505, 2023.
[43] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1411 /784, 2014.
[44] Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, and Kei Sawada. Pslm: Parallel generation of text and speech with lims for low-latency spoken dialogue systems. arXiv preprint arXiv:2406. 12428, 2024.
[45] David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling, Advances in Neural Information Processing Systems, 36, 2024.
[46] Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using ceattrorram-poweree Ilm. arXiv preprint arXiv:2305.15255, 2023.
[47] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20395-20400 2022.
[48] Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar. Can language models learn to listen? In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10083-10093, 2023.
[49] Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Maviyutov, Itai Gat, al. Spirit-im: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755 2024.
13

---

[50] Cristina Palmero, Javier Selva, Sorina Smeureanu, Julio Junior, CS Jacques, Albert Clapés, Alexa Mosegui, Zejian Zhang, David Gallardo, Georgina Guilera, et al. Context-aware per- sonality inference in dyadic scenarios: Introducing the udiva dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1-12, 2021.
[51] Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let's go real talk: Spoken dialogue model for face-to-face conversation. arXiv preprint 20t.:22000.0766,, 2024.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[53] Gan Pei, Jiangning Zhang, Menghan Hu, Zhenyu Zhang, Chengjie Wang, Yunsheng Wu, Guangtao Zhai, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. arXiv preprint arXiv:2403. 17881, 2024.
[54] KR Prajwal, Rudrabha Mukhopadhyay. Vinay Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multinedia, pages 1844492,44 2020.
[55] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International conference machine learning, pages 28492-28518, PMLR, 2023.
[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25,, 1(2):3, 2022.
[57] Fabien Ringeval, Andreas Sonderegger. Juergen Sauer, and Denis Lalanne. Introducing the recola multimodal corpus of remote collaborative and affective interactions. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pages 1-8. IEEE, 2013.
[58] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 12925, 2023.
[59] Hadi Salman, Alaa Khaddaj. Guillaume Leclere, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing. arXiv preprint arXiv:2302.06588 2023.
[60] Siyang Song, Micol Spitale, Xiangyu Kong, Hengde Zhu, Cheng Luo, Cristina Palmero, German Barquero, Sergio Escalera, Michel Valstar, Mohamed Daoudi, et al. React 2025: the third multiple appropriate facial reaction generation challenge. arXiv preprint arXiv:2505.1722: 2025.
[61] Siyang Song, Micol Spitale, Cheng Luo, German Barquero, Cristina Palmero, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth Andre, et al. React2023: The first multiple appropriate facial reaction generation challenge. In Proceedings of the 31st ACM 2023.
International Conference Multimedia, pages
[62] Siyang Song, Micol Spitale, Cheng Luo, Cristina Palmero, German Barquero, Hengde Zhu, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth André, and Hatice Gunes. React 2024: the second multiple appropriate facial reaction generation challenge. In 2024 IEEE 18th International Conference Automatic Face and Gesture Recognition (FG), 2024.
[63] Siyang Song, Micol Spitale, Yiming Luo, Batuhan Bal, and Hatice Gunes. Multiple appropriate facial reaction generation dyadic interaction settings: What, why and how? arXiv preprint 20X1v:2302.06514, 2023.
14

---

[64] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[65] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint MxXi::2405009//1 2024.
[66] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244-260. Springer, 2024.
[67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 13977,, 2023.
[68] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski. and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: / 2018.
[69] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[70] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[71] Xinsheng Wang. Mingqi Jiang, Ziyang Ma, Ziyu Zhang. Songxiang Liu, Lingin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient lim- text-oo-ppecc model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025.
[72] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
[73] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint a409 1140, 2024.
[74] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /40.::22....2228.. 2024.
[75] Haotian Xue, Chumeng Liang, Xiaoyu Wu, and Yongxin Chen. Toward effective protec- tion against diffusion- mimicry through score distillation. In The Twelfth International Conference on Leurning Representations, 2023.
[76] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
[77] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang. Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-moda modal conversational abilities. arXiv preprini arXiv:2305.I 1000, 2023.
[78] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 1904.0965, 2019.
[79] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages $652-8661, 2023. 2023.
15

---

[80] S Zhao, Y Ma, Ni, C Zhang, H Wang, TH Nguyen, K Zhou, J Yip, D Ng, and B Ma. Mossformer2: Combining transformer and mn-free recurrent network for enhanced time- domain monaural speech separation. arxiv 2024. arXiv preprint 2
[81] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with multi modal model. arXiv preprint arXiv:2408. 11039, 2024.
[82] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9299-9306, 2019.
[83] Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. In European Conference on Computer Vision, pages 124 142, 2022.
[84] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), (9(6):1-15, 2020.
[85] Hengde Zhu, Xiangyu Kong, Weicheng Xic, Xin Huang. Linlin Shen, Lu Liu, Hatice Gunes, and Siyang Song. Perfrdiff: Personalised weight editing for multiple appropriate facial reaction generation. In ACM Multimedia 2024, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's main claims are clearly supported by its contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in our supplementary materials (appendix).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification: Our work focuses on applications of multimodel generative model.
Guidelines:
The paper does not include theory assumptions proofs The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed implementation information in the main paper, including model architecture, training objectives, optimization settings, and evaluation protocols and on in our supplementary mateirals.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in the supplemental material?
Question:
Answer: [No]
Justification:
All code and data will be made available upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nnps.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/pplle wttp:ccppppcccceeeeeeeeee..c.ete httpa//gguuiee//eeeuubiisisssss /Cc////oddiiissseeeeeeee ssiisssiiciiiyyyycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provided implementation details our main manuscript and more rele- vant611 information in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed our supplementary materials
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss safeguards in the supplementary materials
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new dataset and codebase as part of this work. Due to ongoing internal approval, the full release will be made publicly available upon paper acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We collected videos from platforms that explicitly grant permission for research usage. The videos do contain personally sensitive or private content, and are used in accordance with each platform's 's terms of service. No direct interaction or compensation with individuals was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve any direct interaction with human subjects. All data were collected from publicly available sources
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We used a large language model solely for minor linguistic improvements.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neuree 22tps//eec.coooeeeeeee///2////// for what should or should be described.
23


Reviews:
[{"review_id": "eH7rpZxCg8", "reviewer": "Reviewer_bqe1", "summary": "This paper introduces Online Multimodal Conversational Response Generation (OMCRG), a new task that advances multimodal conversational AI for real-time human-computer interactions. It presents a Multimodal Large Language Model (MLLM) that generates high-quality multimodal listener responses autoregressively. The paper also introduces ResponseNet, a dataset of 696 dyadic interactions with synchronized split-screen videos, multichannel audio, transcripts, and facial behavior annotations. The model generates synchronized verbal and non-verbal responses based on the speaker’s inputs. Key innovations include Chrono-Text, which aligns text tokens with facial expressions and speech, and TempoVoice, a controllable TTS system that synchronizes speech with facial reactions. Experiments demonstrate the effectiveness of the proposed method.", "questions": "As listed in the Weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n(1) Applicability: This paper proposes a novel task of real-time synchronization, designing a model that addresses the complex challenge of synchronizing multiple modalities—audio, facial expressions, and text—positioning it as a promising candidate for future human-computer interactions.\n(2) Effectiveness: The proposed dataset proves to be highly beneficial for this task, and the model achieves performance that surpasses existing alternatives.\n\nWeaknesses:\n(1) Dependency on Text Modality: Although text offers clarity, relying on it for synchronizing audio and facial reactions may constrain the richness of non-verbal communication. The paper lacks a thorough discussion on the quality of non-verbal generation and the evaluation of text as an intermediary modality.\n(2) Real-Time Applicability: While this paper introduces an online task, it does not address the potential time consumption required for real-time streaming audio-facial generation. Given that the training data includes sequences up to 863 seconds in duration, questions remain about how to efficiently handle visual and textual history when processing long sequences and how to minimize latency in such scenarios.\n(3) Lack of Error Analysis: The paper lacks a detailed error analysis or discussion on failure cases, which would be useful to understand the model's limitations in edge cases.", "was_revised": true, "final_justification": "As the authors solve my main concern, I would like to raise the score.", "rebuttal": "### Rebuttal to Reviewer bqe1\n\n\nWe sincerely thank the reviewer for the positive comments and constructive suggestions. In particular, we are delighted that you recognize our work as **“a novel task”**, that it **“advances multimodal conversational AI for real-time human-computer interactions”**, that it **“positions it as a promising candidate for future human-computer interactions”**, that the **“proposed dataset proves to be highly beneficial”**, and that our model **“generates high-quality multimodal listener responses.”** Your encouragement is greatly appreciated, and we will refine our manuscript carefully in accordance with your suggestions. \n\n\n\n>  ####  **Q1: Dependency on text modality.**\n\nA: Thank you for your comments.  We have conducted experiments  to evaluate whether relying on text would constrain the richness of non-verbal communication (see Table 2 of the main paper and table below).  That is, we build a baseline, namely **w.o Text** (i.e., Audio-visual LLM), which removes text modality from our method. We also provide another baseline, namely, LSTM-based method, employing widely-used network, i.e, a recurrent neural network to learn from only audio and visual modality.\n\n\n\n\n| Method                         | BERTScore F1 ↑ | Distinct-2 ↑ | LSE-D ↓ | UTMOSv2 ↑ | Non-verbal (FD) ↓ | Non-verbal (FVD) ↓ |\n|--------------------------------|----------------|--------------|---------|-----------|-------------------|--------------------|\n| LSTM                           | 0.716          | 0.000        | 9.72    | 1.21      | 6.51              | 320.92             |\n| w.o Text (Audio-visual LLM)    | 0.662          | 0.155        | 10.03   | 1.32      | 580.86            | 681.55             |\n| OmniResponse (Ours)            | **0.806**      | **0.882**    | **9.56**| **1.41**  | **15.46**         | **314.94**         |\n\n\nOur OmniResponse significantly outperforms the w.o Text baseline across all evaluated metrics, including non-verbal ones. These results demonstrate that introducing text as an intermediary modality significantly enhances the naturalness and realistic of non-verbal responses, as indicated by the FD and FVD scores. Similarly, our method outperforms the LSTM-based approach by a considerable margin.\n\nThis superior performance is due to the combination of the intermediate text modality and our method's design:\n\n*  Text removes speaker-specific pitch and timbre, avoiding the negative effects by  noise or variability in the audio while  enabling the LLM to focus on intent understanding.\n*  Unlike raw speech, text does not affected by sampling rate or background audio noise, reducing the training difficulties for large-scale pre-training.\n* We introduce the Vision Projection module and Vision Decoder, and conduct Multimodal Context Modeling, allowing our model to learn from both visual modality itself and cross-modal correlations.\n\nWe will add more discussions in the final version of our paper.\n\n\n\n>  ####  **Q2: How to efficiently handle visual and textual history when processing long sequences and how to minimize latency in such scenarios.**\n\n\nA: Thank you for your comments. Our method  handles long sequences and minimizes latency as follows: \n* We limit temporal inputs to a sliding window spanning the most recent $\\Delta T$ = 2.2s (from time $\\tau$ to $t-1$). All tokens preceding $\\tau$ are distilled into a static textual summary, far cheaper than keeping them as full temporal tokens, shrinking the context length.\n* For the visual stream, only a short sequence of speaker‑listener face tokens is retained. \n\n* With these strategies, our system sustains 15.6 FPS (≈ 64 ms end-to-end delay for audio plus facial animation) on a single NVIDIA A100 80 GB GPU. \n\nTo our knowledge, this is the first online multimodal conversational response generator. We outline additional efficiency improvements as future work, and the revised manuscript will detail the mechanisms that enable our pipeline to approach real-time operation.\n\n>  ####  **Q3: Discussion on Failure cases.**\n\nA: Thank you for your suggestion. Follow your suggestion, we conduct a detailed analysis of failure cases. First, we observe that when the speaker’s speech contains large background noise, our method would  wrongly generate  listener responses.  Second, our model rarely produces head rotations beyond approximately ±40°, and the resulting side-view frames can exhibit noticeable distortions. To quantify these effects, we extracted side-face and front-face frames from our generated videos and performed the following realism evaluation:\n\n| Category          | FVD ↓  |\n| ----------------- | ------ |\n| Side-face frames  | 478.31 |\n| Front-face frames | 312.69 |\n| All frames        | 314.94 |\n\nWe will supplement these analyses and more failure case discussions in our revised version.\n\n>  ####  **Q4. Typo.**\n\nA: Thanks for your suggestion. We apologize for the typos and have corrected these typos."}, {"review_id": "pJi6LrZvJI", "reviewer": "Reviewer_dTxZ", "summary": "This paper introduces Online Multimodal Conversational Response Generation (OMCRG), a novel task focused on generating synchronized verbal and non-verbal listener feedback in real-time based on a speaker's multimodal input. To address the challenges of synchronization between generated audio and facial responses, the authors propose OmniResponse, a Multimodal Large Language Model (MLLM) that uses text as an intermediate modality. OmniResponse incorporates two new components: Chrono-Text, for temporally anchoring text tokens, and TempoVoice, a controllable online Text-to-Speech (TTS) module synchronized with facial reactions. Additionally, the paper presents ResponseNet, a new high-quality dataset of 696 dyadic interactions with synchronized split-screen videos, multichannel audio, transcripts, and facial behavior annotations to support OMCRG research. Evaluations on ResponseNet show that OmniResponse significantly outperforms baseline models in semantic speech content, audio-visual synchronization, and generation quality", "questions": "See the disadvantages.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Advantages\n\n1.This work introduces OmniResponse, the first online model to jointly process and generate synchronized streams of conversational human behavior.\n\n2.This work further propose ResponseNet, an annotated dyadic conversation dataset and benchmark, enabling standardized evaluation of OMCRG models.\n\n3.This work consider a real-world scenario: Online Multimodal Conversational Response Generation.\n\nDisadvantages\n\n1.The baselines in the main experiment could be more comprehensive (just see the GPT-series models)\n\n2.The auxiliary experiments lack depth.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns; I will keep my original score.", "rebuttal": "### Rebuttal to Reviewer dTxZ\n\nWe sincerely thank the reviewers for their generous feedback.\nWe are encouraged that the reviewer agrees that our work introduces **\"a novel task\"**, \"considers a **real-world scenario**: Online Multimodal Conversational Response Generation\",  proposes **\"a new high-quality dataset \"** to support OMCRG research, \"the **first** online model\", and the model \"significantly outperforms baseline models\".\n\n\n\n> #### **Q1. The baselines in the main experiment could be more comprehensive (just see the GPT-series models)**\n\nA: Thanks for your suggestions. We have included the most powerful language-based conversation models (GPT-4o, GPT-4, and GPT-o1), and we also supplement more baselines:\n- Qwen-7B-Chat, \n- Gemini-2.5-Flash, \n- Claude-Sonnet-4-20250514, \n- DeepSeek-R1-200528, \n\nand the results are listed below:\n\n\n| Model                                      | METEOR ↑ | BERTScore F₁ ↑ | ROUGE‑L ↑ | Distinct‑2 ↑ |\n|--------------------------------------------|:--------:|:--------------:|:---------:|:-----------:|\n| **Offline Text Dialogue Generation System**|          |                |           |             |\n| GPT‑4o                                     |  0.167   |     0.805      |   0.079   |    0.928    |\n| GPT‑4                                      |  0.163   |     0.822      |   0.082   |    0.960    |\n| GPT‑o1                                    |  0.189   |     0.822      |   0.113   |    0.948    |\n| Qwen‑7B‑Chat                               |  0.167  |     0.807     |  0.090   |    0.920   |\n| Claude‑Sonnet‑4‑20250514                   |  0.183  |     0.807     |  0.101   |    0.966   |\n| Gemini‑2.5‑Flash                           |  0.175  |     0.824     |  0.085   |    0.932   |\n| DeepSeek‑R1‑20250528                       |  0.173  |     0.815     |  0.078   |    0.981   |\n\n\n\nWe will add these results in the revised version.\n\n> #### **Q2. The auxiliary experiments lack depth.**\n\nA: Thank you for this valuable suggestion. In addition to the ablation study in Table 3, we carried out a comprehensive failure-case analysis to pinpoint where our model underperforms. Specifically, we found that our system seldom generates head rotations beyond ±40°, which can introduce noticeable distortions in side-view frames. To quantify this effect, we extracted side-face and front-face frames from our generated videos and measured realism via Fréchet Video Distance (FVD):\n\n| Category          | FVD ↓  |\n| ----------------- | ------ |\n| Side-face frames  | 478.31 |\n| Front-face frames | 312.69 |\n| All frames        | 314.94 |\n\n\n\n\nWe also conducted a user study (with 49 participants) to assess human preferences in pairwise A/B tests:\n| Criteria                       | Ours vs. LSTM | Ours vs. Audio–Visual LLM |\n| ------------------------------ | -----------: | -----------------------: |\n| Speech Content Appropriateness |       75.5\\% |                   81.6\\%|\n| Audio Speech Quality           |       77.6\\% |                   85.7\\% |\n| Visual Quality                 |       67.3\\% |                   93.4\\% |\n| Audio–Visual Synchronization   |       91.8\\% |                   95.9\\% |\nTable: Percentage of times participants preferred our method over each baseline.\n\n\n\nIn addition, we had built a baseline, namely, Audio-visual LLM, that omits the immediate text modality from our method in Tab. 2 of the main paper, to validate the effectiveness of adding text as an intermediate modality. The Audio-visual LLM baseline directly fuses visual and audio embeddings from both the speaker's and listener's past behavior to predict the listener’s next visual and audio tokens.  Table 2 ( also shown below) demonstrates that introducing text as an intermediary modality significantly enhances the naturalness and realism of non-verbal responses, as indicated by the FD and FVD scores. \n\n\n| Method                         | BERTScore F1 ↑ | Distinct-2 ↑ | LSE-D ↓ | UTMOSv2 ↑ | FD ↓ |  FVD ↓ |\n|--------------------------------|----------------|--------------|---------|-----------|-------------------|--------------------|\n| Audio-visual LLM    | 0.662          | 0.155        | 10.03   | 1.32      | 580.86            | 681.55             |  |\n| OmniResponse (visual-audio-text)            | **0.806**      | **0.882**    | **9.56**| **1.41**  | **15.46**         | **314.94**         |\n\n\nWe will include these auxiliary experimental results into the final version by adding a latency study, a user study, error‑case analysis, and ablations across different LLM variants. We would also highlight that our primary **contribution** is **task formulation**, a **first**, end‑to‑end online generation model,  an **evaluation benchmark**, and a **high‑value dataset** for this novel task."}, {"review_id": "AdDUFH45eY", "reviewer": "Reviewer_FU3z", "summary": "This paper introduces Online Multimodal Conversational Response Generation (OMCRG), a novel task to generate real-time synchronized verbal and non-verbal listener feedback based on the speaker’s multimodal inputs. The proposed OmniResponse framework, a Multimodal Large Language Model (MLLM), addresses the challenges of audio-visual synchronization by using text as an intermediate modality, enhanced with Chrono-Text for temporal anchoring and TempoVoice for synchronized text-to-speech. Additionally, the ResponseNet dataset is presented, comprising 696 high-quality dyadic interactions with synchronized multimodal data. Experiments show that OmniResponse outperforms baselines in semantic content, audio-visual synchronization, and generation quality, laying the foundation for future human-agent interaction research.", "questions": "See Weakness.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. This work introduces a novel task, Online Multimodal Conversational Response Generation (OMCRG), which addresses a critical gap in modeling natural dyadic interactions and holds significant value for advancing human-computer interaction research.\n2. The proposed OmniResponse framework demonstrates robust effectiveness in generating synchronized audio-visual-textual responses, outperforming baselines in semantic alignment and multimodal coherence.\n3. The creation of ResponseNet provides an essential resource for the research community.\n\nWeakness:\n1. A more detailed analysis of temporal dynamics, such as response delay and real-time processing latency, is needed to fully characterize the framework’s performance in online interaction scenarios.\n2.  The paper lacks detailed elaboration on the real-time design mechanisms, particularly in explaining how the framework handles dynamic input streams, manages processing latency, and ensures responsive timing in natural conversational flows.", "was_revised": true, "final_justification": "The authors addressed my concerns. According to the general quality and idea, I keep my score to be slightly positive.", "rebuttal": "Thank you for your thorough review, positive comments, and insightful suggestions. We appreciate your recognition of OMCRG as a “**novel task**” that can “**generate real‑time synchronized verbal and non‑verbal listener feedback**,” and of our OmniResponse framework, which “**outperforms baselines in semantic content, audio‑visual synchronization, and generation quality**.” We are also grateful for your acknowledgement of ResponseNet as “**high‑quality dyadic interactions with synchronized multimodal data**.” Your view that this work is “**laying the foundation for future human‑agent interaction research**” is deeply encouraging, and we will continue refining the manuscript accordingly.\n\n\n\n>  ####  **Q1. More detailed analysis of response delay and real-time processing latency**\n\nThank you for your suggestion. As suggested, we conduct a more detailed analysis of response delay and real-time processing latency,\n\n* *Response delay.* Our raw prototype, which includes preprocessing and postprocessing, runs at **15.6 FPS (≈64ms delay)** on a single NVIDIA Tesla A100 80GB GPU. Note that such inference efficiency can be further improved, since it is obtained without any deployment or inference optimizations such as flash‑attention, multi‑GPU parallelism, distillation, or quantization.\n\n* *Processing latency.* Our method efficiently processes the input stream, taking only **2.94 ms** per frame to handle the video stream even on the CPU and achieving an average of **0.6 real‑time factor (RTF)** for audio processing on the CPU.\n\nSince our work is the first to explore the new OMCRG task, we mainly focus on building a high-quality dataset and providing a solid baseline framework for future research, rather than extensive latency optimizations. Yet, we believe that efficiency and latency can be substantially improved by incorporating effective techniques and insights from existing online methods, which we leave for future work.\n\nWe will includeadd these analysiis in in our final version.\n\n>  ####  **Q2. Detailed elaboration on how the framework handles dynamic input streams, manages processing latency, and ensures responsive timing in natural conversational flows.**\n\nWe thank the reviewer for this valuable comment and provide a more detailed explanation as follows:\n  \n* Handling dynamic input streams.\n\n  * *Visual stream.* Each incoming frame is processed by MediaPipe, which extracts head‑pose and facial‑expression coefficients in **2.94 ms** per frame even on the CPU.\n  * *Audio stream.*  Whisper‑large transcribes audio chunks at an average **0.6 real‑time factor (RTF)** on the CPU, comfortably within live‑stream requirements.\n\n  These stages compress the raw video and audio into compact token sequences, eliminating redundant frames and long waveforms before they are fed into the generation model.\n\n* Managing processing latency.\nWe constrain the context to a short sliding window, and compress earlier text into static embeddings, enabling incremental updates and avoiding the need to reprocess the full temporal context.\n\n * Auto‑regressive multimodal generation.\n  Our OmniResponse model handles the visual and text tokens in an **auto‑regressive** sliding window and enables efficient generation:\n\n   * *audio tokens*  is produced at 0.0069 RTF (≈ **145×** real time) and converted to waveform at 0.0014 RTF (≈ **712×** real time) during de‑tokenization.\n\n   * textual response takes an average per-token latency of **2.2 ms** (≈ 455 tokens/sec) on a single A100 80GB GPU.\n\n   *  visual response.  The facial‑expression coefficients are rendered at >**40 FPS** even on a single RTX 2080 Ti.\n\nThese designs ensure responsive timing in natural conversational flows."}, {"review_id": "rdCmxoBgPx", "reviewer": "Reviewer_c8bb", "summary": "This paper introduced a novel task, Online Multimodal Conversational Response Generation (OMCRG), which aims to generate synchronized audio and facial listener responses in real-time during dyadic interactions. To tackle this challenging task, authors propose d OmniResponse, a MLLM framework that uniquely uses text as an intermediate modality to bridge and synchronize the audio and visual outputs. The core are two novel components: Chrono-Text Markup, which temporally aligns text tokens with visual frames, and TempoVoice, a controllable online Text-to-Speech module that generates speech synchronized with the facial reactions. A dataset of OMCRG is collected for benchmarking the model performance.  Experimental results show OmniResponse outperforms baselines in generation quality and synchronization across modalities.", "questions": "please see the weakness part above.", "limitations": "yes, limitations are discussed in appendix", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. vovel and practical task formulation. The paper defines and formalizes the OMCRG task, a practical step beyond existing work that only focuses on single-modality response generation (e.g., text-only or face-only). This task addresses a more holistic and natural form of human-agent interaction.  besides, the dataset contribution is also worth noting. \n\n2. interesting methodological approach. The core idea of using text as an intermediate modality to synchronize audio and facial generation is novel. This approach cleverly circumvents the difficult problem of direct audio-visual synthesis by leveraging the semantic richness and discrete nature of text, making the problem more tractable. \n\n3. comprehensive evaluation. the authors conduct a thorough evaluation using a wide array of metrics including text, audio, and visual modalities. The inclusion of ablation studies clearly demonstrates the effectiveness and necessity of the proposed Chrono-Text and TempoVoice components, supporting the paper's main claims.\n\nWeakness:\n\n1. small scale of ResponseNet. While ResponseNet is of high quality, its total duration (14.2 hours) is relatively small, which may limit the model's ability to generalize to unseen conversational styles or demographics. A more detailed analysis of the dataset's demographic diversity would be beneficial to understand potential biases.\n\n2. potential oversimplification of temporal dynamicsl. the Chrono-Text Markup uses only two special tokens to align text and vision, which might be too coarse to model highly complex temporal dynamics. This approach may struggle with typical phenomenons like overlapping speech, or subtle non-verbal vocal cues that require more fine-grained temporal controls. \n\n3. lack of user study. The evaluation relies entirely on automated metrics, which may not fully capture the nuances of human perception. A human user study assessing the naturalness and audio-visual synchronization shall be included.\n\n4. there are many typos and grammar errors in current version of manuscript, e.g, line 74, challenge -> challenging, line 116 steaming -> streaming, line 144 accumfd, line 234 frequently -> frequent", "was_revised": true, "final_justification": "this paper proposed novel framework and new datasets, and its performance outperforms baselines in generation quality and synchronization across modalities. and after reading the rebuttal and other reviewers comment, i recommend acceptance for this submission.", "rebuttal": "### Rebuttal to Reviewer c8bb\n\nWe appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that our work is **\"a novel task\"**, **\"the dataset contribution is also worth noting\"**, \"two **novel** components\", \"interesting methodological approach\", and **\"thorough evaluation\"**.\n\n> #### **Q1.1: A more detailed analysis of the dataset's demographic diversity would be beneficial to understand potential biases**\n\nA: Thank you for your constructive suggestions. As suggested, we conduct additional analysis of the dataset's demographic diversity. As indicated in the table below, our dataset covers balanced gender, diverse age distributions, and individuals from multiple ethnic backgrounds.  The properties of our dataset enable us to model varied conversational behaviors, serving as a valuable benchmark for future research.\n\n\n\n\n\n\n**Demographic statistics:**\n\n| Category       | Count / Share                                                                                                                                |\n| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |\n| **Identities** | 161 unique identities                                                                                                                        |\n| **Gender**     | Female: 93 (57.8 \\%),  Male: 68 (42.2 \\%)                                                                                                      |\n| **Ethnicity**  | White: 122 (75.8 \\%), Black: 24 (14.9 \\%), Asian: 15 (9.3 \\%)                                                                                 |\n| **Age bands**  | 10–19: 10 (6.2 \\%),  20–29: 63 (39.1 \\%),  30–39: 51 (31.7 \\%),  40–49: 17 (10.6 \\%), 50–59: 17 (10.6 \\%),  60–69: 2 (1.2 \\%),  70+: 1 (0.6 \\%) |\n\n\n\n> #### **Q1.2: While ResponseNet is of high quality, its total duration is relatively small.**\n\nA: Due to data availability and costly data processing, building datasets for the OMCRG task is challenging. Unlike typical generation tasks, the OMCRG task requires dyadic high‑quality videos where both speaker and listener appear in the same frame, and they must stay in a static and consistent camera view for a long duration, which is scarce.\n\nInstead of simply increasing data scale, recent studies have shown that high‑quality training data is critical for the performance of generative models. We hence perform costly, human‑curated processing to ensure high-quality, diverse, and representative data.  For example, we curated diverse demographics, as well as six conversation genres covering (1) Intimate personal chats, (2) Economic interviews, (3) Interdisciplinary expert panels,  (4) Educational tutoring, (5) News commentary, and (6) Casual small‑talk.\n\n\nThis careful curation allows us to introduce the first high-quality dataset ResponseNet for OMCRG. Despite the modest size of ResponseNet, its high quality and diversity, combined with our effective leverage of pre‑trained LLM, which has been pre-trained on large-scale text data (**billions of text tokens**), enable our method to achieve the best performance.\nMoreover, our pipeline incorporates powerful, pre-trained audio and visual feature extractors.\nThe audio tokenizer has been trained on over **3,000 hours** of diverse speech, supplying generalizable semantic and prosodic encodings.\nFacial expression and head-pose coefficients are extracted via Google’s MediaPipe framework, providing stable, high-fidelity visual features without additional training cost.\nThese audio–visual–textual models furnish strong priors, eliminating the need for vast amounts of data to learn features for each modality.\n\n\n\n\n\n> #### **Q2: The Chrono-Text Markup uses only two special tokens to align text and vision, which might be too coarse to model highly complex temporal dynamics.**\n\nA: Our Chrono‑Text Markup employs only two special tokens to balance temporal fidelity with learnability. Since OMCRG is a highly complex task with multiple requirements, introducing more and fine‑grained special tokens would significantly expand the LLM’s vocabulary and substantially increase training difficulty. An alternative approach is direct audio encoding and decoding with an audio‑based model. However, this introduces format inconsistencies (e.g., sampling rate) and large variations in tone color, pitch, and frequency, making training even more difficult.\n\nIn contrast, our Chrono‑Text Markup is simple yet effective, enabling the model to learn utterance start points, durations, and natural pauses without large complexity. We are unable to upload more video results due to the NeurIPS rebuttal policy. Yet, Fig. 5 in the main paper shows our method effectively handles overlapping speech. \"Table 2 shows our method reaches a BERTScore of 0.806 and an FVD of 314.94,  significantly surpassing the audio‑visual LLM (0.662, 681.55), demonstrating its effectiveness in modeling speech and non‑verbal temporal dynamics. \nOur method takes a step towards this new task, yet, we believe more elegant solutions could be proposed to better address complex temporal dynamics.\n\n\n> #### **Q3: User study**\n\nA: Thanks for your constructive suggestion. We ran a user study with 49 participants (28 male, 21 female). Each subject watched 16 randomly‑ordered clips and assessed\nspeech content appropriateness, audio speech quality, visual quality, and audio-visual synchronisation. \nAll of them were proficient in English (53.1\\% reported advanced proficiency or daily‐communication ability). Educational background was high: 95.9\\% held at least an undergraduate degree, 44.9\\% held a master’s degree, and 18.4\\% held a PhD. Age distribution was as follows: 14.3\\% under 25, 34.7\\% aged 26–35, 24.5\\% aged 36–45, and 26.5\\% aged 46–55.\nIn direct A/B preference, a minimum preference rate of 67.3\\% (i.e., speech content appropriateness Ours vs. LSTM) and a maximum of 95.9\\% (synchronization Ours vs. Audio–Visual LLM). \nWe will add the full protocol and statistics in the revision.\n\n| Criteria                       | Ours vs. LSTM | Ours vs. Audio–Visual LLM |\n| ------------------------------ | -----------: | -----------------------: |\n| Speech Content Appropriateness |       75.5\\% |                   81.6\\%|\n| Audio Speech Quality           |       77.6\\% |                   85.7\\% |\n| Visual Quality                 |       67.3\\% |                   93.4\\% |\n| Audio–Visual Synchronization   |       91.8\\% |                   95.9\\% |\n\n*Table: Percentage of times participants preferred our method over each baseline in pairwise A/B tests.*\n\n> #### **Q4:  Typos and grammar**\n\nA: We appreciate you pointing out those spelling and grammar issues. We have corrected such typos and grammatical errors in the final version."}, {"review_id": "T1yyfmWlea", "reviewer": "Reviewer_tzNS", "summary": "This paper introduces the Online Multimodal Conversational Response Generation (OMCRG) task, which focuses on generating synchronized listener feedback in real-time conversations. To address this, the authors propose OmniResponse, a Multimodal Large Language Model (MLLM) that uses text as an intermediate step to synchronize generated audio and facial reactions. The authors also created a new dataset, ResponseNet, for this task. Experiments show OmniResponse outperforms baselines, creating more natural and temporally aligned conversational responses.", "questions": "Please see the weakness part.", "limitations": "Given the potential for malicious applications of talking-head techniques, the paper should include more discussion on methods for proactive and reactive protection of personal portraits to address potential negative societal impact.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper defines a new task (OMCRG) that requires generating synchronized audio and facial reactions in real-time, which is more advanced than previous work.\n2.\tThe proposed OmniResponse uses text as a bridge to align audio and video, introducing new techniques (Chrono-Text Markup and TempoVoice) for precise temporal synchronization.\n3.\tThis paper introduces ResponseNet, a new, high-quality dataset of synchronized and richly annotated conversational videos.\n\nWeaknesses:\n\n1.\tThe LSE-D scores presented in Table 2 show only a marginal improvement over the baseline. Metrics from SyncNet [1], which is a more reliable indicator of lip-sync accuracy, may be a better choice.\n2.\tHuman evaluation is absent. Since automated scores often fail to capture the nuances of multimodal synchronization, a user study would be essential to validate the model's perceptual performance and strengthen the paper's claims.\n3.\tThe qualitative results in the supplementary materials are not convincing. First, there are noticeable discrepancies and a lack of synchronization between the generated lip movements and the audio. Second, the quality of the synthesized audio itself is poor, suffering from low fidelity and poor intelligibility.\n4.\tOmniResponse is designed for \"online generation\", yet the paper lacks a quantitative analysis of its inference speed or real-time viability.\n5.\tThe phrase \"accumfd udfsalated\" in line 144 is confusing and seems to be a typo.\n\n[1] Prajwal, K. R., et al. \"A lip sync expert is all you need for speech to lip generation in the wild.\" Proceedings of the 28th ACM international conference on multimedia. 2020.", "was_revised": true, "final_justification": "This work serves as a valuable baseline for this new research direction. While the lip-sync quality is relatively low, these initial results provide a solid foundation for future studies in this task. Therefore, I raised my score.", "rebuttal": "### Rebuttal to Reviewer tzNS\n\nWe appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that our work introduces a  **\"new task\"**, which is \"more **advanced** than previous work\",  **\"new techniques\"**,  **\" a new, high-quality dataset\"** with \"**richly annotated** conversational videos\", \"creating more **natural** and **temporally aligned** conversational responses.\"\n\n\n\n> ####  **Q1.  “Metrics from SyncNet [R1], which is a more reliable indicator of lip‑sync accuracy, may be a better choice.”**\n\n\nWe thank the reviewer for this suggestion. **Our evaluation had indeed adopted metrics derived from SyncNet [R1]**. In particular, [R1] introduces two metrics: LSE‑D (Lip Sync Error – Distance) and LSE‑C (Lip Sync Error – Confidence). We adopt LSE‑D in Tables 1 and 2 of the main paper, using the official SyncNet evaluation script [R1]. We did not use LSE‑C because it is not well‑suited for our task. LSE‑C, measuring audio‑visual correlation by averaging confidence scores computed based on SyncNet, is effective for talking‑head generation tasks where the subject speaks continuously. However,  in our task modeling multimodal conversation, listeners are often silent when the speaker is talking. Although silent segments represent appropriate responses, evaluated confidence values drop sharply, making LSE‑C unreliable for our task. \nFollow your suggestion, we also provide LSE‑C results below, where our method outperforms baselines.\n\n\nSimilarly, since the listener often remains silent in multi‑turn conversations, our improvement in LSE‑D may seem small compared with  LSTM, yet,  it actually indicate a large  performance gain in OMCRG task.\n\n\n* **Why we emphasised LSE‑D.**\n  LSE‑D measures the average temporal offset between the predicted video and the reference audio. In conversational footage, speakers alternate with silent listeners (rather than they are always speaking like talking head); during those silent spans SyncNet’s confidence drops sharply (indicate a low average value of LSE-C), making **LSE‑C very noisy**. We therefore used LSE‑D as the primary figure of merit.\n   We also have rerun the official SyncNet script and report both metrics:\n  | Method         | LSE‑D ↓           | LSE‑C ↑           |\n  | -------------- | ----------------- | ----------------- |\n  | LSTM | 9.72             | 0.157           |\n    | Audio-visual LLM | 10.03              | 0.269             |\n  | Ours           | **9.56** | **0.371** |\n\nThe 1.6\\% LSE‑D reduction  confirms a significant improvement, and the gains in LSE‑C further reinforce our method’s effectiveness. We will add these numbers to our revised version and the supplementary materials.\n\n* **Context for the “marginal” gap.**\n  In a turn‑taking dialogue, perfect lip‑syncing is easier to approach because most frames are either silence or low‑motion listener responses. Even a small absolute reduction in LSE‑D therefore corresponds to noticeably fewer mis‑aligned phoneme–viseme pairs ($\\approx$ 200 ms on average).\n\n* **Planned revision.**\n  We will (i) retain LSE‑D as our headline metric, (ii) include the full LSE‑C results in the revised version, and (iii) clarify in Sec. 5 that both metrics originate from SyncNet. We believe this addresses the reviewer’s concern while demonstrating the practical significance of our improvements.\n\n[R1] Prajwal, K. R., et al. \"A lip sync expert is all you need for speech to lip generation in the wild.\" Proceedings of the 28th ACM international conference on multimedia. 2020.\n\n\n\n> ####  **Q2. Human evaluation.**\n\nThank you for your constructive suggestion. We conducted a user study with 49 participants (28 male, 21 female), all of whom were proficient in English (53.1\\% reported advanced proficiency or daily‐communication ability). Educational background was high: 95.9\\% held at least an undergraduate degree, 44.9\\% held a master’s degree, and 18.4\\% held a PhD. Age distribution was as follows: 14.3\\% under 25, 34.7\\% aged 26–35, 24.5\\% aged 36–45, and 26.5\\% aged 46–55. Each participant viewed 16 randomly ordered clips and indicated their preference in pairwise A/B comparisons across four perceptual criteria: speech content appropriateness, audio speech quality, visual quality, and audio–visual synchronization. Our method was preferred in every case, with a minimum preference rate of 67.3\\% (i.e., speech content appropriateness Ours vs. LSTM) and a maximum of 95.9\\% (synchronization Ours vs. Audio–Visual LLM). The table below summarizes these preference rates. We will include these human evaluation results in our final revision.\n\n\n| Criteria                       | Ours vs. LSTM | Ours vs. Audio–Visual LLM |\n| ------------------------------ | -----------: | -----------------------: |\n| Speech Content Appropriateness |       75.5\\% |                   81.6\\%|\n| Audio Speech Quality           |       77.6\\% |                   85.7\\% |\n| Visual Quality                 |       67.3\\% |                   93.4\\% |\n| Audio–Visual Synchronization   |       91.8\\% |                   95.9\\% |\n\n*Table: Percentage of times participants preferred our method over each baseline in pairwise A/B tests.*\n\n\n\n> ####  **Q3: The qualitative results in the supplementary materials.**\n\nOur OMCRG task is much more **challenging** than related tasks (e.g., talking‑head generation), since it requires simultaneously satisfying multiple requirements, such as  (1) online multimodal generation (generate multi-model outputs simultaneously and iteratively, rather than offline single modal generation (generate one modality and the entire sequence at once), and (2) ensuring appropriateness and naturalness of the generated responses.  \n\nMoreover, lip–audio synchronization and online audio-video generation remain a challenging problem.  For example, talking‑head generation works have devoted much effort to improving lip–audio synchronization, introducing modules/losses such as lip‑mask‑aware training with SyncNet supervision [R2] or pre-trained lip synchronization discriminator  [R3].    \n\nHowever, as the first work on the new OMCRG task, we focus on presenting a simple, effective, and scalable framework, instead of introducing overly complex,  subproblem‑specific modules for fine‑grained optimization.\n\nDespite this, our approach outperforms baseline methods by a large margin. (e.g., FVD $\\downarrow$  314.9 vs. 681.6, UTMOS $\\uparrow$ 1.41 vs. 1.32). Moreover, our framework provides a solid foundation for future improvements such as adding lip-mask-aware-training for improving lip–audio synchronization.\n\n| Task                                    | Input Modalities      | Generated Output       | Setting                                                 |\n| --------------------------------------- | --------------------- | ---------------------- | ------------------------------------------------------- |\n| Talking‑Head Generation (e.g., Wav2Lip) | Speaker audio         | Speaker video          | One-to-one mapping                                      |\n| Facial Reaction Generation              | Speaker audio + video | Listener video         | Multi‑modal input → single‑modality output              |\n| **OMCRG (ours)**                        | Speaker audio + video | Listener audio + video | Multi‑modal input → simultaneous multi‑modal generation |\n\n\n\n\n\n[R2] LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision, arXiv, 2025\n\n[R3] SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis, CVPR 2024.\n\n\n\n> #### **Q4. inference speed or real-time viability.**\n\nWe measured the model’s inference speed at 15.6 FPS (64 ms delay) on a single NVIDIA A100 80GB GPU and will report these results in the revised manuscript. Importantly, this benchmark was obtained without any deployment or inference optimizations such as flash‑attention, multi‑GPU parallelism, distillation, or quantization, highlighting substantial potential for further acceleration and more efficient, real‑time deployment.\n\n\n> #### **Q5. Typos**\n\nThanks for your detailed review and catching these mistakes. We apologize for the typos: in line 144, “accumfd udfsalated” should read “accumulated.\nWe have corrected these errors and will conduct a thorough review of the manuscript to eliminate any remaining typos in the final version."}]

Author rebuttal:

Meta-review: This submission initially received a mixed set of ratings. After the rebuttal discussion and subsequent discussions between reviewers and the area chair, several reviewers raised their scores in recognition of the authors' thorough and thoughtful responses. The authors addressed key concerns such as the reliance on the text modality, strategies for real-time processing, and failure case analysis, supported by additional experiments and clarifications. At this point, all reviewers are in favor of acceptance. After carefully reading the paper, the reviews, and the rebuttal, I agree with this decision. I recommend acceptance. I encourage the authors to further polish the camera-ready version by incorporating the reviewers' suggestions to improve clarity and completeness.
Decision: Accept (poster)