From Natural Alignment to Conditional Controllability in Multimodal Dialogue
Authors: Zeyu Jin
Songtao Zhou
Haoyu Wang
Minghao Tian
Kaifeng Yun
Zhuo Chen
Xiaoyu Qin
Jia Jia
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal dialogue dataset
Multimodal conditional dialogue generation
Spoken dialogue generation
Primary area: datasets and benchmarks
TL;DR: We propose an expressive multimodal dialogue dataset with dialogue-level style annotations using an automated pipeline, then introduce explicit and implicit control in multimodal dialogue generation.

Abstract: The recent advancement of Artificial Intelligence Generated Content (AIGC) has led to significant strides in modeling human interaction, particularly in the context of multimodal dialogue. 
While current methods impressively generate realistic dialogue in isolated modalities like speech or vision, challenges remain in controllable Multimodal Dialogue Generation (MDG). 
This paper focuses on the natural alignment between speech, vision, and text in human interaction, aiming for expressive dialogue generation through multimodal conditional control. 
To address the insufficient richness and diversity of dialogue expressiveness in existing datasets, we introduce a novel multimodal dialogue annotation pipeline to curate dialogues from movies and TV series with fine-grained annotations in interactional characteristics.
The resulting MM-Dia dataset (360+ hours, 54,700 dialogues) facilitates explicitly controlled MDG, specifically through style-controllable dialogue speech synthesis. 
In parallel, MM-Dia-Bench (309 highly expressive dialogues with visible single-/dual-speaker scenes) serves as a rigorous testbed for implicit cross-modal MDG control, evaluating audio-visual style consistency across modalities. 
Extensive experiments demonstrate that training on MM-Dia significantly enhances fine-grained controllability, while benchmarks on MM-Dia-Bench reveal limitations in current frameworks to replicate the nuanced expressiveness of human interaction. 
These findings provides new insights and challenges for multimodal conditional dialogue generation.

Published as a conference paper at ICLR 2026
FROM NATURAL ALIGNMENT TO CONDITIONAL CON- TROLLABILITY IN MULTIMODAL DIALOGUE
Zeyu Jin' Songtao Zhou Haoyu Wang', .Minhhao Tian?, Kaifeng Yun?, Zhuo Chen', Xiaoyu Qin Jia Jia 1.5
'Department of Computer Science and Technology, Tsinghua University, "Rice University,
"Tsinghua Shenzhen International Graduate School, Tsinghua University, *ByteDance, SBNRit Tsinghua University
[jinzeyu23, zst24) @ma immalls. 1s. ts wiiinnuuaae. .eggh edu. {xyqin, bjia]Stsinghus edu.
ABSTRACT
The recent advancement of Artificial Intelligence Generated Content (AIGC) has led to significant strides in modeling human interaction, particularly in the con- of multimodal dialogue. While current methods impressively generate re- alistic dialogue in isolated modalities like speech vision, challenges remain in controllable Multimodal Dialogue Generation (MDG). This paper focuses on the natural alignment between speech, vision, and text in human interaction, aiming for expressive dialogue generation through multimodal conditional con- trol. To address the insufficient richness and diversity of dialogue expressive- ness in existing datasets, we introduce a novel multimodal dialogue annotation pipeline curate dialogues from movies and TV series with fine-grained annota- tions in interactional characteristics. The resulting MM-DIA dataset (360+ hours, 54,700 dialogues) facilitates explicitly controlled MDG, specifically through style- controllable dialogue speech synthesis. In parallel, MM-DIA- DEDD-AAANCCE BENCH (309 highly expressive dialogues with visible single-ddual-speaker scenes) serves as rigorous testbed for implicit cross-modal MDG control, evaluating audio-visse visual style con- sistency across modalities. Extensive experiments demonstrate that training on MM-DIA significantly enhances fine-grained controllability, while benchmarks on MM-DIA- BENHH reveal limitations in current frameworks to replicate the nu- anced expressiveness of human interaction. These findings provides new insights and challenges for multimodal conditional dialogue generation.
INTRODUCTION
Dialogue has long been considered one of the most natural forms of human interaction, involving multiple communication channels such text, speech, vision, gestures, and etc. In the AIGC era, multimodal dialogue has become increasingly important for a wide range of applications in human- computer interaction, social computing, and film-making.
Existing research in dialogue primarily falls into two directions: (1) Semantic generation, which emphasizes producing coherent and contextually appropriate responses, as exemplified by large- scale dialogue systems like ChatGPT (OpenAI et al., 2024). (2) Modality mapping, which projects the given semantics into output modalities such as speech (Zhu et al., 2025; Zhang al., 2024) and motion (Kong et al., 2025b). However, both directions prioritize the realistic transmission of modality-isolater dialogue content, while neglecting the systematic modeling of cross-modal inter- actional style. This results in limited expressiveness and controllability in the generated outputs.
Controllable multimodal dialogue generation presents several major challenges: (1) Lack of high- quality native multimodal dialogue data. Existing multimodal dialogue datasets, shown in
"Equal contribution 'Corresponding author
'https:/ : / thub. com http::/gg..obb..o.c..eestt/gtn// htt.://///.ssyyyiy///mmo.m

---

Published as a conference paper at ICLR 2026
Are your spann
sertnwadyy
you meed
ohick Tomes cfferingg to parvi Ning.
Domnt.
A
Transeript wom
OA,, my Gool
No, noest OK, God I'm just whinning you loune aond sokes
Speoker ID
Tanya
Rosie
Tanya
Donnn
Tanya
Donna
Sentence-lewe
Audio
Spoooct
Tisk Sity lec-entralllale Dialigue Speech Synthesis
MAY HyMAy
Now- Harcbal
Cinterrupe
-huughter
A
Vision
Spoaker Fimibitgr
FolMc
Flsible
Fonile
Spenker Josisblle
Pisaiic
V
Movic
Vinsal-andiiined Dislogue Synthesis
Dialogue Clip
A
Dialogue-lew leve--eeve
Dialogue
Relationship
Tonoa playfully badgers a dimissiive Domua with qusstioss, and other Deseription
Style
heerd
Affective Triplet
D
Interaction Mede
iolk with joyjekee,, jukez, the contersation eropay unoo wild, infectious longhter"
Emorional Tone
Tink:: Specch-driven
Kainty
Dialogue Video Generation
Figure 1: An example dialogue clip from MM-DIA and MM-DIA-BENCE with hierarchical (sentence- and dialogue-level) annotations, featuring rich multimodal dialogue interaction details. The right panel depicts three multimodal dialogue generation tasks involving text (T), audio (AA), vision (V) and dialogue style (D), demonstrating both explicit (Task /) & implicit control (Task 2,3).
Tab. 1, suffer from narrow sources and missing modalities, which restricts their expressiveness and ability to portray complex multimodal interactions. This is largely due to the difficulty of collect- ing synchronized text-audio-visual dialogue data. (2) Lack of scalable annotation methods for interaction-level semantics. Human categorically labeled datasets like MELD (Poria et al., 2019) and MC-EIU (Liu et al., 2024b) fail to capture the nuanced, continuous nature of human interaction, while remaining costly and difficult scale. (3) Lack of systematic benchmarks and evaluation protocols. Current dialogue benchmarks (Poria et al., 2019; Liu et al., 2024b) primarily focus on local semantic coherence and modality-isolatee fidelity, leaving a methodological gap in evaluating dialogue-level controllability and cross-modal style consistency.
In this paper, we address these challenges by constructing a large-scale expressive multimodal di- alogue dataset, introducing new annotation paradigms, and establishing systematic benchmarks for controllable multimodal dialogue generation.
Addressing the data scarcity, we develop an automatic data pipeline to curate synchronized mul- timodal dialogues and generate fine-grained interaction-level annotations from in in-the-wild movies and TV series. To overcome the complex scene transitions and audio-visual asynchrony inherent in cinematic data, we propose specialized techniques for robust dialogue boundary segmentation and multimodal speaker identification. To facilitate versatile controllability across diverse scenarios, we formulate two complementary paradigms of dialogue expressiveness: (1) Affective Triplet, a structured schema of (Relationship. Interaction Mode, and Emotional Tone) to jointly encapsulate role identity, social interplay, and emotion dynamics; and (2) Freestyle Description, capturing per- speaker, turn-level style trajectories. Extensive validation demonstrates that our pipeline achieves human-level quality in annotation consistency and reliability.
Applying the proposed pipeline to over 700 hours of movies and TV series, we present MM-DIA, multimodal dialogue dataset characterized by 360.26 hours, 54,700 clips of highly expressive, con- textually rich, and interaction-heavv dialogues. MM-DIA features ane-grained annotation across various dialogue dimensions, including nen-verbal sound, speaker identity and emotional dynamics both the individual (sentence-wide) and collective (dialogue-wide levels. To our best knowledge, it is the first dataset specifically center dialogue expressiveness across multiple modalities.
Leveraging MM-DIA, we formally introduce Multimodal Dialogue Generation (MDG) as a condi- tional generation paradigm. Given multimodal conversational context (text, audio, vision), MDG task aims to synthesize multimodal dialogues that maintain rigorous cross-modal alignment while enabling fine- -me-gnaieed controllability egorize the conditional control into explicit control, where styles are specified via natural language prompts, and implicit control, where conditions are inferred via cross-modal cues.
interaction-level variables. To operationalize this, cat-
2

---

Published as a conference paper at ICLR 2026
Table Comparison the MM-DIA with dialogue-related datasets across domain, scale, modality, annotation, and open-source (OS). Modality includes text (T), vision (V), and audio (A), with audio- visual details on speaker identity (S-ID), non-yerbal cues (N-V), and speaker visibility (S-V).
Scale
Modality
Audio-visull Details
Anmotation
Domain
Dutaset
os
#Clip
#Utt.
#Dur.(h)
V
S-ID
N-V
S-V
Granularity
Label
Spoken Dia.
OpenDialogue sta
IM
6.5M
6.8K
Dialogue
None
Open (011)
5.6M
Dialogue
Textual Dia.
VTD-I8M (lavea 2007)
18M
/
x
V
Dialogue
None
OpenVid-IM
IM
2.1K
Soene
Desc.
Text-to Video
Captain Cinema (Starct 2008
300K
Shot
MELD Divise al, 2009
1.4K
14K
13.6
Sentence
Tag
MM Dia. Und
MC-EIU
5.0K
56K
53.0
Sentence
Movic Gen.
MovieBench (il 2000
16.0K
69.2
ShotfSoene
MM Dia. Gen.
MM-DIA (Ours)
54.7K
449K
360.3
7
Dia/Sent.
Desc/Tag
MM Dia. Gen.
MM-DIA BENCH (Ours)
309
1,851
1.7
/
Dia/Sent.
Desc./Tag
For the explicit prompt control, we introduce Style-controllable Dialogue Speech Synthesis (Fig. Task I) to generate conversational audio consistent with the freestyle descriptions. Through super- vised finetuning MM-DIA. existing models produce high-quality outputs with enhanced intelli- gibility and turn-taking accuracy, while faithfully reflecting the instructed styles.
For the implicit cross-modal control, introduce two tasks: (1) Vision-conditioned Dialogue Speech (Fig. Task 2) which synthesizes contextually accurate speech aligned with turn-taking visual sequences of dialogue; (2) Speech-driven Dialogue Video Generation (Fig. 1 Task 3), which generates expressive video from conversational audio. Both tasks demand modeling implicit alignment in cross-modal interactions, which is difficult quantify. To facilitate standardized eval- uation, we build MM-DIA- -BENHH, a diverse and balanced benchmark of 309 highly expressive dual-speake dialogues with ensured speaker visibility. The benchmark evaluates audio-visual style consistency across dialogue turns, addressing a limitation of existing video evaluation protocols that overlook cross- moosll stylistic alignment. Experiments results highlight the limitations of current frameworks in maintaining audio-visum consistency when replicating the expressiveness of human interaction, offering new insights and challenges in multimodal dialogue generation.
2 RELATED WORKS
2.1 MULTIMODAL DIALOGUE DATASETS
Multimodal dialogue datasets have driven recent progress in multimodal Al systems. Large scale spoken dia- logue corpora (Zhu al., 2025; Jin et al., 2024) support speech-based systems but remain limited uni- modal settings, hindering multimodal alignment and mross-modal interac-
Table 2: Comparison between MM-DIA and existing TV/Movie-ourreed datasets in the annotation framework.
Dataset
Segmentation
Inpat
Anno.
MELD 200
TV
Human
A
Human
MC-EII pisiea casil
TV
Human
V.
Human
MovieBench (oaas
Movie
Vision-based based
ItA+T
GPT-4o
MM-DIA (Ours)
TVMo
Multimodal
4+I+
Gemini 2.5-pro
tion. Web sourced video datasets (Ju et al., 2024; Nan et al., 2025) offer abundant audio-visual data, yet mostly consist of casual chitchat or predefined scenarios, limiting diversity for prompt- controlled dialogue generation. Movie-sourced video datasets (Han et al., 2024; Wu et al., 2025) offer expressive audio-visual content but typically lack clear dialogue boundary delineation. To address these limitations, introduce novel multimodal framework catering for dialogue-level style annotation, as shown in Tab. 2. Unlike prior approaches that rely on isolated key-frame im- age sequences (I), we leverage synchronized audio-visual inputs to enable versatile datasets with fine-grained style annotations for advancing multimodal dialogue modeling.
2.2 DIALOGUE GENERATION FROM MULTIMODALITY
Dialogue serves as the smallest and most structured unit of human interaction. Recent progress in spoken dialogue generation (Nari Labs, 2025; Ju et al.; Zhu et al., 2025) has captured realistic turn-
3

---

Published as a conference paper at ICLR 2026
Wild Movie Datu
Subtiite Synchrenizetion
Dialorue Extraction
Mutttmodaa-heed
Flse-trised Annstalion
Senbenzedrad
Anpstated Dinlogue Cli
Raw Audio
> Oncorrecial calliinttdd Sabible
Spcne deteellttetionn
Spenker Aligameet
Transcript alaiiii
Corrected but Uncellbraned
Subiile
3rt Alignment
Kaune
Speaker Idoutiy
Raw Video
Conrectd califrated Subnde Frog prentampp
Inscens Dislogup-estraction
Bank
Speech Cip
Video Clip
Character Bank
Keyfrume
Koffama
+ Sorvi 25h
25Pg
KATA condle
Keyframe Sequence
Anootating Distogue Clp
Sentence Clip
Sentence-kerel Clip
Diaiguue Captionining
Descriptinn
Transeript
Sparch Nomn
Pranscript alavitiinn
Sperch Clip
Affeciva Triplet
Annotation
Albettr
bacreciin Made
Rulic
Corrected Disoriin
Thasorrger
Keyfrume
Clip
Fidoo Clip
Keyfrume
Espresheere Ansetation
Diialegue-trval Annotaaivo
Eraadian Time
Figure 2: Framework of the Movie/TV- tetthe-wile data curation pipeline for multimodal dialogue extraction with fine-grained interaction-level annotations.
taking and multi-speaker timbres (Boson AI, 2025; Zhou al., 2024), enabling more natural verbal exchanges. In parallel, cinematic video generation (Xiao et al., 2026; Liu et al., 2024a; Blattmann al., 2023) supports high-fidelin multi-shot scenes with consistent character appearances and im- mersive transitions, producing coherent visual narratives. Specialized systems further bridge these modalities through synchronized talking head generation (Cui et al., 2025; Wang al., 2026) and vision-context-awarr speech syathesis(Zhou et al., 2025b; Wang et al., 2025) to ensure expressive voice with coherence visual appearance. Although these advances establish the modality-specific foundations for conveying semantic information, achieving fine-grained tross-modal control across speech, vision, and text for expressive multi-speaker dialogue remains an open challenge. In this paper, we seek to address this gap by introducing the necessary infrastructure (dataset, pipeline, benchmark, and task formulation) for controllable multimodal dialogue generation.
MM-DIA: A LARGE-SCALE EXPRESSIVE MULTIMODAL DIALOGUE DATASET
Movies and TV series two of the richest artistic forms that feature carefully crafted, context- sensitive performances, but their complex audio-visual asynchrony poses significant challenges in data processing. In this section, we describe our pipeline for movic/TV-soureed in-the-wild dia- logue extraction and fine-grained interaction-level annotations, followed by statistical analysis and validation the resulting data.
3.1 PIPELINE ORIENTATION WITH DATA PREPARATION
Cinematic dialogues exhibit stronger emotion, heightened tension, and greater resemblance to ev- eryday interactions. However, the pursuit of strong sensory effects introduce substantial noise. Fre- quent background sounds, dramatic bursts, or ambiguous murmurs hinder the accuracy of Automatic Speech Recognition (ASR), while artistic camera movements create voiceover and flashback, com- plicating dialogue boundary detection and speaker attribution. These factors call for cautious and comprehensive multimodal processing strategy.
In preparation of the dataset, we collect raw movie & TV data from public available sources, noting that official subtitle (.srt) files are often unavailable. Although ASR can generate time-stamped transcriptions, its high word enror rates limit reliability. To ensure high-quality subtitles and further enlarge the dataset while balancing temporal accuracy and textual fidelity, we additionally adopt multi-sourced uncalibrated subtitles and align them with ASR outputs (Fig. Step 1, details in Appendix A. With the corrected timestamps from the calibrated subtitles, we extract a keyframe sequence from each subtitle line representatives for dialogue boundary detection.

---

Published as a conference paper at ICLR 2026
3.2 MULTIMODAL--BSEED DIALOGUE EXTRACTION
Dialogue boundaries differ from shot or scene boundaries, as conversations may span multiple shots and topic shifts within single long scene. To automatically extract continuous dialogue from movies & TVs, we introduce a tolerance-enhanee plies a Vision-Language Model (VLM) identify scene continuity, followed by Large Language Model (LLM) to refine in-scene dialogue boundaries. Unlike traditional frame-to toaaffefemee matching methods (Wu et al., 2025; Xiao et al., 2026), our approach incorporates buffer mechanism with dynamic keyframe pool (Fig. 2 Step 2, details in Appendix A.2), allowing the model to bridge momentary visual disruptions such as rapid camera shifts, flashbacks, or perspective changes. This improves robustness in maintaining dialogue continuity across complex scenes. Based on the sulting scene-level segmentation, we further leverage subtitles and LLM-based semantic filtering to extract meaningful dialogue segments, particularly in long scenes exceeding 90 seconds. By com- bining visual and textual cues, the framework achieves coherent and accurate dialogue extraction, ensuring the integrity of multimodal context.
scene boundary detection method. It first ap-
3.3 SENTANCF-LEVEL FINE-GRAINED ANNOTATION
Based on the identified boundaries, we segment the content into short dialogue clips. Next, we per- form speaker attribution to each dialogue segment. However, conventional automatic tools cannot reliably perform speaker attribution in this setting. Audio-based speaker diarization often suffers from limited accuracy due noisy acoustic conditions. while visual-based active speaker detec- tion is unreliable because speakers are frequently not visible in cinematic content. To address this issue, we employ 2.5--ni-2.5-flsst to perform speaker attribution based on synchronized audio- visual segments and corresponding subtitles (Fig. 2 Step 3). The model is prompted with prede- fined main character bank to recognize speakers; otherwise, it assigns speaker identities based on their on-screen personas. Additionally, we annotate non-verbal sounds and vocalizations to capture fine-grained expressive cues and contextual nuances in dialogue. To support downstream dialogue- related tasks such talking head generation, we further use the Insightface toolkit to annotate the visibility of main speakers in the aligned keyframes.
3.4 DIALOGUE-LEVER EXPRESSIVENERS ANNOTATION
To systematically study complex interaction- level behaviors, we define dialogue expressive- ness cross-modal consistency in interaction that extends beyond semantic content. We pro- pose two complementary paradigms for model- ing dialogue expressiveness:
Table 3: Detailed statistics for MM-DIA and MM-DIA-BENCIS Scored from Gemini/Human.
Statistie
MM-DIA MM-DIA-BENCH
Toral Dialogues
14,200
309
Toral Tums
449.138
Toral Duration
360.26
1.69
Avg. Spk./ Dia.
2,00
(1) Affective Triplet Control, (Relationship, Interaction Mode, Emotional Tone), which jointly model role identity, social interplay, and emotional dynamics. It enables precise control over scenario-level dialogue behavior.
Avg. Dur. / Dia. Avg.
23.71
19.69
Avg. Dur. / Tumm (5)
2.89
3.29
Avg. Turns /Sp..
3.00
Avg. Rounds of Speaker Changes / Dia.
4,28
4.09
Speaker Visibility
Avg. Score on Emotion Intensity
6.76 /
7,81 5.74
(2) Description Control, which captures per- speaker, turn-level style trajectories. It enables the separate control over speakers, and the fine-grained modeling of emotion flow across turns within the same speaker.
Avg. Score Volatilty of Emotion Flow
/.32/4.36
7.45 5.68
Together, these paradigms encompass both structured tag-based control and freestyle description- based natural language control. Given the speakers bank and synchronized audio-visual segments, we use Gemini-2.5-pp to annotate both paradigms of the dialogue expressiveness (Fig. 2 Step 4). Beyond qualitative annotation, we further introduce two complementary quantitative dimensions to measure dialogue expressiveness: global emotional intensity at the dialogue level and local emo- tional volatility at the speaker level. For instance, a consistently high-energy dialogue would be rated as high in emotional intensity but low in emotional volatility.
5

---

Published as a conference paper at ICLR 2026
25.4%
F M
M
(a)
(b)
(c)
Figure 3: Distributions of (a) Dual-speaker Gender, (b) Relationship, and (c) Interaction Mode in MM-DIA.
3.5 MM-DIA WITH MM-DIA MM--IA--ENCIL
Applying our data curation pipeline to over 700 hours of raw content- content-including more than 200 movies and TV series-we construct MM-DIA, multimodal dialogue dataset comprising 360.26 hours, 54,700 clips of expressive, context-rich, and interaction-intensii dialogues. The dataset accompanied by fine-grained annotations covering various dialogue aspects, such as non-verbal sounds, speaker identity and emotional dynamics at both individual and dialogue levels. To our knowledge, MM-DIA is the first dataset specifically centered on dialogue-level expressiveness across modalities. As illustrated in Fig. 3, MM-DIA exhibits a balanced distribution across af- fective triplet categories. Notably, exploratory analysis reveals meaningful associations between Relationships and Interaction Mode. For instance, Commands and Questioning are predominantly linked Workplace settings, while Intimate relationships tend to involve Emotion Release and Ban- ter. Such patterns provide qualitative evidence of distributional consistency between the MM-DIA and real-world social interactions.
Subsequently, construct MM-DIA-BENT carefully selected instances of ibility. The benchmark is designed to support the evaluations of various downstream tasks in cross-modal dialogue generation. As shown in Tab. 3, both Gemini model and human raters con- sistently assign higher expressiveness scores to MM-DIA-BENCH, validating its role high- expressiveness benchmark.
diverse and balanced benchmark comprising 309 dual-speaker dialogues with ensured speaker vis-
3.6 VALIDATION OF THE ANNOTATION SYSTEM AND DATASET
To validate the quality of the annotation system, we conduct series of through evaluation (see Ap- pendix A.3) on each step component, demonstrating that our pipeline achieves human-level quality in annotation consistency and reliability.
MULTIMODAL DIALOGUE GENERATION TASKS
In this section, we first introduce a unified formulation of Multimodal Dialogue Generation (MDG). Subsequently, we present three representative tasks under different control conditions and output modalities, providing instantiations of the MDG framework.
4.1 PROBLEM FORMULATION
To enable systematic study of multimodal dialogue behaviors, formalize MDG conditional generation problem. Given a multimodal context C = {Cuus Cusd, Cvis) the goal to generate multimodal dialogue behaviors y (Yx. Yaud, Yvu) } that are (i) semantically coherent, aligned across modalities, and (iii) controllable. Formally, MDG can be expressed as modeling a condi- tional distribution: P(Y C, 2), where Z denotes explicit/implici control variables for dialogue

---

Published as a conference paper at ICLR 2026
style. This formulation unifies diverse downstream tasks such style-controllable dialogue speech synthesis, conditiodd speech synthesis, and speech-driven dialogue video generation, pro- viding foundation for systematic benchmarking of controllable multimodal dialogue.
4.2 TASK 1: STYLE-CONTROLLABLEE DIALOGUE SPEECH SYNTHESIS
Definition.
Conditioning purely on text, this task synthesizes multi-speaker dialogue audio stream A from transcript T = (....... uN) } (consisting of N utterances) and explicit style condition Zesp The condition Zeep belongs to either a structured affective triplet space or free-form descrip- tions: Zexp Stiplet U Ldese where Suiplet
VR Vz Vg. The synthesis process is modeled as sampling Å A~ P(A T, Zexp). Unlike turn-based concatenation standard TTS (Rong al., 2025; Du et al., 2024), we directly model A as continuous dialogue speech stream, naturally embedding speaker transitions and overlaps without explicit boundaries markers, similar to Zero-Shot Dialogue Generation (ZSDG) (Zhang al., 2024).
Challenges. Compared with conventional Controllable Tex--To-Speech (CTTS) and ZSDG, our task presents several unique challenges: (i) generating continuous single-past end-to-end dialogue dio stream that naturally encodes rich multi multi-speaker interactions beyond turn-level concatenation; (ii) maintaining coherence and consistency across successive speakers, such preserving role iden- tity and interactional dynamics throughout multiple turns; and (iii) supporting multi-level control- lability, ranging from global conditions specified by structured triplets fine-grained per-speaker expressive trajectories.
4.3 TASK 2: VISDONDCONDITIONED DIALOGUE SPEECH SYNTHESIS
Definition. This task operates under implicit controllability, where the control variable is inferred from visual cues.Let input C {Vicy, T} consist of temporally ordered keyframe sequence Viey fUI....,, ux and aligned transcripts T fui...., ug }. the goal is to synthesize multi- speaker dialogue speech A that reflects the interaction-level conditions implied by the visual scene. Formally, the model infers a latent style representation Zimp 15 (Veey) from the visual sequence and generates the audio conditioned on it: ~ P(A T, Zimp) = P(A T, (VVke)).
Challenges. Compared with explicit prompt-based control, this task requires the model to (i) reli- ably infer interactional variables from visual cues such as appearance, posture, and scene compo- sition; (ii) capture temporal dependencies across the keyframe sequence reflect evolving interac- tional dynamics in generated speech; and (iii) align inferred styles with textual content T so that the synthesized audio remains both semantically faithful and contextually expressive.
4.4 TASK 3: SPEECH-DRIVEN DIALOGUE VIDEO GENERATION
Definition. This task instantiates MDG with = [T, and y = (V). Given dialogue audio A and the corresponding transcript T, the objective is to synthesize a dialogue video V that is temporally synchronized with speech and affectively consistent with dialogue semanties: V P(V A, Z). where Z represents the control variables. In explicit control, Z is provided by the user; in implicit control, Z is a latent representation inferred from A and ''' (prosody, turn-taking, and affective dynamics).
Challenges. Compared with text-to-video (T2V) tasks and single talking head generation, our task introduces three key challenges: (i) mulli-speaker identity and scene continuity under rapid shot changes and partial visibility; (ii) multi-granularig audio-visual frig lip-audio sync and utterance-level prosody/gesture to dialogue-level expressiveness (relationship, interaction mode, affective state), often under weak/implici control; and (iii) long-range cinematic reasoning to faithfully stage interactions (who, how, where), requiring shot planning and blocking beyond what standard quality lip-sync metrics specify.
BENCHMARKING IN MULTIMODAL DIALOGUE GENERATION
In this section, we first conduct experiments to verify the effectiveness of MM-DIA in supporting the explicit style control in Dialogue Speech Synthesis (Sec. 5.1). We then leverage MM-DIA-
7

---

Published as a conference paper at ICLR 2026
BENCH to reveal key limitations in maintaining audio-visuie consistency under implicit cross-modal control in conditcond Dialogue Speech Synthesis (Sec. 5.2) and Speech-driven Dialogue Video Generation (Sec. 5.3).
5.1 EXPERIMENTS ON EXPLICIT CONTROL IN DIALOGUE SPEECH SYNTHESIS
A. Evaluation Settings.
Test sets: We evaluate our model on three evaluation splits: Hard, Test, and Out-of- Domain (OOD). The Hard set superset of MM-DIA-BENCH containing 598 clips of highly- expressive data across MM-DIA. The remaining scope of MM-DIA is then randomly split into Train, Valid and Test by 90 5: 5. To further examine generalizability, we additionally construct an OOD set with 60 human-refined dialogue clips from daily scenarios. All experimental inference is performed twice, with the Description and Affective Triplet serving as the style control respectively.
Metries: To evaluate the performance of the synthesized dialogue speech intrigued by MM-DIA, we established evaluation from the speech-, dialogue- and controllability-leee..
Speech Quality: We use word error rate (WER) and UTMOS (Takaaki al., 2022) to access the intelligibility and the overall quality of speech.
Dialogue Quality: We adopt speaker tur-ttaking accuracy (cp-WERR and speaker aware similar- ity (sa-SIM) respectively to represent the intra-speaker similarity and inter-speaker timbre transition accuracy in spoken dialogue generation.
Expressiveness Controllability Since there are appropriate objective metric to reflect consistency between the text prompt and speech, we conduct Human-Mos evaluation on overall Quality and Instruction-Folloot
capability. The Human-Mos score conducted by 10 participants rating 80 audio samples on a 1-5 scale for both dimensions. Following MoonCast (Ju et al.), we further involve Semini-as-Judge protocol to enable large-scale nuanced evaluation across Spontaneity, Coherence, Intelligibility, Timbre Similarity, Quality, and Instruction Following. Additionally, we report the Mean Recall Accuracy the label attributes of Relationship and Interaction Mode.
B. Baseline Models & Implementation Details.
To validate the effectiveness of MM-DIA, perform supervised finetuning of pretrained backbones to enable controllability under both the Triplet and Deseription control. We select two state-of-the art pretrained backbones: (1) Huigg-AAuioo-2-- V2-Base (Boson AI, 2025) and (2) Dia-1.6B (Nari Labs, 2025). Both models support single-pass dialogue speech generation. Notably, Higgs-Audio- allows flexible conditional inputs across multiple tasks, whereas Dia-1.6B is optimized for dialogue synthesis and lacks native support for conditional input. To enable controllability, we introduce a lightweight adapter module that projects explicit style embeddings into Dia-1.6B' decoder.
C. Evaluation Results
Experimental results in Tab. show that Higgs-Audio achieves superior performance style- controllable dialogue generation after fine-tuning on MM-DIA. The fine-tuning significantly re- duces WER (31.3 4.5), and the substantial decrease in ep-WER (104.8 + 33.8) indicates marked improvement in its original capabilities like content accuracy and dialogue tone conversion. A slight reduction in Sa-SIM observed (0.48 - 0.45), suggesting a mild trade-off, where improved style controllability is accompanied by slightly reduced speaker timbre consistency, partially due to the increased speaker and style variability in movie-sourced data. While both backbones benefit from fine-tuning on MM-DIA, the stronger performance of Higgs-Audio Audio demonstrates that MM-DIA ef- fectively enhances dialogue accuracy and controllability, particularly when paired with backbones designed for conditional generation. Results under the Affective Triplet control setting, including evaluations on Hard, Test, and OOD splits, are provided in Appendix A.5.
5.2 EXPERIMENTS ON VISION-CONDITIONED DIALOGUE SPEECH SYNTHESIS
A. Evaluation Settings
Test sets: We use 133 clips from HALDIA-BENCCI which guarantee single speaker visibility in each keyframe for the model to facilitate accurate speaker attribution. The dialogues exhibit
8

---

Published as a conference paper at ICLR 2026
Table 4: Results of Dialogue Speech Synthesis with Description as style control on the Test set.
Speech-Qualiit
Dialogue- Qualitty
Homas-MOS
Gemini-n-Juufg
Model
WERJ
UTMOST
46-SIM7
op-WER)
Qual.
Instr. Follow?
Spont.
Cober
Intllli:
Simila:?
Qualf
last: Follows
Dia-Base
19.991
2,272
0.389
51,713
2,500 2.410go.a4
3.995
4.335
4,446
3,738
4.248
3,807
Dis-STT
29.071
1,974
0.47
57,813
2,890
2,800
3,626
4,071
4,171
3.590
3,911
3,598
31,251
3,093
0.475
104.367
3.580gn seo
3.110g0 on
3,313
3.95
4.276
4.021
3.874
4.012
Higge-Andio- Audio-V2-SFT
4,450
3,280
0.447
33.765
4,440
4.130_...co
4.277
4.881
4.965
4,640
4,851
4,707
Table 5: Results of Vision-conditioned Dialogue Speech Synthesis on MM-DIA- BEENAB.
Speech-Quality
Dialogue-Quality
Labet-Reaal
Gemini-a--Judge
Model
WERL
UTMOS?
sa-IMMt
op st-ER
Aoc. f Spont.
Coher.
Intellig:t
Similart
Qual.
Instr. Follow:
HarmoniVox
21.223
3,5704
0.62
30.981
40.47
1.790
3.390
4,238
1.657
1,895
2.400
Cuscaded Gemini Higgs 5.781
3,3245
0.499
16,267
42.33
3,081
4.129
4,927
2.605
3.21
3.347
Cascaded
Higgs
3,4384
14.583
52.17
3,326
4.000
4.978
3,022
3,587
clear emotional contrasts through visual cues such as facial expressions or body gestures to guide style-consistent speech generation.
Metrics: Since Task shares the same output paradigm as Task 1, we preserve most metrics while slightly modifying prompts in Gemini-as- Judge protocol to analyze the alignment in dialogue expressiveness between the speech and visual sequences.
B. Baseline Models & Implementation Details
We implement several representative baselines for comparison: (1) HarmoniVox (Zhou et al,, 2025a). The model infers the avatar's latent internal states from a visual image and projects it into a talking style representation to conditioned speech synthesis. We adopt sentence-level inference in our experiments and concatenate corresponding utterances into complete dialogue. (2) Cascaded VLM + Higgs-Audio-SFT pro) to first generate descriptive style prompts in human interaction from the visual dialogue context, which are then used to condition Higgs-Audio- Mings-Audio-V22 V2-SFF for speech synthesis.
We employ a strong vision- -snoun-l model (e.g., GPT-5, Gemini-2.5-
C. Evaluation Results
As shown in Tab. 5, the cascaded methods outperform the end-to-end Harmoni Vox in most metrics. Comparing with the explicit syle-prompt setting in in Tab. 4, although most objective results pre- served stable performance related to speech and dialogue quality, the subjective scores in Gemini-as- Judge exhibit noticeable decline, especially in dimensions like Timbre Similarity and Instruction Following. This suggests that while basic speech synthesis performance is preserved, cross-modal style consistency degrades when cues are provided implicitly through visual interaction context. These results reveal an imbalance in existing approaches: they can produce fluent dialogue, yet struggle to maintain coherent interaction-level style alignment across modalities.
5.3 EXPERIMENTS ON SPEECH-DRIVEN DIALOGUE VIDEO GENERATION
A. Evaluation Settings
1. Test sets: We also use the 133 dialogues from MM-DIA- which ensures single-speaker visibility as introduced in Sec. 5.2. This is curated to cover all annotated relationships and interaction modes in MM-DIA, ensuring broad semantic coverage for cross- modsl alignment as- sessment.
2. Metrics: We evaluate along three axes: video quality (Fréchet Video Distance, FVD (Unterthiner al., 2019)), lip-speech synchronization (LSE-C and LSE-D (Chung Zisserman, 2016)), and cross-modal semantics/alignment We adopt the model-as-judge pipeline introduced in Sec. 5.1 to score Spontaneity, Coherence, Intelligibility, Similarity, Overall Quality, and Instruction Following, to quantify how well the generated dialogue videos align with the speech modality-from low- level timing (lip-speech sync) and utterance-leve prosodyllxpreessvvenss to dialogue-Ieve seman- ties (e.g., staging, flow, and instruction following). In addition, we report label accuracy/recall on Relationship and Interaction Mode to test whether generated scenes faithfully reflect dialogue-level interpersonal semantics.

---

Published as a conference paper at ICLR 2026
Table 6: Results of Speech-driven Dialogue Video Synthesis.
Quality
Lip-Sync
Model
FVDJ
LS5-C
LS6-DL
ACC-Rela,
ACC-Enterctt
Spont
Coher.
Inteligg
Similar
Quil1
Instr. Fellow. !
FLOAT (Ki al, 2025)
572.167
4,305
9.502
2,703
2,405
3,050
3.339
2.248
3,050
MukiTaik 2025))
124,543
5.305
5,795
4.524
4.358
4.612
4,922
4.631
Sunie 2025)
117,096
4.986
4,592
4.790
4,833
W. S2V (Win et tl,, 2025)
154,261
4,288
9.873
4,205
4.116
4.357
4,589
4.652
4,384
Valeo (Kong 2025a)
335.591
47.97%
13,926
2.089
4.553
4.099
2.66
4,509
2.29
Wan-22 T2V 2025)
53
18.70%
3,114
4.602
4.423
3
Ground
6.375
100.005
4,892
4.961
5.000
B. Baseline Models & Implementation Details
Because no system currently performs end-to-end Hiulogue-tovvidee generation, we evaluate two practical families:
We split dialogue-Ievel movie clips into sentence-level segments and drive the corresponding speaker images with each utterance, then concatenate per-sentence clips into dialogue videos. Given that SI2V models use reference keyframes, we do not evaluate relationship/scene accuracy here; we focus on lip sync and expressiveness alignment.
SI2V Sppaake--maag-tor
T2V (exteo)-Video). Using sentence-level fine-grained and dialogue-level expressiveness anno- tations in MM-DIA, we construct rich text prompts to condition multi-speaker scene synthesis. Since audio is not explicitly input, do not score lip sync for T2V; instead, emphasize relationship//nteracioon and expressiveness alignment. During model-as-judge, we provide the corresponding audio for Gemini evaluate the cross-modality alignment.
C. Evaluation Results
All experiments are conducted expressiveness to ensure comparable shot complexity across systems.
CMM-DIABENCEE dialogue clips with visible dyads and diverse
Results in Tab. 6 show that current system adequately solves dialogue video generation. Despite rich prompts, T2V models capture only a portion of high-level dialogue semantics; accurate staging of interaction scenes and ceo--inntraatswiiteee with remains unreliable. SI2V systems attain higher bogherence/Innellgibbltty
on average, but Instruction Following and fine-grained Spontaneity alignment fluctuate across long dialogues.
To summarize, SI2V pipelines are complex and depend on keyframes; practical deployment will require coupling with keyframe generation to approach end-to-end usage. Additionally, small face extents and occlusions in natural dialogue shots make lip-syne brittle, often producing artifacts. Meanwhile, T2V systems lack explicit audio conditioning. making it difficult to synchronize with speech timing and match vocal expressiveness; they also underperform at faithfully reconstructing relationships and interaction patterns.
Overall, neither family is yet adequate for dialogue video generation. The results validate our bench- mark design: quality and lip-syne alone are insufficient; cross-modal semantic alignment must be measured explicitly to drive progress. Future work should target: (1) End-to-end dialogue-to-videe modeling that unifies keyframe planning, character visibility, lip/body sync, and scene continuity; (2) Multi-granularity alignment learning using sentence-level and dialogue-level expressiveness la- bels (relationship, interaction mode, affect); (3) Cross-modal semantic discriminators that penalize misalignment during training; and (4) Long-range dependence & shot planning for controllable staging in multi-speaker scenes, consistent with expressiveness schema in MM-DIA.
6 CONCLUSION
In this paper, we propose MM-DIA, the first large-scale highly-expressive multimodal dialogue dataset for the task of Multimodal Dialogue Generation, and the corresponding dual-speaker bench- mark MM-DO AMMDDA--B BENCH for the evaluation of cross-modal conditional generation tasks. Experi- ments demonstrate that MM-DIL AMDDIA enhances the style controllability of dialogue generation model and HIM-DAA-EENT reveals the limitation in current cross-modal style consistency.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China under Grant Nos. 62425604 and 62502256.
REFERENCES
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22563-22575, June 2023.
Boson AL. Higgs Audio V2: Redefining Expressiveness in Audio Generation. https: // 2025. GitHub repository. Release blog available
github. wwtp.//oooooaaaaaaaooaee-ae---ee
at Sittpss: /www wwwp::boooo....booooaao
Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In Asian Conference on Computer Vision, pp. 251-263 Springer, 2016.
Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21086-21099 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijic Yan. Cosy Scalable Multilingual Zero- shot Text-t--ppecc Synthesizer based on Supervised Semantic Tokens, July 2024. URL http: //arxiv.orgge abs 2407. 05407. arXiv:2407.0544 4070440777 [es].
Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, and Youngjae Yu. Champagne: Learning real- world conversation from large-scale web videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15498-15509, October 2023.
Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. Autoad ii:: The prequel back to the pixels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18164-18174, June 2024.
Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming I Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 193-203, 2025.
Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. Speecheraft: A fine-grained expressive speech dataset with natural language description. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1255-1264, 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: large-scale video dataset with long durations and structured captions. In Advances in Neural Information Processing Systems, volume 37, pp. 48955-48970. Curran Associates, Inc., 2024.
Zeqian Ju, Dongchao Yang, Kai Shen, Yichong Leng, Zhengtao Wang, Songxiang Liu, Xinyu Zhou, Tao Qin, Xiangyang Li, Jianwei Yu, al. Mooncast: High-quality zero-shot podcast generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Taekyung Ki, Dongchan Min, and Gyeongsu Chae. Float: Generative motion latent flow matching for audio-driven talking portrait. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14699-14710, 2025.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu. Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, An- dong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song.
11

---

Published as a conference paper at ICLR 2026
Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang. Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhen- tao Yu, Zhiyu He. Zhiyong Xu, Zixiang Zhou, Zumnan Xu, Yangyu Tao, Qinglin Lu, Song- tao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025a. URL https:: /ttp://aaxvvvvv.. org/ abs /ttrg//bb///// 2412.41222220 03603.
Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025b.
Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xie. Intelli- gent grimm open-ended visual storytelling via latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1190, June 2024a.
Rui Liu, Haolin Zuo, Zheng Lian, Xiaofen Xing, Bjöm W. Schuller, and Haizhou Li. Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset, 2024b. URL
http:/ / / /wwxiv..vvv.. org/ abs / 2407. .27551.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang. Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- 1m: A large-scale high-quality dataset for text-to-video generation. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL net htt./forrmidddee j7kdXSrISM.
https: http::s//ppeeeeviiiee
Nari Labs. Dia: A its model capable of generating ultra-realistic dialogue in one pass. https: / //github. com/nari labs/ /abs/dia, 2025. Accessed: 2025-09-20.
OpenAI, Josh Achiam, and Adler. GPT-4 technical report, 2024. URL http::t/axxxi..rrrrtt abs/2303. 08774.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 527-536, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1050.
Yan Rong, Shan Yang, Guangzhi Lei, and Li Liu. Dopamine Audiobook: A Training-free MILM Agent for Emotional and Human-like Audiobook Generation, April 2025. URL http: //a //arxiv.. org/ abs www.aaaa.. 22504. 1 11 100.. arXiv:2504.1 1002 [cs].
Saeki Takaaki, Xin Detai, Nakata Wataru, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. UTMOS: S'ruyo-SaruLab System for VoiceMOS Challenge 2022. In Interspeech 2022, pp. 4521-4525, 2022.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric and challenges, 2019. URL /etts::/arraer
org/ abs /tbs//882... 01717.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025. URL https:: : / a arxiv.... www.vvorg//ooo....oooo...o.ooorr orgg abs / 2500 20314.
Haoyu Wang, Xiaozhe Xin, Xiaoyu Qin, Meiguang Jin, Junfeng Ma, Dan Xu, and Jia Jia. Emotion- conditioned motion sub-spaces with flow matching for real-time audio- talking heads. In Proceedings of the 40th Annual AAAI Conference Artificial Intelligence (AAAI-26), 2026.
12

---

Published as a conference paper at ICLR 2026
Qixin Wang, Songtao Zhou, Zeyu Jin, Chenglin Guo, Shikun Sun, and Xiaoyu Qin. V-cass: Vision-context- expressive speech synthesis for enhancing user understanding of videos. In 2025 International Joint Conference Neural Networks (JJCNN) pp. 1-8, 2025. doi: 10. N0.T109/JCNN644811 22NGCNNN6688.222.11222222 222...02222.111
Shuhe Wang, Yuxian Meng, Xiaoya Li, Xiaofei Sun, Rongbin Ouyang, and Jiwei Li. Open ViDial 2.0: A larger-scale, open-domain dialogue generation dataset with visual contexts, 2021. URL http: wttp:/i..rgg/bb////////
12761.
Weijia Wu, Mingyu Liu, Zeyu Zhu, Xi Xia, Haoen Feng, Wen Wang, Kevin Qinghong Lin, Chun- hua Shen, and Mike Zheng Shou. Moviebench: A hierarchical movie level dataset for long video generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18984-289944, June 2025.
Junfei Xiao, Ceyuan Yang. Lvmin Zhang, Shengqu Cai, Yang Zhao, Yuwei Guo, Gordon Wetzstein, Maneesh Agrawala, Alan Yuille, and Lu Jiang. Captain cinema: Towards short movie generation. In The Fourteenth International Conference on Learning Representations, 2026. URL https: / //openzeview.... net/ um? iididdziiii 1d=21N2BXOIIC..
Leying Zhang, Yao Qian, Long Zhou, Shujie Liu, Dongmei Wang. Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, and Michael Zeng. Covomix: Advancing zero- shot speech generation for human-
multi-talker conversations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 100291-100317. Curran Associates, Inc., 2024.
Songtao Zhou, Xiaoyu Qin, Yixuan Zhou, Qixin Wang, Zeyu Jin, Zixuan Wang, Zhiyong Wu, and Jia Jia. Harmonivox: Painting voices to match the avatar's soul. In Proceedings of the 33rd ACM International Conference on Multimedia, MM '25, 2025a.
Songtao Zhou, Xiaoyu Qin, Yixuan Zhou, Qixin Wang, Zeyu Jin, Zixuan Wang, Zhiyong Wu, and Jia. Harmonivox: Painting voices to match the avatar's soul. Proceedings of the 33rd ACM International Conference on Multimedia, pp. 6720-6729, 2025b.
Yixuan Zhou, Xiaoyu Qin, Zeyu Jin, Shuoyi Zhou, Shun Lei, Songtao Zhou, Zhiyong Wu, and Jia Jia. Voxinstruct: Expressive human instruction-to-speede generation with unified multilin- gual codec language modelling. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 554-563, 2024.
Han Zhu, Wei Kang, Liyong Guo, Zengwei Yao, Fangjun Kuang, Weiji Zhuang, Zhaoqing Li, Zhifeng Han, Dong Zhang, Xin Zhang, Xingchen Song, Long Lin, and Daniel Povey. ZipVeice- dialog: Non-autoregressive spoken dialogue generation with flow matching, 2025. URL http: //t////aaaaiee //arsiv.orr//bs//30700001
13

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide the MM-DIA dataset, a large-scale multimodal dialogue corpus, and the MM-DIA- BENCH benchmark, both of which integral to our research on style-controllable multimodal dia- logue generation. Our experimental code and data curation pipeline will be made publicly available upon acceptance of the paper. The models and algorithms used in this paper can be reproduced us- ing the provided dataset and benchmark, with all necessary details regarding model configurations, training procedures, and evaluation protocols included.
ETHICS STATEMENT
The MM-DIA and MM-DIA- BENCH datasets include multimodal data sourced from movies and TV series, some of which may contain commercial content. We do not release the video or audio clips themselves; instead, we provide annotations (e.g., transeript, affective triplet, dialogue de- scription, speaker identity, keyframe with active speaker, etc.d) and the methods used to generate them. Researchers are encouraged to obtain the corresponding media content independently and align it with the provided timestamps. For any further queries information, readers are welcome to contact us.
We acknowledge the potential for biases inherent in the media content used and are committed to addressing these future versions of the dataset by incorporating more diverse sources and refining annotation methods.
LLM USAGE DISCLOSURE
We used GPT-5 for grammar checking and improving the clarity of sections 1 through 6 in this manuscript. All technical content, experimental design, and analysis are original human work. The LLM suggestions were manually reviewed and modified ensure that they align with the paper's objectives and maintain technical accuracy.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLEMENTATION DETAILS FOR SUBTITLE CALIBRATION
We combine multi-sourced uncalibrated subtitles with ASR results to perform precise synchroniza- tion between the timestamps and content. Selecting the matched specific ASR segments and subtitle entries as anchor points, we perform translation operations to adjust time and duration differences in the uncalibrated subtitle timestamps with minimal discrepancies. Typical cases in multi-sourced movie subtile alignment are shown in Fig. 4. The qualified subtitle with low variance in discrep- ancy are double-checked by human to ensure usability. The alignment not only improves the overall synchronization of subtitles with the spoken content, but also mitigates errors introduced by ASR, enhancing the accuracy and reliability of the subtitle data in audio-visual applications.
(
(b)
(c)
Figure 4: Three bad/good cases of subtitle alignment: (a) edited movie segments, (b) edited movie speed, and (c) potential usability with time translation. In each figure, the upper plot shows the start time discrepancy between anchor point start times in the subtitle and the ASR results, and the lower plot shows the duration discrepancy.
A.2 IMPLEMENTATION DETAILS FOR DIALOGUE EXTRACTION
We introduce a buffer mechanism to dynamically update a keyframe pool of the current scene, with the pseudo-code illustrated in Algorithm
Let P = {P1. P2..... Pm represent the dynamic set of most representative keyframes from the current scene 5 {st-n........ St-1, $t). VLM uses the updated keyframes P perform sparse comparisons of the similarity between the P and the frame after certain buffer interval b, 8144- Whenever match fails, falls back to the subsequent frame 8t+1 through binary search. Once the match is successful, sparse comparisons will start from the end frame, recognizing the passed frames within the same scene. Meanwhile, the keyframe pool is updated the by replacing most similar frame within the pool with the new 81+b-
S' = ( Br4bb, p' p'=
PU (Sc48) \ (Pmost_similar), if VLM(P, 8146) True.
The buffer spanning multiple frames, together with the memory pool, enables the algorithm to "bridge" temporary interruptions instead of triggering incorrect scene boundaries. This allows the algorithm maintain the continuity of dialogue scenes over longer periods, providing greater re- silience to the complex visual dynamics of movies.
A.3 VALIDATION RESULTS THE ANNOTATION SYSTEM AND DATASET
1. Evaluation of the correctness in movie-subtitle synchronization.
As shown Tab. 7, with the official version of subtitle stands for the ground truth of content and human judgment correctness of timestamps boundaries, the calibrated subtitle performs balanced in low word error with high time accuracy. Notably, both ASR and official subtitle tend to present the line slightly earlier than the actual time, while the start time usually correct. As result, slightly extend the audio up to the next starting time the subsequent training.
2. Evaluation the buffer mechanism in boundary detection.
15

---

Published as a conference paper at ICLR 2026
Algorithm Subtitle Scene Segmentation with VLM
Require: Subtitle file srt, Video file video, Step size step, Buffer size buffer Ensure: List of dialogue ranges
1: Load VLM model (Qwen2.5 S-wen2.5-VL-B-B.. B-VV--BB-Instee Instrruttt ParseScriptSit
2: Extract subtitle blocks with index and timecode
3: return list of blocks ExtractFramevideo, timecode
4: Compute midpoint timestamp
5: Use fimpeg to extract frame image
6: return image path ISConiinuaionfraames
7: Prompt VLM with frames to check scene continuity
8: if last frame matches context then
9: return True
else return False
11:
12: end if
13: Initialize ranges list
14: for each block i do
15: Try extend range by comparing future blocks using VLM
16: Allow up to step ahead, using up to buffer context frames
if continuation fails then finalize current segment
18:
end if
20:
end for
return ranges
22: Save ranges to ISON output
Table 8: Completeness and Hallucination of Dialogue Annotation from Qwen-72B, GPT- & Gemini-2.5-pro.
Table 7: TimeStamp accuracy and WER of different subtitle version.
Annotation
Model
Comp.
Hall.
Data Source
TimeStamp Acc.
WER
Qwen
1.25
2.12
Non-verbal Sound
GPT
ASR
0.871
0.34
Gemini
4.66
1.22
SRT-Uncalibrated
0.179
0.43
Qwen
3.45
2.56
SRT-Calibrated
0.857
0.03
Affective Triplet
GPT
3.66
2.20
Gemini
4.76
1.38
SRT-Official
0.870
0.00
Qwen
3.15
2.76
Description
GPT
3.60
2.16
Gemini
4.72
Firstly, conduct human evaluation on random sampled with six movies, with the ported boundary extraction accuracy to be 95.2%, comparing to 86.3% on the traditional frame-by- frame scene continuation detection methods.
As to the ablation study on the proposed buffer mechanism, inspired by the Intersection over Union (IoU) metric commonly used Object Detection, we introduce new metric called F1 Flowertap to represent the similarity between two continuous segmentation of a same sequence of clips, expressed as (A), (B):
Table 9: Ablation study on the buffer with Qwen 7B and Qwen 72B model as VLM.
Overlop
I=1
3
Qwen 7B
0.771
0.866
0.841
0.839
0.836
Qwen 72B
0.947
0.975
0.977
0.978
0.979
16

---

Published as a conference paper at ICLR 2026
Using A the reference segmentation, for the intervals in A, take the corresponding in- terval in B that has the maximum overlap with to calculate the percentage of the total overlap- ping duration of these n overlaps in A, denoted P(A, B). Formally, this can be written B) = Eisg Overlap(A..
Similarly, we reverse the roles of and to compute A))
Zist Duration(A.)
The similarity between the segmentation is then computed using the Fl score of P(A, (3) and P(A.B)xP(B.A)
AA) Floverlap = 2 x P(A.B)+ The Floverlap metric prevents extreme segments, whether excessively dense or sparse, from receiving a high P score based on a single perspective. As shown in Tab. 9, we leverage Qwen 72B with == balance the time and performance.
3. Quality evaluation on the dialogue annotation.
Following MovieBench Wu et al. (2025), we invite two human annotators evaluate the perfor- mance of Gemini annotation in the data curation pipeline, from the perspective of Completeness and Hallucination. Annotators are asked score on a 1-5 scale for the three kinds of annotation of 100 randomly sampled movie/TV clips from MM-DIA. As indicated in Tab. 8, in comparison with Qwen 72B and GPT 5 (which instead takes sequential frames and audio as video input), Gemini- 2.5-pro outperforms in most aspects with the best interpretation of the movie style.
A.4 METRICS EXPLANATION IN TASK 1.
Speech Quality: WER, UTMOS.
We use the official implementation from Zhu et al. (2025) to compute WER and UTMOS, accessing the intelligibility and the overall quality of speech.
Dialogue Quality: cp- WER, sa-SIM.
Speaker Jurn-Taking Accuracy (cp-We (cp-WER) is computed by firstly concatenating all speech utter- ances by the same speaker after processing the speaker diarization to the generated spoken dialogue, then picking up the lowest WER among all the permutations of the generated transcripts with the concatenated ground truth.
Speaker Aware Similarity (sa-SIM) is acquired by computing mean speaker similarity among the permutations of each speaker's utterance after conducting the M
A.5 ADDITIONAL EXPERIMENTAL RESULTS OF DIALOGUE SPEECH SYNTHESIS UNDER AFFECTIVE TRIPLET STYLE CONTROI..
As illustrated in Tab. 10, the results under the Affective Triplet control setting exhibit trends highly consistent with those reported in Tab. 4 under the Description prompt. Across both Test and Hard splits, Higgs-Audio- V2-SFT consistently achieves the best performance in speech quality, dialogue alignment, and controllability metrics. This consistency indicates that MM-DIA enables robust style control under both natural language and structured affective prompts.
As expected, performance on the Hard split (Tab. 11) is generally lower than on the Test split (Tab. 10), particularly in WER and coherence-related metrics. This suggests that highly expres- sive multimodal dialogues introduce increased acoustic variability and emotional volatility, posing additional challenges for speech synthesis. Nevertheless, the relative ranking between models re- mains stable, and controllability metrics (e.g., label recall and Instruction Following) remain strong, indicating preserved control effectiveness under expressive conditions.
Additionally, introduced in Sec. 5.1, manually created Out-of- oooaaann clips of dialogue tent and style annotations to enable variable control in dialogue generation. The dialogues are daily conversation that GPT-generated and human-refined. Specifically, data categorized into every three pieces by the Affective Triplet, with two variables fixed in each group, and the third variable changing. Besides, the three dialogues share a same sentence be presented in different cases. For instance, the sentence 'Please, listen, if you's 'd just give me second, I can clear this up!" could be acted with subtle difference under different speakers relationships, ranging from Lovers, Employer- employee, to Ariice-Ci Please refer to the audio samples on the Demo Page ? which clearly
Phttps://mmdiaa w2t.aillr26...... rtt.giii......ttte wtw.ggtu.b....mmmiiiiio.... htt.:iii/m/miicr///....
17

---

Published as a conference paper at ICLR 2026
demonstrate the model's ability capture the subtle stylistic shifts corresponding to the controlled variable changes.
Overall, these findings confirm that MM-DIA supports both robust style control and generalization across difficulty levels, while highlighting the intrinsic challenges of expressive dialogue speech synthesis.
Table 10: Results of Dialogue Speech Synthesis with Affective Triplet as style control on the Test set.
Speech-Quaiee
DinlogueQuality
Label-Recall
Gemini Sermini-as-Judee
Model
WERJ
UTMOS1
Sa-SIM
p-WER,
Mean Acc.
Spont:
Coher."
Intellig.
Similar.
Qual."
Instr. Follow.?
Dia-Base
19.991
2.272
0.389
51.713
0.210
3.452
4,000
4,161
4.016
3.887
4.113
Dia-SFT
33.178
1,941
0/400
117.947
0.237
3.636
4.118
4.187
3,910
3,962
4,014
Higgs-Audio-V2-4
0.461
3.169
3.843
3,704
HiggsAuddio-2-8F7
5.265
3.286
0,499
33,134
00.28
4,031
4,820
4.967
4,610
4.636
4,809
Table 11: Results of Dialogue Speech Synthesis with Affective Triplet as style control on the Hard
Speech-Quality
Dialogne-Quality
Label-Recall
Gemini-ns-Iudge
Model
WER,
UTMOS1
sa-SIM
cp (p-WER]
Mean Acc.
Spont.
Coher:
Inteligg:
Similar:"
Qual.
Instr. Follow:
Dia-Base
20.266
2.164
0.406
71,956
0.224
4.093
4,407
4.481
3,981
4.381
4.003
Dia-SFT
28.940
1.855
0,409
68.893
0.246
3.719
4.179
4.258
4,018
4.111
4,030
Higgs-Andio-V V2-Base
23.041
3.711
0.463
57.850
0.327
3.190
3,969
4.637
4.151
4.115
4.095
Higes- Audio-u2-dFF
7.206
3.184
0,425
35,690
0.360
3.267
4.667
4.953
4.562
4.611
A.6 EXPLANATION RELATIONSHIP AND INTERACTION MODE CATEGORIES
Table 12: Explanation of Relationship Categories with Typical Labels
Relationship
Explanation and Example
Workplace
Refers to professional relationships and environments, including
people within a work setting.
Example:
Colleague, Boss, Manager, Coworker, Client.
Friends
A relationship between individuals characterized by mutual af-
fection, trust, and companionship outside of family and work.
Example: Buddy, Pal, Companion, Mate, Peer.
Intimate
Relationships of more personal and romantic nature, typically involving emotional and physical closeness.
Example: Boyfriend, Girlfriend, Pariner, Spouse, Fiancé.
Family
Relationships defined by blood ties or marriage, including ex- tended family members.
Example: Mother, Father, Sibling, Uncle, Cousin.
Adversarial
Relationships characterized by opposition or conflict, often in- volving rivalry or animosity.
Example: Enemy, Opponent, Rival, Antagonist, Competitor.
Individual
A relationship with oneself, or solitary state where interaction with others is minimal or nonexistent.
Example: Solo, Loner, Isolated, Monologue.
Social
Encompasses a wide range of social roles and interactions, from professional settings casual encounters.
Example: Teacher, Doctor, Neighbor, Stranger, Host, Cus-
tomer.
Continued next page...
18

---

Published as a conference paper at ICLR 2026
Relationship Authority
Explanation and Example
Relationships based on power and control, typically involving leadership. governance, and decision-making.
Example: King, Judge, Mayor, President, General.
Table 13: Explanation of Interaction Mode Categories with Typical Labels
Interaction Mode
Explanation and Example
Suggesting
The act of convincing someone to believe or do something through reasoning emotional appeal.
Example: Persuasion, Convincing, Negotiation.
Conflict
A state of disagreement or confrontation, often involving tension or hostility.
Example: Argument, Disagreement, Accusation.
Questioning
Asking questions to gain information, clarify doubts, provoke
thought.
Example: Inquiry, Interrogation, Probing.
Narration
The act of narrating story personal experience, often to en- tertain or inform.
Example: Storytelling, Flashback, Monologue.
Explanation
Providing detailed information or clarification on topic to en- understanding.
Example: Justification, Diagnosis. Clarification.
Commands
Issuing direct orders or instructions prompt action.
Example: Orders, Demands, Instruction.
Dynamic Cross-talk
A back-and-forth exchange dynamic dialogue, often with in- terruptions or interjections.
Example: Interjection, Interruption.
Sympathizing
Offering comfort or support someone, often alleviate con- cerns or anxiety.
Example: Comfort, Support, Encouragement.
Rejection
Dismissing refusing request, idea, proposal.
Example: Refusal, Dismissal, Avoidance.
Banter
Playful, often teasing, interaction intended to entertain or create rapport.
Example: Teasing, Flirting, Joke.
Authority Power
The use of authority control direct others' actions, often in a commanding or corrective manner.
Example: Domination, Criticism, Intervention.
Performance
Delivering structured or formal presentation, speech, an- nouncement an audience.
Example: Presentation, Speech, Announcement.
Reflection
Reflecting on one's thoughts, feelings, or experiences, often leading moment of realization.
Example: Introspection, Revelation, Discovery.
Emotion Release
Expressing emotions, often related to frustration, anxiety, or re- lief. Example: Venting, Confession.
Invitation
Extending request for someone join event activity. Example: Invitation, Offer.
19


Reviews:
[{"review_id": "ZuR5T8AhwV", "reviewer": "Reviewer_Nvzm", "summary": "This paper introduces MM-DIA, the first large-scale and highly expressive multimodal dialogue dataset designed for Multimodal Dialogue Generation (MDG). In addition, the authors present MM-DIA-BENCH, a dual-speaker benchmark specifically developed for evaluating cross-modal conditional generation.\nExperiments show that training on MM-DIA significantly enhances controllable dialogue generation, while evaluations on MM-DIA-BENCH reveal notable limitations of current models in achieving consistent multimodal style alignment.", "questions": "Pls refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is well-motivated, and the proposed dataset paves the way for future research on style controllability of multi-modal dialogue generation.\n\n* Strong experimental validation with ablations on both controllability and user satisfaction metrics.\n\n* This paper is clearly structured and easy to follow.\n\n* Although the dataset creation heavily relies on models, the authors try to demonstrate that the proposed pipeline achieves human-level quality in annotation consistency and reliability", "weaknesses": "* The data creation and evaluation partly rely on GPT-based scoring, which could cause an upper bound of future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X1ZXJbhYHq", "reviewer": "Reviewer_jqoy", "summary": "This paper addresses the limitations in scale, expressiveness, and benchmarking of existing datasets for multimodal dialogue generation by proposing a novel data curation and annotation pipeline, resulting in the large-scale and expressive multimodal dialogue dataset MM-DIA. The authors further introduce a unified framework for Multimodal Dialogue Generation (MDG) and define three representative downstream tasks, including style-controllable speech synthesis, vision-conditioned speech synthesis, and speech-driven video generation. Through systematic benchmarks and experiments, the paper demonstrates the effectiveness of the new dataset in enhancing dialogue style controllability and cross-modal consistency, while revealing the shortcomings of current methods in expressiveness and multimodal alignment. This work provides valuable resources and new challenges for future research in conditional multimodal dialogue generation.", "questions": "1. In the evaluation, the authors use Gemini as a judge. Has the accuracy of using large models as judges been tested, and has the model’s performance been compared with human evaluation?\n2. In this paper, the authors achieve \"From Natural Alignment to Conditional Controllability\" from a data perspective. From a methodological standpoint, do you think further improvements at the model level could help achieve this goal?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The unified framework for multimodal dialogue generation proposed in this paper is highly practical and extensible, providing systematic task definitions and benchmarking foundations that will facilitate further advances in the field.\n2. The experimental section is comprehensive, covering various downstream tasks and systematically benchmarking the dataset and methods for style controllability and cross-modal consistency, with convincing results.\n3. The paper is well-structured and clearly articulated, progressing logically from problem motivation, dataset construction, method design, to experimental evaluation, making it easy for readers to follow and understand the research.", "weaknesses": "1. Although the dataset is large and expressive, it is mainly sourced from movies and TV series, which may differ from real-life conversations and affect the generalizability of models to real-world scenarios.\n2. The evaluation methodology in the paper is insufficient for assessing the generalization ability of the trained models. In Section 5.1, the authors mention an out-of-domain dataset, but do not clearly specify whether it comes from different data sources or demonstrate its differences. The model’s performance in real-world scenarios, as well as the potential degradation of its original capabilities after SFT (fine-tuning), require further consideration and analysis.\n3. There are some typographical errors in the paper, such as the table on page 782 not being cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLcv1wdbEz", "reviewer": "Reviewer_KcRL", "summary": "The paper introduces MM-DIA, a large-scale, richly annotated multimodal dialogue dataset from movies and TV series, and MM-DIA-BENCH, a benchmark for evaluating cross-modal conditional generation. Experiments show MM-DIA improves style-controllable dialogue generation", "questions": "1. How do you envision MM-DIA supporting research on more spontaneous, real-world dialogues? \n\n2. How does your approach handle long-range dependencies, such as multi-turn conversations or scenes with complex speaker dynamics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is the first dataset to focus on dialogue expressiveness across multiple modalities. The benchmark (MM-DIA-BENCH) fills a gap for evaluating cross-modal style consistency, which is underexplored in prior work.\n2. The paper provides a unified framework for MDG, with well-defined tasks and evaluation metrics. Experiments are thorough, with both objective and subjective metrics.", "weaknesses": "1. The paper’s main contribution is dataset and benchmark creation; the modeling advances are limited to fine-tuning existing architectures and adapter modules for controllability. No novel end-to-end model for multimodal dialogue generation is proposed or evaluated.\n2. The paper is too dense and at times it is difficult to follow, especially in the technical details of the pipeline and annotation process.\n3. The dataset is sourced primarily from movies and TV series, which may limit the diversity and generalizability to real-world, spontaneous dialogues", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u4YtVU2lGE", "reviewer": "Reviewer_8JZi", "summary": "This paper addresses critical limitations in multimodal dialogue generation—specifically the overemphasis on content transmission over style controllability, scarcity of high-quality datasets, and lack of benchmarks for cross-modal consistency. It focuses on achieving expressive, controllable multimodal dialogue through natural alignment of speech, vision, and text, while constructing a large-scale dataset and systematic benchmarks to advance the field.", "questions": "You claim MM-DIA supports \"broad applications in HCI and social computing\" but exclusively use cinematic data (movies/TV series) for training and testing. Given that movie dialogue is scripted and emotionally exaggerated (e.g., MM-DIA’s average emotion intensity score of 6.76/10 via Gemini; )—a stark contrast to casual real-world interactions—have you tested if models fine-tuned on MM-DIA (e.g., Higgs-Audio-V2-SFT) retain style controllability on real-world multimodal datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces MM-DIA, a dataset curated from 700+ hours of movies/TV series (200+ films, 9 shows) with 360.26 hours of dialogue, 54,700 clips, and 449,138 turns. It features fine-grained annotations across modalities. The paper’s flagship contribution—the MM-DIA dataset—is the first to center on \"multimodal dialogue expressiveness\". To evaluate implicit cross-modal style consistency (a long-overlooked gap), the paper builds MM-DIA-BENCH—a balanced benchmark of 309 dual-speaker dialogues (1.69 hours, 1,851 turns) with guaranteed speaker visibility. Experiments show MM-DIA significantly enhances style controllability.", "weaknesses": "The paper claims MM-DIA and its findings support \"a wide range of applications in human–computer interaction, social computing, and film-making\" but exclusively uses cinematic data (movies/TV series) for dataset construction and experiments. This creates a critical gap: it is unclear if the proposed framework (annotations, tasks, model insights) generalizes to non-scripted, real-world multimodal dialogue—arguably the most impactful use case for HCI and social computing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)