Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
Authors: Zhe Kong
Feng Gao
Yong Zhang
Zhuoliang Kang
Xiaoming Wei
Xunliang Cai
Guanying Chen
Wenhan Luo
Venue: NeurIPS 2025 poster ()
Keywords: aigc
digital human
video generation
Primary area: applications

Abstract: Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.

Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
Zhe Kong 12.3 Feng Gao Yong Zhang Zhuoliang Kang?, Xiaoming Wei", Xunliang Cai², Guanying Chen', Wenhan Luo
Shenzhen Campus of Sun Yat-sen University Meituan
Division of AMC and Department of ECE, HKUST
https://meigenn ai github io/mmlti-taall
Prompt: talking, then they
Conversation Audio
olluu Iimiooon
Prompit: The picked up cup and rabbit's head.
Figure 1: We propose MultiTalk, a novel framework for audio-driven multi-person conversational video generation. Given multi-stream audio input and prompt, MultiTalk generates video containing interactions following the prompt, with consistent lip motions aligned with the audio.
Abstract
Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial move- ments and appealing visual quality videos. However, existing methods primarily focus single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-followee followwig capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Genera- tion, and introduce new framework, MultiTalk,, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training crucial for preserving the instruction -ooloooing ability of the base model. MultiTalk achieves superior performance compared other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.
"Equal Contribution. Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Audio-driven human animation aims to generate natural and vivid numan-centrin videos with syn- chronized facial expressions and body movements from audio control signals. This field has made significant progress recently, and existing methods can be roughly divided into two categories: talking head generation and talking body generation.
5 6 focus on talking head generation. These methods utilize diffusion models to match audio features to visual frames, enabling the synthesis vivid talking head videos with enhanced video quality and realistic facial expressions. However, they are constrained to achieve precise audio-aligned facial movements and often neglect other related motions, such as hand and body. Recently, several methods I7 8 9 10 [1] have utilized video diffusion models [1]13 14] and successfully achieved talking body generation. By leveraging mixed data training strategies using additional hand pose data, they can synchronize body movements with the audio. Despite these advancements, several constraints remain. Existing methods primarily target single-person animation and cannot handle mullti-person scenarios, such as conversational video generation. They lack the capability for dual-stream audio injection. Additionally, they exhibit limitations in cestruction-folloiing capabilities. For instance, generated videos may fail to precisely follow instructions when a text prompt describes a large range of body movement.
Most human animation methods 12
3
conversational video generation. This task has diverse applications, including multi-character movie scenes making and e-retailers' livestreaming. Compared audio-driven single- humann animation, this task presents three main challenges: 1) As conversations involve audio from multiple persons, the model should accommodate multi-stream audio inputs; 2) Each person within the conversation should be driven by only one audio stream to prevent incorrect face and audio binding; 3) Each person in the generated video is dynamic, requiring an adaptive method for person localization. Despite the success of existing methods in achieving subtle expressions and realistic motions for single person, challenges remain in creating multiple-person videos. Specifically, existing methods cannot handle multi-stream input audio and are limited to single audio stream. Additionally, when reference images contain multiple people, the audio tends to drive all individuals speak simultaneously, resulting in consistent lip motions across all persons. This complicates the achievement of alternating speech in conversational video.
In this paper, we propose new task: audio-driven
To complete this new task, we propose novel framework, MultiTalk, for audio-driven multi-person conversational video generation. Multi-stream audio injection often encounters incorrect binding between the audio and the person. We investigate several schemes for audio injection and introduce the Label Rotary Position Embedding (L-RoPE) method. By assigning identical labels to audio embeddings and video latents, effectively activates specific regions within the audio cross-attention map, thereby resolving incorrect binding issues. Furthermore, we explore a set of training strategies, including multi-stage training, partial parameter training, and multi-task training. Our observations highlight the importance of the latter two strategies. After incorporating a multi-event dataset for image-to-vides the
ability of the base model is preserved.
Our main contributions are summarized as follows: (1) We propose a novel task, i.e., audio-driven multi-person conversational video generation, and introduce novel framework to address the challenges. (2) We investigate several schemes for multi- audio injection and propose the Label Rotary Position Embedding method to resolve the inaccurate audio binding problem in nulti-person video generation. (3) We explore a set of training strategies, including multi-stage training, partial parameter training, and multi-task training. We observe that the latter two are crucial for preserving the instruction-followinn ability of the base model, especially with limited compute resources and data. The multi-event dataset for the image-to-vided is quite crucial. (4) We conduct evaluations on various datasets, such as talking face, talking body, and multi-person conversation. The results demonstrate the effectiveness of the proposed method.
2 Related Work
2.1 Audio-driven Human Animation
Pioneering audio-driven human animation works 15 two components. They first employ an audio-to-motion model to transform motion signals into intermediate representations such as 3DMM [22) and FLAME 23 Subsequently, motion-to-video
16
17
18
19
20
21
typically consist of

---

Label Ratory Realtion Embedding (L_Rs)))
Adapter
Wev2Vec
Permatt
Ressnt
Prompt: A man and woman are singing Tegether on stage.
3D VAE Encoderr
Loyar Nar
50 rul Attonfon
Text Cross Atfantin
Audo Austo.ross Cross Attention
Layar Narm
FFN
VATE Daccder
Figure 2: The overall pipeline of the proposed MultiTalk framework. Our framework incorporates an additional audio cross-attention layer to support audio conditions. To achieve multi-person conversational video generation, propose Label Rotary Position Embedding (L-RoPE) for multi-stream audio injection.
rendering techniques, such as GANs, are employed to project these intermediate representations into dynamic portrait animations. Despite notable successes, limitations in audio-to-man to-totion models' ability to capture intricate facial expressions and head movements significantly constrain the authenticity and naturalness of synthesized videos.
omit intermediate representation and directly utilize single diffusion model to integrate audio cues with facial dynamics. These methods demonstrate enhanced potential, exhibiting superior naturalness and consistent portrait animation capability. However, they are constrained to support only head movement. To achieve audio- driven body animation, CyberHost 7 proposes a one-stage audio-driven talking body generation framework equipped with Region Attention Module and Human- Prior-Guide Conditions to address common synthesis degradations half-body animation. EMO2 9 introduces a two-stage framework, first generating hand movements and subsequently using them control signals in the second stage to enable holistic facial expressions and upper body motions. OmniHuman (8 employs a mixed data training strategy with multimodal motion conditioning to overcome the scarcity of high-quality data. Echomimie V2 [10] proposes an Audio-Pose Dynamic Harmonization strategy, requiring an additional hand pose sequence input alongside audio. However, these audio-driven human animations can only animate single person and cannot achieve multi-stream audio-driven image animation.
Recently, end-to-end audio-to-video synthesis methods 1I
24
25
5
2.2 Video Diffusion Model
The success of text-to-image diffusion models and their downstream applications 26 27 28 29 has
sparked considerable interest in exploring their potential for video generation. Video diffusion models be roughly divided into two categories: text-to-video models and image-to-video models. Early typically leverage the U-Net architecture for video generation, text-to-image tasks into 3D generate continuous 33 [4] have adopted a DiT (Diffusion-in- (Diffusion-in--Taasfooneee architecture 34], significantly advancing video generation technology. These DiT-based methods replace the U-Net with a Transformer, incorporating a 3D VAE as the encoder and decoder. By expanding the training dataset, DiT networks learn motion priors for various objects and scenes. Video diffusion models demonstrate substantial potential in tackling intricate video generation tasks and provide strong visual backbone for various downstream tasks [B5] 36 37
video diffusion models (30 31 12
attempting extend the 2D U-Net pretrained
video frames. Recent works 332
38 39). Due its excellent performance in human generation, DiT-based image-to-vides diffusion model is adopted as the backbone of our method to fully leverage its human generative prior.
3 Method
The overall architecture of the proposed method is illustrated in Fig. showcasing audio-driven multi-person conversational video generation framework. Section 3.1 we first briefly describe the network architecture of the video foundational model. Then, in Section 3.2 introduce the

---

integration of audio conditions via an audio cross attssntttnnionn mechanism for single-person animation. Subsequently, in Section 3.3 we present our investigation into multi-stream audio injection and introduce the proposed Ro- method for audio and person binding. In Section 3.4 we explain our training strategy. Finally, we describe our method for long video generation in Section 3.5
3.1 Preliminaries
In this study, we adopt a DiT-based video diffusion model as our foundational model, which is built upon the DiT architecture and incorporates 3D Variational Autoencoder (VAE). This design achieves compression in both spatial and temporal dimensions. A textual encoder is utilized to generate the text-conditioned imput, denoted as Ctext- Additionally, the extracted global context from the CLIP image encoder 40 is injected into the DiT model along with Creat via decoupled cross-attention.
3.2 Audio-Driven Single Person Animation
Our foundational model is image-to-vides diffusion model capable of animating reference image to generate a video. However, it does not natively support audio as an input. To incorporate an additional audio condition, add layers consisting of layer normalization and audio cross- attention mechanism after the text cross-attention in each DiT block.
Audio Embedding Extraction To extract acoustic audio embeddings, employ Wev [4], a widely utilized audio feature extractor. In audio-driven human animation, since current motion influenced by both preceding and succeeding audio frames, we follow [[] and concatenate audio embeddings proximal to the current frames, described as follows:
0₂ Ca [1(] ",,, 0 [$]
where & denotes the context length.
In the audio cross-attention layer, queries derived from video latents, while keys and values originate from audio embeddings. These elements execute frame-by-frame attention calculations. Due to the temporal compression of the 3D VAE, the frame length of video latents is shorter than that of audio embeddings, complicating direct calculations between them. To address this, we propose an audio adapter for audio compression. Specifically, suppose the input audio contains I frames. We first divide the audio embedding into the initial frame and the subsequent frames a:2:1] along the temporal dimension. Next, downsample "[2:[ get (ou))(a2-)). and then encode an with Down(ap.) ((p:4) separately through several MIP layers. After concatenating, encode the concatenated features to obtain the compressed audio condition ca- This process is represented as:
Ca
3.3 Audio-Driven Multi-Perss Animation
Existing methods fail address the problem of multi-human generation driven by multi-audio streams. In this paper, introduce novel task: audio-driven generation. To tackle this challenge, we propose a new framework, MultiTalk, specifically designed to handle multi-stream audio injection and rectify incorrect audio and person binding. The overall architecture of MultiTalk is depicted Fig We first investigate several schemes for multi-stream audio injection. Then, to accurately identify each person's motion region in generated videos, propose an adaptive person localization method. Finally, we introduce the proposed L-RoPE method to effectively bind audio and persons.
conversational video
Multi-stream Audio Injection Schemes. Multi-person conversational video generation, unlike single audio-driven video generation, requires the model to accommodate multi-stream audio inputs. To find an effective method for audio injection, we explore four distinct injection schemes, as illustrated in Fig.
Our first attempt involved directly concatenating the multi-stream audio embeddings 2,1 and 2,2. then calculating the audio cross-attention results with video latent 2t, as shown in Fig. Aa. Another strategy to calculate the multi-stream audio embeddings 2a1 and Fa2 separately with 2;, and then followed by adding operation to calculate these two components, seen in Fig 3b). However,

---

Ast
702
2
Paz
Aden
Peright
Audio Cress Attention
Audio Cross Attuntion
@
Audio Gress Attention
Audio Gross Attention
Audio Cross Attention
Audio Cross Attention
o)
b)
c)
Figure 3: Investigation on different injection strategies for multi-stream audio condition.
these attempts failed to bind the multi-stream audio with its corresponding video latent region. The network cannot learn to bind audio to different persons through training directly. Given that the individuals in the generated video are typically positioned on the left and right sides, we attempted to simplify binding by splitting the video latents into left and right segments, as demonstrated in Fig. (b). Each video latent segment computes attention results with the corresponding audio embedding separately, and the two attention results are concatenated as the final output. Although this attempt successfully binds multi-streat audio to different persons, its generalization capacity is limited. Specifically, it is only effective for videos with minimal movement range. When a person exhibits extensive motion, directly applying this simple operation results in audio binding failures. To address these shortcomings, we propose an adaptive method for multi-stream audio injection, named L-RoPE, as illustrated in Fig. 3d).
Adaptive Person Localization. Before utilizing L.-RPE, the model must adaptively track the localization of each individual. Given a reference image T contains two persons, we first find the subject localization within 7, resulting in the set M {M Mp2- Ms). Here, Mp1 and M,2 represent the mask regions for each person, and Mr denotes the mask covering the background the reference image. Collectively, they satisfy the relation / = MA1 U Ma2 U Mr. The self-attention map reflects the similarity of generated video latents across different frames. In the I2V model, the first frame of the video also serves as the reference image, enabling the creation of pnterence-imager to-video attention map Ar2. E R hww thw as depicted in Fig. 4 Here, f denotes the frame length in latent space, while h and w represent the height and width, respectively. Since the reference image contains multiple subjects within M, we calculate the average similarity of each latent in 2, with the subjects in the reference image, yielding S E Rfwxx3 In this matrix, Sii,j represents the similarity between the i-th token in the video latents and the j-th subject in M. By leveraging the similarity captured in the self-attention map,
adaptively locate each person in the video.
L-RoPE for Audio and Person Binding. Rotary Position Embedding (RoPE) 42 is a relative
positional encoding technique that effectively captures inter-token relationships in large language models (L..Ms). Known for its proficiency in modeling long sequences, RoPE has also been employed in video diffusion models, such VideoX [32] Hunyuan Video [3), and Wan (14) among others, to facilitate multi-resolution, multi-aspect ratio, and variable duration video generation. It utilized to generate position-awart query and key embeddings for time. height, and width within the video latents during the self-attention layer of the DiT block. In this paper, we introduce the Label Rotary Position Embedding (L-RoPE) method, aimed at binding multi-stream audio to multiple persons within the audio cross- atteenttttn layers of the DiT block.
Specifically, take the query g as an example. is a sequence of N vectors ( We compute an angle 0; for each vector qi using its label E R. and rotate qi with 0 obtain q::
8, = 1 Abase
(3)
i I. T.Ro T.RoPF(q..) = 4.e
where Osase a pre-defined base angle.
In the audio cross-attention mechanism, queries are derived from the video latent 21, whereas keys and values originate from the multi-stream audio embeddings 2a1 and 2a2. Appropriately assigning
5

---

Video Token
Video Taken
Papaed Parame Backrrand
Framel; Prame2
Nesso Purson
Frome
Video Token
Fromez
Fersen Bookground
Audio Token
Pareet Ieraatt
Froiaz
a) Self-attention map
Persent Persen Bockyr
Ferune Peroel
Frome3
c) Audio cross-attention map in DIT
Reference
ottection map for human
Audio affection map
afteattinn
Label assign for video latent
Figure 4: Analysis for different components in the DIT. a) We utilize the reference-image-to-iiee self-attention map in DiT for person localization. b) We assign different labels the multiple subjects in the video. c) Assigning a close label for video and audio can activate a specific region the audio cots-attention map.
With Mix- Training
Without Mix- Training
Prompt: The womon holds a red cup of coffee, takes a sip.
Figure 5: Instroction-folllwwing capability comparison between different training strategies.
labels to video and multi-stream audio is crucial. As depicted in Fig. 4 video latents encompass regions corresponding to multiple persons and the background. We adopt specific strategy for label assignment. For person regions, due to varying sensitivity driven by audio in different parts of the body, we first assign a numerical range for each person, (a, b). Then, we determine the category C E fR of each vector in through orgmaar,( (S [i,.)) Finally, taking the first person as an example, the label for personl can be calculated through the normalization function, Norm(S1 [t. [
(b - a) + a. This method is applied for each person in the same manner, but using different label ranges. Specifically, we define the visual label range as {0- - 4) for the first person and (20- 24) for the second person. Conversely, for the background and dual audio, they directly utilize static value as their label. The background should be associated with audio, hence we assign it the label 12. For multi- embedding, as shown in Fig. B1, we first concatenate the multi-stream audio embeddings and subsequently assign different labels Gal and 6a2 them. bind the multi-stream audio with the two persons respectively, we set as 2 and 6,2 as 22.
a. b) mar(S))) mii.))nnn(...
$i, "..z-min(S.)
3.4 Training Strategy
Two-stage training. The training stages and associated data sources are essential for achieving effective multi-person animation. We divide the training process into two stages, progressively enhancing the model's capabilities in audio and lip synchronization. The first stage primarily focuses on developing the model's ability animate single person. Subsequently, in the second stage, we employ training data that contains dual-stream audio to facilitate multi-human animation.

---

Partial Parameter Training, In our method, only the network parameters in the audio cross cross-attention and audio adapter are updated. while all other network parameters are frozen during training. We also compare this strategy with full parameter training. Our findings indicate that network training parameters are crucial; when the compute resources and data are limited, fully parameterized training can lead to not only the degradation in the model's instruction-folloinng ability, especially for motion and interaction, but also cause hand and object distortion. Conversely, training only the audio cross-attention does not result in this issue and the instruction-following ability of the base model can be well preserved.
Multi-task training. During training, we adopt multi-tast hybrid paradigm, dividing model training into multiple tasks, including audio image to video (AI2V) training and image video (I2V) training. Different tasks utilize distinct training data while sharing the same network parameters. For A12V tasks, both the reference image and audio are used as conditions. In the 12V task, the audio condition is removed by zeroing the audio embedding. Additionally, the training data used for the 12V task is unique, comprising mainly of multi-event videos with interactions among human, object, and scene, which is crucial for the alignment between the motion description in the prompt and the generated video.
Multi-task training substantially impacts the results, as shown in Fig 5 Utilizing only talking head and talking body data for AI2V training diminishes the network's 's instruction-following capability. Conversely, incorporating I2V training allows the model to retain its cnrruction-follwiing ability.
3.5 Long Video Generation
Although the model can generate video lengths of up to a few frames, this is still insufficient for real-world applications. To address this issue, we introduce an autoregressive- based method to facilitate long video inference. Specifically, within the I2V model, the first frame of the video is typically used as the condition for inference. contrast, we incorporate the last 5 frames of the previously generated video additional conditions for inference. Following 3D VAE compression, these conditional frames are reduced to 2 frames of latent noise. We pad zeros to the subsequent frames and concatenate them with latent noise and video mask. These are then input into DiT for inference, enabling longer video generation.
4 Experiments
4.1
Settings
Datasets. We collect video dataset of about 2K hours for the first stage training, which covers the face or body of single talking person. We also collect about 200K video clips that contain multiple events and humaa-objecc/envviooneen interactions. The average clip duration is about 10 seconds. For the second stage training, we collect 100 hours of videos consisting of conversations between two persons. For evaluation, we employ three distinct types of testing datasets: the talking head dataset, the talking body dataset, and the dual-human talking body dataset with interactive scenarios. For the talking head dataset, we employ two publicly available datasets, HDTF [43], and CelebV-HQ [4]] for evaluation purposes. For the talking body dataset, we utilize the EMTD [T]] dataset. Since we are the first to propose a dual-human talking body task, no public dataset is available. We collect a dataset containing 40 videos (referred to as MTHM) sourced from the internet.
Evaluation Metries. We utilize the commonly used metrics evaluate the methods. Frechet Inception Distance (FID) [55 and Fréchet Video Distance (FVD) [46] are used to assess the quality of the generated data. Expression-FID (E-FID) is used to evaluate the expressiveness of the facial in the generated video. Sync-C [47] and Sync-D) 47 are utilized to measure the synchronization between audio and lip movements.
Implementation Details. We adopted Wan2. 1an2.1-I2V--1 14B as the foundational video diffusion model for our experiments. The model is trained using constant learning rate of 2e 5, incorporating warm-up strategy, and optimized using the Adam W optimizer. During training, we only fine-tuned the audio cross-attention layer and adapter while keeping other layers frozen. The proposed method was trained using 64 NVIDIA H800-80G GPUs. In stage of the training process, the batch size was set to 64, whereas in stage 2, the batch size was adjusted to 32.
7

---

Table I: Quantitative comparison with other competing methods talking head generation, including HDTF and He datasets.
HDTF
Ce
Methods
Sync-C1
Sync-DI
E-PTD
FIDL
FVD.
Sync-CT
Sync-DI
E-PDD)]
F[D]
FVIJ
AniPortrait 24
3.09
10.94
1.32
32.83
112.21
2.09
11.29
1.66
37.17
250.24
VExpress 21
5.79
8.37
8.92
60.49
200.60
4.30
8.98
10.01
67.34
345.87
Echomimic f
5.36
8.99
1.27
60.82
240.07
4.16
9.55
2.87
63.72
318.08
Hallo3
6.55
8.49
1.12
33,98
153.31
5.57
8.58
1.51
40.81
212.91
Sonic 25
8.35
6,43
1.22
29.53
89.34
6.68
7.31
1.85
39.89
224.48
Fantusy Talking
[1]
3.61
10.78
1.36
32.64
103.01
3.14
10.43
1.77
37.54
218.43
MultiTalk-ssnn single (Ours)
8.54
6.69
1,00
24.01
95.99
7.07
7.13
141
32.31
219.19
Talk multiple (Ours)
8.53
6.81
1.24
27.27
124.06
7.33
7.18
1.48
34.08
184.86
Table 2: Quantitative comparison with other competing methods on talking body generation, including
EMTD dataset.
Methods
Sync-CT
Syne-DI
E-FID
FID_
FVDJ.
Echomimic v2 [10
6.31
8.41
1.91
35.99
163.66
Fantasy Talking [
3.32
11.41
1.98
37.68
284.29
MultiTalk-single (Ours)
8.18
7.28
1.67
32.05
221.86
Multi'alkkmultiile (Ours)
8.34
7.30
1.51
31.93
238.77
4.2 Comparisons with Competing Methods
Quantitative Evaluation. To verify the effectiveness of our method, we compare it with several state-of-the-art human animation methods. For talking head generation, compare with AniPortrait
24.. VExpress [21] EchoMimic 4. Hallo3 (3] Sonic (25 and Fantasy Talking [T. For talking
body comparison, compare with EchoMimie [10] and Fantasy Talking
Quantitative comparisons, including both talking head and talking body analyses, are presented in Table and Table respectively. Our method surpasses most other approaches across majority of metrics, exhibiting superior performance in lip synchronization and video quality, which underscores the effectiveness of our approach.
Qualitative Evaluation. To demonstrate the visual effectiveness of the proposed method, we compare and visualize the results alongside some competitive methods, shown in Fig. Upon providing instructions via a text prompt, only our method successfully responded to the instructions, highlighting its robust cntruuction-olllowiin capability. Additionally, our method generates fewer artifacts in the produced video, attesting to the quality of our approach.
Hallo3
Echominic
Senic
Tolking
Ours
+
clased the dask.
dpppreee
Prompt: a womon eitingg at a table with keadphanes. and then sie takes up the heodphoned
Figure 6: Qualitative comparison with other competing methods.
8

---

Table 3: Ablation study about the label range selection in L-RoPE on MTHM dataset.
Label for video
Label audio person
Variant
Sync-C1
Sync-DJ
E-FID]
FIDL
FVD
personl
person2
person2
0-2
2-4
1
3
7.47
7.22
3.22
52.87
506.49
b)
0-4
20-24
2
22
7.56
7.13
3.16
54.20
508.01
As the first method for multi-person generation, there is no directly comparable approach available. We compare method with the video concatenation technique, which involves generating the left and right video patches separately and subsequently concatenating them. The comparison results are presented in Fig. 7 Our method effectively handles interactive scenarios, avoiding inconsistencies between the left and right segments of the video. Besides, we also visualize the self-attention map for the specific person, highlighted in the red box. Our method can adaptively identify the localization of the person, thereby benefiting the audio binding.
4.3 Analyses
Multi-stream Single-stream. Our initial model for multi-stream audio training is derived from a single human animation model. To investigate whether multi-stream audio training would lead to performance degradation, compared the performance of the single human animation model with multiple human animation models on both the talking head and talking body datasets. The results, presented in Table [ Table show that our multiple human animation models achieve performance comparable to that the single human animation models, indicating that multi-strean audio training does not result in model degradation.
Video
Concat
Self Attention
Map Visuolization
Prompt: An elderly person sitting a wheelchair, Baside him, on elderly women sits at toble, The woman, holding glass of wine, stands up. walks ever, and gives the mon a heartfelt hug.
Figure 7: Qualitative comparison with video concat method in multi-human animation.
To validate the effectiveness of L-RoPE within MultiTalk, conduct an ablation study focusing on label range selection. The evaluation dataset is the collected conversa- tion data, MTHM. The experimental results are presented in Table These results demonstrate that different label choices for various persons yield comparable metrics, indicating that L--RoE is not sensitive label range variations.
Label Selection for L-RoPE
Table 4: Ablation study for different audio inject strategies (Corresponding to Fig. 3
Syne-Cr
Syne-D.
3.49
10.73
3.07
11.26
7.09
8.00
d (Ours)
7.56
7.13
Different Audio Injection Strategies We conducted an additional ablation study to investigate the impact of different audio injection strategies. The results are summarized in Table 4 with each row corresponding to an audio injection strategy as illustrated Fig. 3 Strategies (a) and (b) fail to bind multi-stream audio to the corresponding video latent regions. Strategy (c) employs hard mask-based audio binding approach, which capable of associating multi-stream audio with different persons; however, its effectiveness is limited to videos with minimal motion. When person exhibits extensive

---

movement, this strategy also results in failure cases. In contrast, our proposed L-RoPE method (d) achieves the best results across all tested scenarios, demonstrating the superiority of our approach.
Table 5: Ablation study for different training strategies.
Cross-attention Training
Full Parameter Training
MPST
59.5
40.5
Different Training Strategies To quantitatively evaluate the impact of different training strategies on the model's mnstruction-followiinn ability and hand/object distortion, we conducted an additional ablation study. Specifically, utilized reward model [48] to directly compare the cross- coossattenttion training strategy and full parameter training. using the Multi-dimensional Preference Score (MPS) as evaluation metric. The results, shown in Table demonstrate that training only the cross-attention layers leads to a higher MPS score. This provides clear quantitative evidence that optimizing only the cross-attention layers leads to better performance compared full-parameter training, particularly when computational resources and data are limited.
Conclusion
This paper introduces a novel task: audio-driven multi-person conversational video generation, and presents a new framework, MultiTalk, to accomplish this task. Multi-stream audio conditions are effectively injected using the proposed L-PoRE method, ensuring accurate audio and person binding. Furthermore, our findings demonstrate that partial parameter training and multi-task training are essential for maintaining the cetrruction-ollovviin ability of the base model, equipping our model with powerful instruction-followe
capability.
Limitation. We observe that our method performs better using real audio than using synthesized audio in terms of facial expression. The reason might be that our model is trained exclusively on real audio, which typically contains rich emotional cues and natural prosody. As result, the generated videos exhibit more expressive and realistic facial behaviors when driven by real audio. In contrast, most current TTS generated audio lacks emotional variation and nuanced expressiveness, leading to video outputs that appear less vivid and natural We will explore ways to mitigate the gap between real and synthesized audio for animation future work.
This paper introduces an effective approach for audio-driven multi-person conver- sational video generation to the community. However, this technology also raises ethical concerns. Beyond the risk of generating fake videos of celebrities, there are broader implications, including the potential for misuse in creating deepfakes for misinformation, defamation, fraud, harassment. Such synthetic videos could be used to impersonate individuals, manipulate public opinion, or violate privacy. These risks are not unique our approach, but are common across the broader field of human animation and generative models.
Societal Impacts
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Grant No. 62372480), in part by the Guangdong Basic and Applied Basic Research Foundation (No. 0023A1515012839) in part by 2025 Tencent AI Lab Rhino Bird Focused Research Program, and in part by HKUST-M MetttX Joint Lab Fund (No. METAX24EG01-D),
10

---

References
[1] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In ECCV, pages 244-260. Springer, 2024.
[2] Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu, Hallo: Hierarchical audio- driven visual synthesis for portrait image animation, arXiv preprint arXiv:2406.. 08800, 2024.
[3] Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. arXiv preprinl arXiv:2412.0077 2024.
[4] Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. In AAAI, volume 39, pages 2403-2410, 2025.
[5] Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsyne: Audio conditioned latent diffusion models for lip sync. arXiv preprint arXiv:2412.09262, 2024.
[6] Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprini NoKiv:2409.026334 2024.
[7] Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang. Zerong Zheng, and Yanbo Zheng. Cyberhost: A one-stage diffusion framework for audio-driven talking body generation. In ICLR.
[8] Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang, Omnihuman-I: 1: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint M:Xiv:2502.0101 2025.
[9] Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation. arXiv preprint arXiv:2501. 10687, 2025.
[10] Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimiev2: Towards striking, simplified, and semi-body human animation. arXiv preprint arXiv:2411.10061 2024.
[1I] Mengchao Wang, Qiang Wang. Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.0484. 2025.
[12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
[13] Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. arXiv preprinl arXiv:2501.08316, 2025.
[14] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wa Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng et Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 2025.
[15] Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang. Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. In CVPR, pages 1505-1515, 2023.
[16] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized dudio-driven single image talking face animation. In CVPR, pages 8652-8661, 2023.
11

---

[17] Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In ACM SIGGRAPH Asia, pages 1-9, 2022.
[18] Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement of pose and expression for general video portrait editing. In CVPR, pages 427-436, 2023.
[19] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang. Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV, pages 851101. Springer, 2022.
[20] Yuan Gong, Yong Zhang, Xiaodong Cun, Fei Yin, Yanbo Fan, Xuan Wang, Baoyuan Wu, and Yujiu Yang. Toontalker: Cross-domain face reenactment. In ICCV, pages 7690-7700, 2023.
[21] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.0251/ 2024.
[22] Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In CVPR, pages 7346-7355, 2018.
[23] Linsen Song, Wayne Wu, Chaoyou Fu, Chen Change Loy, and Ran He. Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis. JEEE 1247 1261, 2022.
[24] Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photore- alistic portrait animation. arXiv preprint arXiv:2403. 17694, 2024.
[25] Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonie: Shifting focus to global audio perception in portrait animation. arXiv preprint arXiv:2411.
2024.
[26] Robin Rombach, Andreas Blatmaann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[27] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[28] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driver generation. In CVPR, pages 22500-22510 22510, 2023.
[29] Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Occlusion-friendd personalized multi-concept generation in diffusion models. In ECCV, pages 253-270. Springer, 2024.
[30] Haoxin Chen, Menghan Xia, Yingging He, Yong Zhang. Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19511 2023.
[31] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[32] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint MxXi:22408.06777 2024.
[33] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. 2024.
12

---

[34] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195-4205, 2023.
[35] Sijie Zhao, Wenbo Hu, Xiaodong Cun, Yong Zhang, Xiaoyu Li, Zhe Kong, Xiangjun Gao, Muyao Niu, and Ying Shan. Stereocrafter: Diffusion- based generation of long and high-fidelity stereoscopic 3d from monocular videos. arXiv preprint 2XXv:2449.00744/ 2024.
[36] Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, and Wenhan Luo. Stylemas- ter: Stylize your video with artistic generation and translation. arXiv preprint arXiv:2412.07744, 2024.
[37] Jingyun Xue, Hongfa Wang. Qi Tian, Yue Ma, Andong Wang, Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kaihao Zhang, Heung-Yeung Shum, et al. Towards multiple character image animation through enhancing implicit decoupling. In ICLR.
[38] Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmaacher-Shlizzrnaa Aleksander Holyn- ski, and Steven M Seitz. Generative inbetweening: Adapting image-to-video models for keyframe interpolation. arXiv preprint arXiv:2408. 15239, 2024.
[39] Zhe Kong, Le Li, Yong Zhang, Feng Gao, Shaoshu Yang. Tao Wang, Kaihao Zhang, Zhuoliang Kang, Xiaoming Wei, Guanying Chen, et al. Dam-vsr: Disentanglement of appearance and motion for video super-resolution. In SIGGRAPH, pages 1-11, 2025.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., pages 7748-8763. PmL.R, 2021.
[41] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. 33:12449-12460, 2020.
[42] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
[43] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In CVPR, pages 3661-3670, 2021.
[44] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. 30, 2017.
[46] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[47] Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In ACCV, pages 251-263. Springer, 2017
[48] Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning dimeniiaiannii human preference for text-to-image generation. In CVPR, pages 8018-8027, 2024.
[49] Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. In CVPR, pages 8428-8437, 2025.
[50] Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- 1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint trXiv:2407.0237// 2024.
[51] Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. NeurIPS, 37:48955-48970. 2024.
[52] Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In ACCV, pages 251-263. Springer, 2016
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our our contributions:() propose a novel task (2) propose a framework as solution (3)training strategies, accurately reported in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss the limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We present the implementation details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
[No]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /ttppp.cc/pbllc/guudeeee http://oddbbbbiiiooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Details provided in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We use the commonly used metrics that do not involve error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resource information are presented in the experiment part.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research follows the rules.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Those works are properly cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ttp::///eerrrp.ccccooofrrrree/p for what should or should not be described.
20

---

Appendix
A
Task Definition
Audio-driven multi-pene conversational video generation is defined as follows: Given a reference image containing multiple persons and corresponding audio streams (with one-to-one correspon- dence between each person and their audio), the goal is to synthesize a video sequence in which all persons appear together in the same frame, and each person's lip movements are temporally synchronized with their respective audio input. Unlike previous single-person talking face generation tasks, this new task requires the joint modeling of perti- interactions, spatial consistency, and audio-vinae visual synchronization within unified, end-to-end generative framework, requiring only single diffusion process.
In contrast to approaches that generate videos for each person independently and subsequently composite them, this new task demands integrated modeling multiple individuals, offering several key advantages:
Higher computational efficiency: Only single inference process is required, substantially reducing computational costs.
Global consistency: The unified framework enables better control over the overall coherence of the generated content, such as coordinated camera movements, lighting, and scene dynamics.
Enhanced interaction modeling: This approach is inherently suitable for capturing interactions among individuals, enabling natural and contextually appropriate reactions (e.g.. when one person speaking, others display attentive responsive behaviors).
B Dataset and Implementation Details
B.1 Dataset Details
In this paper, we utilize three distinet testing datasets: the talking head dataset, the talking body dataset, and the dual-human talking body dataset with interactive scenarios. For the talking head and talking body datasets, we employ conventional evaluation techniques for comparison with other methods. However, for the dual-human talking body dataset, where each reference image contains two persons, we evaluate Sync-C, Sync-D, and E-FID by splitting the video into two segments: the left part and the right part. Each segment contains only one person and their corresponding audio. We then average the scores these two segments to derive the final result for this dataset. Fig showcases some examples of our dual-human dataset.
Figure 8: Some examples of our MTHM dataset.
All data used in our experiments were collected from publicly available sources on the internet. Our data collection process follows the best practices established by previous works 49 50, 51) ensuring that our methods are consistent with the standards in the community. All data sources are under CC BY 4.0 International license. Our dataset comprises approximately 2,700 unique subjects, with approximately 71 % male and 29% female. The distributions age and race are presented in Table and 7 respectively.
21

---

Table 6: The distributions of the age of the training dataset.
Age Percentage
0-9
10-19
20-29
30-39
40-49
50-59
60-69
70+
0%
0.02%
20.77%
57.11%
19.03%
2.95%
0.12%
0%
Table 7: The distributions of race of the training dataset.
Race
white
black
middle eastern
asian
latino hispanic
indian
Percentage
61.55%
6.22%
4.89%
21.76%
4.77%
0.81%
For data preprocessing, we closely follow the procedures described 49 and further filter out samples exhibiting large facial movements unsynchronized speech and mouth motion 52 This ensures the high quality and reliability of our dataset.
B.2 Sample Details
In all the experiments and evaluations conducted within this paper, we utilize 40 sampling steps. To filter out undesired variations in diffusion models, we employ the following negative prompt during sampling: "bright tones, overexposed, static, blurred details. subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background. three legs, many people in the background, walking backwards." Additionally, we employ Qwen-VI VL. for reference image captioning.
B.3 Inference Time
Although our method introduces an additional audio condition, the computational time required to pass through the DiT backbone remains the same Wan2.1, and it requires 40 steps for inference. Furthermore, all acceleration strategies available for Wan2. 1onn2.1-stee 1ucch as TeaCache and model distillation are also applicable to our approach. For example, when employing a distilled model (such lightx2v), the total number of inference steps is reduced to steps per video.
C Analyses
C.1 Full Parameter Training vs Cross-attention Training
Full Parometer
Training
Cross-attention Training
Prompt: firmly grasped the and lifted off the table..
Figure 9: Comparison between full parameter training and cross-attention training.
We compare full parameter training with fine-tum tne-tuning only the audio cross-attes aotss-attention layer. Our findings indicate that network training parameters are crucial. When compute resources and data are limited, fully parameterized training can lead not only to degradation the model's instruction-followinn ability, especially for motion and interaction, but also to hand and object distortion. Conversely, training only the audio cross-attention does result these issues, and the instruction-followinn ability of the base model is well preserved. The comparison results between full parameter training
22

---

and cross-attention training are shown in Fig. 2 It can be seen that full parameter training degrades the model's cestructiionfollowinn ability and causes hand distortion.
C.2 Long Video Generation
Utilizing the autoregressive-baset method facilitates the long video generation of our method. The experimental results for long video generation are shown in Fig 10 This example shows generated result containing 305 frames.
(1
Sala
Figure 10: The generation result of long videos.
C.3
Emotional Expressions
We use the same reference image and specify emotional (e.g., angry, sad, happy) via the text prompt. Our model successfully generates videos with the corresponding emotional expressions, as shown in Fig.
Angry
Happy
Sad
Figure 11: Generation results for videos with different emotions using the same reference image.
C.4 Generalization to More Speakers
In the two-speaker setting, we assign distinet, non-overlapping ranges of video and audio labels to each individual (c.g., video labels 0-4 for person and 20-24 for person 2; audio labels 2 for
23

---

person 1 and 22 for person 2). For scenarios involving more individuals, we conducted additional experiments verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40-44 and audio label 42 for third person). This flexible approach enables the model accommodate greater number of speakers simply by expanding the label ranges for both video and audio streams, which demonstrates the generalizability of our L-RoPE framework. The visualization results for scemarios involving three speakers are shown in Fig. [2] Importantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person's 's lip movements accurately aligned with their respective audio streams.
Figure 12: The generation results generalize to scenarios with three persons.
24


Reviews:
[{"review_id": "QK9DhnvXWE", "reviewer": "Reviewer_reUP", "summary": "This paper proposes MultiTalk, a framework for generating multi-person conversational videos driven by multi-stream audio and prompts. The authors tackle the problem of binding each audio stream to the correct person using a novel Label Rotary Position Embedding. They also design partial parameter training and multi-task learning to preserve instruction-following ability. Experiments on talking head, body, and multi-person datasets show improved synchronization and video quality over prior single-person methods.", "questions": "- The instruction-following claims are promising, but current prompts in your experiments are relatively simple. How does the model perform when facing complex, compositional prompts (e.g., involving emotions, multiple actions, overlapping speech, or indirect commands)? Can you provide such examples or failure cases?\n\n- The current experiments focus on two-person scenarios. Does the L-RoPE approach generalize to cases with 3 or more speakers? How does the model handle varying numbers of persons at inference time, especially when the number of audio streams and persons may differ from training?\n\n- The paper mentions a performance gap between real and synthetic audio. Could you elaborate on how significant this gap is, and whether your model could be made robust to TTS-generated speech, which is common in practical applications?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Proposes the new task of multi-person conversational video generation, extending beyond prior single-person animation works.\n\n- The qualitative results shown are impressive. The generated videos demonstrate highly realistic lip-sync, body motion, and person-to-audio alignment, making the visual results very impressive.\n\n- The introduction of the Label Rotary Position Embedding to solve the key challenge of binding each person to their correct audio stream is both novel and effective.\n\n-  The paper provides thorough experimental validation, including detailed ablations on multi-stream audio injection methods and training strategies. The paper also carefully studies how partial parameter tuning and multi-task learning contribute to maintaining performance with limited compute and data.\n\n- The method is validated across multiple datasets and scenarios (talking head, talking body, multi-person), showing consistently better synchronization and video quality than prior methods.\n\nWeaknesses:\n- Although the adaptive person localization is mentioned, it is somewhat unclear how robust this method is in highly occluded or overlapping person scenarios.\n\n- The paper claims instruction-following ability, but most prompts in experiments are relatively simple (“person holds cup,” “person hugs”). It is unclear how well the model handles more abstract or compositional instructions like multi-step narratives, emotional expressions, or simultaneous actions.", "was_revised": true, "final_justification": "The authors have satisfactorily addressed my pre-rebuttal concerns. They demonstrated generalization of the L-RoPE approach to more than two speakers and to scenarios with variable numbers of persons at inference time, with a clear labeling strategy and supporting experiments. They clarified the performance gap between real and TTS-generated audio and proposed concrete strategies to improve robustness.\n\nGiven the novelty of the multi-person conversational video generation task, the impressive qualitative results, and the thorough clarifications provided, I maintain my positive assessment and recommendation for acceptance.", "rebuttal": "We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task, impressive qualitative results, the novelty and effectiveness of our method, thorough experimental validation, and superior performance. Below, we address the specific concerns raised by the reviewers individually.\n\n\n### Q1: Robustness under occlusion and overlap scenarios.\n\nThank you for your question regarding the robustness of our adaptive person localization method in highly occluded or overlapping scenarios.\nWhen individuals’ faces are visible—even if their bodies are partially occluded—our method consistently associates the audio stream with the correct person. In scenarios where a person’s face is fully occluded and not visible in the frame, the corresponding audio does not drive lip motion or appearance for the occluded individual. Notably, if a person is temporarily occluded and subsequently reappears, our model is able to accurately re-associate the audio stream with the correct individual upon their return. The corresponding visualization results will be provided in the supplementary materials of the updated version.\n\n\n\n### Q2: Handling more complex instructions.\n\nThank you for your insightful comment regarding the instruction-following capabilities of our model.\nWe conducted additional evaluations to assess the robustness of our method on more complex and compositional prompts:\n\n\n**Multi-step narratives**: Since Wan2.1 can only generate 81 frames per chunk, it is challenging to complete multiple actions within a single chunk. Hence, we decompose multi-step narratives into several actions, generating each chunk with prompts specifying a few actions. Through leveraging the long video generation mentioned in the paper, our model is able to represent multi-step narratives effectively.\n\n**Emotional expressions**: We use the same reference image, and specify emotional (e.g., angry, sad, happy) via the text prompt. Our model successfully generates videos with the corresponding emotional expressions.\n\n**Simultaneous actions**: In multi-person scenarios, we assign different actions to each person via text prompts. The resulting videos demonstrate that both individuals can perform their respective actions simultaneously.\n\n**Overlapping speech**: By providing separate audio streams for each person, our model generates synchronized lip movements for both speakers without interference, thus achieving overlapping speech.\n\n**Indirect commands**: Our model currently struggles with indirect commands, likely due to the base video model’s limited ability to understand such instructions.\n\nDue to rebuttal format constraints, we are unable to include success and failure visualization results here. However, we will provide these examples in the updated version.\n\n### Q3: Generalization to more speakers and variable person numbers.\n\nThank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time.\nOur L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.\n\nImportantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.\n\nVisualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version.\n\n\n\n### Q4: Performance gap between real and synthetic audio.\n\nThank you for your valuable question regarding the performance gap between real and synthetic (TTS-generated) audio.\nThe observed gap can be primarily attributed to a domain discrepancy: our model is trained exclusively on real audio, which typically contains rich emotional cues and natural prosody. As a result, the generated videos exhibit more expressive and realistic facial behaviors when driven by real audio. In contrast, most current TTS-generated audio lacks emotional variation and nuanced expressiveness, leading to video outputs that appear less vivid and natural. A promising approach to address this issue is to augment the training data with TTS-generated audio, thereby enabling the model to better generalize to synthetic audio inputs. More detailed analysis and experimental results on TTS robustness will be provided in the updated version."}, {"review_id": "o0OYyb6trH", "reviewer": "Reviewer_cNEL", "summary": "The paper introduces a new task: audio-driven multi-person conversational video generation. It proposes a new framework for this task, incorporating a Label Rotary Position Embedding method (L-RoPE) to address the problem of inaccurate audio binding. Additionally, the paper investigates several schemes for multi-stream audio injection and explores a set of training strategies, including two-stage training, partial parameter training, and multi-task training.", "questions": "Please clarify the contents in weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces an interesting task: audio-driven multi-person conversational video generation. The task is also challenging due to multi-stream audio input, the audio binding problem, and person localization in video.\n2. The proposed framework, including audio-video cross-attention and audio-person binding, is reasonable, and the writing is easy to follow.\n3. The video results for multiple persons are promising.\n\nWeaknesses:\n1. The paper investigates several schemes for multi-stream audio injection. Showing the results of these alternative schemes could further strengthen the advantages of L-RoPE.\n2. The proposed task is interesting and inspiring. This may be a minor issue, but since the task focuses on multi-person scenarios, it would be beneficial to discuss how the framework could be extended to more than two persons and the challenges involved in scaling to additional persons.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task, the reasonableness of our method, the clarity of our writing, and the promising results. Below, we address the specific concerns raised by the reviewers individually.\n\n\n### Q1: Ablation study for multi-stream audio injection.\n\nThank you for the valuable comments.\nWe conducted an additional ablation study to investigate the impact of different audio injection strategies. The results are summarized in the table below, with each row corresponding to an audio injection strategy as illustrated in Figure 3. Strategies (a) and (b) fail to bind multi-stream audio to the corresponding video latent regions. Strategy (c) employs a hard mask-based audio binding approach, which is capable of associating multi-stream audio with different persons; however, its effectiveness is limited to videos with minimal motion. When a person exhibits extensive movement, this strategy also results in failure cases. In contrast, our proposed L-RoPE method (d) achieves the best results across all tested scenarios, demonstrating the superiority of our approach. We will provide these results in the supplementary materials of the updated version.\n\n\n|          | Sync-C↑ | Sync-D↓ |\n|:--------:|:-------:|:-------:|\n|     a    |   3.49  |  10.73  |\n|     b    |   3.07  |  11.26  |\n|     c    |   7.09  |   8.00  |\n| d (Ours) |   7.56  |   7.13  |\n\n### Q2: Generalize to more speakers.\n\nThank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time.\nOur L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.\n\nImportantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.\n\nVisualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version."}, {"review_id": "WIn7uTew0o", "reviewer": "Reviewer_M1mM", "summary": "The paper proposed a new task (audio-driven multi-person conversational video generation), a framework MultiTalk for this task employing L-RoPE introduced for multi-stream audio injection. The MultiTalk is evaluated in generating various type of video generation such as talking face, talking body, and multi-person conversation.", "questions": "The paper collects a new dataset for multi-person conversation, but the paper does not mention the dataset is proposed. Why authors do not propose the dataset?\n\nThe time to require to generate videos is not explored at all. Since it is not the main aim and not in the scope, it might be understandable. However, if the generation is computationally expensive and requires time (most probably and as expected), I suggest to mention this in the limitation at least and evaluate the model’s response time. Can you include elapsed time to generate videos?\n\nThe paper investigates different injection strategies for multi-stream audio. An ablation study can be included to see the impact of injection strategies, otherwise the investigation of injection strategies will be incomplete. Can you make such ablation study?\nWhy authors won’t share the collected dataset?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The writing is easy to follow and supported by equations, figures, details of implementation and evaluation well.\n\n2. The model has better scores than SOTA models in terms of the video quality, facial expressiveness, and lip synchronization.\n\n3. The model can also generate multi-person and interactive videos well.\n\nWeaknesses:\n\n1. The definition of the proposed new task (audio-driven multi-person conversational video generation) is brief. It seems incremental as existing talking face models (that are trained to generate talking face videos with a single face) can generate multi face by generating talking face videos independently then combine videos. So, the paper can include the formal definition of the new task in the coloration with existing tasks.\n\n2. The model supports multi-person conversation, but it supports dual-person only. This can be mentioned in the limitation.\n\n3. The proposed model cannot preserve the identity of the persons in the reference image. See the second singing video in the supplementary materials (the male character becomes someone else).\n\nChecklist related:\n\n1. Checklist 4. Although authors provide the details of the experiments, the collected dataset won't be shared, so it questions the reproducibility of the experiments.\n\n2. Checklist 10. The broader impact is not limited to the generating fake videos of celebrities. Please elaborate it. Moreover, I suggest putting the broader impact in the main manuscript instead of supplementary material because it is vital ethical concern of the talking face models.", "was_revised": true, "final_justification": "My concerns are addressed, so I decided to increase my score to Accept from Borderline Accept.", "rebuttal": "We appreciate the reviewers’ positive feedback, including recognition of our clear and detailed writing, superior generation results, and strong capabilities in multi-person and interactive video generation. Below, we address the specific concerns raised by the reviewers individually.\n\n\n\n\n\n### Q1: Definition of the new task and its distinction from single-person talking face task.\n\nThank you for your valuable feedback. We appreciate your suggestion to provide a more formal definition of our proposed task and to clarify its distinction from existing single-person talking face generation tasks.\n\n\nAudio-driven multi-person conversational video generation is defined as follows: Given a reference image containing multiple persons and corresponding audio streams (with a one-to-one correspondence between each person and their audio), the goal is to synthesize a video sequence in which all persons appear together in the same frame, and each person’s lip movements are temporally synchronized with their respective audio input. Unlike previous single-person talking face generation tasks, this new task requires the joint modeling of multi-person interactions, spatial consistency, and audio-visual synchronization within a unified, end-to-end generative framework, requiring only a single diffusion process. \n\nIn contrast to approaches that generate videos for each person independently and subsequently composite them, this new task demands integrated modeling of multiple individuals, offering several key advantages:\n\n1. Higher computational efficiency: Only a single inference process is required, substantially reducing computational costs.\n\n2. Global consistency: The unified framework enables better control over the overall coherence of the generated content, such as coordinated camera movements, lighting, and scene dynamics.\n\n3. Enhanced interaction modeling: This approach is inherently more suitable for capturing interactions among individuals, enabling natural and contextually appropriate reactions (e.g., when one person is speaking, others can display attentive or responsive behaviors).\n\nWe will revise the paper to include a formal definition of the new task and add a dedicated section to compare it with existing single-person talking face tasks. This will help clarify the novelty and technical challenges of our work.\n\n\n\n\n### Q2: Generalize to more speakers.\n\nThank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time.\nOur L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.\n\nImportantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.\n\nVisualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version.\n\n### Q3: Identity degradation in the generated videos.\n\nThank you for pointing out the issue regarding identity preservation in our generated videos.\nWe acknowledge that identity degradation may occur in generated videos, particularly when the video duration is excessively long or when there are substantial changes in facial motion angles. This limitation arises because the video foundation model we employ also suffers from the identity degradation problem. On one hand, as video foundation models continue to advance, the issue of identity degradation is expected to be mitigated. On the other hand, in future work, we plan to investigate explicit identity injection techniques to further enhance identity preservation.\n\nThank you again for your valuable feedback.\n\n\n\n### Q4: Dataset release and experimental reproducibility.\n\nThank you for your comments and concerns regarding the dataset and reproducibility.\nAll data used in our experiments were collected from publicly available sources on the internet. Our data collection process follows the best practices established by previous works [1]-[3].\nFor data preprocessing, we closely follow the procedures described in [1] and further filter out samples exhibiting large facial movements or unsynchronized speech and mouth motion [4], and do not have any unique data processing procedures.\n\nWe acknowledge the importance of reproducibility and are committed to supporting the community.\nWe will release the MTHM test dataset, along with the corresponding code and model weights, to the research community to encourage reproducibility and facilitate further research and development in audio-driven human animation.\n\n\n[1] Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content\n\n[2] Openvid-1m: A large-scale high-quality dataset for text-to-video generation\n\n[3] MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions\n\n[4] Out of time: automated lip sync in the wild\n\n### Q5: Broader impact of the paper.\n\nThank you for highlighting the importance of discussing the broader impact and ethical considerations of our work. This paper introduces an effective approach for audio-driven multi-person conversational video generation to the community. However, this technology also raises significant ethical concerns. Beyond the risk of generating fake videos of celebrities, there are broader implications, including the potential for misuse in creating deepfakes for misinformation, defamation, fraud, or harassment. Such synthetic videos could be used to impersonate individuals, manipulate public opinion, or violate privacy. These risks are not unique to our approach but are shared across the broader field of human animation and generative media.\n\nThank you for your suggestion. We will include the discussion of broader impact in the main manuscript in the updated version.\n\n\n\n\n\n### Q6: Inference time.\n\nThank you for your valuable suggestion regarding the evaluation of video generation time.\nAlthough our method introduces an additional audio condition, the computational time required to pass through the DiT backbone remains the same as in Wan2.1. On an NVIDIA A100 GPU, each inference step in our method takes approximately 35 seconds, resulting in a total generation time of about 1,430 seconds for a 40-step inference. Furthermore, all acceleration strategies available for Wan2.1—such as TeaCache and model distillation—are also applicable to our approach. For example, when employing a distilled model (such as lightx2v), the total inference time is reduced to approximately 170 seconds per video.\n\n### Q7: Audio injection results.\n\nThank you for the valuable comments.\nWe conducted an additional ablation study to investigate the impact of different audio injection strategies. The results are summarized in the table below, with each row corresponding to an audio injection strategy as illustrated in Figure 3. Strategies (a) and (b) fail to bind multi-stream audio to the corresponding video latent regions. Strategy (c) employs a hard mask-based audio binding approach, which is capable of associating multi-stream audio with different persons; however, its effectiveness is limited to videos with minimal motion. When a person exhibits extensive movement, this strategy also results in failure cases. In contrast, our proposed L-RoPE method (d) achieves the best results across all tested scenarios, demonstrating the superiority of our approach. We will provide these results in the supplementary materials of the updated version.\n\n\n\n|          | Sync-C↑ | Sync-D↓ |\n|:--------:|:-------:|:-------:|\n|     a    |   3.49  |  10.73  |\n|     b    |   3.07  |  11.26  |\n|     c    |   7.09  |   8.00  |\n| d (Ours) |   7.56  |   7.13  |"}, {"review_id": "cLjmFAmcV5", "reviewer": "Reviewer_34Jy", "summary": "The paper seeks to use multi-stream audio to animate multi-person from an image. It proposes combining Adaptive Person Localization and L-RoPE to achieve this goal. The model is trained on 100 hours of videos and achieves better results on benchmarks like HDTF, CelebV-HQ, and EMTD than the baselines.", "questions": "Since all cases have a maximum of two person, I wonder how many people this method can scale to at most, and can it handle the situation where the masks of multiple people overlap?", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* This task of driving multiple person has moved forward from the current audio-driven task.\n* The L-RoPE could differentiate to some extent between different people in the given image, enabling them to be driven by multiple audio streams.\n\n**Weaknesses**\n* Regarding the setting, given a reference image with multi-stream audio (only 2 streams in the whole paper) and text, then generating the video. It seems that the correspondence between different audio streams and characters in the image still needs to be manually specified. Considering this, when obtaining the corresponding mask area, it is possible to achieve the same effect directly in the audio cross attention using the mask, and the model trained on single-person speaking data can also directly achieve this. Do authors consider this?\n* For the Adaptive Person Localization, the premise for this to work stably is that there is no collapse in the generated video latents, which would result in a higher similarity. If there is a collapse, the similarity calculation may be affected, which might impact the final effect. Additionally, although the author claims that the Adaptive Person Localization with L-RoPE could handle situations with large movement range motion, neither the main paper nor the supplementary materials contain any related cases; most cases are left-right patterns. In that case, it can be achieved simply by using a mask on the audio cross-attention, as mentioned above.\n* For the training strategy (Partial Parameter Multi-task). The fundamental reason may be that the distribution of the training data is relatively biased, possibly because the training data mostly consists of portrait videos, which do not include hands and bodies, leading to the phenomenon described by the author. Additionally, since only part of the weights have been trained, will there be a conflict between text control and audio control?", "was_revised": true, "final_justification": "Thanks for the author's reply, which has addressed some of my concerns. Finally, I suggest that the author provide more examples of people moving and scenes with more than two persons, as these are sufficient to prove what the paper claims. Currently, due to the lack of these examples, I maintain my rating (3 Borderline Reject), but I am not opposed to accepting this paper.", "rebuttal": "We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task and the affirmation of the L-RoPE. Below, we address the specific concerns raised by the reviewers individually.\n\n### Q1: The difference between L-RoPE and mask-based audio injection strategy.\n\nThank you for your question regarding the correspondence between audio streams and persons, and the potential use of mask-based approaches.\nOur method does not require obtaining a mask for each frame of the generated video. Instead, we only need to establish the correspondence between each person with their audio stream in the reference image, which is a much simpler and more robust requirement.\n\nAs you mentioned, the mask-based method is the same as in Figure 3c of our paper. The mask-based method explicitly binds a specific mask region to an audio stream, requiring a predefined mask for each frame throughout the video. However, this approach faces significant limitations: if the generated person moves outside the predefined mask area, the correspondence between speech and mouth motion can become unsynchronized, leading to degraded performance.\n\nIn contrast, our proposed L-RoPE method leverages visual similarity to dynamically associate each human appearance with its corresponding audio input, without relying on fixed spatial masks for the whole video. This design enables strong generalization and robustness, especially when the generated characters undergo large motions or pose changes. As a result, L-RoPE maintains accurate audio-visual synchronization even in challenging scenarios, which is a key advantage over mask-based and single-person methods. We further validated the effectiveness of L-RoPE through additional experiments comparing these two strategies, as presented in the table, where the mask-based method corresponds to (c) and L-RoPE corresponds to (d). The proposed L-RoPE achieves superior results.\n\n\n|          | Sync-C↑ | Sync-D↓ |\n|:--------:|:-------:|:-------:|\n|     a    |   3.49  |  10.73  |\n|     b    |   3.07  |  11.26  |\n|     c    |   7.09  |   8.00  |\n| d (Ours) |   7.56  |   7.13  |\n\n### Q2: Model collapse and large motion case.\n\nThank you for your comments and suggestions regarding the stability of adaptive person localization.\nWe agree that the stability of similarity-based localization depends on the quality of the underlying video foundation model. With a less robust model, there is indeed a risk of latent collapse. However, in our work, we use Wan2.1, which is currently one of the most stable open-source video generation models. In our extensive experiments, we have not encountered latent collapse or related issues.\n\nRegarding large motion, Figure 7 in our paper already demonstrates cases with significant movement. Additionally, we have conducted further tests involving both camera motion and substantial person movement, and our method is able to handle these scenarios effectively. We will include more such examples in the final version of the paper to better illustrate the robustness of our approach.\n\n### Q3: Training strategy and conflict between different conditions.\n\nThank you for your comments regarding the training strategy and potential conflicts in multi-task control.\nWe agree that the distribution of the training data can affect the generation results. However, our training dataset does include a substantial number of samples featuring hands, bodies, and camera movement. Moreover, our findings are consistent with those reported in [1], which indicate that full parameter sharing in multi-task training can result in image quality degradation and distortions.\n\nRegarding the concern about potential conflicts between text control and audio control, we have conducted extensive testing and have not observed any conflicts. The model is able to follow text-based instructions effectively while maintaining accurate audio-driven lip synchronization. \n\n[1] OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation\n\n### Q4: Generalize to more speakers and overlap scenarios.\n\nThank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and overlap scenarios.\nOur L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.\n\nImportantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.\n\nFor overlap scenarios, when individuals’ faces are visible—even if their bodies are partially occluded—our method consistently associates the audio stream with the correct person. In scenarios where a person’s face is fully occluded and not visible in the frame, the corresponding audio does not drive lip motion or appearance for the occluded individual. Notably, if a person is temporarily occluded and subsequently reappears, our model is able to accurately re-associate the audio stream with the correct individual upon their return. \n\nVisualization results for scenarios involving more speakers and overlap will be provided in the supplementary materials of the updated version."}, {"review_id": "oyo9roDOKb", "reviewer": "Reviewer_s1Za", "summary": "This paper approaches the task of multi-person audio-to-human generation using video Diffusion Transfomers. It specifically aims to address the issues of having multiple people present in the output video, which previous methods do not handle well. This work proposes the use of Label Rotary Position Embedding (L-RoPE) which helps to bind audio streams to the correct tokens (and therefore regions) in the diffusion process.", "questions": "To be accepted, the following questions on data must be answered:\n\n- How is the data obtained? \n- Where is it sourced from?\n- Do all of these sources have a permissive license?\n- How is consent obtained from each of the subjects?\n- How many subjects are there?\n- What is the demographic distribution of the subjects (e.g. age, race, gender etc)? \n\nFurthermore, I have the following questions:\n\n- Do the authors want to claim improved performance in the single speaker setting?\n- If so, what are the differences beyond L-RoPE, how much does the newly collected dataset contribute to the performance, rather than methodological differences?\n- If not, and only multi-speaker is claimed, how well do the other methods perform when trained with the multi-person data collected in this method, and how does the proposed method perform without L-RoPE?\n\nIf the work can be shown to be better in either setting through fair evaluation, and the data concerns are addressed, the paper could be accepted.", "limitations": "More discussion is needed, see weaknesses and questions.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well motivated, it is true that there are difficulties around multi-speaker audio-to-human video generation. This is justified in the paper, and a novel component L-RoPE is proposed to address the issue. L-RoPE is described well and seems to make sense theoretically. The results shown, both quantitatively and qualitatively are very impressive.\n\n**Weaknesses**\n\nThere are major concerns with this paper related to the data. There is a severe lack of discussion on this. It is stated in the paper:\n\n\"We collect a video dataset of about 2K hours for the first stage training, which covers the\nface or body of a single talking person. We also collect about 200K video clips that contain multiple\nevents and human-object/environment interactions. The average clip duration is about 10 seconds.\nFor the second stage training, we collect 100 hours of videos consisting of conversations between two\npersons.\"\n\nThis is problematic on several fronts. From a reproducability perspective, there is not enough information to reproduce this dataset. Furthermore, the following are in the Neurips ethics guidelines:\n\n- Privacy: Datasets should minimize the exposure of any personally identifiable information, unless informed consent from those individuals is provided to do so.\n- Consent: Any paper that chooses to create a dataset with real data of real people should ask for the explicit consent of participants, or explain why they were unable to do so.\n- Copyright and Fair Use: While the norms of fair use and copyright in machine learning research are still evolving, authors must respect the terms of datasets that have defined licenses (e.g. CC 4.0, MIT, etc). \n- Representative evaluation practice:  When collecting new datasets or making decisions about which datasets to use, authors should assess and communicate the degree to which their datasets are representative of their intended population. Claims of diverse or universal representation should be substantiated by concrete evidence or examples. \n\nWithout further information, all of these are significant concerns.\n\nIn terms of evaluations. In the single speaker setting, it is not clear how this work will differ from previous DiT works such as FantasyTalking and Hallo3. It seems to me that the only difference is the L-RoPE module, which is designed for the multi-speaker setting. If the authors want to show that it improves the single speaker case, an ablation study is needed. This is because the data used is different from the baselines, so it is not a fair comparison. If this is not being claimed then Tables 1 & 2 should be removed, and in its place the exitsing models should be shown **not** to work in the multi-speaker case, when trained on multi-speaker data as the proposed method has been. In either case, the evidence is lacking in this work.\n\nIt is not clear what Table 3 is trying to show. It is claimed that it is \"To validate the effectiveness of L-RoPE within MultiTalk\" however, it does not seem to do this. An ablation to show this should be with L-RoPE vs without. The table appears to show the effects of the range parameter, which while interesting, does not support what is claimed. \n\nOverall, I am not convinced that the main improvements in this paper are not just due to better dataset collection.", "was_revised": true, "final_justification": "Thanks to the authors for their clarification. I no longer have any issues with the technical content of the paper, provided the extra experiments are included. However, I still have ethical concerns around the data, as the collection does violate YouTube's ToS. I will allow the ethics reviewers to make the final decision here.", "rebuttal": "We appreciate the reviewers’ positive feedback, including recognition of our well-motivated methodology, the novelty of L-RoPE, and our impressive quantitative and qualitative results. Below, we address the specific concerns raised by the reviewers individually.\n\n### Q1: More details of our dataset.\n\nThank you for raising these important questions regarding our dataset. We appreciate the opportunity to clarify the data-related aspects of our work.\nAll data used in our experiments were collected from publicly available sources on the internet. We only downloaded data from platforms that explicitly permit data download for research purposes, and we strictly avoided any sources that prohibit such use. Our data collection process follows the best practices established by previous works [1]-[3], ensuring that our methods are consistent with the standards in the community.\nAll data sources are under the CC BY 4.0 International license that allows for academic research and non-commercial use. We comply with the platform’s terms of service and applicable privacy regulations during the data collection and usage process. \nOur dataset comprises approximately 2,700 unique subjects, with approximately 71\\% male and 29\\% female. The distributions of age and race are presented in the following tables.\nWe have made efforts to ensure diversity in age, race, and gender to the best extent possible based on the available metadata and annotations. We acknowledge that perfect demographic balance is challenging due to the nature of publicly available data, but we have taken steps to maximize representativeness.\nWe strictly followed the NeurIPS ethics guidelines regarding privacy, consent, and copyright. All data is used strictly for research purposes and will not be commercialized. \n\n| Age        | 0-9 | 10-19 | 20-29  | 30-39  | 40-49  | 50-59 | 60-69 | 70+ |\n|------------|-----|-------|--------|--------|--------|-------|-------|-----|\n| Percentage | 0.00%  | 0.02% | 20.77% | 57.11% | 19.03% | 2.95% | 0.12% | 0.00%  |\n\n\n|    Race    |  white | black | middle eastern |  asian | latino hispanic | indian |\n|:----------:|:------:|:-----:|:--------------:|:------:|:---------------:|:------:|\n| Percentage | 61.55% | 6.22% |      4.89%     | 21.76% |      4.77%      |  0.81% |\n\nFor data preprocessing, we closely follow the procedures described in [1] and further filter out samples exhibiting large facial movements or unsynchronized speech and mouth motion [4]. This ensures the high quality and reliability of our dataset.\n\nWe acknowledge the importance of reproducibility and are committed to supporting the community.\nWe will release the MTHM test dataset, along with the corresponding code and model weights, to the research community to encourage reproducibility and facilitate further research and development in audio-driven human animation.\n\n\n\n[1] Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content\n\n[2] Openvid-1m: A large-scale high-quality dataset for text-to-video generation\n\n[3] MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions\n\n[4] Out of time: automated lip sync in the wild\n\n\n\n### Q2: Clarification of our evaluation.\n\nThere appears to be a misunderstanding regarding our paper. The \"MultiTalk single\" entries in Tables 1 and 2 report results from our pre-trained single-person model on single-person datasets, without using L-RoPE. Similarly, \"MultiTalk multiple\" refers to our fine-tuned model, also evaluated without L-RoPE. In both cases, L-RoPE is not involved in single-person inference; it is only required for multi-person generation. We present both \"MultiTalk single\" and \"MultiTalk multiple\" in the table to demonstrate that L-RoPE does not degrade single-person performance, and our strong results in single-person scenarios are not due to L-RoPE.\n\nRegarding differences between \"MultiTalk single\" and the FantasyTalking and Hallo3:\n\n(1). Model architecture: Our method uses Wan2.1 as the video backbone, while Hallo3 uses CogVideo. Although both are DiT-based, there is a significant performance gap between different foundation models. Hallo3 also incorporates an identity preservation module, which we do not. Compared to FantasyTalking, we employ a different audio encoding method, and FantasyTalking introduces a Face-Cross-Attention mechanism, which our model does not have.\n\n(2). Training strategy: We use partial parameter training and a multi-task training strategy, which are different from those in Hallo3 and FantasyTalking.\n\nOn the issue of data fairness, it is common practice in this field to compare methods using their official open-source models, as each work is trained on its own collected dataset. For example, Hallo3 uses 1,200 hours of YouTube data + 2,346 hours of film video, FantasyTalking leverages large-scale internet data, OmniHuman-1 uses 18.7K hours of data, and Hunyuan-Avatar uses 1,250 hours of collected data. Given these differences, we compare directly with the available open-source checkpoints, following standard practice.\n\n### Q3: Explanation of Table 3.\n\nIn our L-RoPE design, we assign distinct video and audio label ranges to different persons (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). Table 3 investigates the impact of using different label ranges for each person. The results indicate that varying the label assignments yields comparable performance results, which demonstrates that L-RoPE is robust to the choice of label ranges and not sensitive to this hyperparameter.\n\nWe acknowledge that Table 3 does not provide a direct comparison between models with and without L-RoPE. However, such an ablation is not feasible in the multi-person setting, as it is not possible to generate audio and lip synchronized videos for two individuals without the L-RoPE mechanism."}]

Author rebuttal:

Meta-review: "This paper approaches the task of multi-person audio-to-human generation using video Diffusion Transfomers. It specifically aims to address the issues of having multiple people present in the output video, which previous methods do not handle well. This work proposes the use of Label Rotary Position Embedding (L-RoPE) which helps to bind audio streams to the correct tokens (and therefore regions) in the diffusion process."

Strengths
- The paper tackles a challenging problem: generation of multi-person conversational videos driven by multi-stream audio and prompts.

- The paper provides a novel mechanism, label rotary position embedding (L-RoPE) to address the issue to resolve binding between audio speakers and visual persons.

- As mentioned and inquired by Reviewers cNEL and M1mM, the approach can be generalized beyond 2 speakers (as stated by authors in rebuttal)

Weaknesses
- Reviewers (particularly s1Za) raised ethical concerns about the dataset collected by the paper in terms of privacy, consent, copyright and fair use, and representation/bias. The ethics reviewer investigated this, and requested information from the authors on source of the training dataset collected from YouTube, the license of the data, and consent of the subjects. The authors provided the required information in the comments, and the paper seems to follow similar practices for uses of YouTube data of recent NeurIPS 2023/204 papers (as noted by the ethics reviewer). Thus, I think the issue of the dataset is not a rejection factor for this paper, but the authors MUST provide all the information requested by the ethics reviewer, apply the suggested mitigations for 

- Paper lacks reproducibility currently, but authors promise to release the test dataset, code, and model weights.

- More demos of moving people and more than 2 speakers are necessary.

Decision

This paper provides an effective and novel approach for the difficult problem of multi-person audio-drive conversations. Concerns were raised about the training dataset collection, but the authors have provided information requested by ethics reviewer, and this information MUST be included in the final version. During rebuttal, a number of other issues were raised by reviewers, which the authors provided satisfactory responses to, causing reviewers to raise their score. These concerns, as summarized by the authors, are: "generalization to occlusion and overlap scenarios, handling more complex instructions, extension to interactions involving more than two persons, the performance gap between real and synthetic audio, quantitative comparisons of different audio injection methods, the definition of our proposed new task, explanation of the identity degradation issue, the broader impact of this technology, inference time analysis, comparison between L-RoPE and mask-based audio injection methods, model stability, clarification of conflicting conditions, quantitative comparisons of different training strategies, clarification of our evaluation protocols, explanation of Table 3, and clarification of our data collection process."
Decision: Accept (poster)