InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditions
Authors: Zhenzhi Wang
Jiaqi Yang
Jianwen Jiang
Chao Liang
Gaojie Lin
Zerong Zheng
Ceyuan Yang
Yuan Zhang
Mingyuan Gao
Dahua Lin
Venue: ICLR 2026 Poster (2026)
Keywords: talking person video generation
multi-concept video customization
Primary area: generative models
TL;DR: could generate 2-3 people dialogue videos, or single-person talking videos with human-object interactions

Abstract: End-to-end human animation with rich multi-modal conditions, e.g., text, image and audio has achieved remarkable advancements in recent years. However, most existing methods could only animate a single subject and inject conditions in a global manner, ignoring scenarios where multiple concepts could appear in the same video with rich human-human interactions and human-object interactions. Such a global assumption prevents precise and per-identity control of multiple concepts including humans and objects, therefore hinders applications. In this work, we discard the single-entity assumption and introduce a novel framework that enforces strong, region‑specific binding of conditions from modalities to each identity's spatiotemporal footprint. Given reference images of multiple concepts, our method could automatically infer layout information by leveraging a mask predictor to match appearance cues between the denoised video and each reference appearance. Furthermore, we inject local audio condition into its corresponding region to ensure layout-aligned modality matching in an iterative manner. This design enables the high-quality generation of human dialogue videos between two to three people or video customization from multiple reference images. Empirical results and ablation studies validate the effectiveness of our explicit layout control for multi-modal conditions compared to implicit counterparts and other existing methods.

Published as a conference paper at ICLR 2026
INTERACTHUMANT MULTI-CONCEP1 HUMAN ANIMA- TION WITH LAYOUT-ALIGNED AUDIO CONDITIONS
Zhenzhi Wang" Jiaqi Yang"? 2 Jianwen Jiang Jiang Chao Liang?, Gaojie Lin",
Zerong Zheng", Ceyuan Yang". Yuan Zhang'. Mingyuan Gao?, Dahua Lin' 'he Chinese University Hong Kong "ByteDance
ABSTRACT
End-to-end human animation with rich multi-modal conditions, c.g., text, image and audio has achieved remarkable advancements in recent years. However, most existing methods could only animate a single subject and inject conditions in a global manner, ignoring scenarios where multiple concepts could appear in the same video with rich human-human interactions and human-object interactions. Such global assumption prevents precise and per-identity control of multiple concepts including humans and objects, therefore hinders applications. In this work, we discard the single-entity assumption and introduce a novel framework that enforces strong, region-specifer binding of conditions from modalities to each identity's spatiotemporal footprint. Given reference images of multiple concepts, our method could automatically infer layout information by leveraging mask predictor match appearance cues between the denoised video and each reference appearance. Furthermore, we inject local audio condition into its corresponding region to ensure layout-aligned modality matching in an iterative manner. This design enables the nigh-quality generation of human dialogue videos between to three people or video customization from multiple reference images. Empirical results and ablation studies validate the effectiveness of our explicit layout control for multi-modal conditions compared to implicit counterparts and other existing methods. Video demos are available https: /tte::///heeee ht///hhezziiie wang github
INTRODUCTION
By leveraging the priors pretrained Diffusion Transformer-basee (DiT) video diffusion models (Bar
Tal et al. 2024; Blattmann et al. 2023a bj Guo et al. 2023 Zhou et al., 2022; Gupta et al. 2023
Wang et al. 2023: Ho al. 2022, Brooks al. 2022 Wang et al. [2220] Singer et al. [022] Li
et al. 2018 Villegas et al. [022 Lin et al. 20025b) end-to end human animation models, especially
audio-driven approaches (He al. 2023 Tian et al. 20256 Xu et al. 2024a Wang et al. 2024a
Chen et al. 2024 Xu al.] 2024b Stypuikowski et al. 2024 Jiang et al. 2024a] Lin et al. 2024
2025a) have achieved high-quality human-centric video generation and strong controllability from multi-modal conditions, such as text, image and audio. However, most existing methods commonly hold an assumption of a single-identity paradigm: all available conditions should be fused globally and implicitly assumed describe unique subject in the given image. Although this global injection strategy simplifies conditioning by sharing the same condition signal across all regions, fundamentally limits scalability scenarios involving multiple individuals or complex human-object interactions, where each entity requires distinct appearance and voice attributes.
Recent multi-concept video customization methods, such as Video-Alchemist (Chen et al. 2025a), ConceptMaster (Huang et al. (025). Phantom (Liu et al. 2025). and SkyReels-AV (Fei et al. 2025a enable injecting multiple reference images object interaction scenarios. However, these methods face significant challenges when directly applied human animation tasks. In such tasks, animation signals are highly specific to individual identities and demand precise conditioning and alignment with specific spatiotemporal regions. For example, audio signals are exclusively associated with the current speaker and are unrelated to the background
single video, facilitating multi-person or human-
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Figure 1: Video frames generated from audio and multi-concept reference images (human heads/full bodies, objects, scenes) display rich, audio-matched expressions. Our method enables compositional generation including outfit changes, human-object interactions, anime styles, dialogues even without start frame. Red and green wave icons denote speaking and listening, respectively.
or other individuals' concepts. Existing customization methods, akin to end-to-cni human animation approaches, still adopt video-level condition injection. While this approach may suffice for general video generation, it often causes confusion in human-centric video generation, making it difficult to produce satisfactory results, shown in Fig. 3 For end-to-end human-centric video generation, conditioning inputs should ideally include not only global modalities (e.g., reference images and text descriptions) but also local modalities (e.g., audio). As mentioned, existing human animation and multi-concepy customization methods fail to address this critical distinction. This limitation motivates us to propose a new framework that enables the precise injection of local human-related modalities, capability that is both essential and urgently needed for robust multi-concept, human-centee video generation, as illustrated in Fig.
In this paper, we propose InterActHuman, video diffusion framework for spatially aligning multi- modal conditions in multi-concept human video generation. Unlike prior methods (Chen al. 2025a Huang et al. 2025) that rely on feature fusion and attention to implicitly learn relationships between condition signals and concepts, InterActHuman introduces an attention module that explicitly predicts the spatial locations where the reference concepts appear in the video. This explicit layout allows the model to accurately associate local audio conditions with the correct regions via iterative mask prediction and masked audio attention during diffusion inference. Our framework offers key advantages. First, it enforces a stronger layout constraint by precisely binding each condition to its corresponding spatial region. Second, it provides unified interface for synchronously injecting all modalities (e.g., visual and acoustic inputs) through the layout. These features make InterActHuman well suited for multi-modal, multi-concept human animation and establish a baseline for this domain.
Although using an explicit mask for condition injection may seem straightforward, it creates a chicken-and and-egg dilemma: during inference the final video not yet available, leaving the spatial positions of each identity uncertain and making accurate mask prediction impossible; yet without those masks, spatial injection of local audio conditions cannot performed, leading to an incomplete or misaligned generation process. To address this, we leverage the iterative denoising process inherent in diffusion models (Song et al., 2021). Specifically, we introduce a mask- mask--redicttor branch into the diffusion pipeline and adopt an interleaved mask-predictes step guides condition injection at step k+ This iterative refinement progressively finalizes the without
strategy, wherein the mask predicted at
spatial layout, breaking the cyclic dependency and enabling precise spatial alignment ground-nu woound-truth video during inference. In essence, our approach converts the chicken chicken-and-ege -nd--gg problem into sequential, convergent procedure that robustly aligns local audio conditions.
2

---

Published as a conference paper at ICLR 2026
Beyond model design, developed scalable pipeline to automatically assemble high-quality. human-centric animation data to address the lack of suitable multi-concept datasets. This pipeline operates by: 1) accurately tracking individual identities to extract their mask information and images, and 2) aligning audio segments to each identity through lip synchronization. We curated a dataset of over two million video-entity pairs, capturing both human-human and human-object interactions across a wide range of object categories, In summary, our contributions are as follows:
(1) We propose a novel human animation framework capable of synthesizing multi-person and human- object interactions, conditioned on multiple reference images, text descriptions, and audio inputs. The framework also supports long video generation as well as single full -ill-imae conditioning. (2) We highlight the importance of local condition injection for multi multi-concept, multi- multi-modal video generation and introduce a simple yet effective design that enables the model to handle both global and local conditions by automatically localizing the conditioned layout. Experimental results demonstrate that our proposed design significantly outperforms existing baselines.
2
RELATED WORKS
Video Diffusion Models have enabled unprecedented quality text-to-video (T2V) or image-to-vides
(I2V) generation in recent years, thanks diffusion- based generative models (Ho et al., [020] Song
et al. 2021 Karras et al. 2022] Song et al., 2020; Liu et al., 2022). Early T2V approaches either
adapt pretrained text-to-image networks
training- free manner (Singer et al. 2022 Wu et al.
2023b; Qi et al. 2023) fine-tune UNet-based latent diffusion architectures (Guo et al. 2023 Zhou
ct al. 2022 Blattmann et al. 2023a| Wang et al., 2023). To push the frontier, recent works compress
spatiotemporal features with 3D causal
VAEs (Yu et al. 2023), and migrate to Diffusion Transformer
(DiT) backhones (Vaswani et al.
2017 Peebles & Xie
2023 Brooks et al., 2024 Hong et al. 2022).
Through progressive low-to-he high resolution pretraining and fine-tuning (Polyak et al. 2024 Kong
ct al. 2024 Wang et al. 2025a), these models yield longer, more coherent, and high-quality videos.
Our method is also built upon the pretrained DiT video generation models.
Human Animation Models synthesize videos of people driven by text, reference images, human
body poses or audios. Early GAN-based methods (Siarohin et al., 2019; Zhao Zhang 2022
Siarohin et al. 2021; Jiang et al. 2024bb WWng et al. 1021) are trained on small datasets (Naranni
al. 2017 7, Siarohin et al. 2019 Xie al. [2022] et al., 2022) for self-supervised pose transfer.
Diffusion- based approaches (Shao et al. 2024| Zhhng et al.
[0244 Hu 2024 Wang et al. 2024c
20256) now surpass GAN-based methods by conditioning on 2D skeletons, 3D depth, or mesh
sequences. Audio-driven portrait methods (Ye et al. 2022 Zhang et al. 2023 Tian et al. 2025b
Jiang et al.] 2024a; Cui et al. 2024) have been extended toward full-body motion via two-stage
pipelines for improved hand quality (Corona al. 2024 Meng al. 2024| Tian et al. 2025a
Hogue et al. (024) and one-stage unified framework designs (Lin et al. [202]]200 2025a). In summary, none of them have explored multi-concept human animation, and our method is the first to enable
multi-concept human animation with local audio conditions.
Multi-Concept Video Customization Models have received limited attention. Early single-concept identity-preserving methods include Videobooth (Jiang et al. (024c)), which learns coarse-to-fine fine embeddings from WebVid (Bain et al. (021) via Grounded-SAM (Kirillov et al.) 2023 Liu et al.
2023 Ren et al. 2024); ID-Animator (He et al., 2024), which integrates IP-Adapter Yee et al.
2023) into AnimateDiff (Guo et al., 2023); y and ConsisID YYuan et al. 2024). which decouples
frequency signals to preserve facial identity. More recently, Video-Alet Alchemist (Chen et al. 2025a).
ConceptMaster (Huang et al. 2025). and Phantom (Liu et al,, 2025) support multiple reference images and text descriptions via cross- or self-attention injection for customization. BlobGen- Vid (Feng et al. 2025) further enables local layout control of text-specified concepts using user- or --r-vidddeed spatiotemporal masks. Ingredients (Fei et al. (025b) proposes to predict the layout of identities in molti-image customization, yet it does not consider the audio condition and does not support multi- melli-person talking generation. All existing methods rely solely on image and text conditions and lack support for multimodal inputs such as audio, which we argue are essential for truly versatile, human-centric video generation.
multi-concept
3

---

Published as a conference paper at ICLR 2026
Aodio
* -
DID Blocks
Inforence
Aodio WW
Tolee
TTxt Prompt
Text Eacodar
Iduntity 1
Ideadity
and
are talking
aB gwyy
MMDIT Afta
Afta
VAE Decoder
Encoder
MLP
MLP
Figure 2: Illustration of our framework, which adaptively predicts masks as the spatial guidance of audio condition injection. In training. we train the mask predictor (cross-attn w/ MLP) with mask loss; in inference, we collect mask predictions cache and leverage masks predicted from the last denoising step (t - 1) to guide the audio cross-attn in the current denoising step (t).
METHOD
In this section, we present InterActHuman, our multi-concept video generation framework designed to address the challenges of local condition matching for identities in multi-modal conditions. As shown in Fig. [ the framework begins by customizing reference images of multiple concepts into video and predicting the mask regions corresponding each reference appearance in the output video as layout cues with a cross-attention as mask predictor. operates on the features of the noisy video latent and the reference image latent and supervised by ground-trute masks. Finally, local audio conditions are injected into the designated regions guided by an iterative mask prediction procedure.
3.1 PRELIMINARIES
Problem Setting. Given a caption T' that describes target video, alongside of concept reference images {X; i I.....Ny .N} and identity-level audio (Y; i ..... number of distinct concepts, the objective of Multi-Concee Human Animation is to generate high- quality videos that faithfully integrate all image-specified visual concepts with correct lip movements synchronized with each audio signal in accordance with the descriptive caption Each concept should consistently preserve its visual identity depicted in the provided images, while accurately expressing the semantic roles and behaviors described in the caption.
.N .,....N]
where N denotes the
Preliminaries. Transformer-baser text-to-video latent diffusion models have recently demonstrated remarkable capabilities in generating high-quality video content. Our proposed InterActHuman framework is built upon the MMDiT-based video generation model (Pebless & Xie 2023 Esser et al. 2024 Seawead et al., (025) and utilizes a 3D Variational Autoencoder (VAE) (Kingma & Welling 2013), which compresses input videos into compact latent space. For training, adopt the flow matching objective (Lipman et al.] 2022). which formulates the generative process as probability flow ordinary differential equation (ODE). This ODE transports clean latent representations 2o to their noisy counterparts 2 along linear path, defined as = (1 t) + te at timestep t, where is sampled from standard Gaussian distribution. In our setting, which incorporates multi- modal conditions (image and audio), the output of the diffusion transformer is parameterized as Ve 22, t, Congr Caudia). This output is supervised to predict the velocity (2) 20), resulting in the following training objective:
== [1 t, Camgs Coudso) - (21 - 20)1 1..
This formulation ensures the model effectively leams to generate videos conditioned multi-modal inputs while maintaining high fidelity and temporal consistency.
Multi-Concept Reference Image Injection. To handle multiple appearance images, inject appearance cues via self-attention in the original DiT layers. Each reference image X, is encoded into latent tensors x; using the same VAE for noisy video frames. The reference latents
4

---

Published as a conference paper at ICLR 2026
then flattened into token sequences and processed by the DiT along with the noisy latents The reference latents will reuse the parameters of DiT to extract features and interact with noisy latents at the self-attention layer in every DiT block, allowing appearance cues to propagate to the denoising pathway. Notably, no extra networks or additional parameters are required, thus preserving the model's efficiency. Please refer to Appendix Dor more details.
3.2 LAYOUT PREDICTION AND LOCAL CONDITION INJECTION
While reference-image injection is well studied, leveraging spatiotemporal layouts for local conditions, such as precise audio alignment, remains challenging. Mask prediction and audio injection are coupled: if a denoising step is incomplete, the masks are unavailable; if it is complete, it is too late to inject local conditions. We resolve this with the multí-step inference of diffusion models (Song et al.) 202Ip: masks predicted at step k guide mudti- condition injection at step k+1. As the network learns spatiotemporal layouts for each identity, precise masks emerge without user annotations.
In each DiT block, we predict a spatiotemporal mask quantifying how strongly each reference image should influence each video frame. A lightweight mask-predices head is attached to each of the L transformer layers, consisting of: (1) shared linear projection that maps hidden video features h and hidden reference features hy to query, key, and value tensors; (2) LayerNorm; (3) 3D RoPE positional encoding; (4) cross-attention module; and (5) two-layer MLP. After normalization
Mask Predictor.
and RoPE, video tokens attend to a single reference via (1) = softmax
Q'K:"
VT. where d is
the head dimension and indexes the current layer. The attended feature a. is transformed by the MLP and passed through a sigmoid to yield layer-specific mask m Jo, 1)" for reference X;- We then average the predictions from the last few DiT blocks to obtain the final mask. Notably, the mask predictor is trained recover the complete human region, regardless of whether the reference image shows only the upper body, face, or full body. This simplifies mask prediction and stabilizes the conditioning module, yielding robust behavior across diverse scenarios. Because the predictors reuse in-block features and share parameters across references, they add minimal overhead while enabling explicit layout control for each identity.
Mask Prediction by Caching During Inference. During inference, aggregating mask signals across layers is difficult in early denoising steps, when reference concept latents are still ambiguous. We adopt an iterative strategy: at each step, the mask predicted at the previous step is cached and used as a layout prior for the current step. This progressively sharpens spatiotemporal localization, allowing masks to adapt to the evolving content and remain consistent per concept over time, thereby improving overall conditioning quality.
Local Audio Conditioning, To prepare for multi-concept, human-centric video generation, we first pretrain on single-identity audio-conditioned animation by adding cross-attention-basee audio conditioning (without masks) and Concretely, a new cross- aotenntttnnoonn layer injects wav2vee (Baevski et al., (2020) audio features into each DiT block after the MMDIT layer. In the multi-concept setting, this pretrained audio aots-attention already provides strong per-identity audio control. We therefore implement local audio conditioning primarily at inference: instead of updating all noisy video tokens (Lin al. [2025a] 2024), we inject wav2vec features only into tokens assigned to a given identity, using the previous-step mask as guidance. To ensure smooth transitions in latent features and in the final video, blend meaningful and muted audio features per token using the mask confidence, with soft weighting near mask boundaries. This local conditioning also enables multt-person dialogues. Given per-speaker audio tracks as input, the model generates realistic interactions in which speakers take turns, supporting wide range of applications.
training strategy, following (Lin et al. (025a).
Training Loss and Strategies. The overall objective combines flow-matching diffusion loss (Sec. 3.1) with a focal loss (Lin et al., 2017) for mask classification, which stabilizes training compared to binary cross-entropy, likely due to foreground-baccgrond low-quality masks. A crame-alignment flag excludes frames with invalid or low-quality masks from loss computation, strengthening temporal supervision. To mitigate the common "copy-paste" behavior of diffusion models replicatinn subjects from references with little variation in pose viewpoint-wae randomly mask reference images reveal only the head, full body, or clothing,
imbalance and occasional
5

---

Published as a conference paper at ICLR 2026
NOS
os
NE
User Input
Frame
Kling Lip-syne
Humiia-um--FFi
Ours
Figure 3: Qualitative comparison with previous methods on multi-concept audio injection.
promoting appearance diversity. We use a face-to-full-bot appearance conditioning ratio of 0.7 0.3 and weight the diffusion and focal losses equally (1 : 1).
Inference Strategies. Given N reference images and a text prompt, our method uses Qwen2.5- VL (Bai et al. 2025) as a rephraser to extract detailed descriptions from each reference image and integrate them with the original prompt. We apply shared classifier-fres guidance (CFG) across audio and text. For local control, audio conditioning is applied only within the predicted mask regions for each reference. Because early denoising steps yield unreliable masks that may suppress other references, we disable mask-based injection for the first 10 steps, then enable it using the previous-step mask. We inject masked audio only the conditional (positive) branch during CFG sampling, with a CFG scale of 6.5 and 50 denoising steps.
3.3 MULTTI-ONCCEPT HUMAN-CENTRIC VIDEO DATA CURATION
We curate large-scale, human-centric video dataset from large public video dataset (Li et al. 2024) and self-collected videos, filtering out videos that are too short (<4 seconds) or contain too many salient humans (>3) as detected by human pose detector YYagg et al. (023). Unlike
previous multi-concept video generation methods (Chen et al. 2025a Huang et al. (2025) that rely on conventional pipelines like Grounding-DDin DINO (Liu et al. (023) and SAM (Kirillov al. 2023 for
obtaining a single reference image per identity, our multi-stage pipeline leverages advanced vision- language models. First, each raw video is densely captioned using Qwen2-Vi VL (Wang et al. (022bb (distilled from Gemini-2.0-Proo) generating fine-grained descriptions of the environment, subjects' appearance, actions, expressions, interacting salient objects, and inter-subject interactions. Next, these descriptions are parsed by a zero-shot Gemini-2.0-Flaast API to extract structured appearance phrases. For spatial supervision, we use Mounding-SAM.. (Ren et 2024) with the query person to produce accurate, temporally consistent masks, which are used both for extracting foreground reference images (with white background) and ground-truth for our mask predictor. Our corpus comprises over 2.6M triplets of videos, per-fram per-frame masks, and captions, forming the foundation of InterActHuman.
4 EXPERIMENTS
Baselines. Since ConceptMaster (Huang et al. 2025) and Video-Alchemist (Chen et al. 2025a) are not open-sourced, we do not have access their models nor test sets. Therefore, we primarily compare our method with recent multi-concept video customization models through their publicly available APIs or public models, evaluating performance from the perspectives of visual appear- ance and adherence to text prompts, including Vidu2.0 (Bao et al. 2024), Pika2.1 (pik), Kling 1.6 (Kuaishou] 2024) and Phantom (Liu et al. 2025), ) and audio-driven human video generation methods, including DiffTED (Hogue al. 2024). DiffGest (Zhu et al. 2023) Mimiction (Zhang et al. 2024). CyberHost (Lin et al. 2024), OmniHuman (Lin et al.) (025a) and Kling 1.6 (Kuaishou. 2024 with lip synchronization.

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparisons with audio-conditioned full-body animation baselines.
Single-Person Test Set
Multi-Person Test Set
Methods
Syne-Ct
HKV1
HKCt
Sync-DI
IQAt
AEST
FVD4
DifITED
0,926
0.769
DiffGess++Mimis
0,496
23.409
0.833
CyberHost
6.627
24.733
0.884
8.974
4.011
2.856
54.797
Kling .66 + Lip-syne.
4,449
46,490
0.826
8.401
4.716
3,444
33.555
MultiTalk
7.671
4.561
3,248
35,472
OmniHuman w/o mask
7.443
47.561
0.898
9.482
4.768
3,466
33,895
OmniHuman w/ fixed mask
7.068
4.690
3,369
40,239
Ours
7.272
59.635
0.885
6.670
4.757
3,467
22.881
Table 2: User preference evaluation.
means publicly available version with Wan2.1-1.3B.
Audio-Driven
Multi-Concepl Customization
Metric
Kling
OmniHuman
Ours
Pika
Phantom*
Kling
Vidu
Ours
Avg. Score 1.70
1.82
2.48
2.22
2.46
2.90
3,40
4.01
Top-1 (%) 1 14.5%
25.6%
59.9%
4.9%
9.9%
13.6%
22.2%
49.4% %
Evaluation Metries. As audio and appearance are key characteristics of humans, we follow current state-of-the-arr methods (Lin et al. [025a Huang et al. (025) to evaluate audio-driven, multi-concept human-centric video generation. To comprehensively evaluate methods for this task, we consider five dimensions: 1) Concept fidelit:: We leverage image feature extractors like CLIP-I (Radford et al.) 202T) and DINO-I (Oquab et al.] 2023) on subjects in the generated videos to assess whether they aligned with the provided reference images. To get cropped subject images in output videos, randomly sample 5 frames in each video and use Florence-2 (Xiao et al. 2024) to detect the subject bounding boxes. We then crop the detected regions and compute the CLIP-I and DINO-I scores. We also employ Face-Arc (Deng et al. 2019). Face-Cur (Huang et al. 2020) and Face-Glink (Deng et al.) 2019) to evaluate the fidelity of facial features if the concept is human. 2) Prompt following. We utilize video-level CLIP (Wang et al., 2022) to measure the similarity between the input text prompts and the resulting video content. 3) Visual quality: We utilize q-align (Wu al.) (023a). a vision- language model, for no-reference image quality assessment (IQA) and aesthetic score estimation (AES). 4) Audio synchronization and human pose diversity: For lip synchronization, we leverage the widely used Sync-C and Sync-D (Chung & Zisserman| (017)) to compute audio-visual confidence. We also incorporate hand keypoint confidence (HKC) and hand keypoint variance (HKV) (Lin et al.) 2024) to quantify hand-pose accuracy and motion richness, respectively. For simplicity, assume each test video contains exactly two participants: one speaking (with meaningful audio as input) and one listening (with muted audio input). 5) Distribution distance: We employ FVD (nnerrtiierr let a to measure the distance between generated and ground-truth videos.
Test Sets. We three test sets in experiments: 1) Lingle-person audio conditioned human animation test set following OmniHuman (Lin et (025a)) (see Tab. [T] 2) our collected two-person audio conditioned human animation test set where only one person talking (see Tab. and Tab. 4 and 3) multi-concept video customization test set following Phantom (Liu et al. 2025). where we select 100 human-related pairs of reference images and text prompts in our experiments (see Tab.
4.1 COMPARISONS WITH STATE-OF-THE--RTT
Multi-Concept Audio-Driven Human Video Generation. In Tab. our approach achieves state-of- the-art or comparable performance in lip synchronization accuracy and motion diversity, particularly excelling in complex multi-person interactions where baseline methods (Lin et al. [2024] Kuaishou 2024) struggle with accurate audio signal assignments. For single-person scenarios, our method performs on par with specialized models like OmniHuman (Lin et al.) 2025a). In multi-person settings, existing methods including OmniHuman and its extensions as well as leading commercial video generation models with post-processed lip-syne, fail to deliver satisfactory results. Poor lip- sync accuracy highlights their inability to generate accurate lip movements for the correct person. Although HunniHuman with oracle masks (manually setting audio-conditioned regions) improves
7

---

Published as a conference paper at ICLR 2026
Table 3: Quantitative comparison of subject consistency, prompt following and visual quality.
means publicly available version with Wan2.1-1.3B.
Decoupled Concepl Fidelity
Prompt ViCLIP-TI
Video Quality
Methods
CLIP-It
DINO-I1
Face-Aret
Face-Cure Puce-Curt
Face- -rinkG
AESt
IQAT
Vidu2.0
0.696
0.458
0.568
0.597
18.61
3.350
4.689
Pika2.1
0.688
0.459
0.579
0.566
0.607
19.39
3.534
4.791
Kling1.6
0.659
0.420
0.552
0.547
0,582
18.38
3,487
4.787
Phantom"
0.703
0.476
0.589
0.573
0.615
17.73
3.404
4.812
Ours
0.744
0.533
0.598
0.600
0.644
18.87
3.565
4.903
In garage with white door and took in the background, harked searded man a dark T-shirt stands facing the camera, hokdnn and showing small methanical part whiespppingg
dimnly ausitoriam, dress
beside
Mon, adorned elggnnt gloves, stands upright with poised composure, hts posture crucing custtnggcccddiinnt
Reference
Vidu2.0
Pika2.1
Klingg.6 16
Phantom
Ours
Figure 4: Qualitative comparison with previous methods on subject consistency and text following.
lip-sync accuracy, it still significantly degrades overall video quality, reflected in the FVD metric. This clearly demonstrates the limitations of existing methods that rely on single-identity assumptions, hindering their applicability broader scenarios. In contrast, our method achieves strong performance in both lip-sync accuracy and overall video synthesis quality, effectively addressing the challenges of condiiionddi
periiipeeoonnn video generation while remaining compatible with existing settings. Fig. presents qualitative results for multi-perse audio-driven video generation. While Kling w/ lip-sync shows many audio assignment errors and OmniHuman w/ fixed mask shows inflexible audio control with many missing cases, our method consistently assigns audio signals to the correct identity and demonstrates better motion dynamics and more precise audio-driven animation. is worth noting that all previous methods rely reference frame containing complete information to generate talking-person person videos, while our method only needs reference appearance human's head or full- full-body images and audios.
User Study. We conducted a user study to evaluate our method on two tasks: (1) lip synchromization in multi-person talking videos and (2) subject consistency in multi-concept customizations. The lip sync test used 19 videos from three methods, while subject consistency was assessed on 9 videos from five methods. We the same model (labeled 'ours') for evaluation tasks. Ten experienced users ranked each method with scores (higher values indicate superior performance). Tab. reports and top-1 selection percentages. Our method achieved the highest scores and top-1 rates in both tasks, validating the effectiveness of our method.
the average
Multi-Concept Video Customization. Although our major contribution is not on multi-concept video customization, we show that our model is also capable of preserving multi-concept visual appearances. In Tab. 3 our method outperforms existing approaches (Bao al. 2024 pik] Kuaishoul 2024) in preserving identity details and facial features, addressing the common degradation issue in multi-subject generation. Notably, we achieve this without sacrificing audio injection performance, while these methods are incapable of generating videos from audio conditions. It suggests joint optimization framework on video generation and mask prediction successfully balances video generation. Our method ranks second in prompt following, likely due to audio-driven human- centric training data focusing on talking and singing, which limits prompt diversity compared to text-to-video tasks. Despite this, qualitative results (Fig. 4) show that approach maintains natural
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study audio-driven multi-person animation methods.
Variants
Sync-DL
IQAt
AEST
FVD4
Global audio condition
9.482
4.768
3.466
33.895
Embedding
8.627
4.658
3.338
35.665
Fixed Mask
7.068
4.690
3.369
40.239
Predicted Mask (Ours)
6.670
4.757
3.467
22.881
Table 5: Runtime and parameters versus number of reference images.
Component
1 ref
2 refs
3 refs
#Params
DiT Model
6.5s
7.0s
7.7s
7B
Mask Predictor
0.4s
0.8s
1.2s
56M
Full Model
6.9s
7.8s
8.9s
7B
subject consistency and visual quality in both real-world and anime domains, outperforming previous methods that suffer from unnatural compositions and degraded visuals.
4.2 ABLATION STUDY
We validate our local audio injection designs via ablation on three variants: 1) Global audio (Lin et al. 2025ab applies audio across the entire feature map; 2) ID Embedding injects audio features with Tearnable ID embedding without mask prediction; 3) Fixed Mask uses predefined static spatial masks. As shown in Tab. 4 our framework with predicted dynamic masks achieves the best Syne-D and FVD scores. The fixed mask yields decent Syne-D D but suffers motion artifacts (worst FVD), and global audio, while scoring best IQA, offers poor audio-visual alignment. Qualitative results Fig. further reveal that global audio drives all identities uniformly, ID embedding often mismatches audio with identities, and fixed masks lose alignment when characters move. These findings underscore that current methods overlook the need for precise local conditions, highlighting the strength of our dynamic, adaptive mask prediction strategy for nulti-person talking video generation.
Computational Cost and Runtime We measure inference time with the setting of 720p. 109 frames (28 VAE latents) on single A100, and record single forward pass time. The mask predictor adds 56M parameters versus a 7B DiT and incurs small overhead reference image (s=0.013s per DiT block). As more references increase VAE latents from 28 to 28+nn DiT self-attention scales as (1+n/28)².
Failure case analysis of mask prediction Fig. 5 we show the qualitative results on the failure case of mask predictions, where the yellow cat is the current identity image (i.e., ID query of mask predictor). We can see that although the mask prediction achieves good performance in loU measurement from Tables and TO it still makes mistakes when there highly overlapping regions. For example, two girls are standing in front of the yellow cat, and the mask predictor manages to exclude the upper body of the right girl in yellow because of small area of overlapping regions. Yet, the mask predictor fails for the girl in pink in both frames. In the first frame, the mask predictor manages to exclude the face region of the little girl in pink, but mistakenly include another white animal. Besides, due to the high VAE wwwp-sampling ratio of the widely used DiT models, the mask predicted a very low resolution, leading to inaccurate mask boundaries.
Additional analyses in Appendix. We include (i) capability matrix against concurrent systems (Tables 8). (ii) mask IoU versus denoising steps/layers under low/high motion plus motion statistics (iii) mask cache ablation (Table 12). (iv) scaling to >3 people (Table 13]
(Tables 10
5 CONCLUSION
In this paper, we introduced InterActHuman, a novel end-to-end video diffusion framework that supports spatially aligned, multi-modal conditioning for multi-conoe human animation. By integrat- ing an efficient mask-predicte module into pretrained DiT backbone, method automatically infers per-identity
layouts from reference images and uses these masks to guide local audio conditioning. This explicit layout binding enables each reference concept retain its unique

---

Published as a conference paper at ICLR 2026
Generated Frames
Visualized Masks
Figure 5: Qualitative results on the failure case of mask predictions.
Globul Audio
ID Embedding
Fixed Musk
Ours
Reference
Figure 6: Qualitative ablation on audio injection strategies.
appearance and voice, even in complex scenes with multiple interacting entities. To facilitate training procedure our model, we presented high-throughput pipeline for harvesting and annotating over 2.6 million identity-aware video snippets, complete with per-frame masks for humans and objects. Extensive experiments on both single- and multi mulli-pprson benchmarks demonstrate that AnterantHuman achieves state-of-the-art performance lip synchronization, motion diversity and subject appearance fidelity, while maintaining competitive video quality. We hope it could serve as solid baseline for the multi-concept human animation and audio-driven multi-person video generation community, where further improvements could be built upon it.
Limitations. Since the goal of this work for human-centric video generation, the available data domain is inherently narrower than that used for general text-to-video pretraining, limiting the ability follow the diverse text prompts. While our framework is designed to accommodate any number of concept images, the training dataset predominantly consists of videos featuring two to three individuals. This distribution may constrain the generalization of any number of inputs, suggesting further improvements could be achieved by enlarging the diversity and scale of training data.
Acknowledgment. We thank the infrastructure team ByteDance Intelligent Creation for the data curation and annotation support.
Ethics Statement. Our model could be used to generate misinformation by using celebrity images and voices. We will strictly restrict access and add watermarks to prevent misuse.
Reproducibility Statement. We have provided code re-implemented on the publicly available video diffusion pretraining model 1an2.1 (Wang et al. 2025a) show the details of our method. We also provided the dataset processing code, c.g., caption analysis and Grounding-SAA SAM2 (Ren et al. 2024). In the Algorithm provided detailed pseudo-code to show the inference pipeline of method. Based on such information, we believe the reproducibility of method solid enough.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pika art. https://pika... .atptaaattta.a..........
Accessed: 2025-05-12.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 14449-1442,, 2020.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Max Bain, Arsha Nagrani, Güll Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to- video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024.
Omer Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint wrxiv:2401.12945 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22563 -25775, 2023b.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769-31788 2022.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL pattps://opena.: com/ r ld- sitooddddsmmlltoooo
video-generat mww.eeeerriiioomemeeeeeesese
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXlw:2310.0422. 2023.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Yuwei Fang, Kwot Sin Lee, Ivan Sko- rokhodov, Kfir Aberman, Jun- Yan Zhu, Ming-Hsuan Yang, and Sergey Tulyakov. Multi-subject open-set personalization in video generation. arXiv preprint urXiv:2501.06187, 2025a.
Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu. Hunyuanviddeoavvtte High-fidelity audio-driven human animation for multiple characters. arXiv preprint arXiv:2505.20156 2025b.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Life- like audio-driven portrait animations through editable landmark conditions. arXiv preprint MrXiv:2407.08136 2024.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision-ACCT 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pp. 251-263. Springer, 2017.

---

Published as a conference paper at ICLR 2026
Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. arXiv preprint arXiv:2403.08876 2024.
Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks. arXiv preprini arXiv:2412.00775 2024.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, pp. 4690 46900046999000 2019.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, et al. Skyreels-a2: Compose anything in video diffusion transformers. arXiv preprini arXiv:2504.0244:
2025a.
Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, and Mingyuan Fan. Ingredients: Blending custom photos with video diffusion transformers. arXiv preprint arXiv:250100199) 2025b.
Weixi Feng Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, and Weili Nie. Blobgen- vid: Compositional text-to-video generation with blob video representations. arXiv preprint arXiv:2501.076477 2025.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations (ICLR), 2023.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprini arXiv:2312.06662, 2023.
Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et Gaia: Zero-shot talking avatar generation. arXiv preprint arXiv:2311.1522 2023.
Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, and Jie Zhang. Id-animator: Zero-shot identity-prese preserving human video generation. arXiv preprint 2rXiv:2404.15273 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso- www.rinu..... weeeriss.cccpppee cc/ppa/at////2///11... ciates, Inc., 2020. URL htttps::/ppocceelliiiii haw.e//eee...aaaooooaaaaaaa0aaaa
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633-8666 8646, 2022.
Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, and Xiaohu Guo. Diffted: One-shot audio-driven ted talk video generation with diffusion- based co-speech gestures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1922- 1931, 2024.
Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Li Hu. Animate anyone: Consistent and controllable image-to vi-de synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163. 2024.
12

---

Published as a conference paper at ICLR 2026
Yuge Huang, Yuhan Wang. Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5901-5910, 2020.
Yuzhou Huang. Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, and Kun Gai. Conceptmaster: Multi-concept video customization on diffusion transformer models without test-time tuning. arXiv preprint arXiv:2501. 04469, 2025.
Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprini arXiv:2409.02634, 2024a.
Jianwen Jiang, Gaojie Lin, Zhengkun Rong, Chao Liang, Yongming Zhu, Jiaqi Yang, and Tianyun Zhong. Mobileportrait: Real-time one-shot neural head avatars on mobile devices. arXiv preprint arXiv:2407.05772, 2024b.
Tao Jiang. Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. Rim- pose: Real-time multi- person pose estimation based on mmpose. arXiv preprint arXiv:2303.07:33 2023.
Yuming Jiang, Tianxing Wu, Shuai Yang. Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6689-6700, 2024c.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600: 2024.
Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation. arXiv preprint arXiv:2505.2200 226477, 2025.
Kuaishou. Kling video model. https://klis www.:ng..iiihh....ooooo 2024. Accessed: 29 April 2025.
Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, et al. Openhumanvid: large-scale high-quality dataset for enhancing human-centric video generation. arXiv preprint arXiv:2412.001 15, 2024.
Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, and Yanbo Zheng. Cyberhost: Taming audio-drives avatar diffusion model with region codebook attention. arXiv preprint arXiv:2409.0187 01667, 2024.
Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-11 Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061: 2025a.
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. arXiv preprint arXiv:2501. 008306,0,66 2025b.
13

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprini arXiv:2210.02747, 2022.
Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Qian He, and Xinglong Wu. Phantom: Subject-consistet video generation via cross-modal alignment. arXiv preprint arXiv:2502.1107, 2025.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0381 20000000,,0, 2022.
Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimiev2: Towards striking, simplified, and semi-body human animation. arXiv preprint arXiv:2411.1006/ 2024.
A Nagrani, Chung, and A Zisserman. Voxceleb: a large-scale speaker identification dataset. Interspeech 2017, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.095.0 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
8748-8763. PMLR, 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 159, 2024.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, al. Seaweed-7b: 7b: Cost-effective training of video generation foundation model. arXiv preprint arXiv:2504.08683 2025.
Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free-view human video generation with 4d diffusion transformer. arXiv preprint arXiv:2405. 2...22455..4.55 2024.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion rep- resentations for articulated animation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13653-13662, 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 2022.
14

---

Published as a conference paper at ICLR 2026
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Inter- national Conference on Learning Representations, 2021. /tLLhttpss//ppereevi.ee
forum?id=Stlgiarriii
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 I. 1.3456 2020.
Michal Stypulkowski, Konstantinos Vougioukas, Sen Hc, Maciej Zieba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusson models beat gans on aalking-face generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5091-5100, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402 419. Springer, 2020.
Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation. arXiv preprint arXiv:2501.1068) 2025a.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pp. 244-260. Springer, 2025b.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marein Michalski, and Sylvain Gelly. Fvd: A new metric for video generation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022.
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprini arXiv:2503. 2Xv:22503.20314 2025a.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint 24K.::2400.021111 2024a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.06577 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/1/, 2024b.
Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- synthesis for video conferencing. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 10039-10049, 2021.
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. Imaginator: Conditional spatio-temporal gan for video generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1160-1169, 2020.
Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Intervvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.0311 2022.
15

---

Published as a conference paper at ICLR 2026
Zhenzhi Wang. Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, and Dahua Lin. Humanvid: Demystifying training data for camera- controllable human image animation. In Advances in Neural Information Processing Systems, 2024c.
Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Yuwei Guo, Dahua Lin, Tianfan Xue, and Bo Dai. Multi- identity human image animation with structural video diffusion. arXiv preprint arXiv:2504.04126, 2025b.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.170%0 2023a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7623-7633, 2023b.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4818 4829,48292288 2024.
Lianginn Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 657-666, 2022.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406. 08801, 2024a.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang. Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404. 240.060674.6 2024b.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210-4220, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308..000 066221, 2023.
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In The Eleventh International Conference on Learning Representations, 2022.
Lijun Yu, Jos Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:2310.0573: 2023.
Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyuan Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity-peeserving text-to-video generation by frequency decomposition. arXiv preprint 2rXiv:2411.1744/ 2024.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8652-8661, 2023.
Yuang Zhang, Jiaxi Gu. Li- Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-awe aware pose guidance. arXiv preprint arXiv:2406. 2eo::2406.99800 /9680, 2024.
16

---

Published as a conference paper at ICLR 2026
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3657-3666, 2022.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211. 2022.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang. Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scate scale video facial attributes dataset. In European conference on computer vision, pp. 650-667. Springer, 2022.
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10544 10553, 2023.
A ADDITIONAL EXPERIMENTAL RESULTS
A.1 LLM USAGE STATEMENT.
We use LLMs (e.g., Gemini-2.5 and GPT-5) to polish our paragraphs.
A.2 COMPARISON WITH SINGLE SINGLE-PERSON TALKING- HEAD METHODS
Although our method targets multi-person full-body talking video generation, report single-person talking-head results for completeness.
Table 6: CelebV-HQ: higher is better for IQA/ASE/Sync-C; lower
lower is better for FID.
Method
IQAt
ASE1
Sync-Ct
FIDJ
SadTalker (Zhang et al.
2023)
2.953
1.812
3.843
36.648
Hallo (u et al. 2024a
3.505
2.262
4.130
35.961
VExpress (Wang al.
2024a)
2.946
1.901
3.547
65.098
Echo (Chen et al.
2024
3,307
2.128
3.136
35.373
Hallo-3 (Cui et al. 2024
3.451
2.257
3,933
38.481
Loopy (Jang et al. 2024a)
3,780
2.492
4.849
33.204
OmniHuman (Lin et
2025a)
3.875
2.656
5.199
31.435
Ours
3.834
2.553
5.047
30.159
As shown in Table 6 OmniHuman achieves the best single-person talking-head scores on IQA/ASE/Sync-C, while our method yields the best FID. Since our approach targets multi-person, full-body generation rather than single-person heads, these results indicate competitive quality under a setting that ASE, while OmniHuman leads on Sync-C and FID. This mirrors the CelebV-HOC trend and supports our claim that the proposed framework remains competitive on single-person benchmarks while being designed for multi-person, full-body scenarios.
our primary target. On RAVDESS (Table 7 our method attains the best IQA and
A.3 COMPARISON WITH CONCURRENT WORKS
We show comparison with recent multi multi-pprson talking systems which are conditioned on a reference frume only and released concurrently with our method. Our framework supports both concept-only and first-frame modes.
A.4 MASK REFINEMENT ACROSS DENOISING STEPS AND LAYERS
Across both regimes (Tables 9 10 loU steadily improves with denoising steps, validating the iterative refinement strategy. Deeper layers (e.g.. layer 36) outperform shallow layers early on, while the combined mask yields the strongest mid/late-step loUs. Notably, high-motion sequences retain strong loUs and achieve competitive lip-syne (Syne-D 6.921), indicating that masks are not static and remain reliable even under larger motion (see the motion statistics in Table 11)
17

---

Published as a conference paper at ICLR 2026
Table 7: RAVDESS:
higher is better for IQA/ASEE/Snc-C; lower is better for FID.
Method
IQAT
ASE1
Sync-C1
FID
SadTalker (Zhang et al. Hallo (Xu ct al. 2024a
2023)
3.840
2.277
4.304
32.343
4.393
2.688
4.062
19.826
VExpress (Wang al.
2024a)
3.690
2.331
5.001
26.736
Echo.Mimic (Chen et al.
2024
4.504
2.742
3.292
21.058
Hallo-3 (Cui et 2024
4.006
2.462
4.448
28.840
Loopy (Jiag ei al. 20244)
4.506
2.658
4.814
17.017
OmniHuman (Lin 2025a)
4.564
2.815
5.255
16.970
Ours
4.602
2.915
5.132
17.187
Table 8: Qualitative capability comparison. V: supported; not supported.
Methods
Audio-driven full-body
First-frame 12V
Multi-ref images
Multi-person talking
Video-Alchemist (Chen al.
2025g
X
MasteptMaster (Huagg
Phantom (Liu et a. 2025
OmniHunan
Hunyuan Video=Avatar ben al.
2025b
MuliiTalk (Knng et
2025
Ours
A.5 EFFECT OF MASK CACHE
Using the cached mask from step t-1 to gate audio injection at step t markedly improves lip sync (Table [2 6.921 vs 11.046 Syne-D). Without caching, identity assignment degrades toward single-person behavior, confirming the necessity of cross-timestep spatial guidance.
Table 12: Mask cache significantly improves multi multi-person lip sync.
Condition
Sync-D4
With mask cache
6.921
Without mask cache
11.046
A.6 SCALING BEYOND THREE PEOPLE
Table 13 shows stable scaling to 4-5 speakers: Sync-D improves slightly (6.608) and AES increases (3.992) with marginal IQA change. This aligns with our claim that per-identity mask prediction is independent, enabling more entities without collapsing lip-sync quality.
Table 13: Stable performance when scaling to 4-5 subjects.
#People
Sync-DI
AES
IQAT
<3
6.670
3,467
4.757
4-5
6.608
3.992
4.738
B ALGORITHM OF OUR MODEL IMPLEMENTATION
Please refer Algorithm| for our model implementation.
18

---

Published as a conference paper at ICLR 2026
Table 9: Low motion strength: mask IoU across steps/layers; Sync-D for the setting.
step
step 10
step 20
step 30
step 50
Sync-D4
Mask layer 4
0.306
0.436
0.509
0,640
0.858
Mask layer 20
0.386
0.629
0.733
0.890
0.957
Mask layer 36
0.538
0.742
0.850
0.916
0,923
Combine mask
0.376
0.738
0.881
0.949
0.956
7.292
Table 10: High motion strength: mask IoU across steps/layers; Sync-D for the setting.
step
step 10
step 20
step 30
step 50
Sync-D4
Mask layer 4
0.113
0.426
0.529
0.730
0.779
Mask layer 20
0.527
0.816
0.911
0.934
0.945
Mask layer 36
0.694
0.902
0.923
0.931
0.915
Combine mask
0.741
0.916
0.932
0.936
0.937
6.921
C MORE DETAILS OF OUR MODEL
C.I LONG VIDEO GENERATION AND FRAME-ALIGNMEN DETAILS
We follow sliding-window strategies (Jiang et al. 2024a Tian et al. 2025b) by reusing several motion frames from the tail of window the head of window w-11 Reference injection is unchanged; starting frames are optional for window
Frame alignment flag. Per-frame masks have confidence scores from SAM2; frames with confi- dence 0.5 excluded from the mask loss (flow-matching still applies). Videos are retained if other frames have reliable masks, so supervision remains temporally dense.
C.2 Loss DESIGN AND AUDIO STACK CLARIHICATIOSS
Focal loss. We adopt focal loss with 0.25, 7 to mitigate class imbalance (background vs person) and stabilize early training; BCE converges more slowly and is less stable initially.
Audio features. Audio tokens come from wav2vec 2.0; we do not add extra transformer blocks before injection. Spatial attention unrestricted; temporally each latent attends to local +55 token window.
C.3 IMPLEMENTATION DETAILS
Our model was trained for 10,000 steps on 32 A800 GPUs with learning rate of 3e-5. We adopted the PyTorch framework combined with Fully Sharded Data Parallel (FSDP) to finetune the DiT model across multiple GPUs. Specifically, the model was partitioned such that different GPUs handled distinct portions of the model's parameters. We configured the effective batch size that every node-comppi cole--comrising 8 PrUss-prccsssed videos simultancously. With 32 GPUs total (i.c., nodes), this resulted in an overall effective batch size of 8 vidcos.
C.4 ABLATION DETAILS
In Tab. and Fig. of the paper, we validate our local audio injection designs via ablation on three variants: !) Global audio (Lin al. 2025a) applies audio across the entire feature map; 2) ID Embedding injects audio features with a learnable ID embedding without mask prediction; 3) Fixed Mask uses predefined static spatial masks.
For global audio, it is identical to the pretrained audio-driven model (Lin et al. 2025a). For fixed mask, it is manually input rectangle mask upon the audio cross-attention of pretrained audio-driven
19

---

Published as a conference paper at ICLR 2026
Table 11: Statisties used define motion regimes.
Statistic
Low
High
Body keypoints confidence
0.737
0.663
Max body kp moving dist (px)
21.268
216.269
Var body kp moving dist (px)
6.098
34.022
Head keypoints confidence
0.9856
0.9840
Max head kp moving dist (px)
2.258
34.201
Var head kp moving dist (px)
2.961
216.383
2025a). It requires the generated human being static and it needs the user to ensure there is only one person inside a mask.
model (Lin et al.
For ID) embedding, here we provide some implementation details. Inspired by detection transform- ers Carion et al., 2020), we introduce a set of N learnable ID embeddings E Equery E RNxC These embeddings serve as identity tokens for the N individuals. Specifically, we add the same learnable ID embedding Equery E IRC to paired reference image and audio segment. In this way, we expect the model to implicitly match an individual (defined by reference image) and audio segments in the output video to get synchronized lip movements. For the ID embedding dataset preparation, we incorporate multi-person videos into training. We use Active Speaker Detection (ASD) to extract talking persons' identities as a bounding box with identity-number. Then we use the bounding box and SAM2 masks to match the identity-numher of each timestep of audio based on their IoU. Finally we use paired audio with timestep and reference image to generate video, and use flow-matching loss only to supervised No mask information is provided the model and no mask supervision adopted.
In the main paper, our experimental results in ablation study indicate that implicit matching of individuals and audio segments is worse than explicit matching with aayout-aligned mask prediction and audio injection. implicit matching is not necessarily worse than explicit matching. We show that straight-forward implementation of implicit matching cannot generate satisfactory results, yet there could be better implementations to improve this result in the future.
is worth noting that although our implementation shows this evidence.
D DETAILS OF AUDIO-DRIVEN BASE MODEL'S ARCHITECTURE AND TRAINING
This appendix provides a detailed description of the network structure and training specifies for the pretrained audio-driven single-person video generation model, OmniHuman (Lin al.) (025a). As noted in our main paper, the text-to-video base model undergoes post-pretraining for audio conditioning, following the OmniHuman (Lin et al. (025a) methodology. Subsequently, our multi- concept framework is built and trained on this foundation.
D.1 AUDIO-DRIVEN BASE MODEL.
Our foundational model is composed of Variational Autoencoder (VAE) and a Latent Diffusion Transformer (DiT). The VAE includes an encoder, which compresses raw pixel data into a compact latent representation, and decoder, which reconstructs the original pixel inputs from these latent features. The VAE achieves compression ratios of (4, 8, 8) for the (t, h, w) dimensions, respectively. Both the encoder and decoder utilize temporally causal convolutional architecture, facilitating image and video compression actoss both spatial and temporal domains within the joint latent space. The denoising latents possess 16 channels.
The DiT Blocks are based on the dual-stream Diffusion Transformer (DiT) (Esser et al. 2024). This transformer processes video and text tokens through multiple self-attention layers and feedforward networks (FFNs) learn representations that are both shared and modality-specific. SwiGLU employed as the activation function to enhance nonlinear modeling capabilities. Additionally, AdaSingle (Chen al. 2023 used for efficient timestep modulation. Moreover, two-thirds of the
20

---

Published as a conference paper at ICLR 2026
FFN weights in the deeper layers share parameters, creating hybrid-stream design that preserves model capacity while substantially improving parameter efficiency.
D.2 MIXED TRAINING STRATEGY
We utilize a multi-condition training strategy based on the framework established in OmniHuman Lin et al. 2025ap. Our method employs two-stage progressive training scheme develop the base model's 's capabilities. In the initial stage, the model is trained solely on text-to-video (T2V) data build fundamental video generation abilities. The second stage introduces audio-synchronized datasets to expand the model's 's functionality to include audio-driven generation and reference image injection. For this second stage, we follow two core principles from OmniHuman: 1) Tasks with stronger conditioning can leverage tasks with weaker conditioning and their associated data to broaden the effective training dataset; 2) Tasks with stronger conditioning should be assigned proportionally lower training ratios. Guided by these principles, we first train the reference image injection capability before introducing the audio-driven generation objective. This staged methodology facilitates efficient knowledge transfer while ensuring stable training dynamics throughout the multi-condition learning process.
E ADDITIONAL DETAILS FOR AUDIO-DRIVEN BASE MODEL DATASET
E DATASET CURATION
We initially filter the T2V data using rules through Following this, we apply rule further refine the audio-driven data.
Video Clip. We begin by using PySceneDetect ( hwwps///wweebeeiooo///////o/naaa to identify and trim shot transitions and fades within video clips. After this process, all clips are standardized to a duration of 5 to 30 seconds.
2. Human. Utilizing the annotated video captions, we implement rule-based system to detect keywords such as "people", "human", "men", "women", "girl", and "boy". If any of these keywords found, the video is categorized as human-related. This technique ensures the dataset is effectively filtered to comprise only videos pertinent to human activities and interactions.
3. Subtitles. We use PaddleOCR hhtps://github.com/// http://ttuub..cmmaaalllccc htt.Paacllleaall//Puulll.oooo
to detect subtitles in the videos and remove clips where subtitles change. This step guarantees that the dataset emphasizes continuous and consistent visual content, reducing distractions from textual variations and improving the data quality for subsequent tasks.
Visual Quality. We utilize Q-align (Wu al. 2023a) evaluate the visual quality of the videos, filtering out clips that do not meet a predetined threshold. This procedure ensures the dataset maintains high standard of visual clarity, which is essential for accurate analysis and robust model performance. By eliminating low-quality segments, we enhance the overall dependability and utility of the dataset.
5. Aesthetics. We employ Q-align (Wu et al. 2023a) to assess the aesthetic appeal of the videos and discard clips falling below set threshold. Through aesthetic quality assessment, we can filter out videos containing post-prou post-production elements, thereby improving the quality of the training dataset. This measure ensures the dataset is composed natural and unaltered video content, which better represents real-world scenarios and boosts the robustness and generalization of the trained models.
Motion. We use Raft (Teed & Deng 2020) to calculate the optical flow the videos and filter out clips exhibiting overly intense motion. This step ensures that the dataset includes only video clips with moderate and significant motion, which are more appropriate for analysis and model training. By removing clips with extreme motion, we enhance the stability and quality of the dataset, leading to more dependable results.
7. Synenet. For audio-driven content, employ SyncNet (Chung & Zisserman 2017) determine if the lip movements are synchronized with the audio. Videos displaying considerable asynchromization excluded. This step ensures that the dataset consists only of high-quality, synchronized audio-
21

---

Published as a conference paper at ICLR 2026
visual data. By removing out-of-syne segments, improve the overall quality and reliability of the dataset. Ultimately, we gather 2,000 hours of audio-driven data.
E.2 DATASET ANALYSIS
To gain a better understanding of the dataset's distribution, we perform an analysis across three dimensions. Detailed definitions for each dimension are provided below.
Human Size. Human size indicates the portion of the human body visible within the video frame. It is categorized into the following levels: Portrait (head and shoulders). Chest (from head to chest), Waist (from head to waist), Knees (from head to knees), and Full Body (the entire body is visible). To determine this, first detect body keypoints using RTMMose (Jiang et al.) 2023). and then classify the human size based on the confidence scores of these keypoints. This method ensures robust and accurate categorization of the visible human body extent in each video.
Motion Amplitude. After extracting body keypoints from the video, the amplitude of human motion is computed by measuring the displacement of the chest keypoint over time, relative to the width the shoulders. Based on these calculations, we classify motion amplitude into four categories: Slight(< 0.1), Moderate(O.1 0.2), Significan((0.2 0.3), and Extreme(> 0.3).
Scene. Using video captioning, we employ Doubao to categorize videos into the following scenarios: Household, Work, Show, Outdoor Adventure, transportation, Arts and Crafts, Sports, and Others.
Language. For data containing audio, we use language detection tool to identify and count the types of languages
(https://github. www..//ooo/////////aaiiaaao present. These are categorized as English, Chinese, Spanish, French, German, Urdu, Hindi, and Others.
F USER STUDY DETAILS
Here we show our questionnaires used in the user study.
F.1 MULTI-PERSON AUDIO-DRIVER VIDEO GENERATION QUESTIONNAIRE
Please take 10 minutes complete the following ranking questions. For each question, consider the priority of the three videos based on the following dimensions and rank them accordingly:
(1) Lip-syne Accuracy: Based on the audio you hear, do you think the lip movements are accurate? Ideally, only one person's lip movements should correspond to the audio, but it doesn't 't matter specifically who is speaking. It is considered poor if multiple people are speaking simultaneously or if no speaking when there is audio. When multiple methods all correctly show only one person speaking, rank them based on the quality lip-syne with the audio. Perfect synchronization good; missing syllables incorrect lip shapes for certain syllables is poor.
(2) Sense of Dialogue Among Multiple People: there a feeling of conversation between the individuals? It is considered good if it portrays natural scenario where one person speaks and another listens. The judgment here is based on whether their expressions during the interaction appear natural and whether the overall feeling of the conversation is natural.
(3) Video Quality: If all the above criteria are tied, then prioritize the video with higher quality.
The priority of these three criteria decreases in the order listed. Aspect ratio and resolution should not be taken into account; you should not favor a particular aspect ratio or higher resolution, but rather focus the inherent video quality itself.
22

---

Published as a conference paper at ICLR 2026
F.2 MULTI-CONCEPT VIDEO GENERATION QUESTIONNAIRE
Please take minutes to complete the following ranking questions. For each question, consider the priority of the five videos from the perspective of consistency between the reference images and the appearance in the video, and rank them accordingly:
(I) Consistency with Reference Images: Based on the reference images, do you think all the listed reference images appear in the video? And does the appearance of those in the video match the images? Consider the priority in the following order from high low:
Ranked highest: Appearance is consistent, and all reference images appear.
Next: All reference images appear, but the appearance of some reference images is inconsistent.
Ranked lowest: Some reference images do not appear.
(2) Video Quality: If the above criteria are tied, then prioritize the video with higher quality.
(3) Motion Strength: all the above criteria are tied, then prioritize the video with a larger motion strength.
The priority of these criteria decreases in the order listed. Aspect ratio and resolution should not be taken into account; you should not favor particular aspect ratio higher resolution, but rather focus on the inherent video quality itself.
23

---

Published as a conference paper at ICLR 2026
Algorithm I InterActHumaan Audio-Driven Multi-Concept Video Generation Inference
Require:
Text prompt T', reference images {X;} identity-level audio {Xi)
Total diffusion steps S, mask injection threshold step Smask
VAE Encoder/Decoder, diffusion transformer
Ensure: Generated video V
1: Preprocessing:
2: for f- to N do
3: T, f- Rephrase(X, (X,, T)
Generate detailed prompt via Qwen2.5-V VLL Encode reference image. Extract audio features.
4: X; f VAE Encoder (X;)
5: a; wav2vec(Y;)
6: end for
7: Ctext -T,TT....... TN3
Aggregate conditions. Initialize the noisy video latent.
8: 10,
9: Initialize mask eache: {m. 0
10: for k < S downto 1 do
for each DiT block layer 7 that we inject conditions do for i to N do
12:
Reference Injection: Inject {x:} via concatenation and self-attention. Compute hidden video feature h" and reference feature h
14:
15:
Q" + Pror [K; V + Proje. (h,)).
16:
Apply LayerNorm and 3D RoPE to the features.
17:
Compute cross-attention
(K() T
p + softmax
V..
Va
18:
Predict layer mask:
mi (1) t sigmoid (M)P(D."))
end for
20:
end for
21:
A Aggregate Masks:
L mi Σ
m 1
Vie NJ.
22:
Cache current masks: {m. prev + {m:}. if Smask then for i 1 to N do
24:
Audio Injection: inject local audio by updating the noisy latent h" Q" +- Proj, Proj,(h"). (h"). [Kmute Viuute Projk, mure). ). Ki Vil Proj (a.)) (a:).
26:
27:
Compute cross-attention:
p < softmax (*((K))
Q" Q" mule
Vi:
male
+ softmax
Viuuee
Vd
Vd
Bind audio conditions:
h" 0 -m;) o mute
where O denotes element- wise multiplication.
29:
end for
30: end if
31: Update latent via diffusion step via flow-matching formulas or custom samplers.
32: end for
33: Decoding: Decode the final latent via VAE:
+ VAE. Decoder(z))
return V
24


Reviews:
[{"review_id": "ha2Q6d7IEA", "reviewer": "Reviewer_6miK", "summary": "This paper presents InterActHuman, a novel video diffusion framework for multi-concept human animation, addressing the limitations of existing global conditioning methods in multi-person scenarios. The key contribution is a method for enforcing strong, region-specific binding of local conditions, particularly aligning audio to the correct speaker. The model integrates a mask predictor into the diffusion pipeline to explicitly infer the spatiotemporal layout for each reference identity. It solves the problem of mask prediction during inference by adopting an iterative strategy, where the mask predicted at one step guides the local audio injection at the next. The authors also contribute a large-scale dataset of over 2.6 million video-entity pairs to facilitate this task.", "questions": "1: What does \"ID query\" in Figure 2 mean? Is it the tokens of ID 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: The framework introduces the capability for multi-person, audio-driven animation, correctly assigning distinct audio streams to specific individuals in the generated video.\n\n2: The paper proposes a practical iterative mask-caching strategy to solve the \"chicken-and-egg\" problem of local conditioning, using the mask predicted at step $k$ to guide the local audio injection at step $k+1$.\n\n3: Experiments show that the method significantly outperforms existing baselines (like Kling 1.6 w/ lip-sync and OmniHuman w/ fixed mask) on multi-person benchmarks, achieving state-of-the-art lip-sync (Sync-D) and video quality (FVD) metrics.", "weaknesses": "1: **Over-reliance on a Private, Curated Dataset.** The model is trained and evaluated on a new, large-scale dataset (2.6M pairs) curated by the authors. The multi-person test set is also newly collected by the authors. This makes it difficult to assess robustness and generalizability. Since the baselines were not trained on this specific, mask-annotated dataset, it's unclear if the performance gap is due to the model's architecture or its specialized training data, which may be perfectly tailored to its design.\n\n2: **Missing Ablation on Inference Hyperparameters.** The core inference strategy to solve the \"chicken-and-egg\" problem involves disabling mask-based injection for the \"first 10 steps\" because early masks are deemed \"unreliable\". This 10-step threshold is a critical, unexplained hyperparameter. The paper provides no ablation study to justify this specific number or to analyze the sensitivity of the model's performance (e.g., in Sync-D or FVD) to this \"warm-up\" period.\n\n3: **Lack of Failure Case Analysis for Mask Prediction.** The entire method for local conditioning hinges on the accuracy of the predicted mask. While the paper shows average IoU improving over denoising steps (Tables 8 & 9), it provides no qualitative analysis of failure cases.\n\nOverall, this paper proposes a new framework to generate multi-person videos with audio and has comprehensive experiments. Thus, I vote for acceptance at this time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ZN8SJDtYt", "reviewer": "Reviewer_SmPL", "summary": "This paper proposes InterActHuman, a video diffusion framework for multi-concept human animation, incorporating layout-aligned multimodal conditions such as local audio and image references for distinct entities. The approach introduces a mask-predictor module, iteratively integrated into the denoising steps of a diffusion transformer, to spatially align conditioning signals (audio/image) with their corresponding regions. Experiments and ablations on curated large-scale datasets reportedly demonstrate state-of-the-art performance in multi-person talking and human-object interaction video generation.", "questions": "In Line 228-229, what is the meaning of '!'; besides, 'the last few DiT blocks' should be specified to be reproduced.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides relatively comprehensive quantitative evaluations using mainstream avatar-related metrics.\n2. The authors offer detailed descriptions of how they collected and cleaned a large-scale dataset combining reference images, audio, per-frame masks, and captions for diverse multi-human/object interactions (Section 3.3), which could be a valuable resource for empirical studies if released publicly.", "weaknesses": "1. While using layout-based guidance for multi-concept conditioning is a reasonable design, the claimed novelty of this framework is questionable. Similar strategies are already standard in both multi-concept image and video generation. Even within avatar-related tasks, several prior works, including MultiTalk[1], have adopted comparable layout-guided conditioning. Moreover, dynamic layout prediction was first introduced in Ingredients[2], where it serves a clear purpose in ipt2v task. However, for avatar generation tasks, the spatial configuration of characters is typically known beforehand. In such cases, slightly loosening pre-defined masks can often achieve sufficient flexibility without introducing additional model complexity. The paper does not convincingly demonstrate that the proposed learnable mask predictor provides meaningful gains over these simpler, training-free alternatives.\n2. The experimental evaluation relies entirely on a self-collected, closed-source test set. This choice severely limits reproducibility and undermines the credibility of the reported improvements. Without standardized or publicly available benchmarks, it is difficult to assess whether the observed gains generalize beyond the authors’ specific data.\n3. The paper does not specify which base video diffusion model was used for initialization, nor does it provide details about training infrastructure (e.g., number of GPUs, total data scale, or training duration). More importantly, it remains unclear whether the base model natively supports audio conditioning or if such capability was newly introduced by the authors. Without this information, it is difficult to assess how much of the reported performance stems from the proposed mechanism itself versus the underlying backbone or large-scale training setup. Given that the paper reports notably strong results, this omission raises concerns about the fairness and interpretability of the comparisons. A more transparent description of the training setup and backbone configuration is essential to substantiate the claimed contributions.\n[1]. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. \n[2]. Ingredients: Blending Custom Photos with Video Diffusion Transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSbd19lBl1", "reviewer": "Reviewer_xdLa", "summary": "The paper proposes InterActHuman, a diffusion transformer based human video generation framework that aims to synthesize multi person or human object interaction videos in which each identity keeps its own appearance, motion style, and voice, given multiple reference images, a text description, and per speaker audio tracks. The core idea is to drop the usual single identity assumption and instead bind each conditioning signal to the correct spatial region over time. To do this, the model predicts for each reference concept a spatiotemporal mask via a lightweight cross attention head that matches the noisy latent video tokens to that concept. These masks are refined iteratively across denoising steps and then used to inject audio features only into the region of the speaker at the next step, which addresses the chicken and egg problem of needing masks to apply localized audio while the video is still being generated.", "questions": "1. How are individual audio streams associated with the correct identity during training. The paper explains that during inference, audio cross attention is only applied to tokens whose mask corresponds to that speaker, using cached masks from the previous denoising step.\n\n2. The data pipeline aligns audio segments to identities through lip synchronization and produces per-frame masks using Grounding SAM2 with a person query. Could you elaborate on how you ensure temporal identity consistency across frames in multi-person scenes, especially during occlusion or when two people have similar appearances? Do you rely on optical flow tracking or an ID re-identification module?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles a practically important and under explored setting: multi-person, audio-conditioned human animation where each identity must keep its own appearance and voice, instead of the common single identity assumption in prior audio-driven portrait or OmniHuman style models.\n\n2. The proposed iterative mask prediction and cached layout guided audio injection mechanism is elegant. By predicting per identity spatiotemporal masks using cross attention between reference appearance tokens and noisy video latents, then using the previous step mask to gate the next step audio cross attention, the model effectively solves the chicken and egg problem of aligning local audio before the final frames exist.\n\n3. Quantitative and qualitative results are compelling. The user study shows a strong preference for the proposed method in both lip sync realism and subject consistency.", "weaknesses": "1. Runtime cost and scalability claims are mostly deferred to the appendix. The main text asserts minimal overhead and compatibility with long video generation, but does not quantify inference speed or memory usage when conditioning on multiple identities.\n\n2. Multi speaker audio assignment appears to rely on injecting each audio stream only into the spatial region indicated by that speaker’s cached mask at inference time. It is unclear whether the model is ever explicitly trained on multi speaker scenes with multiple simultaneous audio streams, or whether this is essentially a zero shot composition of single speaker training. This could limit robustness when speakers interrupt each other or overlap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8uXpThag6L", "reviewer": "Reviewer_UaXd", "summary": "This paper proposes a human animation framework that synthesizes human-centric videos or images aligned with both audio tracks and text prompts, based on reference images of individuals and multimodal inputs such as audio. Specifically addressing the issue of temporal and identity misalignment between audio tracks and speaking movements of people in the video, the paper introduces a simple yet effective mask prediction mechanism. This mechanism progressively performs local audio condition injection during the denoising process, enabling dynamic audio-spatial alignment without requiring precise masks, thus avoiding incomplete or misaligned generation.", "questions": "The paper focuses on audio and image as the multimodal conditioning cue. Could the same framework, for instance, be adapted to accept additional signals, such as skeletal key-points, so that the generated humans would perform richer, visually matched motions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Per-identity control of multiple concepts is an urgent problem in human animation tasks; this paper offers a simple yet reliable solution that is not limited by the number of people in the video and shows strong scalability.  \n2. The local-audio-conditioning injection strategy is both novel and straightforward, and its effectiveness is verified by experiments and user studies.  \n3. The inference results are excellent; the supplementary videos are particularly impressive.  \n4. The paper explains its core contributions clearly, concisely, and without ambiguity.", "weaknesses": "1. Owing to the limited scope of the training data, the method’s ability to follow detailed instructions appears constrained; it is unclear whether this stems from a fundamental limitation in the approach or simply from the narrow range of scenarios covered by the dataset.\n2. The generated humans exhibit rather subdued body motion. Yet in real conversations, lively gestures are equally important. This could be a valuable direction for the community to explore in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)