Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation
Authors: Jiahao Cui
Hui Li
Yao Yao
Hao Zhu
Hanlin Shang
Kaihui Cheng
Hang Zhou
Siyu Zhu
Jingdong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: image animation
patch-drop augmentation
high-resolution
long-duration
Primary area: generative models

Abstract: Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities.First, we extend the method to produce long-duration videos. To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration.Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension. By integrating a high-quality decoder, we realize visual synthesis at 4K resolution.Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced ''Wild'' dataset. The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes.

Published as a conference paper at ICLR 2025
HALLO2: LONG-DURATION AND HIGH-RESOLUTION AUDIO-DRIVEN PORTRAIT IMAGE ANIMATION
Jiahao Cui Hui : Yao Yao Siyu Zhu't, T Jingdong Wang* FFudan University Baidu Inc. ELahaocui279@gmmn
Hao Zhu?, Hanlin Shang', Kaihui Cheng', Hang Zhou?
"Nanjing University
com, 24110240042 @m. fudan edu. en,
{yaoyao, @nj edu. en, 242102402840m. fudan. edu en, (etuii99991111 siyuzhuß fudan.
chengkaihuil 1999919886 11996126.com com, (zhouhang09, wangiingdorg@@baiduu com
ABSTRACT
Recent advances in latent diffusson--ased generative models for portrait image an- imation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements extend its capabilities. First, we extend the method to produce long-duration videos. To address substantial challenges such appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, introduce patch-drop tech- nique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration. Second, achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization latent codes and apply temporal alignment techniques to maintain coherence across the tempo- ral dimension. By integrating a high-quality decoder, we realize visual synthesis 4K resolution. Third, we incorporate adjustable semantic textual labels for por- trait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image anima- tions enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. The experimental results demonstrate that our approach achieves ooitte-ff-hh-att performance in long-duration portrait video an- imation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes.
INTRODUCTION
Portrait image animation- -he process of creating animated videos from reference portrait using various input signals such as audio PPajwal et al. (2020g Tian et af. (2024); Xu et al. (2024a); Zhang et al. (2023), facial landmarks Wei et al. (2024); Chen et al. (2024). textual descriptions Xu al.
(2024b)- is rapidly evolving field with significant potential across multiple domains. These do- mains include high-quality film and animation production, the development of virtual assistants, personalized customer service solutions, interactive educational content creation, and realistic char- acter animation in the gaming industry. Consequently, the capability generate long-duration high-resolution, audio-driven portrait animations, particularly those assisted by textual prompts. is crucial for these applications. Recent technological advancements, notably in latent diffusion mod- els, have significantly advanced this field.
Several methods utilizing latent diffusion models for portrait image animation have emerged in recent years. For instance, VASA-1 Xu et al. (2024b) employs the DiT model Peebles & Xie
'These authors contributed equally to this work. 1 Cooresponding Author

---

Published as a conference paper at ICLR 2025
Reference Image
Emotion Centrol
1 min
5 min
10 min
hour
Audio
(optional)
"This guy has
scrions expression"
Il his guy shows anger.
This guy feels sad.
Figure 1: Demonstration of the proposed approach. This approach processes a single reference im- age alongside an audio input lasting several minutes. Additionally, optional textual prompts may be introduced at various intervals to modulate and refine the expressions of the portrait. The resulting output is high-resolution 4K video that synchronizes with the audio and is influenced by the op- tional expression prompts, ensuring continuity throughout the extended duration of the video.
2023) a denoiser in the diffuusin process, converting single static image and audio segment into realistic conversational facial animations. Similarly, the EMO framework Tian et al. (024) represents the first end-to-end system capable of generating animations with high expressiveness
and realism, seamless frame transitions, and identity preservation using a U-Net-based diffusion model [Blattmann al. (2023) with only single reference image and audio input. Other signif- icant advancements in this domain include AniPortrait Wei et al. (2024). EchoMimic Chen et al. (2024), V-Express Wang et al. (2024a). Loopy Jiang et al.) (2024), and CyberHost [Lin et al. (2024), each contributing to enhanced capabilities and applications of portrait image animation. Hallo Xu (2024a), another notable contribution, introduces hierarchical audio-driven visual synthesis, building upon previous research to achieve facial expression generation, head pose control, and per- sonalized animation customization. In this paper. we present updates to Hallo Xu et al. (2024a) by introducing several design enhancements to extend its capabilities.
Firstly, we extend Hallo from generating brief, second-long portrait animations to supporting du- ration of up to tens of minutes. As illustrated in Figure 2 two primary approaches commonly employed for long-term video generation. The first approach involves generating audio-driven video clips in parallel, guided by control signals, and then applying appearance and motion constraints be- tween adjacent frames of these clips Wei et al. (2024); Chen et al. (2024). significant limitation of this method the necessity maintain minimal differences appearance and motion across gen- crated clips, which hampers substantial variations in lip movements, facial expressions, and poses, often resulting blurriness and distorted expressions and postures due the enforced continuity constraints. The second approach incrementally generates new video content by leveraging pre- ceding frames as conditional information Xu et al. (2024a); Tian al.) (2024); Wang et al.) (202T). While this allows for continuous motion, it prone to error accumulation. Distortions, deformations relative to the reference image, noise artifacts, or motion inconsistencies in preceding frames can propagate to subsequent frames, degrading the overall video quality.
To achieves high expressiveness, realism, and rich motion dynamics, we follow the second approach. Our method primarily derives the appearance from the reference image, utilizing preceding gener- ated frames solely to convey motion dynamics dynamicc--nncludee ieeelddingnnggg lip movements, facial expressions, and poses. To prevent contamination of appearance information from preceding frames, we implement a patch-drop data augmentation technique that introduces controlled corruption to the appearance information in the conditional frames while preserving motion characteristics. This approach en-
2

---

Published as a conference paper at ICLR 2025
Audio
Constraints
Diitiiser
Difliser
Noise
Audio
(c)
(d)
Figure 2: Comparison parallel and incremental diffusion-basee generative models for long-term portrait image animation. (a) Parallel generation. (b) Incremental generation. The parallel genera- tion approach may lead to blurriness (c) and distorted expressions (d), due to inter-frame continuity constraints. The incremental generation method is susceptible error accumulation in both back- grounds (e) and facial regions (f).
courages that the appearance is predominantly sourced from the reference portrait image, maintain- ing robust identity consistency throughout the animation and enabling long videos with continuous motion. Additionally, to enhance resilience against appearance contamination, we incorporate Gaus- sian noise as an additional data augmentation technique applied to the conditional frames, further reinforcing fidelity to the reference image while effectively utilizing motion information.
Secondly, to achieve 4K video resolution, we extend the Vector Quantized Generative Adversarial Network (VQGAN) Esser et al. (2021) discrete codebook space method for code sequence predic- tion tasks into the temporal dimension. By incorporating temporal alignment into the code sequence prediction network, we achieve smooth transitions in the predicted code sequences of the gener- ated video. Upon applying the high-quality decoder, the strong consistency in both appearance and motion allows our method enhance the temporal coherence of high-resolution details.
Thirdly, to enhance the semantic control of long-term portrait video generation, we introduce ad- justable semantic textual prompt for portrait expressions conditional inputs alongside audio sig- nals. By injecting textual prompts at various time intervals, our method can help to adjust facial expressions and head poses, thereby rendering the animations more lifelike and expressive.
To evaluate the effectiveness of our proposed method, we conducted comprehensive experiments on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. To the best of our knowledge, our approach is the first to achieve 4K resolution in portrait image animation for duration extending up to ten minutes or even several hours. Furthermore, by incorporating adjustable textual prompts that enable precise control over facial features during the generation process, our method ensures high levels realism and diversity in the generated animations.
2 RELATED WORK
Video Diffusion Models. Diffusion-based models have demonstrated remarkable capabilities in generating high-quality and realistic videos from textual and image inputs. Stable Video Diffu- sion [Blatmann et al. (2023) emphasizes latent video diffusion approaches, utilizing pretraining. fine-tuning, and curated datasets to enhance video quality. Make-A-Vides Singer et al.] (2022) leverages text-to-image synthesis techniques to optimize text-to-video generation without requir- ing paired data. Magic Video Zhou et al. (2022a) introduces an efficient framework with a novel 3D U-Net design, reducing computational costs. AiimateDifff Guo et al. (2023) enables animation of personalized text-to-image models via plug-and- as VideoComposer Wang et al. (2024b) and VideoCrafter Chen et al. (2023a), emphasize controlla- bility and quality in video generation. Cog VideoX Yang et al. (2024) enhances text-video alignment through expert transformers, and MagicTime Yuan et al. (2024) addresses the encoding of physical knowledge with metamorphic time-lapse model. Building upon these advancements, approach adopts superior pretrained diffusion models tailored specifically for portrait image animation, focus- ing on long-duration and high-resolution synthesis.
motion module. Further contributions, such
3

---

Published as a conference paper at ICLR 2025
Exooder
Spacial-Alintiinn
Ando-Altttion
Tesmua Contell
Deep Dava Aigmenaation
MLP
Attertins
Selt
MilP
Chea- Attertion
Image
Pace Encoder
Augmentatioon
High Resolation
Eshuncemene
Paschify
Tesnall Contool
Nake
Ap
Text Erasdler
Donoiiing
Video
Surplins
Acdio
Asdio
Gassiian
Spocil-Attttion
Audio-Attorion
Crookilationn
Toctal-Cortral
Tanpeml-Allaation
Concatenaioo
Figure 3: The framework of the proposed approach. The details of the proposed patch drop data augmentation is shown on the right side. Zret and znot denote the features of the reference image and motion frames, respectively. represents the latent features. Caudio and Ctext correspond to the features of the audio condition and textual prompts, respectively.
Portrait Image Animation. Significant progress has been made in audio-driven talking head gen- eration and portrait image animation, emphasizing realism and synchronization with audio inputs. Animate Anyone Hu et al. (2023) and MagicAnimate Xu et al.] (2024c) utilize reference network and temporal attention modules produce consistent and coherent human image animations. Lip- SyncExpert Prajwal et al. (2020) improved lip-sync accuracy using discriminators and novel evalu- ation benchmarks. Subsequent methods like Talker Zhang et al. (2033) and Vivid Sun et al. 2023) incorporated 3D motion modeling and head pose generation to enhance expressiveness and temporal synchronization. ReSyncer Guan et al.) (2024) revisited and rewired the btyle-based gen- erator to efficiently adopt 3D facial dynamics predicted by a principled style-injected Transformer. (hen
Diff Dife et al. (2023) and Dream Talk Ma et al. (2023) improved video quality while maintain- ing synchronization across diverse identities. VASA-1 Xu et al. (2024b) and AniTalker Liu et al. (2024) integrated nuanced facial expressions and universal motion representations, resulting in life- like and synchronous animations. AniPortrait Wei al. (2024). EchoMimic Chen et al.) (2024). V- Express Wang et al. (2024a), Loopy Jiang et al. (2024), CyberHost Lin et al. (2024), and EMO al. (2024) have contributed to enhanced capabilities, focusing on expressiveness, realism, and identity preservation. Despite these advancements, generating long-duration, high-resolution por- trait videos with consistent visual quality and temporal coherence remains a challenge. Our method builds upon Hallo Xu et al, (2024a) to address this gap by achieving realistic, high-resolution motion dynamics in long-term term portrait image animations.
Long-Tene Term and High- Resolutin Video Generation. Recent advances in video diffusion mod- els have significantly enhanced the generation long-duration, high-resolution videos. Frame- works like Flexible Diffusion Modeling Harvey et al. (2022) and Gen-L-Videoo Harvey et al. (2022) improve temporal coherence and enable text-driven video generation without additional training- Methods such as SEINE Chen et (2023b) and StoryDiffusion Zhou et al. (2024) introduce gen- crative transitions and semantic motion predictors for smooth scene changes and visual storytelling. Approaches like Streaming T2V Henschel et al. (2024) and MovieDreamer Zhao et al. (2024) use autoregressive strategies and diffusion rendering for extended narrative videos with seamless transi- tions. In this paper, we employ patch-drop and Gaussian noise augmentation to enable long-duration portrait image animation.
Discrete prior representations with learned dictionaries have proven effective for image restora- tion. VQ-VAE Razavi et al.) (2019) enhances VAEs by introducing discrete latent spaces via vector quantization, addressing posterior collapse. Building on this, VQ-GAN Lee et al. (2022) combines CNNs and Transformers to create context-rich vocabulary of image components, achieving state- of-the-art results. CodeFormer Zhou et al. (2022b) uses a learned discrete codebook and employs Transformer-based network for enhanced robustness against degradation. This paper proposes vector quantization latent codes with temporal alignment techniques to maintain high-resolution coherence temporally for 4K synthesis.
4

---

Published as a conference paper at ICLR 2025
Temporal Aligament
CodeBook
Self Attention
HQ Decoder
Latent Feature
xF
Option
Option
Decoder
LOQ Encoder
Diffuser
Diffuser
Latent Fealure
Latent l'eature
The illustration of the proposed high-resolution enhancement module. Two alternative Figure 4:
designs for extracting input latent features demonstrated.
METHOD
In this section, we introduce an extended technique for portrait image animation that effectively addresses the challenges of generating long-duration, high-resolution videos with intricate motion dynamics, as well as enabling audio-driven and textually prompted control. Our proposed method derives the subject's appearance primarily from single reference image while utilizing preceding generated frames as conditional inputs to capture motion information. To preserve appearance de- tails of the reference image and prevent contamination from preceding frames, introduce patch drop data augmentation technique combined with Gaussian noise injection (see Section Addi- tionally, extend the VQGAN discrete codebook prediction into the temporal domain, facilitating high-resolution video generation and enhancing temporal coherence (see Section 3.2).. Furthermore, integrate textual conditions alongside audio signals to enable diverse control over facial pressions and motions during long-term video generation (see Section 3.3). Finally, we detail the network structure along with the training and inference strategies in Section 3.4
3.1 LONG-DURATION ANIMATION
Patch-Drop Augmentation. To generate long-duration portrait videos that maintain consistent ap- pearance while exhibiting rich motion dynamics, we introduce patch drop data augmentation tech- nique applied to the conditioning frames. The core idea is corrupt the appearance information in preceding frames while preserving their motion cues, thereby ensuring that the model relies primar- ily on the reference image for appearance features and utilizes preceding frames to capture temporal dynamics.
Let Tef denotes the reference image, and let I I--N] represent the preceding gen- erated frames at time steps t 1 to N. To mitigate the influence of appearance information from preceding frames, we apply a patch drop augmentation to each frame It-i. for = 1,2.. N. Specifically, each frame is partitioned into K non-overlapping patches of size p p, yielding (1 where indexes the patches. For each patch, binary mask M((A) generated as fol- lows: A1 L, if (1)) 2 r: (M 0, if ((() r. Here g(k) u((0, 1) is uniformly distributed random variable, and E [0, 1] the patch drop rate controlling the probability of retaining each patch. The augmented frame I, is then constructed by applying the masks to the corresponding patches: (() = (k) (k) for k = 1,2...., K. This random omission of patches effectively disrupts detailed appearance information while preserving the coarse spatial structure necessary for modeling motion dynamics.
Gaussian Noise Augmentation. During the incremental generation process, previously generated video frames may introduce contamination in both appearance and dynamics, such as noise in facial regions and the background, or subtle distortions in lip movements and facial expressions. As this process continues, these contaminations can propagate to subsequent frames, leading to the gradual accumulation and amplification of artifacts. mitigate this issue, we incorporate Gaussian noise into the motion frames, enhancing the denoiser's ability in the latent space to recover from contam- inations in appearance and dynamics. Specifically, we introduce Gaussian noise to the augmented latent representations: 2,-; - + 7h-i- 7h- A/ I where controls the noise level, and denotes the identity matrix. The corrupted latent representations {2,-1, 1. 2, 2--2 2,- are then used motion condition inputs the diffusion model.
5

---

Published as a conference paper at ICLR 2025
Method
FID.
FVD.
Sync-Ci
Syac-D]
E-FID
AcBuliaal
Aedio[Head
41.753
246.041
8,051
7.117
10.190
SadTeleer
21,924
293.084
7.399
7.812
6.881
EchoMianic
47.331
532.733
5,930
9.143
11,051
AniPortrait
26.241
361,978
3,912
10.264
11,253
Hallo
16.748
366, 366,066
7.268
7.714
7.081
Ours
16.616
239.5117
7.379
7.697
6.702
Real video
8,377
6,809
Table 1: The quantitative comparisons with existed portrait image animation approaches on the HDTF dataset. Our evaluation focuses on generated videos with duration of 4 minutes, maintaining consistent settings across subsequent quantitative experiments.
Figure 5: FID metrics of different methods as inference time increases.
These augmmeted motion frames incorporated into the diffusion process via attss-att mechanisms within the denoising U-Net. At each denoising step t. the model predicts the noise component 60(2,, 1, c), where 2, the current noisy latent, and c represents the set of conditioning inputs: c 60(2,, 1, c) = Eo (Z,, f, Zref, {2--i), Caudio: Crea) ) Here, Zie = (Ther) is the latent representa- tion of the reference image, and Candio. Ctext are the encoded audio features and textual embeddings, respectively, By leveraging the noise-augmented motion frames, the model effectively captures tem- poral dynamics while mitigating the influence of accumulated artifacts. This approach encourages that the subject's appearance remains stable, derived from the reference image, throughout the gen- erated video sequence.
3.2 NIGH-RESOLUTION ENHANCEMENT
To enhance temporal coherence in high-resolution video generation, we adopt codebook prediction approach ZZho et al. 2022c), incorporating an introduced temporal alignment mechanism.
[hou
As illustrated in Figure we propose two methods for extracting input latent features. The first method directly uses latent features from the diffusion model. While straightforward, this approach requires end-to-end training of the entire super-resolution module. Alternatively, the second method processes the latent features through the diffusion model's ss decoder, followed by a low-quality en- coder that maps them into continuous latent space. This method leverages pretrained encoder, decoder, and codebook components, thereby simplifying the training process. Subsequently, code index prediction transformer module is then employed predict the codebook index and select corresponding features from it, which are fed into a high-quality decoder to generate the super- resolution video. In order to keep the temporal consistency of super-resolution video, grate the temporal alignment within the code index prediction transformer module. Specifically, let Xyelr E R (I--)--CCCCC denotes the hiddden state following the self-attention layer in the code predic- tion transformer module, where N represents the frame number, H/ and W correspond the height and width dimensions, and C' denotes the channel dimension. Subsequently, we reshape X into the shape of (H W) x N C and apply conventional scaled dot-product attention. By integrat- ing temporal alignment, the network effectively captures intra-frame and inter- frame dependencies, enhancing both temporal consistency and visual fidelity in high-resolution video outputs.
inte-
3.3 TEXTUAL PROMPT CONTROL
To enable precise modulation of facial expressions and motions based on textual instructions, we incorporate an adaptive layer normalization mechanism into the denoising U-Net architecture. Given text prompt, a text embedding etext is extracted using the CLIIP text encoder Radford et al. (2021) This embedding is processed through a zero-initialized multilayer perceptron (MLP) produce scaling (7) and shifting (5) parameters: 7, MLP(eex.)
The adaptive layer normalization applied between the cross-attention layer and the audio attention layer within the denoising U-Net. Specifically, the intermediate features X from the cross- attention layer are adjusted as follows: X = LayerNorm (X Xadapes = @ X + + X where @ denotes element- wise multiplication. This adaptation conditions the denoising process on the textual input, enabling fine- ggained control over the synthesized expressions and motions in the generated video frames.

---

Published as a conference paper at ICLR 2025
Ampoont
SeeThaleer
ELS
Avehoal beet
Hollo
Refersmos image
Ouas
Figure 6: The qualitative comparison with exited approaches HDTF data-set.
3.4 NETWORK
Network Architecture. Figure the proposed approach's 's architecture. The ReferenceNet embeds the reference image Zeet, capturing the visual appearance of both the portrait and the cor- responding background. To model temporal dynamics while mitigating appearance contamination from preceding frames, the motion frames {&--i} } are subjected to patch dropping and Gaussian noise augmentation. Our extended framework utilizes denoising U-Net architecture that processes noisy latent vectors Z, at each diffusion timestep t. The embedding of the input audio Caudio is de- rived from 12-layer wav2vec network Schneider
(019)), while the textual prompt embedding Crext is obtained through CLIP Radford et al. (2021) By synthesizing these diverse conditioning inputs via gross-attention layers within the denoising U-Net (latttann et al. (2023), the model gen- erates frames that maintain visual coherence with the reference image while dynamically exhibiting nuanced and expressive lip motions and facial expressions. Finally, the high-resolution enhance- ment module employs vector quantization of latent codes in conjunction with temporal alignment techniques to produce final videos at 4K resolution.
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Implementation. All experiments were conducted on a GPU server equipped with 8 NVIDIA A100 GPUs. The training process was executed in two stages: the first stage comprised 30,000 steps with a step size of 4, targeting a video resolution of 512 x 512 pixels. The second stage involved 28,000 steps with batch size of 4, initializing the motion module with weights from Animatediff. Approximately 160 hours of video data were utilized across both stages, with learning rate set at le-5. For the super-resolution component, training for temporal alignment was extended to 550,000 steps, leveraging initial weights from CodeFormer and a learning rate of 1e-4, using the VFHQ dataset as the super-resolution training data. Each instance in the second stage generated 16 video frames, integrating latents from the motion module with the first ground truth frames, designated as motion frames. During inference, the output video resolution is increased to a maximum of 4096 4096 pixels.
Regarding textual control, we employed the vision- language model MiniCPM Hu et al. (2024) to generate textual prompts. These prompts were refined using Llama 3.1, which extracted expression and emotion descriptions from the original captions. The final textual prompt was formatted as (human) (expression)). During training, we randomly dropped the textual prompt condition with

---

Published as a conference paper at ICLR 2025
Method
FID
FVD.
Sync-C1
Sync-D]
E-FIDI
6.269
8.325
38.981
Patch size
Drup
FID
IVDL
Sync-C]
Sync-D]
SadTalker
24,000
380.366
6.384
8.169
44.596
1
0.50
39.642
513.314
6.687
8,515
EchoMimic
50.994
854.836
5,082
9.675
35.806
38.518
AniPeetrait
301
544 000
3,975
10.171
41.307
2 0.20
38.477
492.216
6.762
8.413
Halla
991
8.156
36.793
4
0.30
37.756
44.172
756.517
198.957
6.754
6,431
8,371
8.517
7,088
7.726
Table 3: The quantitative comparison of different settings patch drop augmentation. The statistics are obtained on the CelebV dataset.
Table 2: The quantitative comparisons with existed approaches on the proposed "Wild"
data-set.
Ref. Image
Audio2Head
SadTalker
EchoMimic
AmiPotrait
Hallo
Ours
GT
Figure 7: The qualitative comparison with existed approaches on the proposed "Wild" data-set.
a probability of 0.05. During inference, the classifier-free guidance (CFG) scale was uniformly set to 3.5 for the textual prompt, the reference image, and the audio. When comparing against other baseline methods, we used null textual prompt as the condition.
Baseline Approaches. We evaluate our framework against leading state-of-ther art techniques, in- cluding both non-diftusion and diffusion-based models. Non-diffusion models, such as Aedio2Head and SadTalker, are compared with diffusion- counterparts like EchoMimic, AniPortrait, and Hallo. Notably, EchoMimic and AniPortrait employ a parallel generation approach for long- durationn outputs, while Hallo utilizes an incremental formulation. Unlike previous studies that focused on short-duration videos of only a few seconds, our evaluation is conducted on generated videos lasting 4 minutes, using looped audio from the benchmark dataset the driving audio. To ensure fair com- parison, we have excluded the high-resolution enhancement module, maintaining the same output video resolution (512 512 pixels) as the existed approaches across all quantitative comparisons.
4.2 COMPARISON WITH STATE-OF-THE-ART
Comparison on HDTF Dataset. Table D and Figure 6 present quantitative and qualitative com- parisons on the HDTF dataset. Our framework achieves the lowest FID of 16.616 and E-FID of 6.702, demonstrating superior fidelity and perceptual quality. Additionally, our synchroniza- tion metrics, Sync-C (7.379) and Sync-D (7.697), further validate the effectiveness of our method. As illustrated in Figure []] the extended inference duration significantly impacts FID metrics in existing diffusion-basee approaches, leading to notable declines compared to their short-duration performance. In terms of lip and expression motion synchronization, parallel methods such as EchoMinic and AniPortrait exhibit marked deterioration. In contrast, our extended approach con- sistently demonstrates superior and stable performance across image and video quality, as well as motion synchronization, even as inference time increases.
Comparison on the Proposed "Wild" Dataset. Table and Figure offers additional quantitative and qualitative comparison results of the introduced "Wild" dataset. Our method achieves an FID of 24.072 and E-FID of 33.316, both indicative of high image quality. We also register Sync-C
8

---

Published as a conference paper at ICLR 2025
drop
Syne-C]
SyneDD
Reference Image
Media Fromes
F10.
FVD.
Symo-C"
Synce
80.715
37,961
477,122
6.928
8.307
8.387
Table 4: Ablation study of the patch drop and Gaussian noise augmentation the CelebV data-set.
Table 5: Analysis of different combinations of patch augmentations applied to the reference image and motion frames.
(a)
(b)
Effectiveness of Gaussian noise augmentation: (a) No augmentation is used for motion frames. (b) Gaussian noise augmentation is introduced to the motion frames.
Figure 8:
(d)
Figure 9: Different combinations of patch augmentation for the reference image and motion frames are as follows: (a) No patch-drop augmentation is used for either the reference or motion frames. (b) Patch-drop augmentation is introduced only to the reference image. (c) Patch-drop augmentation is added to both the reference image and motion frames. (d) Patch-drop augmentation applied only to the motion frames.
score of 6.760 and Sync-D Men--Dor of 8. 1,65,, alongside the highest FVD of 360. demonstrating superior coherent video structure.
4.3 ABLATION STUDIES AND DISCUSSION
Different Patch Drop Sizes and Rate. Table presents a comparative analysis of varying patch sizes and drop rate applied to motion frames. Our results indicate that using patch sizes of 1-4 pixels with appropriate patch drop augmentation ratios of 20%-30% yields comparable performance sync metrics (Sync-C and Sync-D) and image and video quality metrics (FID and FVD). Too large patch size too large drop rate not only destroys the appearance details of motion frames, but also damages the facial motion of motion frames and reduces the quality of the generated video, resulting in high FID and FVD scores.
lip
Effectiveness of Augmentation Strategies. Table evaluate different augmentation strategies. Gaussian noise alone results in high FID of 82.715 and FVD of 1088.158, indicating subopti- mal quality. As shown in Figure 8 without Gaussian noise augmentation, errors tend to accumulate in the background over time. contrast, applying this augmentation strengthens the model's ability to suppress these етrоrs, resulting in clearer background. The patch drop strategy significantly improves these metries, reducing FID to 38.518 and FVD 491.338, as it effectively suppresses artifacts in portrait appearances, illustrated in Figure 9 Notably, the combined strategy further en- hances performance, achieving the lowest FID of 37.944 .44 and FVD 477.412, alongside the highest Sync-C score of 6.928. Thus, the combined augmentation method proves to be the most effective in generating high-quality motion frames.
To assess the effectiveness of patch augmentation. we apply it to the reference and motion im- ages, as presented in Table and Figure 9 Our experiments demonstrate that without the proposed augmentation, the generated portraits (Figure I()) exhibit numerous artifacts. Augmenting only the reference image contaminates its identity features (see Figure 9b) because noise accumulates across the motion frames. Similarly, applying augmentation both the reference image and motion

---

Published as a conference paper at ICLR 2025
Method
LPIPS
FID.
FVD.
MUSIQT
MagVTTv2
0.2647
91.46
425.68
0.6841
ESRGAN
0.2532
88.78
418.06
0.7033
Ours w/ HRE
0.2310
66.92
360.71
0.7187
Table 6:
Quantitative evaluations of the super-resolution results from different methods. HRE represents the proposed high-resolution enhancement.
Figure 10: Comparison of the portrait image ani- mation results with and without high-resolution en- hancement.
Prompt: "this guy shows anger."
Prompt: "this is happy."
Prompt: "the feels sad."
Prompt: "the man has neutral expression."
Before
Afler
Figure 11: Comparison of portrait animation before and after applying the textual prompts.
frames introduces noise into the portraits (Figure c)) due to the compromised reference image. To address this issue, we exclusively augment the motion frames, aiming for them to provide only facial motion while the reference image supplies the identity appearance. This approach as shown in Figure d) preserves the identity appearance of the reference image while effectively capturing the facial motion.
Effectiveness of High-Resolution Enhancement. The effectiveness of high-resolution enhance- ment techniques is illustrated in Figure I0 which demonstrates improved animation quality via video super-resolution. We compare the proposed super-resolution method with previous models, specifically ESRGAN Wang et al. (2018) and MagVITv2 Yu et al. (2023). The corresponding statis- ties are provided in Table 6 The results indicate that although the introduced super-resolution ap- proaches enhance visual details measured by MUSIQ, our proposed high-reso high-resolution enhancement (HRE) achieves superior video quality, as evidenced by better FVD scores. This improvement is due to the incorporation of the temporal alignment module, which enhances temporal coherence.
Effectiveness of Textual Prompt. The integration of textual prompts into our portrait image an- imation framework significantly enhances the control over generated animations, illustrated in Figure [1 The comparative analysis demonstrates that textual prompts facilitate precise manipu- lation of facial expressions and emotional nuances, allowing for more tailored animation output. By providing explicit instructions regarding desired emotional states, the model exhibits improved responsiveness in generating animations that align closely with the specified prompts.
5 CONCLUSION
This paper presents advancements in portrait image animation through the enhanced capabilities of the Hallo framework. By extending animation durations to of minutes while maintaining high- resolution 4K output, our approach addresses significant limitations of cxisting methods. Specif- ically, innovative data augmentation techniques, including patch-drop and Gaussian noise, ensure robust identity consistency and reduce appearance contamination. Furthermore, we implement vec- tor quantization of latent codes and employ temporal alignment techniques to achieve temporally consistent 4K videos. Additionally, the integration of audio-driven signals with adjustable seman- tie textual prompts enables precise control over facial expressions and motion dynamics, resulting in lifelike and expressive animations. Comprehensive experiments conducted on publicly available datasets validate the effectiveness of our method, representing a significant contribution to the field of long-duration, high-resolution portrait image animation.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
This project is sponsored by Natural Science Foundation of Shanghai under Grant No. 242R1407200 and Shanghai Oriental Talents Project under Grant No. QNKJ2024060.
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Haoxin Chen, Menghan Xia, Yingging He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310. 195/2, 2023a.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations, 2023b.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Life- like audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08116 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Tamiigg transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, 2021.
Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, et al. Resyncer: Rewiring style-based generator for unified audio-visually synced facial performer. In European Conference on Computer Vision, pp. 348-367. Springer, 2024.
Yuwei Guo, Ceyuan Yang. Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 00fT::2307.04755. 2023.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flex- ible diffusion modeling of long videos. Advances Neural Information Processing Systems (NeurIPS), 35:27953-27965. 2022.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.1 2403.14773.. 2024.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang. and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:231..77117, 2023.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404. 063955 2024.
Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint 00X2v:2409.026344, 2024.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11 11532, 2022.

---

Published as a conference paper at ICLR 2025
Gaojie Lin, Jianwen Jiang. Chao Liang, Tianyun Zhong, Jiaqi Yang, and Yanbo Zheng. Cyber- host: Taming audio-driven avatar diffusion model with region codebook attention. arXiv preprint arXiv:2409.01877 2024.
Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: Animate vivid and diverse talking faces through deentity-deooplee facial motion encoding. arXiv preprint arXiv:2405.033 03121, 2024.
Yifeng Ma. Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:23/2.09767, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205, 2023.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. In Proceedings the 28th ACM international conference on multimedia, pp.
2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning (ICML), pp. 8748-8763, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Steffen Schneider, Alexci Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint 1xXiv:194.008800 2019.
Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li. Zheng Zhu. Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1992-1991, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without tixt-video data. In The Eleventh International Conference on Learning Representations (ICLR), 2022.
Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Dai- heng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive- alive-generatie generating pressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint 1724452402.17485. 2024.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprini arXiv:2406.0051// 2024a.
S Wang, L Li, Y Ding, C Fan, and X Yu. Audio2head: Audio-drivan one-shot talking-head gen- eration with natural head motion. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 1098-1105, 2021.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion trollability. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024b.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 0-0, 2018
12

---

Published as a conference paper at ICLR 2025
Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-drived synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403. 17694, 2024.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024a.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang. Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404. 2024b.
Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1481-1490, 2024c.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-tovvideo diffusion models with an expert transformer. arXiv preprint 47X10:2408.06072, 2024.
Lijun Yu, José Lezama, Nitesh В Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo. Magictime: Time-lapse video generation models as metamorphic simulators. arXiv preprint arXiv:2404.05014 2024.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8652-8661, 2023.
Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024.
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11016 2022a.
Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. Advances in Neural Information Processing Sys- tems, 35:30599-30611, 2022b.
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022c.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01434, 2024.
13


Reviews:
[{"review_id": "oLy6e6CThJ", "reviewer": "Reviewer_VuDm", "summary": "This paper presents a pipeline for generating long-duration, high-resolution, audio-driven facial animations. To enable extended video generation, the authors introduce patch-drop and Gaussian noise augmentation techniques during training. For high-resolution output, the method incorporates a VQ-based super-resolution stage and temporal alignment attention to ensure temporal consistency. Additionally, a norm-based textual prompt control mechanism is used to enhance the expressiveness of facial animations.\n\nExperiments across multiple datasets demonstrate the effectiveness of the proposed approach.", "questions": "- For patch augmentation, could the authors analyze the “effective augmentation ratio”? Since the purpose of patch augmentation is to encourage the model to disregard appearance information and focus on motion, an analysis of which patches contain motion information and which contain appearance information, along with a calculation of the effective augmentation ratio for the video clip, would provide valuable insight into the effectiveness of this design.\n    \n- How is the textual condition used during inference? Since it serves as an additional conditioning factor, the authors should clarify how this information is labeled, details of training behavior (e.g., random dropping for CFG), and how CFG is applied during inference. It would also be useful to include an analysis of the impact of conflicts between non-textual and text-audio conditions.\n    \n- On the HDTF dataset (i.e., Table 1), the proposed method outperforms most methods except Audio2Head. Could the authors provide an explanation for this? The paper should include further explanation of the evaluation metrics used to help readers understand which capabilities are being measured and improved.\n    \n- In the third image on the bottom right side of Figure 6, there appear to be “water droplet-like” artifacts. Could the authors explain the cause of these artifacts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- An effective augmentation strategy  which do not need to change model structure and support long-duration\n    \n- The textual condition injection propose a novel method to inject text prompt to a reference-net based animation structure.", "weaknesses": "- For high-resolution enhancement, the paper lacks a comparison with video VQ models (e.g., MagVITv2) and does not include quantitative evaluations of the super-resolution results from the SR encoder-decoder.\n    \n- For textual prompt control, the paper does not provide a detailed description of how prompts are used in comparisons with baseline methods or how prompts are incorporated during training data preparation. Additionally, details regarding the CFG process for managing multiple conditions (e.g., text, audio, reference image) are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QIbrnSK9X", "reviewer": "Reviewer_MzNt", "summary": "This paper proposes Hallo2 for audio-driven face image animation, which mainly includes three training strategies. Firstly, this paper proposes a patch-drop data augmentation technique to prevent contamination of appearance information from preceding frames. Secondly, this paper extends the Vector Quantized Generative Adversarial Network discrete codebook space method for code sequence prediction tasks into the temporal dimension. Finally, this paper also introduces additional textual prompts for semantic control.", "questions": "Please see above. If the author solves my problems, I will consider raising the score. Thanks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper discusses how to generate high-quality animated videos driven by audio, which is a very interesting topic.\n2. The structure of this paper is well-organized and easy to follow. \n3. The experimental results show the effectiveness of the proposed method.", "weaknesses": "There are some questions and suggestions,\n1. This paper proposes three strategies to improve the performance of audio-driven face image animation; however, the motivation for each strategy is not well explained. For example, in Section 3.2, a codebook prediction approach is used to enhance temporal coherence. It would be helpful if the authors provided more insights into why temporal coherence is insufficient without this codebook prediction approach.\n2. This paper presents three distinct strategies; however, the connections between each contribution are weak, making it difficult for reviewers to grasp the central contribution. This lack of coherence also gives the impression that each contribution is only incremental.\n3. From lines 238 to 241, the author uses a masking strategy to mask frames in the video. The author claims that this approach improves consistency. However, I’m a bit puzzled. In video generation, the masking strategy is generally beneficial for generating long videos autoregressively, so why would masking improve consistency? Could you please give more insights?\n4. Some descriptions in the HIGH-RESOLUTION ENHANCEMENT section are a bit confusing. In the first half of this section, it seems that a Transformer is used to model the video in a continuous latent space. However, Figure 4 actually shows a codebook. Since it appears the authors are using a codebook, I suggest reorganizing the first half of this section to reflect this.\n5. I observed flickering in some of the videos in the experimental results, especially with the man wearing a checkered shirt. Could this be due to limitations in the pre-trained VAE? Given that there is temporal alignment in the HIGH-RESOLUTION ENHANCEMENT section, why hasn’t this temporal alignment helped reduce the flickering effect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLzieVvbb8", "reviewer": "Reviewer_a9qA", "summary": "This paper proposes the first method to generate an hour-long, audio-driven portrait image animation at 4K resolutions coupled with textual prompts. To accomplish this goal, the paper applies patch-drop augmentation on Gaussian noise, a vector quantization on latent codes with a high-quality decoder, and uses enhanced textual labels as the conditional inputs. These improvements constitute the proposed contributions.", "questions": "Please carefully address the above questions. I think there are many details that need to be clarified, particularly in the individual components constituting the proposed method.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "The paper offers several contributions that are deemed to be the strengths of this paper. This work is an extension of a previous work on Hallo, thus termed Hallo2. It still attempts to propose a few enhancements that can further address the limitations of the previous work, mainly by extending it to long-hour portrait image animation with high resolutions. \n\nThe patch-drop data augmentation looks interesting. It intentionally adds corruption to the appearance of the conditional frames while preserving the motion information. Meanwhile, the appearance contamination is resolved using the Gaussian noise as the other data augmentation. \n\nThe 4K resolution was achieved by leveraging the VQ-GAN and code sequence prediction (existing but applied to a new task) to ensure smooth transitions among the temporal frames. \n\nTherefore, the main strengths of this paper would lie in the patch-drop augmentation, gaussian noise augmentation and the code sequence prediction.", "weaknesses": "The main idea of patch-drop data augmentation is to separate the motion from the appearance, where the former is controlled by temporal dynamics in the preceding frames. The patch-augmentation was applied to individual latent images z_{t-i}, where it was further partitioned into the K non-overlapping patches. Each patch was randomly masked. The question here is if we generate random masking and apply that to the time step t-1, would the same masking positions be enforced with the t-2 to t-N? Or the t-2 to t-N time steps still use a random masking operation. Could you clarify that and also show the motivation or the merits of each design? The other question is that it makes sense the omission of patches can still preserve the coarse structure. The motion dynamics on the other hand are not affected by the masking operations. Could you provide more details on why this is the case? \n\nAnother intriguing part is related to the choice of patch size for dividing the latent image. Based on the results given in Table 3, the best patch size is 1*1, which means that a single pixel is dropped. This seems to contradict the intuition of using a patch. I don't mean 1 pixel is not a patch, but it does not conform to the normal choice of patch size. Could you offer more insights on why 1 pixel is considered to be more effective in this task? This would be very important. \n\nRegarding the preceding generated frames, which approach was used? Somehow, I am not able to find that detail in the paper, or maybe I miss it. Was that achieved by generating the second frame based on the first one, and then continually generating the third frame based on the first two frames? \n\nThe design of Gaussian noise augmentation is simple and easy to understand, which was implemented by adding the Gaussian noise to the augmented latent representations. This was used as one of the conditional inputs. Could you provide more insights on why such an operation works? Ensuring temporal consistency is difficult. The paper seems to offer a much simpler and non-complex solution. Therefore, it would be more necessary to provide stronger validations on such a design. \n\nThe set of conditional inputs used in this work includes four parts, which are the audio, textual, noise-augmented motion frames, and reference image encoding. Based on the flowchart given in Figure 3, the cross-attention conditions occur in different layers of the encoder and decoder of the U-Net. Could you justify how to select the corresponding layer for the designated condition? Are they all implemented via cross-attention? \n\nThe high-resolution enhancement is still unclear to me. The paper does not seem to apply the super-resolution module but relies on the output from the diffusion model’s decoder and then a low-quality decoder. The technical part description in Sec 3.2 looks more like normal QKV operations. How the 4K resolution was achieved is still unclear, nor it was depicted in Figure 3. \n\nThe other issue might be related to the over-claim of the proposed approach by saying the \"hour-long\". The evaluation was done on minutes duration (e.g., 4 minutes in evaluation). More evidences are required for this \"hour-long\" claim. \n\nThe textual prompt implementation looks just like a regular conditional injection. This paper claims it to be one of the strengths of this work. \n\nThe paper needs to add some more comparisons such as the EMO work. This has been mentioned in the Introduction but appears to be not compared in the experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFgjaISs5z", "reviewer": "Reviewer_rhVg", "summary": "This work presents an enhanced version of the prior work Hallo. This work improves upon the previous model in terms of long-term video generation, 4K high-resolution video generation, and semantic textual label condition. This work is the first one to study 4K audio-driven portrait image animation. Experiment results show that this method achieves state-of-the-art performance in long-term portrait video animation, which can generate rich and controllable 4K content of tens of minutes.", "questions": "No questions, please see the weaknesses for rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This is the first work which attempts to animate portrait videos under 4K resolution. The qualitative results shown in the supplementary material demonstrate the improvements against the baseline.\n2. The one hour length portrait video animation is interesting. The Gaussian noise augmentation technique is effective in alleviating the error accumulation issue in the autoregressive long video animation process.\n3.  The textual prompt condition helps maintain the emotion and expression of the talking person, which is simple yet effective.", "weaknesses": "1. This work is a combination of existing works. The autoregressive animation pipeline has already been explored in EMO. The noise simulation trick is commonly used in the autoregressive generation models to alleviate error accumulation. The 4K resolution generation is an extension of VQGAN in the audio-driven portrait animation. Therefore, this work is a combination of prior works from the engineering perspective, which is not novel or significant enough.\n2. Compared with Hallo, the animation framework remains unchanged and is quite similar to prior works like EMO. The reference unet borrows ideas from human image animation but the authors didn’t cite them properly. Please consider cite [1,2] in the final version.\n3. The long video results shown in the supplementary do not contain too many translation and rotation of the talking head, which cannot demonstrate the temporal consistency for long-term animation. It is recommended to conduct such an experiment with diverse translation and rotation of the talking head to prove the effectiveness of this proposed method.\nThe ablation of HRE is not comprehensive enough. Comparisons with the 512x512 model cannot show the necessity of developing such a VQGAN-based superresolution in the pipeline. It is recommended to compare the proposed method with previous superresolution models, such as ESRGAN[3].\n\n[1] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation\n[2] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model\n[3] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)