Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

## Abstract

Abstract Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis.
In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities.
First, we extend the method to produce long-duration videos.
To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration.
Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension.
By integrating a high-quality decoder, we realize visual synthesis at 4K resolution.
Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs.
This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content.
To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts.
We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced “Wild” dataset.
The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes.
Project page: https://fudan-generative-vision.github.io/hallo2

## 1 Introduction

Portrait image animation—the process of creating animated videos from a reference portrait using various input signals such as audio Prajwal et al. (); Tian et al. (); Xu et al. (); Zhang et al. (), facial landmarks Wei et al. (); Chen et al. (), or textual descriptions Xu et al. ()—is a rapidly evolving field with significant potential across multiple domains.
These domains include high-quality film and animation production, the development of virtual assistants, personalized customer service solutions, interactive educational content creation, and realistic character animation in the gaming industry.
Consequently, the capability to generate long-duration, high-resolution, audio-driven portrait animations, particularly those assisted by textual prompts, is crucial for these applications. Recent technological advancements, notably in latent diffusion models, have significantly advanced this field.

Several methods utilizing latent diffusion models for portrait image animation have emerged in recent years.
For instance, VASA-1 Xu et al. () employs the DiT model Peebles & Xie () as a denoiser in the diffusion process, converting a single static image and an audio segment into realistic conversational facial animations.
Similarly, the EMO framework Tian et al. () represents the first end-to-end system capable of generating animations with high expressiveness and realism, seamless frame transitions, and identity preservation using a U-Net-based diffusion model Blattmann et al. () with only a single reference image and audio input.
Other significant advancements in this domain include AniPortrait Wei et al. (), EchoMimic Chen et al. (), V-Express Wang et al. (), Loopy Jiang et al. (), and CyberHost Lin et al. (), each contributing to enhanced capabilities and applications of portrait image animation.
Hallo Xu et al. (), another notable contribution, introduces hierarchical audio-driven visual synthesis, building upon previous research to achieve facial expression generation, head pose control, and personalized animation customization.
In this paper, we present updates to Hallo Xu et al. () by introducing several design enhancements to extend its capabilities.

Firstly, we extend Hallo from generating brief, second-long portrait animations to supporting duration of up to tens of minutes.
As illustrated in Figure 2, two primary approaches are commonly employed for long-term video generation.
The first approach involves generating audio-driven video clips in parallel, guided by control signals, and then applying appearance and motion constraints between adjacent frames of these clips Wei et al. (); Chen et al. ().
A significant limitation of this method is the necessity to maintain minimal differences in appearance and motion across generated clips, which hampers substantial variations in lip movements, facial expressions, and poses, often resulting in blurriness and distorted expressions and postures due to the enforced continuity constraints.
The second approach incrementally generates new video content by leveraging preceding frames as conditional information Xu et al. (); Tian et al. (); Wang et al. ().
While this allows for continuous motion, it is prone to error accumulation.
Distortions, deformations relative to the reference image, noise artifacts, or motion inconsistencies in preceding frames can propagate to subsequent frames, degrading the overall video quality.

Figure: Figure 1: Demonstration of the proposed approach. This approach processes a single reference image alongside an audio input lasting several minutes. Additionally, optional textual prompts may be introduced at various intervals to modulate and refine the expressions of the portrait. The resulting output is a high-resolution 4K video that synchronizes with the audio and is influenced by the optional expression prompts, ensuring continuity throughout the extended duration of the video.
Refer to caption: extracted/5925503/fig/teaser.jpg

To achieves high expressiveness, realism, and rich motion dynamics, we follow the second approach.
Our method primarily derives the appearance from the reference image, utilizing preceding generated frames solely to convey motion dynamics—including lip movements, facial expressions, and poses.
To prevent contamination of appearance information from preceding frames, we implement a patch-drop data augmentation technique that introduces controlled corruption to the appearance information in the conditional frames while preserving motion characteristics.
This approach encourages that the appearance is predominantly sourced from the reference portrait image, maintaining robust identity consistency throughout the animation and enabling long videos with continuous motion.
Additionally, to enhance resilience against appearance contamination, we incorporate Gaussian noise as an additional data augmentation technique applied to the conditional frames, further reinforcing fidelity to the reference image while effectively utilizing motion information.

Secondly, to achieve 4K video resolution, we extend the Vector Quantized Generative Adversarial Network (VQGAN) Esser et al. () discrete codebook space method for code sequence prediction tasks into the temporal dimension.
By incorporating temporal alignment into the code sequence prediction network, we achieve smooth transitions in the predicted code sequences of the generated video.
Upon applying the high-quality decoder, the strong consistency in both appearance and motion allows our method to enhance the temporal coherence of high-resolution details.

Thirdly, to enhance the semantic control of long-term portrait video generation, we introduce adjustable semantic textual prompt for portrait expressions as conditional inputs alongside audio signals.
By injecting textual prompts at various time intervals, our method can help to adjust facial expressions and head poses, thereby rendering the animations more lifelike and expressive.

To evaluate the effectiveness of our proposed method, we conducted comprehensive experiments on publicly available datasets, including HDTF, CelebV, and our introduced “Wild” dataset.
To the best of our knowledge, our approach is the first to achieve 4K resolution in portrait image animation for duration extending up to ten minutes or even several hours.
Furthermore, by incorporating adjustable textual prompts that enable precise control over facial features during the generation process, our method ensures high levels of realism and diversity in the generated animations.

Figure: (a) Parallel Generation
Refer to caption: extracted/5925503/fig/ParallelGeneration.jpg

## 2 Related Work

Video Diffusion Models.
Diffusion-based models have demonstrated remarkable capabilities in generating high-quality and realistic videos from textual and image inputs Hu et al. (); Zhu et al. (); Zhang et al. ().
Stable Video Diffusion Blattmann et al. () emphasizes latent video diffusion approaches, utilizing pretraining, fine-tuning, and curated datasets to enhance video quality.
Make-A-Video Singer et al. () leverages text-to-image synthesis techniques to optimize text-to-video generation without requiring paired data.
MagicVideo Zhou et al. () introduces an efficient framework with a novel 3D U-Net design, reducing computational costs. AnimateDiff Guo et al. () enables animation of personalized text-to-image models via a plug-and-play motion module.
Further contributions, such as VideoComposer Wang et al. () and VideoCrafter Chen et al. (), emphasize controllability and quality in video generation.
VideoComposer integrates motion vectors for dynamic guidance, while VideoCrafter offers open-source models. CogVideoX Yang et al. () enhances text-video alignment through expert transformers, and MagicTime Yuan et al. () addresses the encoding of physical knowledge with a metamorphic time-lapse model.
Building upon these advancements, our approach adopts superior pretrained diffusion models tailored specifically for portrait image animation, focusing on long-duration and high-resolution synthesis.

Figure: Figure 3: The framework of the proposed approach. The details of the proposed patch drop data augmentation and textual prompt control are shown on the right side. $c_{\mathtt{ref}}$ and $c_{\mathtt{mot}}$ refer to the feature of reference image and motion frames.
Refer to caption: extracted/5925503/fig/overview.png

Portrait Image Animation.
Significant progress has been made in audio-driven talking head generation and portrait image animation, emphasizing realism and synchronization with audio inputs.
LipSyncExpert Prajwal et al. () improved lip-sync accuracy using discriminators and novel evaluation benchmarks.
Subsequent methods like SadTalker Zhang et al. () and VividTalk Sun et al. () incorporated 3D motion modeling and head pose generation to enhance expressiveness and temporal synchronization.
Diffusion-based techniques have further advanced the field. DiffTalk Shen et al. () and DreamTalk Ma et al. () improved video quality while maintaining synchronization across diverse identities.
VASA-1 Xu et al. () and AniTalker Liu et al. () integrated nuanced facial expressions and universal motion representations, resulting in lifelike and synchronous animations.
AniPortrait Wei et al. (), EchoMimic Chen et al. (), V-Express Wang et al. (), Loopy Jiang et al. (), CyberHost Lin et al. (), and EMO Tian et al. () have contributed to enhanced capabilities, focusing on expressiveness, realism, and identity preservation.
Despite these advancements, generating long-duration, high-resolution portrait videos with consistent visual quality and temporal coherence remains a challenge.
Our method builds upon Hallo Xu et al. () to address this gap by achieving realistic, high-resolution motion dynamics in long-term portrait image animations.

Long-Term and High-Resolution Video Generation.
Recent advances in video diffusion models have significantly enhanced the generation of long-duration, high-resolution videos.
Frameworks like Flexible Diffusion Modeling Harvey et al. () and Gen-L-Video Harvey et al. () improve temporal coherence and enable text-driven video generation without additional training.
Methods such as SEINE Chen et al. () and StoryDiffusion Zhou et al. () introduce generative transitions and semantic motion predictors for smooth scene changes and visual storytelling.
Approaches like StreamingT2V Henschel et al. () and MovieDreamer Zhao et al. () use autoregressive strategies and diffusion rendering for extended narrative videos with seamless transitions.
Video-InfinityTan et al. () optimizes long video synthesis through distributed inference, while FreeLong Lu et al. () integrates global and local video features without training for consistency.
In this paper, we employ patch-drop and Gaussian noise augmentation to enable long-duration portrait image animation.

Discrete prior representations with learned dictionaries have proven effective for image restoration.
VQ-VAE Razavi et al. () enhances VAEs by introducing discrete latent spaces via vector quantization, addressing posterior collapse, and enabling high-quality image, video, and speech generation.
Building on this, VQ-GAN Lee et al. () combines CNNs and Transformers to create a context-rich vocabulary of image components, achieving state-of-the-art results in conditional image generation.
CodeFormer Zhou et al. () uses a learned discrete codebook for blind face restoration, employing a Transformer-based network for enhanced robustness against degradation.
This paper proposes vector quantization of latent codes with temporal alignment techniques to maintain high-resolution coherence temporally for 4K synthesis.

## 3 Preliminaries

### 3.1 Latent Diffusion Models

Latent Diffusion Models (LDMs), introduced by Rombach et al. (), represent a significant advancement in generative modeling by conducting diffusion and denoising processes within a compressed latent space rather than directly in the high-dimensional image space.
This approach substantially reduces computational complexity while maintaining the quality of generated images.

Specifically, a pre-trained Variational Autoencoder (VAE) Kingma & Welling () is employed to encode input images into lower-dimensional latent representations.
Given an input image $\mathbf{I}$, the encoder $\mathcal{E}(\cdot)$ maps it to a latent vector: $\mathbf{z}_{0}=\mathcal{E}(\mathbf{I})$.
A forward stochastic diffusion process Sohl-Dickstein et al. (); Ho et al. (); Song et al. () is then applied to the latent vector $\mathbf{z}_{0}$, adding Gaussian noise over $T$ time steps to produce a sequence of noisy latent variables $\{\mathbf{z}_{t}\}_{t=1}^{T}$.
The process is defined by:

$$ $\mathbf{z}_{t}=\sqrt{\bar{\alpha}_{t}}\,\mathbf{z}_{0}+\sqrt{1-\bar{\alpha}_{t }}\,\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$ (1) $$

where $t\in\{1,2,\ldots,T\}$ denotes the diffusion steps, $\alpha_{t}=1-\beta_{t}$ with $\beta_{t}\in(0,1)$ being the variance schedule, and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$ is the cumulative product of $\alpha_{t}$.
As $t$ approaches $T$, the distribution of $\mathbf{z}_{T}$ converges to a standard normal distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$ due to the accumulated noise.

The reverse diffusion process aims to reconstruct the original latent vector $\mathbf{z}_{0}$ by sequentially denoising $\mathbf{z}_{T}$. At each timestep $t$, a noise prediction network $\bm{\epsilon}_{\theta}$, typically parameterized using a U-Net architecture Ronneberger et al. (), estimates the noise component in $\mathbf{z}_{t}$ using optional conditioning information $\mathbf{c}$. The network is trained to minimize the expected mean squared error between the true noise $\bm{\epsilon}$ and the predicted noise $\bm{\epsilon}_{\theta}$:

$$ $\mathcal{L}=\mathbb{E}_{\mathbf{z}_{0},\mathbf{c},\bm{\epsilon},t}\left[\omega (t)\left\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathbf{c}) \right\|_{2}^{2}\right],$ (2) $$

where $\omega(t)$ is a weighting function that balances the loss contribution across different timesteps.

Once trained, the model can generate new samples by starting from a random Gaussian latent vector $\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and iteratively applying the denoising process:

$$ $\mathbf{z}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{z}_{t}-\frac{1- \alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\,\bm{\epsilon}_{\theta}(\mathbf{z}_{t}, t,\mathbf{c})\right)+\sigma_{t}\,\mathbf{n},\mathbf{n}\sim\mathcal{N}(\mathbf{ 0},\mathbf{I}),$ (3) $$

for $t=T,T-1,\ldots,1$, where $\sigma_{t}$ is the standard deviation of the noise added at step $t$. The final latent vector $\mathbf{z}_{0}$ is then decoded to reconstruct the image: $\mathbf{I}=\mathcal{D}(\mathbf{z}_{0})$,
where $\mathcal{D}(\cdot)$ is the decoder of the Variational Autoencoder (VAE).

### 3.2 Incorporating Motion Conditions via Cross-Attention

Incorporating conditioning information is crucial for controlling the generative process in latent diffusion models. Cross-attention mechanisms Vaswani () are employed to effectively integrate motion conditions into the model. The attention layers process both the noisy latent variables $\mathbf{z}_{t}$ and the embedded motion conditions $\mathbf{c}$ to guide the denoising process.
The cross-attention operation is formulated as:

$$ $\text{CrossAttn}(\mathbf{z}_{t},\mathbf{c})=\text{softmax}\left(\mathbf{Q} \mathbf{K}^{\top}/\sqrt{d_{k}}\right)\mathbf{V},$ (4) $$

where $\mathbf{Q}=\mathbf{W}_{Q}\mathbf{z}_{t}$, $\mathbf{K}=\mathbf{W}_{K}\mathbf{c}$ and $\mathbf{V}=\mathbf{W}_{V}\mathbf{c}$ are the queries;
$\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ are learnable projection matrices;
and $d_{k}$ is the dimensionality of the keys.
The softmax function ensures that the attention weights sum to one, focusing on the most relevant components of the conditioning information.
By integrating cross-attention into the denoising network, the model dynamically adjusts its focus based on the current latent state and the provided conditions. This mechanism enables the generation of images that are coherent with the conditioning inputs, enhancing the expressiveness and realism of the animated portraits.

In our work, the motion conditions $\mathbf{c}$ include the reference image embedding $\mathbf{c}_{\text{image}}$, audio features $\mathbf{c}_{\text{audio}}$, and textual embeddings $\mathbf{c}_{\text{text}}$ obtained via Contrastive Language-Image Pretraining (CLIP) Radford et al. (). The combination of these modalities allows for nuanced control over facial expressions, lip movements, and head poses in the generated animations.

## 4 Method

Figure: Figure 4: The illustration of the proposed high-resolution enhancement module. Two alternative designs for extracting input latent features are demonstrated.
Refer to caption: extracted/5925503/fig/overview_module.png

In this section, we introduce an extended technique for portrait image animation that effectively addresses the challenges of generating long-duration, high-resolution videos with intricate motion dynamics, as well as enabling audio-driven and textually prompted control.
Our proposed method derives the subject’s appearance primarily from a single reference image while utilizing preceding generated frames as conditional inputs to capture motion information.
To preserve appearance details of the reference image and prevent contamination from preceding frames, we introduce a patch drop data augmentation technique combined with Gaussian noise injection (see Section 4.1).
Additionally, we extend the VQGAN discrete codebook prediction into the temporal domain, facilitating high-resolution video generation and enhancing temporal coherence (see Section 4.2).
Furthermore, we integrate textual conditions alongside audio signals to enable diverse control over facial expressions and motions during long-term video generation (see Section 4.3).
Finally, we detail the network structure along with the training and inference strategies in Section 4.4.

### 4.1 Long-Duration Animation

Patch-Drop Augmentation.
To generate long-duration portrait videos that maintain consistent appearance while exhibiting rich motion dynamics, we introduce a patch drop data augmentation technique applied to the conditioning frames.
The core idea is to corrupt the appearance information in preceding frames while preserving their motion cues, thereby ensuring that the model relies primarily on the reference image for appearance features and utilizes preceding frames to capture temporal dynamics.

Let $\mathbf{I}_{\text{ref}}$ denote the reference image, and let $\{\mathbf{I}_{t-1},\mathbf{I}_{t-2},\dotsc,\mathbf{I}_{t-N}\}$ represent the preceding $N$ generated frames at time steps $t-1$ to $t-N$.
To mitigate the influence of appearance information from preceding frames, we apply a patch drop augmentation to each frame $\mathbf{I}_{t-i}$, $\text{for }i=1,2,\dotsc,N$.
Specifically, each frame is partitioned into $K$ non-overlapping patches of size $p\times p$, yielding $\{\mathbf{I}_{t-i}^{(k)}\}_{k=1}^{K}$, where $k$ indexes the patches. For each patch, a binary mask $M_{t-i}^{(k)}$ is generated as follows:

$$ $M_{t-i}^{(k)}=\begin{cases}1&\text{if }\xi^{(k)}\geq r\\ 0&\text{if }\xi^{(k)}<r\end{cases}$ (5) $$

Here $\xi^{(k)}\sim\mathcal{U}(0,1)$ is a uniformly distributed random variable, and $r\in[0,1]$ is the patch drop rate controlling the probability of retaining each patch.

The augmented frame $\tilde{\mathbf{I}}_{t-i}$ is then constructed by applying the masks to the corresponding patches:

$$ $\tilde{\mathbf{I}}_{t-i}^{(k)}=M_{t-i}^{(k)}\cdot\mathbf{I}_{t-i}^{(k)},\; \text{for }k=1,2,\dotsc,K.$ (6) $$

This random omission of patches effectively disrupts detailed appearance information while preserving the coarse spatial structure necessary for modeling motion dynamics.

Gaussian Noise Augmentation.
During the incremental generation process, previously generated video frames may introduce contamination in both appearance and dynamics, such as noise in facial regions and the background, or subtle distortions in lip movements and facial expressions.
As this process continues, these contaminations can propagate to subsequent frames, leading to the gradual accumulation and amplification of artifacts.
To mitigate this issue, we incorporate Gaussian noise into the motion frames, enhancing the denoiser’s ability in the latent space to recover from contaminations in appearance and dynamics.
Specifically, we introduce Gaussian noise to the augmented latent representations:

$$ $\hat{\mathbf{z}}_{t-i}=\tilde{\mathbf{z}}_{t-i}+\bm{\eta}_{t-i},\bm{\eta}_{t-i }\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I}),$ (7) $$

where $\sigma$ controls the noise level, and $\mathbf{I}$ denotes the identity matrix.
The corrupted latent representations $\{\hat{\mathbf{z}}_{t-1},\hat{\mathbf{z}}_{t-2},\dotsc,\hat{\mathbf{z}}_{t-N}\}$ are then used as motion condition inputs to the diffusion model.

These noise-augmented motion frames are incorporated into the diffusion process via cross-attention mechanisms within the denoising U-Net. At each denoising step $t$, the model predicts the noise component $\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathbf{c})$, where $\mathbf{z}_{t}$ is the current noisy latent, and $\mathbf{c}$ represents the set of conditioning inputs:

$$ $\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathbf{c})=\bm{\epsilon}_{\theta} \left(\mathbf{z}_{t},t,\mathbf{z}_{\text{ref}},\{\hat{\mathbf{z}}_{t-i}\}, \mathbf{c}_{\text{audio}},\mathbf{c}_{\text{text}}\right).$ (8) $$

Here, $\mathbf{z}_{\text{ref}}=\mathcal{E}(\mathbf{I}_{\text{ref}})$ is the latent representation of the reference image, and $\mathbf{c}_{\text{audio}}$, $\mathbf{c}_{\text{text}}$ are the encoded audio features and textual embeddings, respectively.
By leveraging the noise-augmented motion frames, the model effectively captures temporal dynamics while mitigating the influence of accumulated artifacts.
This approach encourages that the subject’s appearance remains stable, derived from the reference image, throughout the generated video sequence.

### 4.2 High-Resolution Enhancement

To enhance temporal coherence in high-resolution video generation, we adopt a codebook prediction approach Zhou et al. (), incorporating an introduced temporal alignment mechanism.

Given the generated video frames, we first encode them using a fixed encoder $\mathcal{E}$ to obtain latent representations $\mathbf{z}\in\mathbb{R}^{N\times H\times W\times C}$, where $N$ denotes the number of frames, while $H$, $W$, and $C$ represent the height, width, and number of channels, respectively.
Each Transformer block comprises a spatial self-attention layer followed by a temporal alignment layer. The operations of the spatial self-attention layer are defined as follows.
Let $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ be learnable projection matrices. Given the input $\mathbf{z}$ to this Transformer block, we compute the queries, keys, and values as follows:

$$ $\mathbf{Q}_{\text{self}}=\mathbf{W}_{Q}\mathbf{z},\;\;\mathbf{K}_{\text{self}} =\mathbf{W}_{K}\mathbf{z},\;\;\mathbf{V}_{\text{self}}=\mathbf{W}_{V}\mathbf{z}.$ (9) $$

Subsequently, the output of the spatial self-attention layer, denoted as $\mathbf{X}_{\text{self}}$, is computed using the softmax function:

$$ $\mathbf{X}_{\text{self}}=\text{Softmax}\left({\mathbf{Q}_{\text{self}}\mathbf{ K}_{\text{self}}^{T}}/{\sqrt{d_{k}}}\right)\mathbf{V}_{\text{self}}+\mathbf{z},$ (10) $$

where $d_{k}$ is the dimensionality of the keys.
Following this, the hidden state $\mathbf{X}_{\text{self}}\in\mathbb{R}^{N\times(H\cdot W)\times C}$ is reshaped into $\mathbf{X}_{\text{temp}}\in\mathbb{R}^{(H\cdot W)\times N\times C}$ to facilitate temporal attention across frames:
$\mathbf{X}_{\text{temp}}=\text{ReshapeToTemporal}(\mathbf{X}_{\text{self}})$.
In this context, let $\mathbf{W}^{\prime}_{Q}$, $\mathbf{W}^{\prime}_{K}$, and $\mathbf{W}^{\prime}_{V}$ be additional learnable projection matrices. The queries, keys, and values for the temporal alignment layer are computed as follows:

$$ $\mathbf{Q}_{\text{temp}}=\mathbf{W}^{\prime}_{Q}\mathbf{X}_{\text{temp}},\;\; \mathbf{K}_{\text{temp}}=\mathbf{W}^{\prime}_{K}\mathbf{X}_{\text{temp}},\;\; \mathbf{V}_{\text{temp}}=\mathbf{W}^{\prime}_{V}\mathbf{X}_{\text{temp}}.$ (11) $$

The output of the temporal attention mechanism, denoted as $\widetilde{\mathbf{X}}_{\text{temp}}$, is computed similarly:

$$ $\widetilde{\mathbf{X}}_{\text{temp}}=\text{Softmax}\left({\mathbf{Q}_{\text{ temp}}\mathbf{K}_{\text{temp}}^{T}}/{\sqrt{d_{k}}}\right)\mathbf{V}_{\text{ temp}}+\mathbf{X}_{\text{temp}}.$ (12) $$

Finally, $\widetilde{\mathbf{X}}_{\text{temp}}$ is reshaped back to the original dimensions of $\mathbf{z}\in\mathbb{R}^{N\times H\times W\times C}$:

$$ $\mathbf{z}=\text{ReshapeBack}(\widetilde{\mathbf{X}}_{\text{temp}}).$ (13) $$

As shown in Figure 4, we propose two implementations for extracting input latent features.
The first approach directly utilizes latent features from the diffusion model for the super-resolution module, which, while simple, requires end-to-end training of the entire module.
The second approach processes latent features through the diffusion model’s decoder and then a low-quality decoder, necessitating only the training of a lightweight temporal alignment module.
Given the sparsity of super-resolution video data, the second approach demonstrates superior performance under limited training conditions.

By integrating spatial and temporal attention mechanisms within the Transformer module, the network effectively captures intra-frame and inter-frame dependencies, enhancing both temporal consistency and visual fidelity in high-resolution video outputs.

### 4.3 Textual Prompt Control

To enable precise modulation of facial expressions and motions based on textual instructions, we incorporate an adaptive layer normalization mechanism into the denoising U-Net architecture. Given a text prompt, a text embedding $\mathbf{e}_{\text{text}}$ is extracted using the CLIP text encoder Radford et al. (). This embedding is processed through a zero-initialized multilayer perceptron (MLP) to produce scaling ($\gamma$) and shifting ($\beta$) parameters:
$\gamma,\beta=\mathrm{MLP}(\mathbf{e}_{\text{text}}).$

The adaptive layer normalization is applied between the cross-attention layer and the audio attention layer within the denoising U-Net. Specifically, the intermediate features $\mathbf{X}_{\text{cross}}$ from the cross-attention layer are adjusted as follows:
$\mathbf{X}_{\text{norm}}=\mathrm{LayerNorm}(\mathbf{X}_{\text{cross}}),\mathbf
{X}_{\text{adapted}}=\gamma\odot\mathbf{X}_{\text{norm}}+\beta+\mathbf{X}_{
\text{cross}},$
where $\odot$ denotes element-wise multiplication. This adaptation conditions the denoising process on the textual input, enabling fine-grained control over the synthesized expressions and motions in the generated video frames.

Figure: Figure 5: The quantitative comparisons with existed portrait image animation approaches on the HDTF dataset. Our evaluation focuses on generated videos with a duration of 4 minutes, maintaining consistent settings across subsequent quantitative experiments.
Refer to caption: extracted/5925503/fig/fid.png

Figure: Figure 7: The qualitative comparison with exited approaches on HDTF data-set.
Refer to caption: extracted/5925503/fig/comparison_hdtf.jpg

### 4.4 Network

Network Architecture.
Figure 3 illustrates the proposed approach’s architecture.
The ReferenceNet embeds the reference image $\mathbf{z}_{\text{ref}}$, capturing the visual appearance of both the portrait and the corresponding background.
To model temporal dynamics while mitigating appearance contamination from preceding frames, the motion frames $\{\hat{\mathbf{z}}_{t-i}\}$ are subjected to patch dropping and Gaussian noise augmentation. Our extended framework utilizes a denoising U-Net architecture that processes noisy latent vectors $\mathbf{z}_{t}$ at each diffusion timestep $t$.
The embedding of the input audio $\mathbf{c}_{\text{audio}}$ is derived from a 12-layer wav2vec network Schneider et al. (), while the textual prompt embedding $\mathbf{c}_{\text{text}}$ is obtained through CLIP Radford et al. ().
By synthesizing these diverse conditioning inputs via cross-attention layers within the denoising U-Net Blattmann et al. (), the model generates frames that maintain visual coherence with the reference image while dynamically exhibiting nuanced and expressive lip motions and facial expressions.
Finally, the high-resolution enhancement module employs vector quantization of latent codes in conjunction with temporal alignment techniques to produce final videos at 4K resolution.

Training.
This study implements a two-stage training process aimed at optimizing distinct components of the overall framework.

In the initial stage, the model is trained to generate video frames using a reference image, input-driven audio, and a target video frame.
During this phase, the parameters of the Variational Autoencoder (VAE) encoder and decoder, as well as those of the facial image encoder, are held constant.
The optimization process focuses on the spatial cross-attention modules within both the ReferenceNet and the denoising U-Net, with the objective of enhancing the model’s capabilities for portrait video generation.
Specifically, a random image is selected from the input video clip to serve as the reference image, while adjacent frames are designated as target images for training purposes.
Additionally, motion modules are introduced to improve the model’s temporal coherence and smoothness.

In the second stage, patch drop and Gaussian noise augmentation techniques are applied to the motion frames to train the model for generating long-duration videos characterized by temporal coherence and smooth transitions.
This stage refines the modeling of temporal dynamics by incorporating corrupted motion frames into the conditioning set, thereby enhancing the model’s ability to capture motion continuity over extended sequences.
Concurrently, textual prompts are utilized at this stage to facilitate precise modulation of facial expressions and motions based on textual instructions.
For the super-resolution model, the parameters of the VAE encoder are optimized, with a focus on refining the weights responsible for codebook prediction.
Temporal alignment is employed within the Transformer-based architecture to ensure consistency and high-quality outputs across frames, thereby enhancing temporal coherence in high-resolution details.

Inference.
During inference, the video generation network receives a single reference image, driving audio, an optional textual prompt, and motion frames augmented using patch dropping and Gaussian noise techniques as inputs.
The network generates a video sequence that animates the reference image in accordance with the provided audio and textual prompt, synthesizing realistic lip movements and expressions synchronized with the audio output.
Subsequently, the high-resolution enhancement module processes the generated video to produce high-resolution frames, thereby enhancing visual quality and fine facial details.

## 5 Experiments

### 5.1 Experimental Setups

Implementation.
All experiments were conducted on a GPU server equipped with 8 NVIDIA A100 GPUs.
The training process was executed in two stages: the first stage comprised 30,000 steps with a step size of 4, targeting a video resolution of 512 × 512 pixels.
The second stage involved 28,000 steps with a batch size of 4, initializing the motion module with weights from Animatediff.
Approximately 160 hours of video data were utilized across both stages, with a learning rate set at 1e-5.
For the super-resolution component, training for temporal alignment was extended to 550,000 steps, leveraging initial weights from CodeFormer and a learning rate of 1e-4, using the VFHQ dataset as the super-resolution training data.
Each instance in the second stage generated 16 video frames, integrating latents from the motion module with the first 4 ground truth frames, designated as motion frames.
During inference, the output video resolution is increased to a maximum of 4096 × 4096 pixels.

Datasets.
To evaluate our proposed method, we employed several publicly available datasets, including HDTF, CelebV, and our introduced “Wild” dataset.
The “Wild” dataset comprises 2019 clips, totaling approximately 155.9 hours of video content, featuring a diverse array of lip motions, facial expressions, and head poses.
This extensive dataset provides a solid foundation for training and testing our portrait image animation framework, facilitating a comprehensive assessment of its ability to generate high-quality and expressive animations across various scenarios.

Evaluation Metrics.
We employ several evaluation metrics to rigorously evaluate our portrait image animation framework.
The Fréchet Inception Distance (FID) measures the statistical distance between generated and real images in feature space, with lower values indicating higher quality.
The Fréchet Video Distance (FVD) extends this concept to video, assessing the similarity between generated and real videos, where lower values signify superior visual quality.
The Sync-C metric gauges lip synchronization consistency with audio, with higher scores reflecting better alignment.
Conversely, the Sync-D metric evaluates the temporal consistency of dynamic lip movements, where lower values denote improved motion fidelity.
Finally, the Expression-FID (E-FID) quantifies expression synchronization differences between generated content and ground truth videos, providing a quantitative assessment of expression accuracy.

Figure: Figure 8: Qualitative comparison with exited approaches on CelebV data-set.
Refer to caption: extracted/5925503/fig/comparison_celebv.jpg

**Table 1: The quantitative comparisons with existed portrait image animation approaches on the CelebV data-set.**
| Method | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Audio2Head | 57.879 | 495.421 | 7.069 | 7.916 | 60.538 |
| SadTalker | 41.852 | 588.173 | 7.026 | 7.931 | 21.806 |
| EchoMimic | 60.252 | 805.067 | 5.499 | 9.482 | 19.680 |
| AniPortrait | 49.626 | 583.709 | 3.810 | 10.930 | 22.220 |
| Hallo | 82.715 | 1088.158 | 6.683 | 8.420 | 15.616 |
| Ours | 37.944 | 477.412 | 6.928 | 8.307 | 14.682 |
| Real video | - | - | 7.109 | 7.938 | - |

**Table 2: The quantitative comparisons with existed approaches on the proposed “Wild” data-set.**
| Method | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Audio2Head | 50.449 | 448.695 | 6.269 | 8.325 | 38.981 |
| SadTalker | 24.600 | 380.866 | 6.384 | 8.169 | 44.596 |
| EchoMimic | 50.994 | 854.826 | 5.082 | 9.675 | 35.806 |
| AniPortrait | 24.301 | 344.000 | 3.975 | 10.171 | 41.307 |
| Hallo | 28.186 | 571.991 | 6.610 | 8.181 | 36.793 |
| Ours | 24.072 | 360.192 | 6.760 | 8.156 | 33.316 |
| Real video | - | - | 7.088 | 7.726 | - |

Figure: Figure 9: The qualitative comparison with existed approaches on the proposed “Wild” data-set.
Refer to caption: extracted/5925503/fig/figure7.jpg

Figure: Figure 10: Portrait image animation results given different portrait styles.
Refer to caption: extracted/5925503/fig/styles.jpg

Baseline Approaches.
We evaluate our framework against leading state-of-the-art techniques, including both non-diffusion and diffusion-based models.
Non-diffusion models, such as Audio2Head and SadTalker, are compared with diffusion-based counterparts like EchoMimic, AniPortrait, and Hallo.
Notably, EchoMimic and AniPortrait employ a parallel generation approach for long-duration outputs, while Hallo utilizes an incremental formulation.
Unlike previous studies that focused on short-duration videos of only a few seconds, our evaluation is conducted on generated videos lasting 4 minutes, using looped audio from the benchmark dataset as the driving audio.
To ensure a fair comparison, we have excluded the high-resolution enhancement module, maintaining the same output video resolution (512 × 512 pixels) as the existed approaches across all quantitative comparisons.

### 5.2 Comparison with State-of-the-Art

Comparison on HDTF Dataset.
Table 6 and Figure 7 present quantitative and qualitative comparisons on the HDTF dataset.
Our framework achieves the lowest FID of 16.616 and an E-FID of 6.702, demonstrating superior fidelity and perceptual quality. Additionally, our synchronization metrics, Sync-C (7.379) and Sync-D (7.697), further validate the effectiveness of our method.
As illustrated in Figure 6, the extended inference duration significantly impacts FID metrics in existing diffusion-based approaches, leading to notable declines compared to their short-duration performance.
In terms of lip and expression motion synchronization, parallel methods such as EchoMimic and AniPortrait exhibit marked deterioration.
In contrast, our extended approach consistently demonstrates superior and stable performance across image and video quality, as well as motion synchronization, even as inference time increases.

Comparison on CelebV Dataset.
Table 1 and Figure 8 present the quantitative and qualitative comparisons for the CelebV dataset.
Our method achieves the lowest FID of 37.944 and an E-FID of 14.682, indicating superior animation quality. The FVD metric is reported at 477.412, suggesting a coherent video structure.
Additionally, our Sync-C score of 6.928 demonstrates competitive performance relative to real video standards.
Notably, the increased inference duration has resulted in a significant deterioration in both FID and FVD scores among existing methods, particularly with EchoMimic and Hallo, which exhibit marked degradation in FVD metrics.
Additionally, Aniportrait demonstrates notable declines in lip synchronization and expression metrics.

Comparison on the Proposed “Wild” Dataset.
Table 2 and Figure 9 offers additional quantitative and qualitative comparison results of the introduced “Wild” dataset.
Our method achieves an FID of 24.072 and an E-FID of 33.316, both indicative of high image quality.
We also register a Sync-C score of 6.760 and a Sync-D of 8.156, alongside the highest FVD of 360.192, demonstrating superior coherent video structure.

Animation of Different Portrait Styles.
Figure 10. This figure illustrates that our method is capable of processing a wide range of input types, including oil paintings, anime images, and portraits from generative models.
These findings highlight the versatility and effectiveness of our approach in accommodating different artistic styles.

**Table 3: Quantitative comparison on the CelebV dataset given different patch sizes of patch drop augmentation. A patch size of 0 indicates no patch drop.**
| patch size | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ |
| --- | --- | --- | --- | --- |
| 0 | 82.715 | 1088.158 | 6.683 | 8.420 |
| 1 | 38.518 | 491.338 | 6.766 | 8.387 |
| 4 | 39.615 | 504.287 | 6.712 | 8.411 |
| 16 | 44.172 | 756.517 | 6.431 | 8.517 |

**Table 5: Ablation study of the patch drop and Gaussian noise augmentation on the CelebV data-set.**
| Gaussian noise | Patch drop | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
|  |  | 82.715 | 1088.158 | 6.683 | 8.420 |
| ✓ |  | 78.283 | 984.876 | 6.701 | 8.415 |
|  | ✓ | 38.518 | 491.338 | 6.766 | 8.387 |
| ✓ | ✓ | 37.944 | 477.412 | 6.928 | 8.307 |

### 5.3 Ablation Studies

Different Patch Drop Size.
Table 4 illustrates the effects of varying patch drop sizes on performance metrics.
A patch size of 0 signifies no patch drop, while our implementation employs a patch size of 1.
The results indicate that patch drops enhance visual outcomes, as evidenced by improvements in FID and FVD, and contribute to a degree of enhancement in motion synchronization capabilities.

Different Patch Drop Rate. Table 4 and Figure 12 present a comparative analysis of varying drop rates applied to motion frames. A drop rate of 0.25 achieves the lowest FID score of 38.518 and FVD of 491.338, indicating improved image quality and coherence.

Effectiveness of Augmentation Strategies.
Table 5 and Figure 12 evaluate different augmentation strategies. Gaussian noise alone results in a high FID of 82.715 and FVD of 1088.158, indicating suboptimal quality.
The patch drop strategy significantly improves these metrics, reducing FID to 38.518 and FVD to 491.338. Notably, the combined strategy further enhances performance, achieving the lowest FID of 37.944 and FVD of 477.412, alongside the highest Sync-C score of 6.928.
Thus, the combined augmentation method proves to be the most effective in generating high-quality motion frames.

Figure: Figure 11: Qualitative comparison of different patch drop rate applied to motion frames on the CelebV data-set.
Refer to caption: extracted/5925503/fig/ablation_mask_rate1.jpg

Effectiveness of High-Resolution Enhancement.
The effectiveness of high-resolution enhancement techniques is illustrated in Figure 13, which demonstrates improved animation quality via video super-resolution.

Figure: Figure 13: Qualitative comparison of the portrait image animation results with and without high-resolution enhancement.
Refer to caption: extracted/5925503/fig/figure11.jpg

Figure: Figure 14: Qualitative comparison between different high-resolution enhancement methods.
Refer to caption: extracted/5925503/fig/ablation_temp_align.jpg

Figure: Figure 15: Qualitative comparison of portrait animation before and after applying the textual prompts.
Refer to caption: extracted/5925503/fig/emotion.jpg

Figure: (a) Reference image
Refer to caption: extracted/5925503/fig/att_ref.png

Comparison between Different High-Resolution Enhancement Methods.
Figure 14 provides a qualitative comparison of other image-based enhancement methods.
The analysis reveals that integrating super-resolution with temporal alignment significantly enhances visual fidelity, reduces artifacts, and increases image sharpness, resulting in a more coherent and realistic representation of facial features and expressions.

Effectiveness of Textual Prompt.
The integration of textual prompts into our portrait image animation framework significantly enhances the control over generated animations, as illustrated in Figure 15.
The comparative analysis demonstrates that textual prompts facilitate precise manipulation of facial expressions and emotional nuances, allowing for a more tailored animation output.
By providing explicit instructions regarding desired emotional states, the model exhibits improved responsiveness in generating animations that align closely with the specified prompts.

Attention Map Visualization.
Figure 16 presents the attention map visualization, which highlights both the reference image and the temporal attention associated with the motion frames.
The results indicate that the reference image indeed influences the overall appearance of the portrait and background due to the implementation of patch drop augmentation.
In contrast, the motion frames predominantly focus on regions related to facial motion, underscoring their role in capturing dynamic attributes in the generated animation.

### 5.4 Limitations and Future Work

Our method for long-duration, high-resolution portrait image animation has several limitations.
(1) Reliance on a single reference image constrains the diversity of generated expressions and poses, indicating a need for multiple references or advanced models capable of synthesizing varied facial features.
(2) While the patch-drop data augmentation technique effectively preserves motion dynamics, it may introduce artifacts; thus, future research should investigate alternative strategies or adaptive mechanisms for content-specific corruption.
(3) The substantial computational demands of generating 4K resolution videos necessitate optimization and hardware acceleration to enable real-time applications.

## 6 Conclusion

This paper presents advancements in portrait image animation through the enhanced capabilities of the Hallo framework.
By extending animation durations to tens of minutes while maintaining high-resolution 4K output, our approach addresses significant limitations of existing methods.
Specifically, innovative data augmentation techniques, including patch-drop and Gaussian noise, ensure robust identity consistency and reduce appearance contamination.
Furthermore, we implement vector quantization of latent codes and employ temporal alignment techniques to achieve temporally consistent 4K videos.
Additionally, the integration of audio-driven signals with adjustable semantic textual prompts enables precise control over facial expressions and motion dynamics, resulting in lifelike and expressive animations.
Comprehensive experiments conducted on publicly available datasets validate the effectiveness of our method, representing a significant contribution to the field of long-duration, high-resolution portrait image animation.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - For patch augmentation, could the authors analyze the “effective augmentation ratio”? Since the purpose of patch augmentation is to encourage the model to disregard appearance information and focus on motion, an analysis of which patches contain motion information and which contain appearance information, along with a calculation of the effective augmentation ratio for the video clip, would provide valuable insight into the effectiveness of this design.
    
- How is the textual condition used during inference? Since it serves as an additional conditioning factor, the authors should clarify how this information is labeled, details of training behavior (e.g., random dropping for CFG), and how CFG is applied during inference. It would also be useful to include an analysis of the impact of conflicts between non-textual and text-audio conditions.
    
- On the HDTF dataset (i.e., Table 1), the proposed method outperforms most methods except Audio2Head. Could the authors provide an explanation for this? The paper should include further explanation of the evaluation metrics used to help readers understand which capabilities are being measured and improved.
    
- In the third image on the bottom right side of Figure 6, there appear to be “water droplet-like” artifacts. Could the authors explain the cause of these artifacts?
soundness: 3
strengths: - An effective augmentation strategy  which do not need to change model structure and support long-duration
    
- The textual condition injection propose a novel method to inject text prompt to a reference-net based animation structure.
summary: This paper presents a pipeline for generating long-duration, high-resolution, audio-driven facial animations. To enable extended video generation, the authors introduce patch-drop and Gaussian noise augmentation techniques during training. For high-resolution output, the method incorporates a VQ-based super-resolution stage and temporal alignment attention to ensure temporal consistency. Additionally, a norm-based textual prompt control mechanism is used to enhance the expressiveness of facial animations.

Experiments across multiple datasets demonstrate the effectiveness of the proposed approach.
weaknesses: - For high-resolution enhancement, the paper lacks a comparison with video VQ models (e.g., MagVITv2) and does not include quantitative evaluations of the super-resolution results from the SR encoder-decoder.
    
- For textual prompt control, the paper does not provide a detailed description of how prompts are used in comparisons with baseline methods or how prompts are incorporated during training data preparation. Additionally, details regarding the CFG process for managing multiple conditions (e.g., text, audio, reference image) are missing.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see above. If the author solves my problems, I will consider raising the score. Thanks.
soundness: 3
strengths: 1. This paper discusses how to generate high-quality animated videos driven by audio, which is a very interesting topic.
2. The structure of this paper is well-organized and easy to follow. 
3. The experimental results show the effectiveness of the proposed method.
summary: This paper proposes Hallo2 for audio-driven face image animation, which mainly includes three training strategies. Firstly, this paper proposes a patch-drop data augmentation technique to prevent contamination of appearance information from preceding frames. Secondly, this paper extends the Vector Quantized Generative Adversarial Network discrete codebook space method for code sequence prediction tasks into the temporal dimension. Finally, this paper also introduces additional textual prompts for semantic control.
weaknesses: There are some questions and suggestions,
1. This paper proposes three strategies to improve the performance of audio-driven face image animation; however, the motivation for each strategy is not well explained. For example, in Section 3.2, a codebook prediction approach is used to enhance temporal coherence. It would be helpful if the authors provided more insights into why temporal coherence is insufficient without this codebook prediction approach.
2. This paper presents three distinct strategies; however, the connections between each contribution are weak, making it difficult for reviewers to grasp the central contribution. This lack of coherence also gives the impression that each contribution is only incremental.
3. From lines 238 to 241, the author uses a masking strategy to mask frames in the video. The author claims that this approach improves consistency. However, I’m a bit puzzled. In video generation, the masking strategy is generally beneficial for generating long videos autoregressively, so why would masking improve consistency? Could you please give more insights?
4. Some descriptions in the HIGH-RESOLUTION ENHANCEMENT section are a bit confusing. In the first half of this section, it seems that a Transformer is used to model the video in a continuous latent space. However, Figure 4 actually shows a codebook. Since it appears the authors are using a codebook, I suggest reorganizing the first half of this section to reflect this.
5. I observed flickering in some of the videos in the experimental results, especially with the man wearing a checkered shirt. Could this be due to limitations in the pre-trained VAE? Given that there is temporal alignment in the HIGH-RESOLUTION ENHANCEMENT section, why hasn’t this temporal alignment helped reduce the flickering effect?

---

confidence: 5
contribution: 1
initial_rating: 6
presentation: 4
questions: Please carefully address the above questions. I think there are many details that need to be clarified, particularly in the individual components constituting the proposed method.
soundness: 3
strengths: The paper offers several contributions that are deemed to be the strengths of this paper. This work is an extension of a previous work on Hallo, thus termed Hallo2. It still attempts to propose a few enhancements that can further address the limitations of the previous work, mainly by extending it to long-hour portrait image animation with high resolutions. 

The patch-drop data augmentation looks interesting. It intentionally adds corruption to the appearance of the conditional frames while preserving the motion information. Meanwhile, the appearance contamination is resolved using the Gaussian noise as the other data augmentation. 

The 4K resolution was achieved by leveraging the VQ-GAN and code sequence prediction (existing but applied to a new task) to ensure smooth transitions among the temporal frames. 

Therefore, the main strengths of this paper would lie in the patch-drop augmentation, gaussian noise augmentation and the code sequence prediction.
summary: This paper proposes the first method to generate an hour-long, audio-driven portrait image animation at 4K resolutions coupled with textual prompts. To accomplish this goal, the paper applies patch-drop augmentation on Gaussian noise, a vector quantization on latent codes with a high-quality decoder, and uses enhanced textual labels as the conditional inputs. These improvements constitute the proposed contributions.
weaknesses: The main idea of patch-drop data augmentation is to separate the motion from the appearance, where the former is controlled by temporal dynamics in the preceding frames. The patch-augmentation was applied to individual latent images z_{t-i}, where it was further partitioned into the K non-overlapping patches. Each patch was randomly masked. The question here is if we generate random masking and apply that to the time step t-1, would the same masking positions be enforced with the t-2 to t-N? Or the t-2 to t-N time steps still use a random masking operation. Could you clarify that and also show the motivation or the merits of each design? The other question is that it makes sense the omission of patches can still preserve the coarse structure. The motion dynamics on the other hand are not affected by the masking operations. Could you provide more details on why this is the case? 

Another intriguing part is related to the choice of patch size for dividing the latent image. Based on the results given in Table 3, the best patch size is 1*1, which means that a single pixel is dropped. This seems to contradict the intuition of using a patch. I don't mean 1 pixel is not a patch, but it does not conform to the normal choice of patch size. Could you offer more insights on why 1 pixel is considered to be more effective in this task? This would be very important. 

Regarding the preceding generated frames, which approach was used? Somehow, I am not able to find that detail in the paper, or maybe I miss it. Was that achieved by generating the second frame based on the first one, and then continually generating the third frame based on the first two frames? 

The design of Gaussian noise augmentation is simple and easy to understand, which was implemented by adding the Gaussian noise to the augmented latent representations. This was used as one of the conditional inputs. Could you provide more insights on why such an operation works? Ensuring temporal consistency is difficult. The paper seems to offer a much simpler and non-complex solution. Therefore, it would be more necessary to provide stronger validations on such a design. 

The set of conditional inputs used in this work includes four parts, which are the audio, textual, noise-augmented motion frames, and reference image encoding. Based on the flowchart given in Figure 3, the cross-attention conditions occur in different layers of the encoder and decoder of the U-Net. Could you justify how to select the corresponding layer for the designated condition? Are they all implemented via cross-attention? 

The high-resolution enhancement is still unclear to me. The paper does not seem to apply the super-resolution module but relies on the output from the diffusion model’s decoder and then a low-quality decoder. The technical part description in Sec 3.2 looks more like normal QKV operations. How the 4K resolution was achieved is still unclear, nor it was depicted in Figure 3. 

The other issue might be related to the over-claim of the proposed approach by saying the "hour-long". The evaluation was done on minutes duration (e.g., 4 minutes in evaluation). More evidences are required for this "hour-long" claim. 

The textual prompt implementation looks just like a regular conditional injection. This paper claims it to be one of the strengths of this work. 

The paper needs to add some more comparisons such as the EMO work. This has been mentioned in the Introduction but appears to be not compared in the experimental section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: No questions, please see the weaknesses for rebuttal.
soundness: 3
strengths: 1. This is the first work which attempts to animate portrait videos under 4K resolution. The qualitative results shown in the supplementary material demonstrate the improvements against the baseline.
2. The one hour length portrait video animation is interesting. The Gaussian noise augmentation technique is effective in alleviating the error accumulation issue in the autoregressive long video animation process.
3.  The textual prompt condition helps maintain the emotion and expression of the talking person, which is simple yet effective.
summary: This work presents an enhanced version of the prior work Hallo. This work improves upon the previous model in terms of long-term video generation, 4K high-resolution video generation, and semantic textual label condition. This work is the first one to study 4K audio-driven portrait image animation. Experiment results show that this method achieves state-of-the-art performance in long-term portrait video animation, which can generate rich and controllable 4K content of tens of minutes.
weaknesses: 1. This work is a combination of existing works. The autoregressive animation pipeline has already been explored in EMO. The noise simulation trick is commonly used in the autoregressive generation models to alleviate error accumulation. The 4K resolution generation is an extension of VQGAN in the audio-driven portrait animation. Therefore, this work is a combination of prior works from the engineering perspective, which is not novel or significant enough.
2. Compared with Hallo, the animation framework remains unchanged and is quite similar to prior works like EMO. The reference unet borrows ideas from human image animation but the authors didn’t cite them properly. Please consider cite [1,2] in the final version.
3. The long video results shown in the supplementary do not contain too many translation and rotation of the talking head, which cannot demonstrate the temporal consistency for long-term animation. It is recommended to conduct such an experiment with diverse translation and rotation of the talking head to prove the effectiveness of this proposed method.
The ablation of HRE is not comprehensive enough. Comparisons with the 512x512 model cannot show the necessity of developing such a VQGAN-based superresolution in the pipeline. It is recommended to compare the proposed method with previous superresolution models, such as ESRGAN[3].

[1] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
[2] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
[3] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Meta-review: The paper addresses the problem of audio-driven portrait image animation and proposes a series of improvements which enable long duration video, high resolution (4k), and incorporate adjustable semantic textual labels for portrait expressions. The long duration video is enabled through a new patch-drop technique. The high resolution is done through vector quantization of latent codes as well as a high quality decoder. The paper also introduces a new "Wild" dataset.

The core strength of the paper is the strong qualitative results and the very high resolution and very long duration generations. The main weakness appears to be the somewhat incremental nature of the paper.

I agree with all reviewers and advocate for acceptance believing that the strengths outweigh the weaknesses.

Decision: Accept (Poster)