Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency

## Abstract

Abstract With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion.
In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

## 1 Introduction

Due to the rapid advancements GAN and diffusion models in the field of video synthesis (Bar-Tal et al., ; Blattmann et al., ; ; Guo et al., ; Zhou et al., ; Gupta et al., ; Wang et al., ; Ho et al., ; Brooks et al., ; Wang et al., ; Singer et al., ; Li et al., ; Villegas et al., ), human video synthesis (Siarohin et al., ; ; Xu et al., ; Hu, ; Corona et al., ) has gradually approached the threshold of practical usability in terms of quality, attracting significant attention in recent years. Among these, zero-shot audio-driven portrait synthesis has seen an explosion of research (He et al., ; Tian et al., ; Xu et al., ; Wang et al., ; Chen et al., ; Xu et al., ; Stypulkowski et al., ) since around 2020, due to its ability to generate talking head videos with minimal barriers to entry. Starting last year, diffusion model techniques were introduced, with end-to-end audio-driven models (Tian et al., ; Xu et al., ; Chen et al., ) demonstrating more vivid synthesis results compared to existing methods.

However, due to the weak correlation between audio and portrait motion, end-to-end audio-driven methods typically introduce additional conditions related to spatial motion to ensure temporal stability in the synthesized videos. Conditions such as face locators and speed layers (Tian et al., ; Xu et al., ; Chen et al., ) restrict the range and velocity of portrait movements, potentially reducing the expressiveness of the final output, as illustrated in Figure 1. While the introduction of preset motion templates may mitigate this issue, it also introduces complications related to template selection, audio-motion synchronization, and repetitive template movements. From a model perspective, these limitations also hinder the full potential of video diffusion models in generating vivid motion. This paper aims to address this issue by proposing an audio-only conditioned portrait diffusion model, enabling the model to learn natural motion patterns from data without the need for spatial templates.

Figure: Figure 1: Visual comparisons with existing methods. Existing methods struggle to generate natural movements; compared to reference images, their motion, posture, and expressions often resemble the reference or remain nearly static due to the auxiliary spatial conditions. In contrast, Loopy effectively generates natural anthropomorphic movements solely from audio, including detailed head movements and facial expressions. Video results are provided in the supplementary materials.
Refer to caption: x1.png

We first attempted to remove structures such as the face locator and speed layer. This resulted in more frequent undesirable motion patterns in the synthesized videos, such as sudden temporal jitter, frame corruption, increased autoregressive degradation, and unnatural temporal details, leading to a decrease in overall quality. In practice, in current diffusion-based frameworks, the conditions influencing motion include not only audio (which maps to motion patterns in a many-to-many manner and is hard to fully define motions) but also motion frames. Motion frames provide appearance information from preceding clips, strongly influencing motion generation. However, current methods usually generate based on 4 motion frames from the last clip and over ten target frames from the current clip. At a typical 25 fps, this covers about 0.5 seconds in total, with motion frames covering only 0.2 seconds. Such short motion frames cause the model to primarily extract appearance information from the last clip rather than temporal motion information, such as motion style. For example, in the case of eye blinking, 0.2 seconds of preceding information is insufficient for the model to determine whether blinking should occur (it cannot know if blinking has occurred previously, making blinking a probabilistic event, which may lead to long periods without blinking in the generated video). When motion style is hard to determine by audio and motion frames, it exhibits randomness, necessitating additional guidance from spatial conditions such as face box and movement speed. We attempted to increase the length of motion frames and found that this approach can generate larger and more flexible movements to some extent, although it may lead to increased instability. This observation reminds us that increasing the temporal receptive field in an appropriate manner may help capture motion patterns more effectively and aid in generating natural motion. Additionally, current methods inject audio features into the diffusion model directly through cross-attention, which makes it difficult for the model to learn the relationship between audio and portrait motion, instead modeling the relationship between audio and all video pixels. This phenomenon is also mentioned in the Hallo (Xu et al., ) work. The weak correlation between audio and portrait also increases the likelihood of generating random and unsatisfactory movements, making it difficult to generate natural motion that corresponds well with the audio.

Based on the above observations and considerations, we propose Loopy, an end-to-end template-free audio-conditioned diffusion model for portrait video generation, leveraging long-term motion dependency to generate vivid portrait videos. Specifically: for temporal aspect: We designed inter- and intra-clip temporal modules. Motion frames are modeled with a separate temporal layer to capture inter-clip temporal relationships, while the original temporal module focuses on intra-clip temporal modeling. Additionally, we introduced a temporal segment module to the intra-clip layer, extending the receptive field to over 100 frames (covering approximately 5 seconds at 25 fps, 30 times the original). For audio aspect: We introduced the audio-to-motion latents module, which transforms audio and facial motion-related features (landmarks, head motion variance, expression motion variance) into motion latents based on a shared feature space. These latents are inserted into the denoising net as conditions. During testing, motion latents are generated using only audio. This approach allows weakly correlated audio to leverage strongly correlated motion conditions, enhancing the modeling of the relationship between audio and portrait motion. Extensive experiments validate that our design effectively improves the naturalness of motion and the robustness of video synthesis. In summary, our contributions include:

(1) We propose a template-free audio-conditioned diffusion model for portrait video generation, featuring inter- and intra-clip temporal modules that leverage long-term motion dependency to learn natural motion patterns. Additionally, the audio-to-latents module enhances the correlation between audio and portrait motion by using strongly correlated conditions during training.

(2) We validated the effectiveness of our method on public datasets and evaluated the model’s capabilities across various application scenarios (including multiple types of images and audio), demonstrating that our model achieves more vivid and stable synthesis results compared to existing methods.

## 2 Related Works

Audio-driven portrait video generation has attracted significant attention in recent years, with numerous works advancing the field. These methods can be categorized into GAN-based and diffusion-based approaches based on their video synthesis techniques.

GAN-based methods (Zhou et al., ; Prajwal et al., ; Zhang et al., ; Liang et al., ) typically consist of two key components: an audio-to-motion model and a motion-to-video model. These models are usually implemented independently. For example, MakeItTalk (Zhou et al., ) uses an LSTM module to predict the landmark coordinates corresponding to the audio, and then a warp-based GAN model converts the landmark signals into video images. SadTalker (Zhang et al., ) utilizes the existing FaceVid2Vid (Wang et al., ) method as the image synthesizer, employing ExpNet and PoseVAE to transform audio features into the inputs required by FaceVid2Vid, thereby completing the audio-to-video generation. With the introduction of diffusion techniques, some methods have implemented the audio-to-motion module using diffusion models while retaining the independent implementation of the motion-to-video module. For instance, GAIA (He et al., ) uses a VAE to represent motion as motion latents and implements a motion latents-to-video generation model. Additionally, it designs a diffusion model to achieve audio-to-motion latents generation, thereby enabling audio-to-video generation. DreamTalk (Ma et al., ), Dream-Talk (Zhang et al., ), and VASA-1 (Xu et al., ) propose similar ideas, using PIRender (Ren et al., ), FaceVid2Vid (Wang et al., ), and MegaPortrait (Drobyshev et al., ) as their motion-to-video models, respectively, and designing audio-to-motion representation models to complete the audio-to-portrait video generation process.

Apart from the above types, EMO Portrait (Tian et al., ) achieves audio-to-portrait video generation using a single diffusion model, replacing the two-stage independent design of the audio-to-motion module and the motion-to-video model. Hallo (Xu et al., ), EchoMimic (Chen et al., ) and VExpress (Wang et al., ) have improved their audio-to-video modeling based on a similar audio-to-video diffusion framework. Although these end-to-end methods can generate vivid portrait videos, they need to introduce spatial condition module, like face locator and speed layer, to constrain head movements for stability, limiting the diversity of motion models in practical applications and hindering the full potential of diffusion models.

## 3 Method

Figure: Figure 2: The framework of Loopy. it removes the commonly used face locator and speed layer modules found in existing methods. Instead, it achieves flexible and natural motion generation through the proposed inter/intra-clip temporal layer and audio-to-latents module.
Refer to caption: extracted/5835222/imgs/framework.png

In this section, we will introduce our method, Loopy. First, we will provide an overview of the framework, including the input, output, and key components of Loopy. Second, we will focus on the design of the inter-/intra-temporal modules, including the temporal segment module. Third, we will detail the implementation of the audio condition module. Finally, we will describe the implementation details during the training and testing of Loopy.

### 3.1 Framework

Our method is built upon Stable Diffusion (SD) and uses its initialization weights. SD is a text-to-image diffusion model based on the Latent Diffusion Model (LDM) (Rombach et al., ). It employs a pretrained VQ-VAE (Kingma, ; Van Den Oord et al., ) $\mathcal{E}$ to transform images from pixel space to latent space. During training, images are first converted to latents, i.e., $z_{0}=\mathcal{E}(I)$. Gaussian noise $\epsilon$ is then added to the latents in the latent space based on the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., ) for $t$ steps, resulting in a noisy latent $z_{t}$. The denoising net takes $z_{t}$ as input to predict $\epsilon$. The training objective can be formulated as follows:

$$ $L=\mathbb{E}_{z_{t},c,\epsilon\sim\mathcal{N}(0,1),t}\left[\lVert\epsilon- \epsilon_{\theta}(z_{t},t,c)\rVert_{2}^{2}\right],$ (1) $$

where $\epsilon_{\theta}$ represents the denoising net, including condition-related attention modules, which is the main part Loopy aims to improve. $c$ represents the text condition embedding in SD, and in Loopy, it includes audio, motion frames, and other additional information influencing the final generation. During testing, the final image is obtained by sampling from Gaussian noise and removing the noise based on DDIM (Song et al., ) or DDPM.

As demonstrated in Figure 2, in the Loopy, the inputs to the denoising net include noisy latents, i.e., the VQ-VAE encoded image latents $z_{t}$. Unlike original SD, where the input is a single image, here it is a sequence of images representing a video clip. The inputs also include reference latents $c_{ref}$ (encoded reference image latents via VQ-VAE), audio embedding $c_{audio}$ (audio features of the current clip), motion frames $c_{mf}$ (image latents of the M frames sequence from the preceding clips), and timestep $t$. During training, additional facial movement-related features are involved: $c_{lmk}$ (facial keypoint sequence of the current clip), $c_{mov}$ (head motion variance of the current clip), and $c_{exp}$ (expression variance of the current clip). The output is the predicted noise $\epsilon$. The denoising network employs a Dual U-Net architecture (Hu, ; Zhu et al., ). This architecture includes an additional reference net module, which replicates the original SD U-Net structure but utilizes reference latents $c_{ref}$ as input. The reference net operates concurrently with the denoising U-Net. During the spatial attention layer computation in the denoising U-Net, the key and value features from corresponding positions in the reference net are concatenated with the denoising U-Net’s features along the token dimension before proceeding with the attention module computation. This design enables the denoising U-Net to effectively incorporate reference image features from the reference net. Additionally, the reference net also takes motion frames latents $c_{mf}$ as input for feature extraction, allowing these features to be utilized in subsequent temporal attention computations.

### 3.2 Inter/Intra- Clip Temporal Layers Design

Here, we introduce the design of the proposed inter-/intra-clip temporal modules. Unlike existing methods (Tian et al., ; Xu et al., ; Chen et al., ; Wang et al., ) that process motion frame latents and noisy latents features simultaneously through a single temporal layer, Loopy employs two temporal attention layers: the inter-clip temporal layer and the intra-clip temporal layer. The inter-clip temporal layer first handles the cross-clip temporal relationships between motion frame latents and noisy latents, while the intra-clip temporal layer focuses on the temporal relationships within the noisy latents of the current clip.

First, we introduce the inter-clip temporal layer, initially ignoring the temporal segment module in Figure 2. We first collect the $m$ image latents from the preceding clip, referred to as motion frames latents $c_{mf}$. Similar to $c_{ref}$, these latents are processed frame-by-frame through the reference network for feature extraction. Within each residual block, the features $x_{c_{mf}}$ obtained from the reference network are concatenated with the features $x$ from the denoising U-Net along the temporal dimension. To distinguish the types of latents, we add learnable temporal embeddings. Subsequently, self-attention is computed on the concatenated tokens along the temporal dimension, referred to as temporal attention. The intra-clip temporal layer differs in that its input does not include features from motion frames latents, it only processes features from the noisy latents of the current clip. By separating the dual temporal layers, the model can better handle the aggregation of different semantic temporal features in cross-clip temporal relationships.

Figure: Figure 3: The illustration of the temporal segment module and the inter/intra- clip temporal layers. The former allows us to expand the motion frame to cover over 100 frames, while the later enables the modeling of long-term motion dependency.
Refer to caption: extracted/5835222/imgs/TS.png

Due to the independent design of the inter-clip temporal layer, Loopy is better equipped to model the motion relationships among clips. To enhance this capability, we introduce the temporal segment module before $c_{mf}$ enters the reference network. This module not only extends the temporal range covered by the inter-clip temporal layer but also accounts for the variations in information due to the varying distances of different clips from the current clip, as illustrated in Figure 3. The temporal segment module divides the original motion frame into multiple segments and extracts representative motion frames from each segment to abstract the segment. Based on these abstracted motion frames, we recombine them to obtain motion frame latents for subsequent computations. For the segmentation process, we define two hyperparameters: stride $s$ and expand ratio $r$. The stride $s$ represents the number of abstract motion frames in each segment, while the expand ratio $r$ is used to calculate the length of the original motion frames in each segment. The number of frames in the $i$-th segment, ordered from closest to farthest from the current clip, is given by $s\times r^{(i-1)}$. For example, with a stride $s=4$ and an expand ratio $r=2$, the first segment would contain 4 frames, the second segment would contain 8 frames, and the third segment would contain 16 frames. For the abstraction process after segmentation, we default to uniform sampling within each segment. In the experimental section, we investigate different segmentation parameters and abstraction methods. Since segmentation and abstraction directly affect the learning of long-term motion dependency, different methods have a significant impact on the results. The output of the temporal segment module, $c^{o}_{mf}$, can be defined as:

$$ $c^{o}_{mf,i}=c_{mf,\left\lfloor\frac{\text{start_index}_{i}+\text{end_index} _{i}}{2}\right\rfloor}$ (2) $$

where $\text{start_index}_{i}=\sum_{j=0}^{k-1}r^{j}\cdot s$ and $\text{end_index}_{i}=\text{start_index}_{i}+r^{k}\cdot s$, with $k=\left\lfloor\frac{i}{s}\right\rfloor$. $c^{o}_{mf,i}$ represents the mean value of the $i$-th element of the output, calculated from the sequence $c_{mf}$.

The temporal segment module rapidly expands the temporal coverage of the motion frames input to the inter-clip temporal layer while maintaining acceptable computational complexity. For closer frames, a lower expansion rate retains more details, while for distant frames, a higher expansion rate covers a longer duration. This approach helps the model better capture motion style from long-term information and generate temporally natural motion without spatial templates.

### 3.3 Audio Condition Module

For the audio condition, we first use wav2vec (Baevski et al., ; Schneider et al., ) for audio feature extraction. Following the approach in EMO, we concatenate the hidden states from each layer of the wav2vec network to obtain multi-scale audio features. For each video frame, we concatenate the audio features of the two preceding and two succeeding frames, resulting in a 5-frame audio feature as audio embeddings $c_{a}udio$ for the current frame. Initially, in each residual block, we use cross-attention with noisy latents as the query and audio embedding $c_{audio}$ as the key and value to compute an attended audio feature. This attended audio feature is then added to the noisy latents feature obtained from self-attention, resulting in a new noisy latents feature. This provides a preliminary audio condition.

Figure: Figure 4: The audio-to-latents module.
Refer to caption: extracted/5835222/imgs/A2M.png

Additionally, as illustrated in Figure 4, we introduce the Audio-to-Latents module. This module maps conditions that have both strong and weak correlations with portrait motion, such as audio, to a shared motion latents space. These mapped conditions serve as the final condition features, thereby enhancing the modeling of the relationship between audio and portrait movements based on motion latents.
Specifically, we maintain a set of learnable embeddings.
For each input condition, we map it to a query feature using a fully connected (FC) layer, while the learnable embeddings serve as key and value features for attention computation to obtain a new feature based on the learnable embeddings. These new features, referred to as motion latents, replace the input condition in subsequent computations. These motion latents are then dimensionally transformed via an FC layer and added to the timestep embedding features for subsequent network computation.
During training, we sample an input condition for the Audio-to-Latents module with equal probability from audio embeddings and facial movement-related features such as landmarks, face absolute motion variance, and face expression motion variance. During testing, we only input audio to generate motion latents. By leveraging features strongly correlated with portrait movement, the model utilizes learnable embeddings to control motion. Consequently, transforming audio embeddings into motion latents can also allows for more direct influence on portrait motion.

### 3.4 Training Strategies

Conditions Mask and Dropout. In the Loopy framework, various conditions are involved, including the reference image $c_{ref}$, audio features $c_{audio}$, preceding frame motion frames $c_{mf}$, and motion latents $c_{ml}$ representing audio and facial motion conditions. Due to the overlapping nature of the information contained within these conditions, to better learn the unique information specific to each condition, we used distinct masking strategies for the conditions during the training process. During training, $c_{audio}$ and motion latents are masked to all-zero features with a 10% probability. For $c_{ref}$ and $c_{mf}$, we design specific dropout and mask strategies due to their conflicting relationship. $c_{mf}$ also provides appearance information and is closer to the current clip compared to $c_{ref}$, leading the model to heavily rely on motion frames rather than the reference image. This can cause color shifts and artifacts in long video sequences during inference. To address this, $c_{ref}$ has a 15% probability of being dropped, meaning the denoising U-Net will not concatenate features from the reference network during self-attention computation. When $c_{ref}$ is dropped, motion frames are also dropped, meaning the denoising U-Net will not concatenate features from the reference network during temporal attention computation. Additionally, motion frames have an independent 40% probability of being masked to all-zero features.

Multistage Training. Following AnimateAnyone (Hu, ) and EMO (Tian et al., ), we employ a 2-stage training process. In the first stage, the model is trained without temporal layers and the audio condition module. The inputs to the model are the noisy latents of the target single-frame image and reference image latents, focusing the model on an image-level pose variations task. After completing the first stage, we proceed to the second stage, where the model is initialized with the reference network and denoising U-Net from the first stage. We then add the inter-/intra-clip temporal layers and the audio condition module for full training to obtain the final model.

Inference. During Inference, we perform class-free guidance (Ho & Salimans, ) using multiple conditions. Specifically, we conduct three inference runs, differing in whether certain conditions are dropped. The final noise $e_{final}$ is computed as:

$$ $e_{final}=\text{audio_ratio}\times(e_{audio}-e_{ref})+\text{ref_ratio}\times (e_{ref}-e_{base})+e_{base}$ $$

where $e_{audio}$ includes all conditions $c_{ref}$, $c_{audio}$, $c_{mf}$, $c_{ml}$, $e_{ref}$ but mask $c_{audio}$ to all-zero features, and $e_{base}$ masks $c_{audio}$ to all-zero features and removes the concatenation of reference network features in the denoising U-Net’s self-attention. This approach allows us to control the model’s final output in terms of adherence to the reference image and alignment with the audio. The audio ratio is set to 5 and the reference ratio is set to 3. We use DDIM sampling with 25 denoising steps to complete the inference.

### 3.5 Experiments

Datasets. For training Data, We collected talking head video data from the internet, excluding videos with low lip sync scores, excessive head movement, extreme rotation, or incomplete head exposure. This resulted in 160 hours of cleaned training data. Additionally, we supplemented the training data with public datasets such as HDTF (Zhang et al., ). For the test set, we randomly sampled 100 videos from CelebV-HQ (Zhu et al., ) (a public high-quality celebrity video dataset with mixed scenes) and RAVDESS () (a public high-definition indoor talking scene dataset with rich emotions). To test the generalization ability of diffusion-based models, we also collected 20 portrait test images, including real people, anime, side face, and humanoid crafts of different materials, along with 20 audio clips, including speeches, singing, rap and emotionally rich speech. We refer to this test set as the openset test set.

Implementation Details. We trained our model using 24 Nvidia A100 GPUs with a batch size of 24, using the AdamW (Loshchilov & Hutter, ) optimizer with a learning rate of 1e-5. The generated video length was set to 12 frames, and the motion frame length was set to 124 frames, representing the preceding 124 frames of the current 12-frame video. After temporal squeezing, this was compressed to 20 motion frame latents. During training, the reference image was randomly selected from a frame within the video clip. For the facial motion information required to train audo-to-motion module, we used DWPose (Yang et al., ) to detect facial keypoints for the current 12 frames. The variance of the absolute position of the nose tip across these 12 frames was used as the absolute head motion variance. The variance of the displacement of the upper half of the facial keypoints (37 keypoints) relative to the nose tip across these 12 frames was used as the expression variance. The training videos were uniformly processed at 25 fps and cropped to 512x512 portrait videos.

Metrics. We assess image quality with the IQA metric (Wu et al., ), video motion with VBench’s smooth metric (Huang et al., ), and audio-visual synchronization using SyncC and SyncD (Prajwal et al., ). For the CelebvHQ and RAVDESS test sets, which have corresponding ground truth videos, we will also compute FVD (Unterthiner et al., ), E-FID (Tian et al., ) and FID metrics for comparison. Additionally, to compare the global motion (denoted as Glo) and dynamic expressions (denoted as Exp) of the portrait, we calculated the variance values based on key points of the nose and the upper face, specifically excluding the mouth area. We also computed the values for the ground truth video as a reference for comparison. For the openset testset, which lacks ground truth video references, we conducted subjective evaluations. Ten experienced users were invited to assess six key dimensions: identity consistency, video synthesis quality, audio-emotion matching, motion diversity, naturalness of motion, and lip-sync accuracy. For each case, participants were required to identify the top-performing method in each dimension.

#### 3.5.1 Results and Analysis

Performance in Complex Scenarios. CelebV-HQ includes videos of celebrities speaking in various scenarios, including films and interviews, both indoors and outdoors, with diverse portrait poses. This makes testing on this dataset effectively simulate real-world usage conditions. As shown in Table 1, our method significantly outperforms the compared methods in most metrics, as evidenced by the comparison videos provided in the supplementary materials. Regarding motion-related metrics, although not the best, our results are similar to those of the ground truth in terms of smoothness. In the dynamic expression metric (Exp), our method closely matches GT, more so than the compared methods. For global motion (Glo), our performance is similar to that of EchoMimic. However, it is evident that our method has a distinct advantage in video synthesis quality and lip-sync accuracy.

**Table 1: Comparisons with existing methods on the CelebV-HQ test set.**
| Method | IQA$\uparrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | FVD-Res$\downarrow$ | FVD-Inc$\downarrow$ | FID$\downarrow$ | Smooth<br>$\uparrow$ | Glo | Exp | E-FID$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SadTalker | 2.953 | 3.843 | 8.765 | 171.848 | 1746.038 | 36.648 | 0.9964 | 0.554 | 0.270 | 2.248 |
| Hallo | 3.505 | 4.130 | 9.079 | 53.992 | 742.974 | 35.961 | 0.9946 | 0.499 | 0.255 | 2.426 |
| VExpress | 2.946 | 3.547 | 9.415 | 117.868 | 1356.510 | 65.098 | 0.9957 | 0.020 | 0.166 | 2.414 |
| EchoMimic | 3.307 | 3.136 | 10.378 | 54.715 | 828.966 | 35.373 | 0.9926 | 2.259 | 0.640 | 3.018 |
| GT | - | - | - | - | - | - | 0.9937 | 3.249 | 0.506 | - |
| Loopy | 3.780 | 4.849 | 8.196 | 49.153 | 680.634 | 33.204 | 0.9949 | 2.233 | 0.452 | 2.307 |

**Table 2: Comparisons with existing methods on the RAVDESS test set.**
| Method | IQA$\uparrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | FVD-Res$\downarrow$ | FVD-Inc$\downarrow$ | FID$\downarrow$ | Smooth<br>$\uparrow$ | Glo | Exp | E-FID$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SadTalker | 3.840 | 4.304 | 7.621 | 22.516 | 487.924 | 32.343 | 0.9955 | 0.604 | 0.120 | 3.270 |
| Hallo | 4.393 | 4.062 | 8.552 | 38.471 | 537.478 | 19.826 | 0.9942 | 0.194 | 0.080 | 3.785 |
| VExpress | 3.690 | 5.001 | 7.710 | 62.388 | 982.810 | 26.736 | 0.9962 | 0.007 | 0.039 | 3.901 |
| EchoMimic | 4.504 | 3.292 | 9.096 | 54.115 | 688.675 | 21.058 | 0.9924 | 0.641 | 0.184 | 3.350 |
| GT | - | - | - | - | - | - | 0.9917 | 3.335 | 0.317 | - |
| Loopy | 4.506 | 4.814 | 7.798 | 16.134 | 394.288 | 17.017 | 0.9923 | 2.962 | 0.343 | 3.132 |

Figure: Figure 5: User voting comparisons on the openset test set. The first row includes experimental results for various categories of input images, while the second row includes experimental results for various categories of input audio.
Refer to caption: x2.png

**Table 3: Experiments on the effectiveness of the proposed modules.**
| Method | IQA$\uparrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | smooth$\uparrow$ |  |
| --- | --- | --- | --- | --- | --- |
| Full Model | 4.507 | 6.303 | 7.749 | 0.9932 |  |
| w/o inter-clip temp. | 4.335 | 6.104 | 8.129 | 0.9942 |  |
| w/o TSM | 4.386 | 6.054 | 8.235 | 0.9922 |  |
| w/o A2L | 4.428 | 5.999 | 8.351 | 0.9922 |  |
| single temp. + 20 mf | 4.072 | 6.201 | 8.309 | 0.9752 |  |
| $s=1,\ r=2$ | 4.461 | 5.919 | 8.245 | 0.9940 |  |
| $s=2,\ r=2$ | 4.453 | 5.855 | 8.326 | 0.9930 |  |
| $s=3,\ r=2$ | 4.443 | 6.083 | 8.161 | 0.9930 |  |
| $s=4,\ r=1$ | 4.424 | 6.219 | 8.004 | 0.9931 |  |
| mean sample | 4.452 | 5.907 | 8.199 | 0.9931 |  |
| random sample | 4.438 | 6.098 | 8.144 | 0.9932 |  |

Performance in Emotional Expression. RAVDESS is a high-definition talking scene dataset containing videos with varying emotional intensities. It effectively evaluates the method’s performance in emotional expression. As demonstrated by the E-FID (Tian et al., ) metric, our method outperforms the compared methods. This is corroborated in the motion dynamics metrics Glo and Exp, where our results are closer to the ground truth. Although our lip-sync accuracy is slightly inferior to VExpress, it is important to note that the results generated by VExpress generally lack dynamic motion, as indicated by the Glo, Exp, and E-FID metrics. This static nature can provide an advantage when measured with SyncNet for lip-sync accuracy.

Performance in Openset Scenarios. We compared different input styles (real people, anime, humanoid crafts and sife face) and various types of audio (speech, singing, rap, and emotional audio) to evaluate the robustness of the methods. As shown in Table 3, Loopy consistently outperforms the compared methods across these diverse scenarios.

#### 3.5.2 Ablation Studies

Analysis of Key Components. We analyzed the impact of the two key components of the Loopy, the inter/intra-clip temporal layer and the audio-to-latents module. For the former, we conducted two experiments: (1) removing the dual temporal layer design and retaining a single temporal layer to handle the temporal relationships of both the last and current clips, similar to methods like EMO and Hallo; (2) removing the temporal segment module and retaining the design of 4 motion frames as in other methods. For the latter, we removed the audio-to-latents module and retained only cross-attention for audio feature injection. The results, listed in Table 3, show that the dual temporal layer design improves temporal stability and image quality, and its removal leads to performance degradation. Removing the temporal segment module prevents the model from learning motion style information from long-term motion dependency, resulting in a decline in overall motion quality, including expressiveness and temporal stability. Removing the audio-to-latents module also reduces the overall quality of synthesized visuals and motion. This is because, during the training of the audio-to-latents module, spatial conditions are added for mixed training. Spatial conditions provide clearer motion guidance compared to audio alone, facilitating easier model convergence. These results validate the effectiveness of our method.

Impact of Long-Term Temporal Dependency. We also investigated the impact of long-term motion dependency on the results and list the results in Table 3. Initially, we compared the effects of extending the motion frame length to 20 under a single temporal layer setting. We observed that while this approach enhances the dynamics of the model’s output, it significantly degrades the overall image quality. In contrast, on the full model with 20 motion frames ($s=4,\ r=1$), the addition of inter/intra-clip temporal layers indeed improved the overall results. Regarding the settings of s and r, we experimented with various values. Starting with a fixed $r=2$, we found that smaller values of s resulted in poorer overall performance. We attribute this to the disparity in FPS between the target and motion frames, which complicates inter-clip temporal modeling due to the lower proportion of motion frames. As s increases, the number of motion frames also increases, allowing the inter-clip temporal layer to effectively model cross-clip temporal relationships, thereby significantly enhancing overall performance.
We also compared the effects of r with the same $s=4$, where $r=1$. A smaller r indicates a narrower temporal coverage, which slightly reduces performance compared to the full model ($s=4,\ r=2$).
Finally, we explored different sampling strategies for motion frames within the temporal segment module, comparing methods such as average pooling and random single-frame sampling. The uniform sampling approach of the full model proved more effective, likely because it provides more stable interval information, which benefits the inter-clip temporal layer in learning long-term motion information.

Figure: Figure 6: Visualization of videos generated by Loopy in different scenarios.
Refer to caption: x3.png

#### 3.5.3 Visual Results Analysis

We provide visual analysis for the openset scenarios in Figure 6. Compared to other methods, Loopy demonstrates significant advantages in ID preservation, motion amplitude, and image quality. It also performs well with uncommon images. Additional video results are available in the supplementary materials.

## 4 Conclusion

In this paper, we propose LOOPY, an end-to-end audio-driven portrait video generation framework that does not require spatial conditions and leverages long-term motion dependency to learn natural motion patterns from data. Specifically, we introduce the inter/intra-clip temporal layer design and the audio-to-latents module, which enhance the model’s ability to learn the correlation between audio and portrait motion from temporal and audio dimensions, respectively. Extensive experiments validate the effectiveness of our method, demonstrating significant improvements in temporal stability, motion diversity, and overall video quality over existing methods.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: See weaknesses above.
soundness: 4
strengths: 1. The proposed method is solid, with enough technical contributions to address the long-term dependency between motions and audio conditions.

2. The experiment results are strong enough compared to prior works and baselines, in particular on FVD metrics and DExp metrics.

3. Both qualitative results and the demos shown in the supplementary webpage are appealing and convincing enough, where the long-term dependencies and correlations between audio and portrait motions are consistently maintained.

4. Overall, the paper is well-written and easy to follow, albeit having many technical details.

5. The human study results clearly show that the proposed method perceptually outperforms other baselines and prior arts.
summary: The paper proposes an audio-only conditioned video diffusion model. The model consists of three key components: an inter- and intra-clip temporal module, and an audio-to-latents module. These modules are designed to facilitate long-term movement modeling, enhancing the correlation between audio and motion. During inference, a single reference image as well as the audio is sent as input to autoregressively generate future frames window by window.
weaknesses: 1. For audio-to-latent module, why replacing it with cross-attention module leads to largest performance drop as seen in Table 3. What are missing from cross-attention that makes it fail to perform as good.

2. During inference, audio ratio and ref ratio are manually set for classifier guidance, an ablation study is suggested to their impact on the final quality of generated video to have some insights about this weighting scheme.

3. Could the proposed method be further optimized and adapted to real-time settings, where the audio is being played and video follows interactively?

4. What are limitations of the proposed method and what could be improved? Are there failure cases where the generated motions cannot follow the audio closely?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Are there specific cases where Loopy struggles to maintain natural motion or facial expressions? An analysis of these limitations would provide a more complete understanding of the model’s strengths and weaknesses.

2. In the experiments section, the baseline models compared with Loopy were not trained using the collected dataset. It would be helpful to see how these baseline models perform when trained on the same dataset. This could further validate the effectiveness of the proposed modules and confirm that the performance gains are due to the model’s design, rather than advantages inherent to the dataset itself.

3. Will the collected dataset be made publically available?
soundness: 4
strengths: 1. The paper introduces an end-to-end audio-only conditioned video diffusion model, which moves beyond traditional methods that rely on spatial constraints for motion stabilization.

2. The proposed novel modules like inter- and intra-clip temporal modules and audio-to-latents module are well-designed, resulting in more natural and consistent portrait movements and leading to better synchronization and more expressive facial movements.

3. The paper includes extensive experiments that demonstrate Loopy’s superiority over other audio-driven portrait diffusion models both quantitatively and qualitatively, with evidence of more lifelike and stable video outputs in the supplemental website.

4. The paper is well-written, the proposed components and architecture are described clearly.
summary: The paper presents Loopy, an innovative audio-driven diffusion model for generating portrait videos that addresses limitations in current methods related to motion naturalness and dependency on auxiliary spatial signals. Existing approaches often compromise the natural freedom of movement by using preset spatial constraints like movement regions or face locators to stabilize motion, leading to repetitive and less dynamic results.

Loopy stands out by adopting an end-to-end audio-only conditioning framework, leveraging two main components: 1. Inter- and Intra-clip Temporal Modules: These modules are designed to extend the model’s temporal receptive field, enabling it to utilize long-term motion dependencies and generate consistent, natural motion across video frames without external movement constraints; 2. Audio-to-Latents Module: This module enhances the correlation between audio input and portrait motion by converting audio features and motion-related characteristics into latent space representations that guide the synthesis process.

Experiments show that Loopy outperforms existing methods, generating lifelike and stable videos with natural facial expressions.
weaknesses: 1. While the audio-to-latents module improves the audio-motion correlation, there is no mention of how different audio characteristics (e.g., background noise, varying loudness) might impact the model’s performance, which could be critical for real-world applications.

2. The paper lacks a detailed analysis of potential failure modes or scenarios where Loopy may struggle. Highlighting these cases would provide a more balanced view of the model's robustness and limitations.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Currently, end-to-end audio-driven portrait generation is typically trained on training sets of varying sizes, which is crucial for a good model. How can we reasonably evaluate the performance of the method?
2. In Table 3, the metrics for audio-visual synchronization related to Loopy w/o TSM and w/o ASL still outperform other methods. Does this indicate that the performance improvement of the method primarily comes from the self-collected data?
3. Regarding training A2L, how do head movements and expressions individually affect the results?
soundness: 4
strengths: 1. The motivation is clear. The authors focus on the weak correlation between audio and portrait motion in end-to-end audio-driven methods.
2. Overall, this paper is easy to follow. The proposed TSM module is technically sound in its design, and the experimental validation is effective.
3. Many synchronized and vivid portrait videos are generated.
summary: This paper proposes an end-to-end audio-driven portrait video generation method. This method introduces an inter- and intra-clip temporal module and an audio-to-latent module to establish long-term natural correlations between audio and portrait movements. Many lifelike and impressive results are presented.
weaknesses: 1. In the A2L module, the effects of Movement and Expression on the method have not been thoroughly validated. The audio inputs shown in Fig. 4 are somewhat confusing. I assume they refer to audio features from wav2vec. 
2. Human expressions are closely related to many facial details, but the implementation in the paper is rather trivial. 
    1) the detected landmarks are too sparse and not accurate enough (DWPose), which makes it difficult to capture a person's expression accurately. 
    2) using the variance of keypoints to calculate head movement and expression changes presents several practical issues, 
such as the entanglement of head movement and camera movement. Why not use FLAME coefficients or results from other emotion estimation methods? 
3. The TSM module needs a deeper discussion on its impact on overall computational efficiency.
4. In Tables 1 and 2, the methods perform worse than others on some metrics, especially those related to Glo and Exp. The authors do not provide detailed analysis or discussion on this.
5. The paper has several writing issues. Some symbols and abbreviations are introduced without explanation, such as TSM in Fig. 2. Additionally, some text in the figures is too small to read, such as "other computational layers" in Fig. 3. The main paper does not reference Table 2. There are also some typos, such as in Line 302, where there is an error with punctuation.
6. The paper does not include a discussion of the limitations of the proposed method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. During inference what if motion frames are provided? How would they influence the results?
2. Can the overall head motion be controlled?
3. (L291) Is there any analysis of the strong correlation between the head movement and expression variances? Can the type of expression be controlled?
soundness: 4
strengths: 1. The results are good. 
2. The introduction of two modules (Temporal and Audio) is reasonable and interesting. Ablation study supports the benefits of these modules.
summary: This paper introduces an audio2video model for co-speech human portrait video synthesis. A novel temporal module is proposed to enable natural movement generation. A joint audio, movement, and expression latent space is learned to achieve better head pose and facial expression control from speech. Experiments and demonstrations show better performance and more realistic results.
weaknesses: 1. Lack of ablation of stand-alone intra- / inter-temporal model. Is both of them necessary or only the inter-clip temporal layer is enough?
2.  The functionality of the Temporal Segment Model is unclear. Is it for capturing the appearance of the character under different expressions? If so, why (L478) longer motion frames lead to worse results?
3. Similar to the above issue. I watched the video samples of the ablated model. Seems to me the ablation of either part leads to similar degradations — lack of head pose variance and subtle expression. This makes me unclear about the different roles of the two proposed modules.

Meta-review: The paper presents Loopy, an audio-driven portrait animation pipeline that demonstrates natural talking-head motion. The key innovation in Loopy against prior works is the use of diffusion-based synthesis using long-range motion frames that leads up to the currently synthesized frame, thus allowing access to the context for producing subtle, consistent, and continuous head movements. The model is trained on a very large dataset and experiments show state-of-the-art results. The provided videos show high-quality in talking head generation. 

The paper received four reviews, all recommending accept. The reviewers liked the quality of the generated videos, quality of the contributions, organization of the paper, and the extensive experimental results presented demonstrating state-of-the-art results.

Decision: Accept (Oral)