<figure><img src="image_1.png" /></figure>

HAL
open science

# LEO: Generative Latent Image Animator for Human Video Synthesis

Yaohui Wang, Xin Ma, Xinyuan Chen, Cunjian Chen, Antitza Dantcheva, Bo

Dai, Yu Qiao

► To cite this version:

Yaohui Wang, Xin Ma, Xinyuan Chen, Cunjian Chen, Antitza Dantcheva, et al. LEO: Generative Latent Image Animator for Human Video Synthesis. International Journal of Computer Vision, 2024, 10.1007/s11263-024-02231-3. hal-04837899

HAL Id: hal-04837899

https://inria.hal.science/hal-04837899v1

Submitted on 14 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LEO: Generative Latent Image Animator for Human Video Synthesis

Yaohui Wang¹, Xin Ma¹,², Xinyuan Chen¹, Cunjian Chen², Antitza Dantcheva³,
Bo Dai¹, Yu Qiao¹

¹Shanghai Artificial Intelligence Laboratory, Shanghai, China.

²Monash University, Melbourne, Australia.

³Inria, Université Côte d'Azur, Valbonne, France.

## Abstract

Spatio-temporal coherency is a major challenge in synthesizing high quality videos, particularly in synthesizing human videos that contain rich global and local deformations. To resolve this challenge, previous approaches have resorted to different features in the generation process aimed at representing appearance and motion. However, in the absence of strict mechanisms to guarantee such disentanglement, a separation of motion from appearance has remained challenging, resulting in spatial distortions and temporal jittering that break the spatio-temporal coherency. Motivated by this, we here propose LEO, a novel framework for human video synthesis, placing emphasis on spatio-temporal coherency. Our key idea is to represent motion as a sequence of flow maps in the generation process, which inherently isolate motion from appearance. We implement this idea via a flow-based image animator and a Latent Motion Diffusion Model (LMDM). The former bridges a space of motion codes with the space of flow maps, and synthesizes video frames in a warp-and-inpaint manner. LMDM learns to capture motion prior in the training data by synthesizing sequences of motion codes. Extensive quantitative and qualitative analysis suggests that LEO significantly improves coherent synthesis of human videos over previous methods on the datasets TaichiHD, FaceForensics and CelebV-HQ. In addition, the effective disentanglement of appearance and motion in LEO allows for two additional tasks, namely infinite-length human video synthesis, as well as content-preserving video editing. Project page: https://wyhsirius.github.io/LEO-project/

**Keywords:** Video generation, diffusion models, generative modeling

## 1 Introduction

Deep generative models such as generative adversarial networks (GANs) [1] and Diffusion Models [2, 3] have fostered a breakthrough in video synthesis [4–14], elevating tasks such as text-to-video generation [12, 13], video editing [15], as well as 3D-aware video generation [16]. While existing work has demonstrated promising results w.r.t. frame-wise visual quality, synthesizing videos

of strong spatio-temporal coherency, tailored to human videos, containing rich global and local deformations, remains challenging.

Motivated by this, we here propose an effective generative framework, placing emphasis on spatio-temporal coherency in human video synthesis. Having this in mind, a fundamental step has to do with the *disentanglement* of videos w.r.t. appearance and motion. Previous approaches have tackled such disentanglement by two jointly trained
<figure><img src="image_3.png" /><figcaption>**Fig. 1:** Our framework caters a set of video synthesis tasks including (i) unconditional video generation (first and second row), (ii) conditional generation based on one single image (fourth row) and (iii) video editing from the starting image (third and fifth row). Results pertain to our model being trained on the datasets TaichiHD, FaceForensics and CelebV-HQ.</figcaption></figure>

distinct networks, respectively providing appearance and motion features [5, 7–9, 17], as well as by a two-phase generation pipeline that firstly aims at training an image generator, and then at training a temporal network to generate videos in the image generator’s latent space [11, 18, 19]. Nevertheless, such approaches encompass limitations related to spatial artifacts (e.g., distortions of body structures and facial identities in the same sequence), as well as temporal artifacts (e.g., inter-frame semantic jittering), even in short generated videos of 16 frames. We argue that such limitations stem from incomplete disentanglement of appearance and motion in the generation process. Specifically, without predominant mechanisms or hard constraints to guarantee disentanglement,

even a minor perturbation in the high-level semantics will be amplified and will lead to significant changes in the pixel space.

Deviating from the above and towards disentangling videos w.r.t. appearance and motion, in this paper we propose a novel framework for human video generation, referred to as LEO, streamlined to ensure strong spatio-temporal coherency. At the core of this framework is a sequence of flow maps, representing motion semantics, which inherently isolate motion from appearance. Specifically, LEO incorporates a latent motion diffusion module (LMDM), as well as a flow-based image animator. In order to synthesize a video, an initial frame is either provided externally for conditional generation, or obtained
<figure><img src="image_4.png" /><figcaption>**Fig. 2: Inference stage.** At the inference stage, LMDM firstly accepts a starting motion code $\alpha_1$ and a sequence of noise-vectors as input, in order to generate a sequence of motion codes **a**, further utilized to synthesize a sequence of flow maps $\phi_i$ by the pre-trained image animator. The output video is obtained in a warp-and-inpaint manner based on $x_1$ and $\phi_i$.</figcaption></figure>

by a generative module for *unconditional generation*. Given such initial frame and a sequence of motion codes sampled from the LMDM, the flow-based image animator generates a sequence of flow maps, and proceeds to synthesize the corresponding sequence of frames in a *warp-and-inpaint* manner.

The training of LEO is decomposed into *two phases*. Firstly, we train the flow-based image animator to encode input images into low-dimensional latent motion codes, and map such codes to flow maps, which are used for reconstruction via warp-and-inpaint. Therefore, once trained, the flow-based image animator naturally provides a space of motion codes that are strictly constrained to only containing motion-related information. At the *second stage*, upon the space provided by the image animator, we train the LMDM to synthesize sequences of motion codes and capture *motion prior* in the training data. To endow LEO with the ability to synthesize videos of arbitrary length beyond the short training videos, we adopt a Linear Motion Condition (LMC) mechanism in LMDM. As opposed to directly synthesizing sequences of motion codes, LMC enables LMDM to synthesize sequences of residuals w.r.t. a starting motion code, in order for longer videos to be easily obtained by concatenating additional sequences of residuals.

To evaluate LEO, we conduct extensive experiments pertained to three human video datasets, including TaichiHD [20], FaceForensics [21], and CelebV-HQ [22]. Compared to previous video

synthesis methods, LEO demonstrates a significantly improved spatio-temporal coherency, even on synthesized videos of length of 512 frames. In addition, LEO shows great potential in two extended tasks, namely *infinite-length video synthesis*, as well as *video editing* of a style in a synthesized video, while maintaining the content of the original video.

## 2 Related Works

Unconditional video generation aims to generate videos by learning the full distribution of training dataset. Most of the previous works [4, 5, 7, 8, 23–26] are built upon GANs [1, 27–30] towards benefiting from the strong performance of the image generator. Approaches [31–34] based on VAEs [35] were also proposed while only show results on toy datasets. Recently, with the progress of deep generative models (e.g., VQVAE [36], VQGAN [37], GPT [38] and Denoising Diffusion Models [2, 3, 39]) on both image [40, 41] and language synthesis [42], as well as the usage of large-scale pre-trained models, video generation also started to be explored with various approaches.

MoCoGANHD [18] builds the model on top of a well-trained StyleGAN2 [30] by integrating an LSTM in the latent space towards disentangling content and motion. DIGAN [9] and StyleGAN-V [10] and MoStGAN-V [43], inspired by NeRF [44], proposed an implicit neural representation approach to model time as a continuous signal aiming for long-term video generation. VideoGPT [19] and TATS [11] introduced to first train 3D-VQ models to learn discrete spatio-temporal codebooks, which are then be refined temporally by modified transformers [45]. Recently, several works [46–48] have shown promising capacity to model complex video distribution by incorporating spatio-temporal operations in Diffusion Models. While previous approaches have proposed various attempts either in training strategies [11, 18, 19] or in model architectures [7–10] to disentangle appearance and motion, due to the lack of strong constraints, it is still difficult to obtain satisfying results.

In contrast to unconditional video generation, conditional video generation seeks to produce high-quality videos, following image-to-image generation pipeline [49–51]. In this context,
<figure><img src="image_5.png" /><figcaption>**Fig. 3: Overview of LEO.** Our framework incorporates two main parts, (i) an image animator, aiming to generate flow maps and synthesize videos in the pixel space, and (ii) Latent Motion Diffusion Model (LMDM), focusing on modeling the motion distribution in a latent motion space. Our framework requires a two-phase training. In the first phase, we train the image animator in a self-supervised manner towards mapping latent codes to corresponding flow maps $\phi_i$. Once the image animator is well-trained, motion codes **a** are extracted from a frozen encoder and used as inputs of LMDM. In the second phase, LMDMs are trained to learn the motion distribution by providing the starting motion $\alpha_1$ as condition. Instead of directly learning the distribution of **a**, we adopt a Linear Motion Condition (LMC) mechanism in LMDM towards synthesizing sequences of residuals with respect to $x_1$. At the inference stage, given a starting image $x_i$ and corresponding motion code $\alpha_i$, LMDM firstly generates a motion code sequence, which is then used by the image animator to generate flow maps to synthesize output videos in a warp-and-inpaint manner.</figcaption></figure>

additional signals such as semantic maps [52–54], human key-points [54–60], motion labels [17], 3DMM [61, 62] and optical flow [63, 64] have been exploited to guide motion generation. In addition, text description, has been used in large-scale video diffusion models [12, 14, 46, 65–72] for high-quality video generation. Our framework also supports for conditional video generation based on a single image. However, unlike previous approaches, our method follows the image animation pipeline [20, 73, 74] which leverages the dense flow maps for motion modeling. We introduce our method in details in the following.

## 3 Method

Fig. 3 illustrates the training of LEO, comprising of two-phases. We firstly train an image animator towards learning high-quality latent motion codes of the datasets. In the second phase, we train the Latent Motion Diffusion Model (LMDM) to learn a motion prior over the latent motion codes. To

synthesize a video, the pre-trained image animator takes the motion codes to generate corresponding flow maps, which are used to warp and inpaint starting frame. The warp-and-inpaint operation is conducted in two modules inside image animator. The warping module firstly produces flow fields based on motion codes to warp starting frame, then the inpainting module learns to fill in the holes in the warped starting frame and refine the entire image. Each video sequence is produced frame by frame.

We formulate a video sequence $v = \{x_i\}_{i=1}^L, x_i \sim \mathcal{X} \in \mathbb{R}^{3 \times H \times W}$ as $v = \{\mathcal{T}(x_1, G(\alpha_i))\}_{i=2}^L, \alpha_i \sim \mathcal{A} \in \mathbb{R}^{1 \times N}$, where $x_i$ denotes the $i^{th}$ frame, $\alpha_i$ denotes a latent motion code at timestep $i$, $G$ represents the generator in the image animator aiming to generate a flow map $\phi_i$ from $\alpha_i$.

### 3.1 Learning Latent Motion Codes

Towards learning a frame-wise latent motion code, we adopt the state-of-the-art image animator
LIA [74] as it enables to encode input images into corresponding motion codes. LIA consists of two modules, an encoder E and a generator G. During training, given a source image $x_s$ and a driving image $x_d$, E encodes $x_s, x_d$ into a motion code $\alpha = E(x_s, x_d)$, and G generates a flow field $\phi = G(\alpha)$ from the code. LIA is trained in a self-supervised manner with the objective to reconstruct the driving image.

Training LIA in such a self-supervised manner brings two notable benefits for our framework, (i) it enables LIA to achieve high-quality perceptual results, and (ii) as a motion code is strictly equivalent to flow maps, there are guarantees that $\alpha$ is only motion-related without any appearance interference.

## 3.2 Leaning a Motion Prior

Once LIA is well-trained on a target dataset, for any given video $v = \{x_i\}_{i=1}^L$, we are able to obtain a motion sequence $\mathbf{a} = \{\alpha_i\}_{i=1}^L$ with the frozen E. In the second phase of our training, we propose to learn a motion prior by temporal Diffusion Models.

Unlike image synthesis, data in our second phase is a set of sequences. We firstly apply a temporal Diffusion Model for modeling the temporal correlation of $\mathbf{a}$. The general architecture of this model is a 1D U-Net adopted from [2]. To train this model, we follow the standard training strategy with a simple mean-squared loss,

$$L_{\text{LMDM}} = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,1), t} \left[ \| \epsilon - \epsilon_{\theta}(\mathbf{a}_t, t) \|_2^2 \right], \quad (1)$$

where $\epsilon$ denotes the unscaled noise, $t$ is the time step, $\mathbf{a}_t$ is the latent noised motion code to time $t$. During inference, a random Gaussian noise $\mathbf{a}_T$ is iteratively denoised to $\mathbf{a}_0 = \{\alpha_i\}_{i=1}^L$, and the final video sequence is obtained through the generator.

At the same time in our experiments, we found that learning motion sequences in a complete unconditional manner brings to the fore limitations, namely (i) the generated codes are not consistent enough for producing smooth videos, as well as (ii) the generated motion codes can only be used to produce fixed length videos. Hence, towards addressing those issues, we propose a conditional Latent Motion Diffusion Model (cLMDM) which aims for high-quality and long-term human videos.

One major characteristic of LIA has to do with the linear motion space. Any motion code $\alpha_t$ in $\mathbf{a}$ can be re-formulated as

$$\alpha_i = \alpha_1 + m_i, i \ge 2, \quad (2)$$

where $\alpha_1$ denotes the motion code at the first timestep and $m_i$ denotes the motion difference between timestep 1 and $i$, so that we can re-formulate $\mathbf{a}$ as

$$\mathbf{a} = \alpha_1 + \mathbf{m}, \quad (3)$$

where $\mathbf{m} = \{m_i\}_{i=2}^L$ denotes the motion difference sequence. Therefore, Eq. 2 and 3 indicate that a motion sequence can be represented by $\alpha_1$ and $\mathbf{m}$. Based on this, we propose a Linear Motion Condition (LMC) mechanism in cLMDM to condition the generative process with $\alpha_1$. During training, at each time step, we only add noise onto $\mathbf{m}_t$ instead of the entire $\mathbf{a}$ and leave $\alpha_1$ intact. The objective function of cLMDM is

$$L_{\text{cLMDM}} = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,1), t} \left[ \| \epsilon - \epsilon_{\theta}(\mathbf{m}_t, \alpha_1, t) \|_2^2 \right], \quad (4)$$

where $\alpha_1$ denotes the condition signal and $\mathbf{m}_t$ stands for the noised $\mathbf{m}$ to time $t$. $\alpha_1$ is first added on $\mathbf{m}_t$ and then concatenated along temporal dimension. LMC will be applied at each time step until we reach $\mathbf{m}_0$. The final motion sequence is obtained as $\mathbf{a}_0 = [\alpha_1, \mathbf{m}_0]$. We find that following this, a related generated motion sequence is more stable and contains fewer artifacts, as $\alpha_1$ serves as a strong signal to constrain the generated $\mathbf{m}$ to follow the initial motion.

While the results from cLMDM outperforms previous models, the groundtruth $\alpha_1$ is necessitated during both, training and inference stage. Towards unconditional generation, we train an additional simple DM to fit the distribution $p(\alpha_i)$ in a frame-wise manner. We refer to the cLMDM and such simple DM jointly as Latent Motion Diffusion Model (LMDM). By this way, LMDM are able to work in both conditional and unconditional motion generation.

Towards generating videos of arbitrary length, we propose an autoregressive approach based on proposed LMDM. By taking the last motion code from the previous generated sequence as
<figure><img src="image_6.png" /><figcaption>**Fig. 4: Qualitative Comparison.** We qualitatively compare LEO with DIGAN, TATS, StyleGAN-V on short video generation. The results indicate that on both (a) TaichiHD (128 and 256 resolutions) and (b) FaceForensics datasets, our proposed method achieves the best visual quality and is able to capture the human structure well. Other approaches either modify the facial structure (e.g., StyleGAN-V) or fail to generate a complete human body (e.g., TATS and DIGAN).</figcaption></figure>

the $\alpha_1$ in the current sequence, with a randomly sampled noise, LMDM are able to generate an infinite-length motion sequence. By combining such sequence in pre-trained LIA with a starting image, LEO can synthesize photo-realistic and long-term videos.

## 3.3 Learning Starting Frames

In our framework, a starting image $x_1$ is required to synthesize a video. As image space is modeled independently, here we propose two options to obtain $x_1$.

**Option 1: existing images.** The first option is to directly take the images either from a real distribution or from an image generation network.
<figure><img src="image_7.png" /><figcaption>**Fig. 5:** Comparison on long-term video generation. We compare with TATS by generating 512-frame videos. Videos from TATS start crashing around 50 frames while our model is able to continue producing high-quality frames with diverse motion.</figcaption></figure>

In this context, our model is a conditional video generation model, which learns to predict future motion from $x_1$. Starting motion $\alpha_1$ is obtained through $\alpha_1 = E(x_1)$.

**Option 2: conditional Diffusion Models.** The second option is to learn a conditional DDPM [2] (cDDPM) with $\alpha_1$ as a condition to synthesize $x_1$. By combining LEO with LMDM as well as cDDPM, we are able to conduct unconditional video synthesis.

# 4 Experiments

In this section, we firstly briefly describe our experimental setup, introducing datasets, evaluation metrics and implementation details. Secondly, we qualitatively demonstrate generated results on both, short and long video synthesis. Then we show quantitative evaluation w.r.t. video quality, comparing LEO with SoTA. Next, we conduct an ablation study to prove the effectiveness of proposed conditional mechanism LMC. Finally, we provide additional analysis of our framework, exhibiting motion and appearance disentanglement, video editing and infinite-length video generation.

**Datasets.** As we focus on human video synthesis, evaluation results are reported on three human-related datasets, TaichiHD [20], FaceForensics [21] and CelebV-HQ [22]. We use both 128 × 128 and 256 × 256 resolution TaichiHD datasets, and only 256 × 256 resolution FaceForensics and CelebV-HQ datasets.

* **TaichiHD** [20] comprises 3100 video sequences downloaded from YouTube. In train and test splits, it contains 2815 and 285 videos, respectively. We conducted all our experiments on the

train split and used both 128×128 and 256×256 resolutions in our experiments.

* **FaceForensics** [21] includes 1000 video sequences downloaded from YouTube. Following the preprocessing of previous methods [6, 10], face areas are cropped based on the provided per-frame meshes. We resized all videos to 256 × 256 resolution.

* **CelebV-HQ** [22] comprises 35666 high-quality talking head videos of 3 to 20 seconds each. In total, it represents 15653 celebrities. We resized the original videos to 256 × 256 resolution, in order to train our models.

**Evaluation metric.** For quantitative evaluation, we apply the commonly used metrics FVD and KVD, in order to compare with other approaches on video quality and apply Average Content Distance (ACD) towards evaluating the identity consistency of faces and bodies in the generated videos. In addition, we conduct a user study with 20 users towards comparing with objective quantitative evaluation.

* **Frechet video distance (FVD) and Kernel Video Distance (KVD).** We use I3D [75] trained on Kinetics-400 as feature extractor to compute FVD and KVD. However, we find FVD is a very sensitive metric, which can be affected by many factors such as frame-rate, single image quality, video length and implementation, which also mentioned in [10]. Therefore, towards making a fair comparison, on the TaichiHD dataset, we adopt the implementation from DIGAN [9]. As for FaceForensics and CelebV-HQ, we chose to follow the implementation of StyleGAN-V [10].

* **Average Content Distance (ACD).** ACD measures the content consistency in generated videos. To evaluate results from FaceForensics
and TaichiHD, we extract features from each generated frame and proceed to extract a per-frame feature vector in a video. The ACD was then computed using the average pairwise L2 distance of the per-frame feature vectors. We follow the implementation in [18] to compute ACD for FaceForensics. As for TaichiHD, we employ the pre-trained person-reID model [76] to extract person identity features.

*   **User study.** We asked 20 human raters to evaluate generated video quality, as well as video coherency. In each user study, we show paired videos and ask the raters, to rate 'which clip is more realistic / which clip is more coherent'. Each video-pair contains one generated video from our method, whereas the second video is either *real* or generated from other methods.

**Implementation details.** Our framework requires two-phase training. In the first phase, we follow the standard protocol to train LIA [74] to encode input images into low-dimensional latent motion codes, and map such codes to flow maps, which are used for reconstruction via warp-and-inpaint. Therefore, once trained, LIA naturally provides a space of motion codes that are strictly constrained to only containing motion-related information. In the second phase, we only train LMDM on the extracted motion codes from Encoder. We note that the LMDM is a 1D U-Net adopted from [39], we set the input size as 64×20, where 64 is the length of the sequence and 20 is the dimension of the motion code. We use 1000 diffusion steps and a learning rate of 1e-4. As the training of LMDM is conducted in the latent space of LIA, the entire training is very efficient and only requires one single GPU.

## 4.1 Qualitative Evaluation

We qualitatively compare LEO with SoTA by visualizing the generated results. We firstly compare our method with DIGAN, TATS and StyleGAN-V on the FaceForensics and TaichiHD datasets for short video generation. As shown in Fig. 1 and 4, the visual quality of our generated results outperforms other approaches w.r.t both, appearance and motion. For both resolutions on TaichiHD datasets, our method is able to generate complete human structures, whereas both, DIGAN and TATS fail, especially for arms and

legs. When compared with StyleGAN-V on FaceForensics dataset, we identify that while LEO preserves well facial structures, StyleGAN-V modifies such attributes when synthesizing large motion.

Secondly, we compare with TATS for long-term video generation. Specifically, 512 frames are produced for the resolution 128 × 128 per-tained to the TaichiHD dataset. As shown in Fig. 5, the subject in the videos from TATS starts crashing around 50 frames and the entire video sequence starts to fade. On the other hand, in our results, the subject continues to perform diverse actions whilst well preserving the human structure. We note that our model is only trained using a 64-frame sequence.

## 4.2 Quantitative evaluation

In this section, we compare our framework with five state-of-the-art for both, conditional and unconditional short video generation, as well as unconditional long-term video generation.

**Unconditional short video generation.** In this context, as described in Sec. 3.3, Option 2, the $x_1$ is randomly generated by a pre-trained cDDPM. We compare with SoTA by generating 16 frames. To compare with DIGAN on high-resolution generation, we also generate videos of 256×256 resolution. Related FVDs and KVDs are reported in Tab. 1. LEO systematically outperforms other methods w.r.t. video quality, obtaining lower or competitive FVD and KVD on all datasets. On high-resolution generation, our results remain better than DIGAN.

However, by comparing the results between StyleGAN-V and ours, we find FVD is not able to represent the quality of generated videos veritably. We observe that StyleGAN-V is not able to preserve facial structures, whereas LEO is able to do so, see Fig. 4. We additionally compute ACD, in order to further analyze the identity consistency in 16-frame videos. Tab. 1 reflects on the fact that our method achieves significantly better results compared to other approaches. In addition, we conduct user study w.r.t. video quality and coherency of generated videos among different methods. Results in Tab. 3 showcase that as nearly all users rated for our generated results to be superior than other approaches. Hence, we conclude that a metric, replacing FVD is in urgent need in the context of video generation.
<table><caption>**Table 1:** Evaluation for unconditional and conditional short video generation. LEO systematically outperforms other approaches on conditional video generation, and achieves better or competitive results on unconditional generation w.r.t. FVD, KVD and ACD. (*results are reproduced based on official code and released checkpoints.)</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">TaichiHD128</th><th colspan="3">TaichiHD256</th><th colspan="2">FaceForensics</th><th>CelebV-HQ</th></tr><tr><th>FVD<sub>16</sub></th><th>KVD<sub>16</sub></th><th>ACD<sub>16</sub></th><th>FVD<sub>16</sub></th><th>KVD<sub>16</sub></th><th>ACD<sub>16</sub></th><th>FVD<sub>16</sub></th><th>ACD<sub>16</sub></th><th>FVD<sub>16</sub></th></tr></thead><tbody><tr><td>MoCoGAN-HD</td><td>144.7 ± 6.0</td><td>25.4 ± 1.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>111.8</td><td>0.33</td><td>212.4</td></tr><tr><td>DIGAN</td><td>128.1 ± 4.9</td><td>20.6 ± 1.1</td><td>2.17</td><td>156.7 ± 6.2</td><td>-</td><td>-</td><td>62.5</td><td>-</td><td>72.9</td></tr><tr><td>TATS</td><td>136.5 ± 1.2*</td><td>22.2 ± 1.0*</td><td>2.28</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>StyleGAN-V</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>47.4</td><td>0.36</td><td>69.1</td></tr><tr><td>MoStGAN-V</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>39.7</td><td>0.38</td><td>132.1</td></tr><tr><td>Ours (uncond)</td><td>100.4 ± 3.1</td><td>11.4 ± 3.2</td><td>1.83</td><td>122.7 ± 1.1</td><td>20.49 ± 0.9</td><td>-</td><td>52.3</td><td>0.28</td><td>-</td></tr><tr><td>Ours (cond)</td><td><b>57.6 ± 2.0</b></td><td><b>4.0 ± 1.5</b></td><td><b>1.22</b></td><td><b>94.8 ± 4.2</b></td><td><b>13.47 ± 2.3</b></td><td>-</td><td><b>35.9</b></td><td><b>0.27</b></td><td><b>40.2</b></td></tr></tbody></table>

<table><caption>**Table 2:** Evaluation for unconditional long-term video generation. LEO outperforms other methods on long-term (128 frames) video generation w.r.t. FVD, KVD and ACD.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">TaichiHD128</th><th colspan="2">FaceForensics</th></tr><tr><th>FVD<sub>128</sub></th><th>KVD<sub>128</sub></th><th>ACD<sub>128</sub></th><th>FVD<sub>128</sub></th><th>ACD<sub>128</sub></th></tr></thead><tbody><tr><td>DIGAN</td><td>-</td><td>-</td><td>-</td><td>1824.7</td><td>-</td></tr><tr><td>TATS</td><td>1194.58 ± 1.1</td><td>462.03 ± 8.2</td><td>2.85</td><td>-</td><td>-</td></tr><tr><td>StyleGAN-V</td><td>-</td><td>-</td><td>-</td><td><b>89.34</b></td><td>0.49</td></tr><tr><td>Ours</td><td><b>155.54 ± 2.6</b></td><td><b>48.82 ± 5.9</b></td><td><b>2.06</b></td><td>96.28</td><td><b>0.34</b></td></tr></tbody></table>

**Unconditional long video generation** We evaluate our approach for long-term video generation w.r.t. FVD and ACD. In this context, we compare LEO with StyleGAN-V on the FaceForensics dataset, and both DIGAN and TATS on the TaichiHD. We report results based on 128-frame generation in Tab. 2, which clearly shows that our method outperforms others in such context. We hypothesize that consistent and stable motion codes produced by our LMDM are key to producing high-quality long-term videos.

**Conditional short video generation** As described in Sec. 3.3, Option 1, our framework additionally caters for conditional video generation by taking an existing image to hallucinate the following motion. Specifically, we randomly select 2048 images from both, TaichiHD and FaceForensics datasets as $x_1$ and compute corresponding $\alpha_1$ as input of LMDM. As depicted in Tab. 1, results conditioned on the real images achieve the lowest FVD, KVD and ACD values, suggesting that the quality of a starting image is pertinent for output video quality, which further signifies that in the setting of unconditional generation, training a better cDDPM will be instrumental for improving results.

<table><caption>**Table 3:** User study. We conduct user studies pertaining to the datasets TaichiHD and FaceForensics w.r.t. video quality (up) as well as coherency (down).</caption><thead><tr><th rowspan="2">Method</th><th>TaichiHD (%)</th><th>FaceForensics (%)</th></tr><tr><th>Ours / TATS</th><th>93.00 / 7.00</th></tr></thead><tbody><tr><td>Ours / StyleGAN-V</td><td>-</td><td><b>91.33</b> / 8.67</td></tr></tbody></table><table><thead><tr><th rowspan="2">Method</th><th>TaichiHD (%)</th><th>FaceForensics (%)</th></tr><tr><th>Ours / TATS</th><th>98.60 / 1.40</th></tr></thead><tbody><tr><td>Ours / StyleGAN-V</td><td>-</td><td><b>93.20</b> / 6.80</td></tr></tbody></table>

<table><caption>**Table 4:** Ablation study of proposed LMC. Models with LMC achieved the lowest FVD on both datasets.</caption><thead><tr><th></th><th>TaichiHD</th><th>FaceForensics</th></tr></thead><tbody><tr><td>w/o LMC</td><td>118.6</td><td>60.03</td></tr><tr><td>with LMC</td><td><b>100.4</b></td><td><b>52.32</b></td></tr></tbody></table>

## 5 Ablation Study

In this section, we place emphasis on analyzing the effectiveness of proposed Linear Motion Condition (LMC) in LMDM. We train two models, with and without LMC on both TaichiHD and FaceForensics datasets. As shown in Tab. 4, using LMC significantly improves the generated video quality, which proves that our proposed LMC is an effective mechanism for involving $\alpha_1$ in LMDM.

## 6 Additional Analysis

**Motion and appearance disentanglement.** We proceed to combine the same $x_1$ with different $m$, aiming to reveal whether $m$ is only motion-related. Fig. 6 illustrates that different $m$ enables the same subject to perform different motion - which proves that our proposed LMDM
<figure><img src="image_8.png" /><figcaption>**Fig. 6: Disentanglement of motion and appearance.** The first and second row share the same appearance, with different motion codes. Results display that our model is able to produce diverse motion from the same content.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Fig. 7: Video editing.** We show video editing results by combining LEO with off-the-shelf image editing model ControlNet. We are able to edit the appearance of the entire video sequence through only editing the starting image.</figcaption></figure>

is indeed learning a motion space, and appearance and motion are clearly disentangled. This experiment additionally indicates that our model does not overfit on the training dataset, as different noise sequences are able to produce diverse motion sequences.

**Video Editing.** As appearance is modeled in $x_1$, we here explore the task of video editing by modifying the semantics in the starting image. Compared to previous approaches, where image-to-image translation is required, our framework simply needs an edit of the semantics in an one-shot manner. Associated results are depicted in Fig. 1 and Fig. 7. We apply the open-source approach ControlNet [77] on the starting frame by entering various different prompts. Given that the motion space is fully disentangled from the appearance space, our videos maintain the original temporal consistency, uniquely altering the appearance.

**Infinite-length video generation.** In addition to presented settings, our framework is able to generate infinite-length videos. To generate long-term FaceForensics, as shown in Fig. 9, we

provide the last generated code from the previous sequence as the starting code of the current sequence. The entire long-term video is generated in an *autoregressive* manner. Surprisingly, we find that such a simple approach is sufficient to produce more than 1000 frames. We note that for TaichiHD dataset, due to limited motion patterns, this setting yields repeated motion. Towards addressing this limitation, as shown in Fig. 8, we design an additional *Transition Diffusion Model (Transition DM)* aimed at generating transition motion between the last code from original generated sequence and a new motion code generated from the *simple DM*. Doing so, the Transition DM enforces the network to exit the original motion pattern and transit to new pattern. To evaluate the effectiveness of the proposed method, we generate long videos *with* and *without* Transition DM and request human raters to watch respective videos and answer the question ‘Does the clip contain repeated motion?’. Results are reported in Tab. 5, which shows the effectiveness of Transition DM to prevent repeated motion.
<figure><img src="image_10.png" /><figcaption>Fig. 8: Infinite-length video generation for TaichiHD.</figcaption></figure>

<table><caption>**Table 5: User study of repeated motion.** We show the occurrence of repeated motion with and without the usage of Transition DM.</caption><thead><tr><th></th><th>Occurrence (%)</th></tr></thead><tbody><tr><td>w/o Transition DM</td><td>0.45</td></tr><tr><td>with Transition DM</td><td>0.02</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 9: Infinite-length video generation for Face-Forensics.</figcaption></figure>

Compared to current diffusion-based methods, our approach is very efficient in long video generation. We autoregressively run LMDM for the generation of long motion code sequences in latent

space. Given that LMDM is a small-scale network, which only focuses on generating 1-D motion code, even for very long sequence generation, it only requires few seconds during inference stage. In addition, the image animator itself is a one-step inference model which enables our proposed method to be significantly efficient.

## 7 Limitations

We list several limitations in current framework and proposed potential solutions for future work.

* *Geometry ambiguity and temporal coherency.* Since we use a 2D generator to predict 2D flow maps, LEO is not able to handle human body occlusion very well especially in Taichi dataset. One solution would be to incorporate the architecture of NeRF or Tri-plane into our generator to support 3D-aware generation. We think in this way, the issues of geometry ambiguity and human body occlusion could be addressed.

* *Generalizability.* Since the pre-trained image animator focuses on talking head and human bodies, our proposed framework currently performs better on human-centric videos. However, to analyze the generalizability of LEO, we conducted a small-scale experiment on UCF101 and report quantitative evaluation in Tab. 6. The results show that under current model design, LEO achieves competitive results with previous
GAN-based methods but still has large performance gap compared with large-scale video diffusion models.

We believe our framework is pushing the boundaries of video generation, as it solves a challenge, which constitutes generation of long human-centric videos. While this is a first step, the proposed method has the potential to generalize onto additional settings such as text-to-video generation. However, achieving such goals requires scaling up and re-designing (a) the original LIA, as well as (b) LMDM, and (c) training the entire system on larger-scale well-curated video datasets, which requires extremely expensive computational resources. We will explore such research directions in our future work.

<table><caption>**Table 6:** Quantitative evaluation on UCF101 w.r.t. FVD.</caption><thead><tr><th>Methods</th><th>FVD<sub>16</sub></th></tr></thead><tbody><tr><td>MoCoGAN-HD</td><td>1729.6</td></tr><tr><td>DIGAN</td><td>1630.2</td></tr><tr><td>StyleGAN-V</td><td>1431.0</td></tr><tr><td>Make-A-Video</td><td>367.23</td></tr><tr><td>Video LDM</td><td>550.61</td></tr><tr><td>LaVie</td><td>540.30</td></tr><tr><td>Ours</td><td>1356.2</td></tr></tbody></table>

* *Architecture.* Current architect of LEO still relies on convolutional networks in both image animator and latent motion diffusion models. Advanced techniques such as transformers have not been explored yet. Future work would be involving novel architecture design and training LEO on larger-scale dataset to explore the limits of current approach.

## 8 Conclusions

In this paper, we introduced LEO, a novel framework incorporating a Latent Image Animator (LIA), as well as a Latent Motion Diffusion Model (LMDM), placing emphasis on spatio-temporal coherency in human video synthesis. By jointly exploiting LIA and LMDM in a two-phase training strategy, we endow LEO with the ability to disentangle appearance and motion. We quantitatively and qualitatively evaluated proposed method on both, human body and talking head datasets and

demonstrated that our approach is able to successfully produce photo-realistic, long human videos. In addition, we showcased that the effective disentanglement of appearance and motion in LEO allows for two additional tasks, namely infinite-length human video synthesis by autoregressively applying LMDM, as well as content-preserving video editing (employing an off-the-shelf image editor (e.g., ControlNet)). We postulate that LEO opens a new door in design of generative models for video synthesis and plan to extend our method onto more general videos and applications.

## References

[1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)

[2] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS 33 (2020)

[3] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: ICLR (2021)

[4] Vondrick, C., Pirsiavash, H., Torralba, A.: Generating videos with scene dynamics. In: NIPS (2016)

[5] Tulyakov, S., Liu, M.-Y., Yang, X., Kautz, J.: MoCoGAN: Decomposing motion and content for video generation. In: CVPR (2018)

[6] Saito, M., Saito, S., Koyama, M., Kobayashi, S.: Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. IJCV (2020)

[7] Wang, Y., Bilinski, P., Bremond, F., Dantcheva, A.: G3AN: Disentangling appearance and motion for video generation. In: CVPR (2020)

[8] Wang, Y., Bremond, F., Dantcheva, A.: Inmodegan: Interpretable motion decomposition generative adversarial network for video generation. arXiv preprint arXiv:2101.03049 (2021)

[9] Yu, S., Tack, J., Mo, S., Kim, H., Kim, J., Ha, J.-W., Shin, J.: Generating videos with
dynamics-aware implicit generative adversarial networks. In: ICLR (2022)

[10] Skorokhodov, I., Tulyakov, S., Elhoseiny, M.: Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In: CVPR (2022)

[11] Ge, S., Hayes, T., Yang, H., Yin, X., Pang, G., Jacobs, D., Huang, J.-B., Parikh, D.: Long video generation with time-agnostic vqgan and time-sensitive transformer. In: ECCV (2022)

[12] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., Taigman, Y.: Make-a-video: Text-to-video generation without text-video data. In: ICLR (2023)

[13] Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., Erhan, D.: Phenaki: Variable length video generation from open domain textual descriptions. In: ICLR (2023)

[14] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)

[15] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. In: ECCV (2022)

[16] Bergman, A., Kellnhofer, P., Yifan, W., Chan, E., Lindell, D., Wetzstein, G.: Generative neural articulated radiance fields. NeurIPS 35 (2022)

[17] Wang, Y., Bilinski, P., Bremond, F., Dantcheva, A.: Imaginator: Conditional spatio-temporal gan for video generation. In: WACV (2020)

[18] Tian, Y., Ren, J., Chai, M., Olszewski, K., Peng, X., Metaxas, D.N., Tulyakov, S.: A good image generator is what you need for high-resolution video synthesis. In: ICLR (2021)

[19] Yan, W., Zhang, Y., Abbeel, P., Srini-vas, A.: Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 (2021)

[20] Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: First order motion model for image animation. In: NeurIPS (2019)

[21] Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics: A large-scale video dataset for forgery detection in human faces. arXiv preprint arXiv:1803.09179 (2018)

[22] Zhu, H., Wu, W., Zhu, W., Jiang, L., Tang, S., Zhang, L., Liu, Z., Loy, C.C.: CelebV-HQ: A large-scale video facial attributes dataset. In: ECCV (2022)

[23] Saito, M., Matsumoto, E., Saito, S.: Temporal generative adversarial nets with singular value clipping. In: ICCV (2017)

[24] Wang, Y.: Learning to Generate Human Videos. Theses, Inria - Sophia Antipolis ; Université Cote d'Azur (September 2021)

[25] Clark, A., Donahue, J., Simonyan, K.: Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571 (2019)

[26] Brooks, T., Hellsten, J., Aittala, M., Wang, T.-C., Aila, T., Lehtinen, J., Liu, M.-Y., Efros, A.A., Karras, T.: Generating long videos of dynamic scenes. (2022)

[27] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

[28] Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: ICLR (2019)

[29] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)

[30] Karras, T., Laine, S., Aittala, M., Hellsten,
J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR (2020)

[31] Denton, E.L., Birodkar, v.: Unsupervised learning of disentangled representations from video. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) NeurIPS (2017)

[32] Li, Y., Mandt, S.: Disentangled sequential autoencoder. ICML (2018)

[33] Bhagat, S., Uppal, S., Yin, Z., Lim, N.: Disentangling multiple features in video sequences using gaussian processes in variational autoencoders. In: ECCV (2020)

[34] Xie, J., Gao, R., Zheng, Z., Zhu, S.-C., Wu, Y.N.: Motion-based generator model: Unsupervised disentanglement of appearance, trackable and intrackable motions in dynamic patterns. In: AAAI (2020)

[35] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)

[36] Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. NeurIPS (2017)

[37] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: CVPR (2021)

[38] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)

[39] Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: ICML (2021)

[40] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: ICML (2021)

[41] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv

preprint arXiv:2204.06125 (2022)

[42] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)

[43] Shen, X., Li, X., Elhoseiny, M.: Mostgan-v: Video generation with temporal motion styles. In: CVPR (2023)

[44] Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: CVPR (2019)

[45] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)

[46] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. arXiv preprint arXiv:2204.03458 (2022)

[47] Yu, S., Sohn, K., Kim, S., Shin, J.: Video probabilistic diffusion models in projected latent space. In: CVPR (2023)

[48] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., Tan, T.: Videofusion: Decomposed diffusion models for high-quality video generation. In: CVPR (2023)

[49] Chu, C., Zhmoginov, A., Sandler, M.: CycleGAN: a master of steganography. arXiv preprint arXiv:1712.02950 (2017)

[50] Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-Image Translation with Conditional Adversarial Networks. In: CVPR (2017)

[51] Huang, X., Liu, M.-Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: ECCV (2018)

[52] Pan, J., Wang, C., Jia, X., Shao, J., Sheng, L., Yan, J., Wang, X.: Video generation from single semantic label map. arXiv preprint arXiv:1903.04480 (2019)

[53] Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G.,
Tao, A., Kautz, J., Catanzaro, B.: Video-to-video synthesis. In: NeurIPS (2018)

[54] Wang, T.-C., Liu, M.-Y., Tao, A., Liu, G., Kautz, J., Catanzaro, B.: Few-shot video-to-video synthesis. In: NeurIPS (2019)

[55] Jang, Y., Kim, G., Song, Y.: Video Prediction with Appearance and Motion Conditions. In: ICML (2018)

[56] Yang, C., Wang, Z., Zhu, X., Huang, C., Shi, J., Lin, D.: Pose guided human video generation. In: ECCV (2018)

[57] Walker, J., Marino, K., Gupta, A., Hebert, M.: The pose knows: Video forecasting by generating pose futures. In: ICCV (2017)

[58] Chan, C., Ginosar, S., Zhou, T., Efros, A.A.: Everybody dance now. In: ICCV (2019)

[59] Zakharov, E., Shysheya, A., Burkov, E., Lempitsky, V.: Few-shot adversarial learning of realistic neural talking head models. In: ICCV (2019)

[60] Wang, T.Y., Ceylan, D., Singh, K.K., Mitra, N.J.: Dance in the wild: Monocular human animation with neural dynamic appearance synthesis. In: 3DV (2021)

[61] Zhao, L., Peng, X., Tian, Y., Kapadia, M., Metaxas, D.: Learning to forecast and refine residual motion for image-to-video generation. In: ECCV (2018)

[62] Yang, Z., Li, S., Wu, W., Dai, B.: 3dhuman-gan: Towards photo-realistic 3d-aware human image generation. arXiv preprint (2022)

[63] Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Flow-grounded spatial-temporal video prediction from still images. In: ECCV (2018)

[64] Ohnishi, K., Yamamoto, S., Ushiku, Y., Harada, T.: Hierarchical video generation from orthogonal information: Optical flow and texture. In: AAAI (2018)

[65] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: CVPR (2023)

[66] Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al.: Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103 (2023)

[67] Zhang, D.J., Wu, J.Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D., Shou, M.Z.: Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818 (2023)

[68] Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.-S., Kag, A., Fang, Y., Stoliar, A., Ricci, E., Ren, J., et al.: Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In: CVPR (2024)

[69] Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In: CVPR (2024)

[70] Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F., Chen, C., Qiao, Y.: Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 (2024)

[71] Chen, X., Wang, Y., Zhang, L., Zhuang, S., Ma, X., Yu, J., Wang, Y., Lin, D., Qiao, Y., Liu, Z.: Seine: Short-to-long video diffusion model for generative transition and prediction. In: ICLR (2023)

[72] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023)

[73] Siarohin, A., Woodford, O., Ren, J., Chai, M., Tulyakov, S.: Motion representations for articulated animation. In: CVPR (2021)

[74] Wang, Y., Yang, D., Bremond, F., Dantcheva, A.: Latent image animator: Learning to animate images via latent space navigation. In:
[75] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR (2017)

[76] Zheng, Z., Zheng, L., Yang, Y.: A discriminatively learned cnn embedding for person reidentification. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) (2018)

[77] Zhang, L., Agrawala, M.: Adding Conditional Control to Text-to-Image Diffusion Models (2023)