Latent Image Animator: Learning to Animate Images via Latent Space Navigation
Authors: Yaohui Wang
Di Yang
Francois Bremond
Antitza Dantcheva
Venue: ICLR 2022 Poster (2022)
Keywords: Video generation
Generative Adversarial Network

Abstract: Due to the remarkable progress of deep generative models, animating images has become increasingly efficient, whereas associated results have become increasingly realistic. Current animation-approaches commonly exploit structure representation extracted from driving videos. Such structure representation is instrumental in transferring motion from driving videos to still images. However, such approaches fail in case the source image and driving video encompass large appearance variation. Moreover, the extraction of structure information requires additional modules that endow the animation-model with increased complexity. Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation. LIA is streamlined to animate images by linear navigation in the latent space. Specifically, motion in generated video is constructed by linear displacement of codes in the latent space. Towards this, we learn a set of orthogonal motion directions simultaneously, and use their linear combination, in order to represent any displacement in the latent space. Extensive quantitative and qualitative analysis suggests that our model systematically and significantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t. generated quality.

Published as a conference paper at ICLR 2022
LATENT IMAGE ANIMATOR: LEARNING TO ANIMATE IMAGES VIA LATENT SPACE NAVIGATION
Yaohui Wang, Di Yang, Francois Bremond & Antitza Dantcheva
Inria, Université Côte d' Azur
{yachui. wang, di. yang, franceis bremond, antitza waw.aaaaaaacceeeeaaria....
ABSTRACT
Due to the remarkable progress of deep generative models, animating images has become increasingly efficient, whereas associated results have become increas- ingly realistic. Current commation-approcchie
commonly exploit structure repre- sentation extracted from driving videos. Such structure representation instru- mental in transferring motion from driving videos to still images. However, such approaches fail in case the source image and driving video encompass large ap- pearance variation. Moreover, the extraction of structure information requires ad- ditional modules that endow the animation-mode viating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation. LIA is streamlined to animate images by linear navigation in the latent space. Specifi- cally, motion in generated video is constructed by linear displacement of codes in the latent space. Towards this, learn set of orthogonal motion directions si- multaneously, and use their linear combination, in order to represent any displace- ment in the latent space. Extensive quantitative and qualitative analysis suggests that our model systematically and significantly outperforms state-of-art methods VoxCeleb, Taichi and TED-talk datasets w.r.t generated quality. Source code and pre-trained models are publicly available!
with increased complexity, De-
Source image
Output images
Figure 1: LIA animation examples. The two images of Marilyn Monroe and Emmanuel Macron animated by LIA,, which transfers motion of driving video (smaller images the top) from VoxCeleb dataset (Chung et al., 2018) onto the still images. LIA is able to successfully animate these two images without relying on any explicit structure representations, such as landmarks and region representations.

---

Published as a conference paper at ICLR 2022
1 INTRODUCTION
In the series of science fiction books Harry Potter (Rowling et al., 2016; Rowling, 2019), wizards and witches were able to magically enchant portraits, bringing them life. Remarkable progress of deep generative models has recently turned this vision into reality. This work examines the scenario where a framework animates a source image by motion representations learned from a driving video. Existing approaches for image animation are classically related to computer graphics (Cao et al., 2014; Thies et al., 2016; 2019; Zhao et al., 2018) exploit motion labels (Wang et al., 2020b) and structure representations such semantic maps (Pan al., 2019; Wang et al., 2018; 2019), human keypoints (Jang et al., 2018; Yang et al., 2018; Walker et al., 2017; Chan et al., 2019; Zakharov et al., 2019; Wang et al., 2019; Siarohin et al., 2019), 3D meshes (Liu et al., 2019; Chen et al., 2021), and optical flows (Li et al., 2018; Ohnishi et al., 2018). We note that the ground truth of such structure representations has been computed a-priori for the purpose of supervised training, which poses constraints on applications, where such representations of unseen testing images might be fragmentary or difficult to access.
Self-supervieed motion transfer approaches (Wiles et al., 2018; Siarohin et al., 2019; 2021) accept raw videos as input and learn to reconstruct driving images by warping source image with pre- dicted dense optical flow fields. While the need for domain knowledge or labeled ground truth data has been obviated, which improves performance on in-the-wild testing images, such methods entail necessity of explicit structure representations as motion guidance. Prior information such as key- points (Siarohin et al., 2019; Wang et al., 2021a) or regions (Siarohin et al., 2021) are learned in an end-to-end training manner by additional networks intermediate features, in order to predict target flow fields. Although online prediction of such representations less tedious than the acquisition of ground truth labels, it still strains the complexity of networks.
Deviating from such approaches. we here aim to fully eliminate the need of explicit structure rep- resentations by directly manipulating the latent space of a deep generative model. To the best of our knowledge, this constitutes new direction in the context of image animation. Our work is motivated by interpretation of GANs (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al., 2020; Voynov & Babenko, 2020), showcasing that latent spaces of StyleGAN (Karras et al., 2019; 2020b) and BigGAN (Brock et al., 2019) contain rich semantically meaningful directions. Given that walking along such directions, basic visual transformations such as zooming and rotation can be induced in generated results. As in image animation, we have that motion between source and driving images can be considered as higher-level transformation, a natural question here arises: can we discover a set of directions in the latent space that induces high-level motion transformations collaboratively?
Towards answering this question, we introduce LIA, novel Latent Image Animator constituting of an autoencoder for animating still images via latent space navigation. LIA seeks to animate source image via linearly navigating associated source latent code along a learned path to reach a target latent code, which represents the high-level transformation for animating the source image. We introduce a Linear Motion Decomposition (LMD) approach aiming to represent a latent path via linear combination of set of learned motion directions and associated magnitudes. Specifically, we constrain the set as an orthogonal basis, where each vector indicates a basic visual transformation. By describing the whole motion space using such learned basis, LIA eliminates the requirement of explicit structure representations.
In addition, we design LIA to disentangle motion and appearance within a single encoder-generane architecture. Deviating from existing methods using separate networks to learn disentangled fea- tures, LIA integrates both, latent motion code, as well as appearance features in a single encoder, which highly reduces the model complexity and simplifies training.
We provide evaluation on multiple datasets including VoxCeleb (Chung et al., 2018), TaichiHD (Siarohin et al., 2019) and TED-talk (Siarohin et al., 2021). In addition, we show that LIA outperforms the statc-of-the-art in preserving the facial structure in generated videos in the setting of one-shot image animation on unseen datasets such as FFHQ (Karras et al., 2019) and GermanPublicTVV (Thies et al., 2020).
2 RELATED WORK
Video generation GAN-based video generation is aimed mapping Gaussian noise to video, di- rectly and in the absence of prior information (Vondrick et al., 2016; Saito et al., 2017; Tulyakov et al,, 2018; Wang et al., 2020a; Wang, 2021). Approaches based on deep probabilistic models (Den-
2

---

Published as a conference paper at ICLR 2022
Warping
Zg-ad + Wy-ad dl
4--d
@ s
E
G
d2
Z or
Xs
5-
Image Space
Latent Space
Flow Field Space
Figure 2: General pipeline. Our objective is to transfer motion via latent space navigation. The entire training pipeline consists of two steps. Firstly, we encode source image I, into a latent code Zz-or. By linearly navigating Σg-ar along a path wy-+d. we reach target latent code 2z-ad. The latent paths are represented by a linear combination between a set of learned motion directions (e.g., di and d2). which is an orthogonal basis, and associated magnitudes. In the second step, we decode Ze-ad to a target dense optical flow field Ø.-dd which is used to warp I, into the driving image Td-- While we train our model using images from the same video sequence, in the testing phase, I, and Fy generally pertain to different identities.
ton & Birodkar, 2017; Li & Mandt, 2018; Bhagat al., 2020; Xie al., 2020) were also proposed to tackle this problem, however only show results on toy datasets with low resolution. Recently, with the progress of GANs in photo-realistic image generation (Brock et al., 2019; Karras et al., 2019; 2020a). a series of works (Clark et al., 2019; Wang et al., 2021c) explored production of high- resolution videos by incorporating the architecture of image generator into video GANs, trained jointly with RNNs. Tian al. (2021) directly leveraged the knowledge of a pre-trainn pre-trained StyleGAN to produce videos of resolution up to 1024 x 1024. Unlike these approaches, which generate random videos based on noise vectors in an unconditional manner, in this paper, we focus on conditionally creating novel videos by transferring motion from driving videos to input images.
Latent space editing In an effort to control generated images, recent works explored the discovery of semantically meaningful directions in the latent space of pre-trained GANs, where linear navi- gation corresponds to desired image manipulation. Supervised (Shen et al., 2020; Jahanian et al., 2020; Goetschalckx et al., 2019) and unsupervised (Voynov & Babenko, 2020; Peebles et al., 2020; Shen & Zhou, 2021) approaches were proposed to edit semantics such as facial attributes, colors and basic visual transformations (e.g., rotation and zooming) in generated or inverted real images (Zhu et al., 2020; Abdal al., 2020). In this work, as opposed to finding directions corresponding to individual visual transformations, we seek to learn a set of directions that cooperatively allows for high-level visual transformations that can be beneficial in image animation.
Image animation Related approaches (Chan et al., 2019; Wang et al., 2018; Zakharov et al., 2019; Wang al., 2019; Yang et al., 2020) in image animation required strong prior structure labels as mo- tion guidance. In particular, Chan et al. (2019), Yang et al. (2020) and Wang al. (2018) proposed to map representations such as human keypoints and facial landmarks videos in the setting of image-to-image translation proposed by Isola et al. (2017). However, such approaches were only able to learn an individual model for single identity. Transferring motion on new appearances requires retraining the entire model from scratch by using videos of target identities. Several recent works (Zakharov et al., 2019; Wang et al., 2019) explored meta learning fine-tuning models on target identities. While only few images target identities were required during inference time, it was still compulsory to input pre-computed structure representations in those approaches, which usually are hard to access in many real-world scenarios. Towards addressing this issue, very recent works (Siarohin age animation in self-supervised manner, only relying on RGB videos for both, training and testing without any priors. They firstly predicted dense flow fields from input images, which were then uti- lized to warp source images, in order to obtain final generated results. Inference only required one image of target identity without any fine-tuning step pre-trained models. While priors were required, state-of-the-art methods still followed the idea of using explicit structure representations. FOMM (Siarohin et al., 2019) proposed a first order motion approach to predict keypoints and local transformations online to generate flow fields. Siarohin et al. (2021) developed this idea to model articulated objects by replacing keypoints predictor by a PCA-based region prediction module. Wang et al. (2021b) extended FOMM by predicting 3D) keypoints for view-free generation. We
al., 2019; 2021; Wang et al., 2021b; Wiles et al., 2018) proposed to learn im-
3

---

Published as a conference paper at ICLR 2022
Linear Motion Decomposition
a
Ar-d
E
MLP
m
aM
dy ds
Driving image (xd)
W
W
a2 di
Share parameters
t-]
s-dd
s-r
E
G
Reconstruction
Source image (x₂)
Source features (x
Output image (xg-dd)
Driving image (xa)
Figure 3: Overview of LIA. LIA is an autoencoder consisting of two networks, an encoder E and generator G. In the latent space, apply Linear Motion Decomposition (LMD) towards learning motion dictionary Dm- which is an orthogonal basis where each vector represents basic visual transformation. LIA takes two frames sampled from the same video sequence as source image V, and driving image Ia respectively during training. Firstly, it encodes Vs into a source latent code Ex-rr and Zdd into magnitude vector Ar-d. Then, it linearly combines Ar-d and trainable Dm using LMD to obtain latent path wy-rd. which is used to navigate z target code 25-ad- Finally, G decodes 2x-ad into a target dense flow field and warps Ig an output image Tz-+d- The training objective is to reconstruct dd using Iz-+d-
note though that in all approaches, given that keypoints regions are inadequately predicted, the quality of generated images drastically decreases. In contrast to such approaches, our method does not require any explicit structure representations. We dive into the latent space of the generator and self-learn to navigate motion codes in certain directions with the goal to reach target codes, which then decoded to flow fields for warping.
METHOD
Self-supervised image animation aims at learning to transfer motion from a subject a driving video to subject in a source image based on training with a large video dataset. In this work, we propose to model such motion transformation via latent space navigation. The general pipeline is illustrated in Fig. 2. Specifically, for training, our model takes in pair of source and driving images, randomly sampled from one video sequence. These two images are encoded into a latent code which used to represent motion transformation in the image space. The training objective is to reconstruct the driving image by combining source image with learned motion transformation. For testing, frames of driving video are sequentially processed with the source image to animate the source subject.
We provide an overview of the proposed model in Fig, Our model is autoencoder, consisting of two main networks, an encoder and generator G. general, our model requires two steps to transfer motion. In the first step, E encodes source and driving images 2s, F. X' E 3 into latent codes in the latent space. The source code is then navigated into target code, which used to represent target motion transformation, along a learned latent path. Based on proposed Linear Motion Decomposition (LMD), we represent such path as linear combination of set of learned motion directions and associated magnitudes, which are learned from In the second step, once the target latent code is obtained, G decodes it as a dense flow field é.-ed ~ & E 2 H XXW xW and uses 2 d to warp v. and then to obtain the output image. In the following, we proceed to discuss the steps in detail.
3.1 LATENT MOTION REPRESENTATION
Given a source image I. and a driving image za, our first step constitutes of learning a latent code Za-ad Z - to represent the motion transformation from H, Zg. Due the uncertainty of two images, directly learning 2--d puts forward high requirement on the model to capture a complex distribution of motion. Mathematically, it requires modeling directions and norms of the vector 25- simultaneously, which is challenging. Therefore, instead of modeling motion transfor- mation I, - Fdt assume there exists reference image v, and motion transfer be modeled

---

Published as a conference paper at ICLR 2022
Es - Zr Fds, where 2g->d is learned in an indirect manner. We model 2x-rd as a target point in the latent space, which can be reached by taking linear walks from starting point Rg-ay along linear path Wr-rd (see Fig. 2), given by
2 28- + Wr-sd:
(1)
where 2x-or and wy indicate the transformation I, - I,, and 2, za respectively. Both 2x-or and Wr-+d are learned independently and ~ is obtained by passing =, through E.
We learn wy->d via Linear Motion Decomposition (LMD). Our idea is to learn a set of motion directions Dm = .d....., d to represent any path in the latent space. We constrain Dm as an orthogonal basis, where each vector indicates a motion direction d, We then combine each vector the basis with vector Ar = {"I,.... aM) J. where a, represents the magnitude of dj. Hence, any linear path in the latent space can be represented using a linear combination
M
Wp-rd
a,
(2)
where d; RN and a; R for all E fl...., M). Semantically, each d; should represent basic visual transformation and a; indicates the required steps to walk in d; towards achieving wr-d. Due to D, entailing an orthogonal basis, any two directions d; dj follow the constrain
< <.. dd >- 0
} 3-
(3)
We implement Dm as a learnable matrix and apply the Gram-Schmidt process during each forward pass, in order to meet the requirement of orthogonality. Ar is obtained by mapping zd-ar. which is the output of za after E, through 5-layer MLP. The final formulation of latent motion represen- tation for each vs and Fa thus given
Re-ad He-tr
a.d
3.2 LATENT CODE DRIVEN IMAGE ANIMATION
Once we obtain Za-ad in our second step, we use to decode a flow field d.-ad and warp w.. Our G consists of two components, a flow field generator G2 and a refinement network G, (we provide details in App. A).
Towards learning multi-scale features, G is designed to produce a pyramid of flow fields Øs-e { } in different layers of Gy. Multi-scale source features x.nc = {-t"e J are obtained from E' and are warped in Gy.
residual network containing N models
However, as pointed out by Siarohin et al. (2019), only relying on c.--d to warp source features is insufficient to precisely reconstruct driving images due the existing occlusions in some positions of .... In order to predict pixels in those positions, the network required to inpaint the warped feature maps. Therefore, we predict multi-scale masks {m;)² V along with { in G. to mask out the regions required to be inpainted. In each residual module, we have
(5) where denotes the Hadamard product and T denotes warping operation, whereas ay: signifies the masked features. We generate both dense flow fields, as well as masks by letting each residual module output a 3-channel feature map in which the first two channels represent Φ; and the last channel m, Based on an inpainted feature map f(((). as well as an upsampled image g( provided by the previous module in G,, the RGB image from each module is given by
x/- = ((0,,;70)
T(o,, (nC) m,,
o, f(z)) + 9(0,-)),
(6)
where f and g denote the inpainting and upsampling layers, respectively. The output image on from the last module constitutes the final generated image Iz-ed = ON.
3.3 LEARNING
We train LIA in self-supervised manner to reconstruct F. using three losses, i.e., reconstruction loss Lrecon, a perceptual loss Long (Johnson et al., 2016) and an adversarial loss Lade- We use Lrecon to minimize the pixel-wise L1 distance between 2a and r,-id, calculated as
Lrecon (rrcon(xs-pn (xx-rd, Tt) Bl[lar xs
5

---

Published as a conference paper at ICLR 2022
Towards minimizing the perceptual distance, we apply VGG19-based Logs multi-scale feature maps between real and generated images, written as
Logg (d, Td) = -E Σ ([ (x
(8)
where Fg denotes the n layer in a pre-trained VGG19 (Simonyan & Zisserman, 2015). In prac- tice, towards penalizing real and generated images in multi-scale images, we use pyramid of four resolutions, namely 256 x 256, 128 128, 64 x 64 and 32 32 as inputs of VGG19. The final perceptual loss the addition of perceptual losses in four resolutions.
Further, towards generating photo-realistic results, we incorporate a non- cotturtuaa adversarial loss Ladv on Fg which is calculated as
Ludo (Ze-d) -e ( (I
(9)
where D is a discriminator, aimed at distinguishing reconstructed images from the original ones. Our full loss function is the combination of three losses with À as a balanced hyperparameter
L(I Lrecon (I==d; (. + XLrgg (T F.a) + Lidd((s-aa))
(10)
3.4 INFERENCE
In inference stage, given a driving video sequence V22 {x+TT. we aim to transfer motion from Va Ts. in order to generate novel video Va { If Va and x, stem from the same video sequence, i.e., Vs = 21, our task comprises of reconstructing the entire original video sequence. Therefore, we construct the latent motion representation each frame using absolute transfer, which follows the training process, given
Ze-+0 Zg-tr + Wr-+e E (}.
(11)
However, in real world applications, interest is rather placed on the scenario, where motion transfer between I, and Va, the latter stemming from different identities, i.e., I, * I1. Taking talking head video an example, in this setting, beyond identity, 3j and T, might also differ in pose and expression. Therefore, we propose relative transfer to eliminate the motion impact of wy-1 and involve motion of wy-s in the full generated video sequence. Owing to a linear representation of the latent path, we can easily represent d5 for each frame as
Zr (2,-tr wr + (Wr++ Wr-+1
(12)
2 + (Wr+t wy-1), tE (E,....,
The first term in Eq. (12), Z5-クo indicates the reconstruction of while the second term (Wy-st Wy-+1) represents the motion from I1 St. This equation indicates that the original pose preserved in Za at the same time motion transferred from Va. We note that order to
completely replicate the position and pose in Vd, requires x, and Tj to contain similar poses in relative motion transfer.
EXPERIMENTS
In this section, we firstly deseribe our experimental setup including implementation details and datasets. Secondly, we qualitatively demonstrate generated results based on testing datasets. Then, we provide quantitative evaluation w.r.f. image quality on (a) same-identity reconstruction, (b) cross- video motion transfer, presenting (c) user study. Next, we conduct ablation study that demon- strates (d) the effectiveness of our proposed motion dictionary, as well (e) associated size. Finally, we provide an in-depth analysis our (f) latent codes and (g) motion dictionary to interpret their semantic meanings.
Datasets Our model is trained on the datasets VoxCeleb, TaichiHD and TED-talk. We follow the pre-processing method in (Siarohin et al., 2019) to crop frames into 256 x 256 resolution for quantitative evaluation.
Implementation details Our model is implemented in PoToh (Paszke et al., 2019). All models are trained on four 16G NVIDIA V100 GPUs. The total batch size 32 with 8 images per GPU. We use learning rate of 0.002 to train our model with the Adam optimizer (Kingma & Ba, 2014). The dimension of all latent codes, as well as directions in D,,, is set to be 512. In our loss function, we use À 10 in order to penalize more on the perceptual loss. It takes around 150 hours to fully train framework.

---

where we Published as a conference paper at ICLR 2022
Source (%)
Driving (4)
Oura
MRAA
FOMM
Source (s))
Driwing (4)
Source (K.)
FOMM
MRAA
Ours
Starting (k)
Driving (4)
Ours
MILAA
FOMM
Figure 4: Qualitative results. Examples for same-dataset absolute motion transfer on TaichiHD (top-right) and TED-talk (bottom-right). On VoxCeleb (left), demonstrate cross-dataset relative motion transfer. We successfully transfer motion between #1 and my from videos VoxCeleb to I, from FFHQ, the latter not being used for training.
Evaluation metrics We evaluate our model w.r.f. (i) reconstruction faithfulness using L1, LPIPS, (ii) generated video quality using video FID, as well as (iii) semantic consistency using average keypoint distance (AKD), missing keypoint rate (MKR) and average euclidean distance (AED). Details are available in App. B.2.
4.1 QUALITATIVE RESULTS
Firstly, we evaluate the ability of LIA to generate realistic videos and compare related results with four state-of-the methods. For TaichiHD and TED-talk datasets, conduct an experiment lated to cross- cross-video generation. Corresponding results (see Fig. 4) confirm that method is able correctly transfer motion on articulated human bodies, the absence of explicit structure repre- sentations. For the VoxCeleb dataset, we conduct a cross-dataset generation-experinee transfer motion from VoxCeleb to images of the FFHQ dataset. We observe that our method out- performs FOMM and MRAA w.r.t. image quality, both approaches visibly deform the shape of the original faces. This is specifically notable the case that souree and driving images entail large pose variations. At the same time, LIA is able to successfully tackle this challenge and no similar deformations are visible.
4.2 COMPARISON WITH OFATE-OF-THE-ARE METHODS
We quantitatively compare our method with the state-of ofte--of-thearr approaches X2Face, Monkey-Ner FOMM and MRAA on two tasks, namely (a) same-identity reconstruction and (b) cross-video mo- tion transfer. Additionally, we conduct (c) user study.
(a) Same-identity reconstruction We here evaluate the reconstruction ability of our method. Specifically, reconstruct each testing video by using the first frame 2, and the remain- ing frames as Zz. Results on three datasets are reported Table 1. Focusing on foreground- reconstruction, our method outperforms the other approaches w.r.. all metrics. More results are presented in App. B.3, discussing mackground-ecconsttuatiio..
(b) Cross-video motion transfer Next, we conduct experiments, where source images and driv- ing videos stem from different video sequences. In this context, we mainly focus on evaluating talking head videos and explore two different cases. In the first case, generate videos using the VoxCeleb testing set to conduct motion transfer. In the second case, source images are from an unseen dataset, namely the GermanPublicTT dataset, as we conduct cross cross-dataset motion transfer. In both experiments, we randomly construct source and driving pairs and transfer motion from driv- ing videos to source images to generate novel manipulated dataset. Since ground truth data for generated videos is not available,
video FID (as initialized by Wang al. (2020a)) to

---

Published as a conference paper at ICLR 2022
Table 1: Same-identity reconstruction. Comparison with state-of-the-art methods on three datasets for same-identity reconstruction.
VoxCeleb
TachiHD
TED-alks
Method
L1
AKD
AED
L.PISS
L1
(AKD), MKR) AED L.IPS
41
(AKD, MKR) AED LPIPS
X2Face
0.078
7.687
0.405
0.080
(17.654, 0.109)
Monkey-Net
0.049
1.878
0.199
0.077
(10.798, 0.059)
FOMM
0.046
1.395
0,141 0.136
0.063
(6.472. 0.032) 0.495
0.191
0.030
(3.759, 0.0090)
0,422
0.13
MRAA w/o bg
0.043
1.307
0.140
0.127
0.063
(5.626,
0.025) 0,460
0.189
0.029
(3.126, 0.0092)
0.396
0.12
Ours
0.041
1.353
0.138
0.123
0.057
(4.823, 0.020)
0.431 0.180
0.027
(3,141, 0.0095)
0.399
0.11
Table 2: Cross-video generation. We report video FID for both inner- and cross-datasct tasks on VoxCeleb and GermanPublicTV.
Table study. We ask 20 human raters to conduct subjective video quality evalua- tion.
VoxCeleb GermanPublieI
FOMM 0.323
0.436
(oxCole)(%)
(aichiHD(%))
TED-
MRAA
0,308
0.454
OUrS/FOMM
92,9/7.1
64.5/33..5
71.4/28.6
Ours/MRAA
89.7/10.3
60.7/39.9
54.8/45.2
Ours
0.161
0.406
compute the distance between generated and real data distributions. As shown in Tab. 2, our method outperforms all other approaches w.E.t. video FID, indicating the best generated video quality.
(c) User study We conduct a user study to evaluate video quality. Towards this, we displayed paired videos and asked 20 human raters 'which clip is more realistic?'. Each video-pair contains generated video from our method, well video generated from FOMM or MRAA. Results suggest that our results are more realistic in comparison to FOMM and MRAA across all three datasets (see Tab. 3). Hence, the obtained human preference is in accordance with our quantitative evaluation.
Table Ablation study on Dm size. We con- duct experiments on three datasets with different Dm. size and show reconstruction results.
Table 4: Ablation study motion dictionary. We conduct experiments on three datasets with and without Dm and show reconstruction results.
VoxCelcb
TaichiHD Li LPIPS
TED-talks
VoxCeleb
TED-talks C LPIPS
TaichiHD
M
Li
LPIPS
Li LPIPS
Method C LPIPS
C LPIPS
5 0.051
0.15
0.070
0.22
0.037
0.15
w/ Dm 0.049 0.165 0.062 0.186 0.031 0.12 Full 0.041 0.123 0.057 0.180
10
0.043
0.13
0.065
0.20
0.036
0.13
20
0.041
0.12
0.057
0.18
0.028
0.11
0.028 0.11
40
0.042
0.12
0.060
0.19
0.030
0.12
100
0.041
0.12
0.058
0.18
0.028
0.11
4.3 ABLATION STUDY
We here analyze our proposed motion dictionary and focus on answering following two questions.
(d) Is the motion dictionary Dm beneficial? We here explore the impact of proposed Dms by training our model without Dm Specificallyy, we output wy-sd directly from MIP, without using LMD to learn an orthogonal basis. From the evaluation results reported in Tab. 4 and qualitative results in App. B.5, we observe that in the absence of Dm model fails to generate high-quality images, which proves the effectiveness of Dm,, consistently on all datasets.
(e) How many directions are required in Dmm Towards finding an effective size of Dmm we empirically test three different M, viz. 5, 10, 20, 40 and 100. Quantitative results in Tab. 5 show that when using 20 directions, the model achieves the best reconstruction results.
4.4 FURTHER ANALYSIS
(I) Latent code analysis. While our method successfully transfers motion via latent space navi- gation, we here aim at answering the question what does I, represent? Towards answering this question, we proceed to visualize z,. We firstly decode ž into dense flow field d.-rr, which is then used to warp x, (we show details in App. B.4). Fig. 5 shows examples of F, and x,. Inter- estingly, we observe that x,, represents the canonical pose of xx, regardless of original poses of the subjects. And for all datasets, reference images resemble each other w.r.f. pose and scale. As such
8

---

Published as a conference paper at ICLR 2022
Figure 5: Visualization of reference images. Example source (top) and reference images (down) from VoxCeleb, TaichiHD and TED-talk datasets. Our network learns reference images of a consis- tently frontal pose, systematically for all input images of each dataset.
4,+
-4,-
de+
dig
Figure 6: Linear manipulation of four motion directions on the painting of Mona Lisa. Manip- ulated results indicate that dg represents eye movement, ds represents head nodding, whereas d19 and d7 represent facial expressions.
reference images can be considered as a normalized form of v.s learning transformations between I, and xy using Vs - 2, - Zg considerably more efficient than v, + Fz, once v, is fixed.
Noteworthy, we found the similar idea of learning a 'reference image' has also been explored by Siarohin et al. (2019) (FOMM) and Wiles et al. (2018) (X2Face). However, deviating from our visualized "reference image', the "reference image 'in FOMM refers to non-visualized and ab- stract concept. addition, LIA only requires a latent code zx-or, rather than the 'reference image' for both, training and testing, which contrast to X2Face.
(g) Motion dictionary interpretation. Towards further interpretation of directions in Dm- we conduct linear manipulations on each d,, Images pertained to manipulating four motion directions are depicted in Fig. 6. The results suggest that the directions in Dm are semantically meaningful, as they represent basic visual transformations such head nodding (dg), eye movement (dg) and facial expressions (d19 and d7). More results can be found on our project webpage?.
5 CONCLUSIONS
In this paper, we presented a novel self-supervised autoencoder LIA, aimed at animating images latent space navigation. By the proposed Linear Motion Decomposition (LMD), we were able to formulate the task of transferring motion from driving videos to source images as learning lin- ear transformations in the latent space. We evaluated proposed method on real-world videos and demonstrated that our approach is able successfully animate still images, while eliminating the necessity of explicit structure representations. In addition, showed that the incorporated motion dictionary is interpretable and contains directions pertaining to basic visual transformations. Both quantitative and qualitative evaluations showed that LIA outperforms state-of-art algorithms on all benchmarks. We postulate that LIA opens new door in design of interpretable generative models for video generation.
hhttps://uhhsirris.iiiiii http://is....iitii.o.ooo//ooopp

---

Published as a conference paper at ICLR 2022
ETHIC STATEMENT
In this work, we aim to synthesize high-quality videos by transferring motion on still images. Our approach be used for movie production, making video games, online education, generating synthetic data for other computer vision tasks, etc. We note that our framework mainly focuses on learning how to model motion distribution rather than directly model appearance, therefore is not biased towards any specific gender, race, region, or social class. It works equally well irrespective of the difference in subjects.
REPRODUCIBILITY STATEMENT
We assure that the results shown in the paper and supplemental materials can reproduced. We intend to open-source our code, as well as trained models.
ACKNOWLEDGEMENTS
This work granted access the HPC resources of IDRIS under the allocation 1627RR was supported by the French Government, by the National Research Agency (ANR) under Grant ANR-18- 18--18-C992-0022 project RESPECT and through the 31A Côte d' Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19. P3IA-0002.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded im- ages? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. Openface: general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, CMU School of Computer Science, 2016.
Sarthak Bhagat, Shagun Uppal, Zhuyun Yin, and Nengli Lim. Disentangling multiple features in video sequences using gaussian processes in variational autoencoders. In ECCV, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In /CLR, 2019.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In ICCV, 2017.
Chen Cao, Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on graphics (TOG), 33(4):1-10, 2014.
Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi- person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, 2019.
Hao Chen, Yaohui Wang, Benoit Lagadec, Antitza Dantcheva, and Francois Bremond. Joint gener- ative and contrastive learning for unsupervised person re- re-identification. In CVPR, 2021.
S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: SPEECH, 2018.
Deep speaker recognition. In INTER-
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv preprint arXiv: 1907 2019.
10

---

Published as a conference paper at ICLR 2022
Emily L Denton and vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), NeurIPS, 2017.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. In ICCV, pp. 5744-5753, 2019.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history 2d cnns and imagenet? In CVPR, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NIPS, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In ICLR, 2020.
Yunseok Jang, Gunhee Kim, and Yale Song. Video Prediction with Appearance and Motion Condi- tions. In ICML, 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In CVPR, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and improving the image quality of stylegan. In CVPR, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, 2012.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In ECCV, 2018.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. ICMI., 2018.
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In CVPR, 2019.
Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Science and Lunguage, 2019.
Katsunori Ohnishi, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Hierarchical video generation from orthogonal information: Optical flow and texture. In AAAI, 2018.
Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. arXiv preprint arXiv: 044804 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems. 2019.
William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
11

---

Published as a conference paper at ICLR 2022
JK Rowling. Harry potter. The 100 Greatest Literary Characters, pp. 183, 2019.
Joanne Kathleen Rowling, John Tiffany, and Jack Thorne. Harry Potter and the Cursed Child-Parts One and Two (Special Rehearsal Edition). Pottermore Publishing, 2016.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In ICCV, 2017.
Yujun Shen and Bolei Zhou. Closed-fome factorization of latent semantics in gans. In CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Garnett (eds.), NeurlPS. Curran Associates, Inc., 2019.
Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In CVPR, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In International Conference on Learning Representations, 2015.
Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of RGB videos. In CVPR, 2016.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4): 2019.
Justus Thics, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. ECCV, 2020.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In ICLR, 2021.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, 2018.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NIPS, 2016.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. arXiv preprint arXiv:2002.03754. 2020.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video fore- casting by generating pose futures. In ICCV, 2017.
Ting-Chun Wang, Ming-Yi Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. In NeurlPS, 2018
Ting-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few- shot video-to-video synthesis. In NeurlPS, 2019.
Ting- Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- head synthesis for video conferencing. In CVPR, 2021a
Ting- Chen Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021b.
Yaohui Wang. Learning to Generate Human Videos. Theses, Inria Sophia Antipolis Université Cote Azur, September 2021.
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3AN: Disentangling appearance and motion for video generation. In CVPR, 2020a.
Yaohui Wang, Piotr Bilinski, Francois F Bremond, and Antitza Dantcheva. ImaGINator: Conditional Pratio-Tempora GAN for Video Generation. In WACV, 2020b.
12

---

Published as a conference paper at ICLR 2022
Yaohui Wang, Francois Bremond, and Antitza Dantcheva. Inmodegan: Interpretable motion decom- position generative adversarial network for video generation. arXiv preprint arXiv:2101.0004 2021c.
Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pp. 670-686, 2018.
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Motion-based gener- ator model: Unsupervised disentanglement of appearance, trackable and intrackable motions in dynamic patterns. In AAAI, 2020.
Ceyuan Yang. Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. In ECCV, 2018.
Zhuoqian Yang, Wentao Zhu, Wayne Wu, Chen Qian, Qiang Zhou, Bolei Zhou, and Chen Change Loy. Transmomo: Invariance-driven unsupervised video motion retargeting. In CVPR, 2020.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Long Zhao, Xi Peng. Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning to forecast and refine residual motion for image-to- to-ideo generation. In ECCV, 2018.
Zhedong Zheng, Tao Ruan, Yunchao Wei, Yi Yang. and Tao Mei. Vehiclenet: Learning robust visual representation for vehicle re-identification. IEEE Transaction Multimedia (TMM), 2020.
Jiapeng 7hu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference Computer Vision (ECCV), 2020.
13

---

Published as a conference paper at ICLR 2022
A DETAILS OF MODEL ARCHITECTURE
We proceed to describe the model architecture in this section. Fig. 7 shows details of our E. In each ResBlock in E, spatial size of input feature maps are downsampled. We take feature maps of spatial sizes from x 8 to 256 256 as our appearance features x! We use 5-layer MLP to predict a magnitude vector Ad-or from 2d-rr. Fig. 8 (a) shows the general architecture which consists of two components, flow field generator G₁ and a refinement network G,.. We apply SryleConv (Upsample + Conv3 3), which is proposed by StyleGAN2, in Gj. StyleCom takes latent representation Σg-sz as style code and generates flow field o; and corresponding mask my. G, uses UpConv (Convl + Upsample) to upsample and refine inpainted feature maps to larget resolution. We show details pertaining to G block in Fig. 8 (b). Each G block is used to upsample x the previous resolution. We stack 6 blocks towards producing 256 resolution images.
our G,
Xs
x
Conv3 x (64)
256 x 256
X6
ResBlock (64)
256 x 256
CTCC
ResBlock (128)
128 x 128
x4
ResBlock (256)
ResBlock (512)
32 32
X2
ResBlock (512)
x
Conv 3
Conv 1x1
x1
ResBlock (512)
x
Conv 3 x 3
ResBlock (512)
x
Downsample
Downsample
Conv4 x (512)
x
Linear (512)
Linear (512)
Linear ($12)
Linear (512)
Linear (20)
A
Z
Zg-or
(a) Encoder
(b) ResBlock
Figure 7: Encoder architecture. We show details of architecture of E (a) and ResBlock in (b).
B EXPERIMENTS
We proceed to introduce details of datasets and evaluation metrics used in our experiments.
B. DATASETS
VoxCeleb (Nagrani et al., 2019) consists a large amount of interview videos different celebri- ties. Following the process of FOMM (Siarohin et al., 2019), we extract frames and crop them into 256 x 256 resolution. In total, VoxCeleb contains training set of 17928 videos and a test set of 495 videos.
14

---

Published as a conference paper at ICLR 2022
-5-
G
Gr
512
StyleCoom
xx 4, 512
Eg-pt
Gy
G
UpConv
Upsample
Cone 3x3
Gome
StyleConv
8, 512
warping
UpConv
700
Cont
x
258
StyleCoov
(b) Block
warping
Xg
(a) Generator architecture
Figure 8: Generator architecture. We show details about architecture of G in (a) and G block in (b).
TaiChiHD (Siarohin et al., 2019) consists of videos of full human bodies performing Chi actions. We follow the original pre-processing of FOMM (Siarohin et al., 2019) and utilize its 256 x 256 version. TaiChiHD contains 1096 training videos and 115 testing videos.
TED-talk is
dataset proposed in MRAA (Siarohin et al., 2021). It comprises number of TED-talk videos, where the main subjects have been cropped out. We resize the original version into 256 256 resolution to train our model. This dataset includes 1124 training videos and 130 testing videos.
B.2 EVALUATION METRICS
We use five different metrics to evaluate our experimental results, namely L1, LPIPS, AKD, MKR and AED that quantify the reconstructed results. In addition, compute video FID evaluate video quality motion transferring tasks.
L1 represents the mean absolute pixel difference between reconstructed and real videos.
LPIPS (Zhang et al., 2018) aims at measuring the perceptual similarity between reconstructed and real images by leveraging the deep features from AlexNet (Krizhevsky et al., 2012).
Video FID is modified version of the original FID (Heusel et al., 2017). We here follow the same implementation as Wang al. (2020a) and utilize pre-trained ResNext101 (Hara et al., 2018) to extract spatio-temporal features compute the distance between real and generated videos distributions. We take the first 100 frames of each video as input of the feature-exee -extractett to compute the final scores.
Average keypoint distance (AKD) and missing keypoint rate (MKR) evaluate the difference be- tween keypoints of reconstructed and ground truth videos. We extract landmarks using the face alignment approach (Bulat & Tzimiropoulos, 2017) and extract body poses for both TaiChiHD and TED-talks using OpenPose (Cao et al., 2019). AKD is computed as the average distance be- tween corresponding keypoints, whereas MKR is the proportion of keypoints present in the ground- truth that are missing in reconstructed video.
Average Euclidean distance (AED) measures the ability of preserving identity in reconstructed video. We use a person re-identification pretrained model (Zheng et al., 2020) for measuring human bodies (TaichiHH and TED-talk) and OpenFace (Amos et al., 2016) for faces to extract identity em- beddings from reconstructed and ground truth frame pairs, then we compute MSE of their difference for all pairs.
15

---

Published as a conference paper at ICLR 2022
B.3 COMPARISON WITH FULL MRAA
We show quantitative evaluation results with the full MRAA model in Tab. We observe that our method achieves competitive results in reconstruction and keypoint evaluation. While we do not explicitly predict keypoints, w.r.. the TaichiHD dataset, interestingly we outperform MRAA in both, AKD and MKR. Such results showcase the effectiveness of our proposed method on modeling articulated human structures. However, reconstruction evaluation cannot provide completely fair comparison on how well the main subjects (e.g,, faces and human bodies) are generated in videos. This is in particular the case for TaichiHD and TED-talk, where backgrounds have large contribu- tions to the final scores.
Table 6: Comparison with full MRAA.
VoxCeleb
TaichHD
TED-taks (AKD, MKR)
Method 4 AKD AED LPIPS
4
(AKD, MKR)
AED LPIPS
AED LPIPS
MRAA 0.041
1.303
0.135
0.124
0.045
(5,551, 0.025)
0.431 0.178
0.027 (3,107, 0,0093)
0.379 0.11 0.0095)
0.123
(4.823, 0.020)
B.4 REFERENCE IMAGE GENERATION.
To produce æ,, we use G to decode z into the flow field d.-a.. Reference image T.. is obtained by warping using 0s-rr. The entire process is shown in Fig. 9.
Z s s-r
E
G
Source features (x
Source image (x₂)
Reference image (xy-)
Figure 9: Reference image generation.
B.5 QUALITATIVE RESULTS ON EFFECTIVENESS USING MOTION DICTIONARY
Fig. 10 illustrates the generated results on transferring motion from VoxCeleb to GermanPublicTV with and without motion dictionary. We observe that without the motion dictionary, appearance information is undesirably transferred from driving videos to generated videos.
B.6 LIMITATIONS
For human body, one limitation of our method is dealing with body occlusion. We observe in Fig. 11 that in taichi videos, in case of occlusion cause by legs and arms, motion is not transferred successfully. In addition, in TED-talks, transferring hand motion is challenging, hands of small size, articulated and sometimes occluded by human bodies.
16

---

Published as a conference paper at ICLR 2022
X
w/o dictionary
with dictionary
X
Figure 10: Generated results with and without Dm- We observe that the disentanglement of appearance and motion is much better by using Dm.
TaichiHD
TED-talks
wein WWW
C
WWW
weia
WWW vein
Driving
resuuts
Figure 11: Failure cases. We observed that it is still challenging for LIA to handle arm-leg occlusion (Taichi) and hand motion (TED-talk).
17


Reviews:
[{"review_id": "xjT8iUvtCRD", "reviewer": "Reviewer_PbBc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n - The idea is interesting and well motivated based on recent literature on interpretable direction in the latent space of generative models.\n\n - The experimental results highlight the improvement over various methods using a variety of metrics. I would be interested to see comparison with different approaches for face re-enactment, for example audio-driven approaches (e.g., [1])\n\n - The paper is generally well-written.\n\nWeaknesses: \n\n - The main weakness is the limited novelty of the method, since similar approaches have been utilised before, albeit not in the exact same way. For example, linear displacement of the latent codes using a motion basis is also used in [2] for video generation (not re-enactment). Similarly, a pre-trained encoder and generator are used in [3], where motion is synthesised again by shifting the latent code. However, I believe that the significance of the paper lies beyond the novelty of the method itself. Therefore, I would like to see some experimental comparison with some of these works, as well as discussion regarding the differences and advantages of the proposed method. \n\n - It would be helpful to see some qualitative results containing failure cases.\n\n - The impact of the orthogonality constraint on D is not studied. It would be interesting to see how it relates to the number of directions.\n\n\n[1] End-to-end speech-driven facial animation with temporal gans\n\n[2] A good image generator is what you need for high-resolution video synthesis \n\n[3] Style Generator Inversion for Image Enhancement and Animation\n\n", "summary_of_the_paper": "The paper proposes a method to animate images using a guide video. The novelty of the method lies in the absence of any form of structural representation. The authors pose the problem of synthesising motion as that of traversing the latent space. To this end,  a motion basis is learnt and is utilised to linearly shift each latent code. The experiments showcase the efficacy of the proposed model on a number of datasets.\n", "summary_of_the_review": "Overall, this is an interesting paper with significant results. I lean towards accepting this work if the issues that were raised are resolved.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t3d8Pf6ziNi", "reviewer": "Reviewer_EXdL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper presented a good idea for learning image motion representation. The motion itself could be tricky to represent, especially within the image space for animation purposes. Previously used concepts such as key points often have limitations to sufficiently disentangle motion from other attributes such as shape. Therefore it is desirable to have a more interpretable and disentangled representation. \n\nStrength:\n1. The idea is novel to me. The motivation is clear that the paper wants to get rid of explicit structural representation for motion. The linear motion decomposition module is interesting and novel.\n\n2. Experimental results are generally supportive. Compared to existing work, results seem to have less distortion and artifacts. The quantitative results also suggest the proposed method outperforms previous art.\n\nI have some concerns about the paper as follows:\n\n1. As explicit structural representation has its limitations. It also has the advantages of being very interpretable and can be learned in an unsupervised manner. The interpretability of the proposed method is not very clear to me. For example, how interpretable is the motion dictionary? The paper selectively showed some dimensions of the motion dictionary in Fig 6 and Appendix B.5. I think it might also be beneficial to show all of them. Do they also exhibit physical disentanglement such as 3D pose, expression? What about for human body?\n\n2. The size of the dictionary is decided empirically, as described in 4.3. It is said to be empirically decided between 5, 10, and 20, where 20 yields the best reconstruction. I wonder why not go higher to 40 or even 100? Meanwhile, how does M value affect the learned motion dictionary in terms of disentanglement and interpretability? \n\n3. In Figure 4, the proposed method doesn't seem to be able to replicate the driving video's pose very well. The paper should probably also compare with [Wang et al 2021a], which has an online demo.\n\n4. Some concepts are not clarified the first time its mentioned, leading to confusion. For example, what is \"r\" in \"Zs->r\" is not explained until the experiments. In the generator, T is also not explained.\n\n5. Some Typos exist in the paper. For example,  Section 3.4 , first paragraph, \"In reference\" --> \"In inference\".", "summary_of_the_paper": "This paper presented a method for animate images of a certain category, e.g., faces or human bodies. Different from some previous work with explicit structural representations, this paper focuses on learning a set of linear latent space directions that controls image animation without explicit structural representations. The network learns a set of orthogonal linear basis of latent codes that can be linearly combined into a latent code that represents a motion from a \"reference.\" The paper conducted a re-animation experiment on the human face and body motion dataset. Results are state-of-the-art by qualitative and quantitative measures.", "summary_of_the_review": "I think the paper presented an interesting idea for learning latent motion representation. The results are also very encouraging. It would be great if there are more in-depth analysis about the learned representation in terms of interpretability and disentanglement. Some writing improvement would also help the readers to understand the details better.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "k6qruk_3xBx", "reviewer": "Reviewer_jq2i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is well-written and insights of this work are clearly illustrated.\n+ This work proposes to directly manipulate the latent space without need of explicit structure representation, such as keypoints or regions. The framework is elegant, relatively novel, and technically sound.\n+ Qualitative and quantitative experiments demonstrate the superiority of the proposed method compared with previous arts. High-fidelity images (512x512 and 256x256)are shown.\n\nWeaknesses:\n- Though technically sound, the disentanglement of motion and appearance in an unsupervised manner is not entirely new, e.g. \nMoCoGAN: Decomposing Motion and Content for Video Generation, CVPR 2018\nMotion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns, AAAI 2020\n- Some implementation details are missing, e.g. model architectures, latent code dimensions, etc. \n- In terms of qualitative experiments, a sequence of generated images is expected. The model should not only produce realistic images  that match motion flow of the driving image, but should also maintain the temporal consistency within the driving video. Such performance should also be demonstrated.\n- Visualization of some basic transformation $d_i$ or the generated optical flow can also be beneficial in showing the effectiveness of the motion dictionary.", "summary_of_the_paper": "This work proposes LIA, a self-supervised AE that animates images by linear navigation in the latent space. LIA employs a two-step training paradigm. The first step encodes a source image into a latent code and then navigates to the target latent code along with a linear path. The second step decodes the target code to dense optical flow and warp with the source image. The framework is trained in a self-supervised manner. Experiments demonstrate the effectiveness of the proposed method on high-fidelity datasets.", "summary_of_the_review": "This work proposes a relatively novel method that aims to animate still image via latent space navigation. Experiments show the state-of-the-art performance compared with previous works. This work, nevertheless, can further be improved by providing more supporting experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UzP8dW-kmZ", "reviewer": "Reviewer_UntW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found the paper quite interesting and the results are quite impressive. Some parts of the presentation could be improved. I’ll list below some of the strengths and weaknesses of the paper in the current form. \n\nSTRENGTHS\n\n+ The proposed method is significantly more streamlined w.r.t. to the main competitor that relies on breaking the motion to animate into smaller local motions. The result is that the model itself as well as the training regime is significantly simpler to follow and (probably) to reimplement based on the information in the paper.\n\n+ The quality of the results shown is impressive and the quantitative analysis confirms the improvement w.r.t. previously published methods.\n\nWEAKNESSES\n\n1. The proposed technique does not explicitly disentangle the motion of the foreground object from the motion of the background, while one of the contributions of the main competitor (i.e., MRAA) was handling exactly this. This can generate unwanted artifacts where the background seems to stick to the motion of the foreground subjects. This effect is clearly visible in the  `realworld_vox.mp4` video in the supplementary material.\n\n2. The method presentation in section 3 could be improved. \n\n     **a.** While reading, for me, it was not clear what $z_{s \\leftarrow r}$ was supposed to represent till I read section 4.4. I would suggest moving part of this explanation directly in Section 3 in the form of intuitions about what the network it is supposed to model, and leave the experimental verification to Section 4.4. \n\n    **b.** It’s not clear why the two images $x_s$ and $x_d$ should be processed by the same encoder with shared parameters. Per my understanding, the outcome of $E(x_d)$ would be a latent transformation $z_{d \\rightarrow r}$ which the following MLP needs to “invert” and express as a combination of the learned basis vectors. Have the author considered using a different encoder for directly predicting $z_{r \\rightarrow d}$?\n\n3. The motion dictionary expressed as an orthogonal basis in the latent space is a pretty cool idea, but it could have been evaluated more deeply in the ablation study. In particular:\n\n    **a.** Table 5 seems to suggest an increase in performance connected with an increase of dimensions in the learned dictionary, however the evaluation stops at 20. Is there anything preventing the authors from trying more than 20 dimensions?\n\n    **b.** Tab 4 shows the impact of using a dictionary to model the motion, however there isn’t an ablation study testing whether the orthogonality constraint is needed and/or is beneficial to performance. I wonder what would happen if you express the motion as a linear combination of motions without constraints.\n\n4. The fact that nothing is specifically designed in the network to force $z_{s \\rightarrow r}$ to represent a transformation that brings the source image in a learned reference pose is both a strength and a weakness of the method as it may expose it to unexpected failures when applying it to a new set of data where such canonical pose cannot be clearly learned from the data. \n\n5. Failure cases of the model are not discussed neither in the main paper nor in the supplementary material. This should be a good practice for any research work, but in particular for this model as moving from a more structured intermediate representation of the motion, to a completely implicit one could expose the model to very unexpected and catastrophic failures. I would encourage the authors to discuss some of these in the paper.\n\nTYPOS & ERRORS\n\n* Page 5 after equation 2: “Semantically each $d_i$ should represent an basic” -> “a basic”\n* Beginning of Section 3.4.: “In reference” -> “At inference” ??\n* First paragraph of section 4: (c) is mentioned two times, the first one might be a typo\n* Is the driving image in the second row of Fig. 4 left correct? The poses of the generated images seem completely off.\n* Tab 6 of the supplementary, please highlight with bold even numbers from the competitor.\n", "summary_of_the_paper": "This work introduces a novel method for self-supervised image animation using a source image for the identity of the subject to animate (e.g., a face) and a driving video, involving another subject, showing the motion to replicate (e.g., another face speaking). \nThe main novelty of the work is to avoid explicit representation for the subject motion. The previous works decompose the subject into known parts (e.g., keypoints or regions) and defy local trajectories for them. Instead, the authors rely on an implicit representation of the motion as a trajectory in a learned latent space. The proposed architecture, given two frames, a source and a target one, generates an implicit transformation that brings the source one in a learned canonical reference pose and another transformation to go from canonical pose to target one. The two transformations combined can be decoded into the desired animation. This formulation greatly simplifies the network architecture and training regime w.r.t. previous works,without sacrificing the reconstruction quality. The proposed system achieves SOTA results in commonly used benchmarks for this field.  \n", "summary_of_the_review": "This work is quite interesting as it shows how it is possible to achieve compelling results in image animation without requiring any explicit intermediate structured representation for the motion of the main subject. The work builds on recent findings on the expressive power of the latent spaces of deep generative models for image manipulation and repurpose these for the task of animating images.\nAs a side effect of this drastically different approach, the method simplifies quite significantly the network architecture and the training regime while achieving equal or better results on standard benchmarks and user validation wrt the sota. The method presentation could be improved but the paper does not present major flaws in terms of methodology and experimental validation. For this reason I’m currently suggesting acceptance and rate  this paper at 8.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "77jXyHNrlg", "reviewer": "Reviewer_gSih", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ves: \n1. The idea of using learned linear orthogonal basis to replace representation to replace structure representations and re-use encoder is interesting. \n2. Overall, the paper is clear, well written and easy to follow.\n3. The comparison and ablation studies are intensive and well structured. The video for the real world case is impressive. \n\nConcerns: \n1. I am not fully convinced that the proposed linear orthogonal basis motion representation would be better than implicit structure landmark representations used in FOMM.  What is the dimension of this latent representation? And how to use it to generate dense flow? For a highly compact latent representation might not be easy to represent the complex motion and pose. In Fig. 4 row 2 and 3, the generated results’ pose seems less consistent with driving video. Also, the left-bottom video in ”comparison_taichi&ted.m4v”, the quality of video generated by MRAA seems better than the proposed method, especially for large poses of the face. \n \n2. Since the encoder is shared for appearance and motion, how to ensure these two could be well disentangled? A deeper analysis of the proposed method would have been nice. \n \n3. The main contribution of the proposed method is to use a learned linear motion dictionary, and can improve the generated image quality. On the other hand, would the linear assumption limit to generate large and complex motion?\n", "summary_of_the_paper": "This paper proposes a self-supervised auto-encoder based pipeline for animating images via latent space navigation. To enable animation, they designed a Latent Image Animator (LIA) module to model motion using a learned linear orthogonal basis. Given an input source and target image, their pipeline first encodes images into latent space, further estimates an offset vector from the target image, and produces a dense flow field/ The final output image is generated by warping the source image and a multi-scale refinement network.\nThe main contribution of this paper is the introduction of a learned linear orthogonal basis motion representation to replace structure representations like landmarks. Besides this, they use a single encoder to extract appearance and motion information to reduce model size. Studies are performed to show the superiority of their proposed method over two SOTA approaches, including FOMM and MRAA on 3 datasets. Results are shown on the task of (1) source reconstruction from video. (2) motion transfer. (3). User study.\n", "summary_of_the_review": "On balance I'm somewhat positive about the paper.  However, I have found some considerable issues as described above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a self-supervised auto-encoder latent image animator that animates images via latent space navigation. The task of transferring motion from driving videos to source images is formulated as learning linear transformations in the latent space. Experiments conducted on real-world videos demonstrate that the proposed framework can successfully animate still images. The proposed framework is novel, the experimental results are supportive and promising. However, some related works are still missing and might need to be added to the current paper for discussion and comparison. 

The rebuttal has addressed all major concerns raised by all 5 reviewers. The revised paper also included some feedback from the reviewers, except those discussions and comparisons with some missing related works pointed out by reviewers. After the rebuttal, all reviewers tend to accept the paper. AC agrees with the reviewers and recommends accepting the paper as a poster. Lastly, AC urges the authors to further improve their paper by incorporating the discussion on other missing related works suggested by the reviewers.
Decision: Accept (Poster)