Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presented a method for animate images of a certain category, e.g., faces or human bodies. Different from some previous work with explicit structural representations, this paper focuses on learning a set of linear latent space directions that controls image animation without explicit structural representations. The network learns a set of orthogonal linear basis of latent codes that can be linearly combined into a latent code that represents a motion from a "reference." The paper conducted a re-animation experiment on the human face and body motion dataset. Results are state-of-the-art by qualitative and quantitative measures.
main_review: I think the paper presented a good idea for learning image motion representation. The motion itself could be tricky to represent, especially within the image space for animation purposes. Previously used concepts such as key points often have limitations to sufficiently disentangle motion from other attributes such as shape. Therefore it is desirable to have a more interpretable and disentangled representation. 

Strength:
1. The idea is novel to me. The motivation is clear that the paper wants to get rid of explicit structural representation for motion. The linear motion decomposition module is interesting and novel.

2. Experimental results are generally supportive. Compared to existing work, results seem to have less distortion and artifacts. The quantitative results also suggest the proposed method outperforms previous art.

I have some concerns about the paper as follows:

1. As explicit structural representation has its limitations. It also has the advantages of being very interpretable and can be learned in an unsupervised manner. The interpretability of the proposed method is not very clear to me. For example, how interpretable is the motion dictionary? The paper selectively showed some dimensions of the motion dictionary in Fig 6 and Appendix B.5. I think it might also be beneficial to show all of them. Do they also exhibit physical disentanglement such as 3D pose, expression? What about for human body?

2. The size of the dictionary is decided empirically, as described in 4.3. It is said to be empirically decided between 5, 10, and 20, where 20 yields the best reconstruction. I wonder why not go higher to 40 or even 100? Meanwhile, how does M value affect the learned motion dictionary in terms of disentanglement and interpretability? 

3. In Figure 4, the proposed method doesn't seem to be able to replicate the driving video's pose very well. The paper should probably also compare with [Wang et al 2021a], which has an online demo.

4. Some concepts are not clarified the first time its mentioned, leading to confusion. For example, what is "r" in "Zs->r" is not explained until the experiments. In the generator, T is also not explained.

5. Some Typos exist in the paper. For example,  Section 3.4 , first paragraph, "In reference" --> "In inference".
summary_of_the_review: I think the paper presented an interesting idea for learning latent motion representation. The results are also very encouraging. It would be great if there are more in-depth analysis about the learned representation in terms of interpretability and disentanglement. Some writing improvement would also help the readers to understand the details better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes LIA, a self-supervised AE that animates images by linear navigation in the latent space. LIA employs a two-step training paradigm. The first step encodes a source image into a latent code and then navigates to the target latent code along with a linear path. The second step decodes the target code to dense optical flow and warp with the source image. The framework is trained in a self-supervised manner. Experiments demonstrate the effectiveness of the proposed method on high-fidelity datasets.
main_review: Strengths:
+ The paper is well-written and insights of this work are clearly illustrated.
+ This work proposes to directly manipulate the latent space without need of explicit structure representation, such as keypoints or regions. The framework is elegant, relatively novel, and technically sound.
+ Qualitative and quantitative experiments demonstrate the superiority of the proposed method compared with previous arts. High-fidelity images (512x512 and 256x256)are shown.

Weaknesses:
- Though technically sound, the disentanglement of motion and appearance in an unsupervised manner is not entirely new, e.g. 
MoCoGAN: Decomposing Motion and Content for Video Generation, CVPR 2018
Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns, AAAI 2020
- Some implementation details are missing, e.g. model architectures, latent code dimensions, etc. 
- In terms of qualitative experiments, a sequence of generated images is expected. The model should not only produce realistic images  that match motion flow of the driving image, but should also maintain the temporal consistency within the driving video. Such performance should also be demonstrated.
- Visualization of some basic transformation $d_i$ or the generated optical flow can also be beneficial in showing the effectiveness of the motion dictionary.
summary_of_the_review: This work proposes a relatively novel method that aims to animate still image via latent space navigation. Experiments show the state-of-the-art performance compared with previous works. This work, nevertheless, can further be improved by providing more supporting experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to animate images using a guide video. The novelty of the method lies in the absence of any form of structural representation. The authors pose the problem of synthesising motion as that of traversing the latent space. To this end,  a motion basis is learnt and is utilised to linearly shift each latent code. The experiments showcase the efficacy of the proposed model on a number of datasets.

main_review: Strengths: 

 - The idea is interesting and well motivated based on recent literature on interpretable direction in the latent space of generative models.

 - The experimental results highlight the improvement over various methods using a variety of metrics. I would be interested to see comparison with different approaches for face re-enactment, for example audio-driven approaches (e.g., [1])

 - The paper is generally well-written.

Weaknesses: 

 - The main weakness is the limited novelty of the method, since similar approaches have been utilised before, albeit not in the exact same way. For example, linear displacement of the latent codes using a motion basis is also used in [2] for video generation (not re-enactment). Similarly, a pre-trained encoder and generator are used in [3], where motion is synthesised again by shifting the latent code. However, I believe that the significance of the paper lies beyond the novelty of the method itself. Therefore, I would like to see some experimental comparison with some of these works, as well as discussion regarding the differences and advantages of the proposed method. 

 - It would be helpful to see some qualitative results containing failure cases.

 - The impact of the orthogonality constraint on D is not studied. It would be interesting to see how it relates to the number of directions.


[1] End-to-end speech-driven facial animation with temporal gans

[2] A good image generator is what you need for high-resolution video synthesis 

[3] Style Generator Inversion for Image Enhancement and Animation


summary_of_the_review: Overall, this is an interesting paper with significant results. I lean towards accepting this work if the issues that were raised are resolved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised auto-encoder based pipeline for animating images via latent space navigation. To enable animation, they designed a Latent Image Animator (LIA) module to model motion using a learned linear orthogonal basis. Given an input source and target image, their pipeline first encodes images into latent space, further estimates an offset vector from the target image, and produces a dense flow field/ The final output image is generated by warping the source image and a multi-scale refinement network.
The main contribution of this paper is the introduction of a learned linear orthogonal basis motion representation to replace structure representations like landmarks. Besides this, they use a single encoder to extract appearance and motion information to reduce model size. Studies are performed to show the superiority of their proposed method over two SOTA approaches, including FOMM and MRAA on 3 datasets. Results are shown on the task of (1) source reconstruction from video. (2) motion transfer. (3). User study.

main_review: +ves: 
1. The idea of using learned linear orthogonal basis to replace representation to replace structure representations and re-use encoder is interesting. 
2. Overall, the paper is clear, well written and easy to follow.
3. The comparison and ablation studies are intensive and well structured. The video for the real world case is impressive. 

Concerns: 
1. I am not fully convinced that the proposed linear orthogonal basis motion representation would be better than implicit structure landmark representations used in FOMM.  What is the dimension of this latent representation? And how to use it to generate dense flow? For a highly compact latent representation might not be easy to represent the complex motion and pose. In Fig. 4 row 2 and 3, the generated results’ pose seems less consistent with driving video. Also, the left-bottom video in ”comparison_taichi&ted.m4v”, the quality of video generated by MRAA seems better than the proposed method, especially for large poses of the face. 
 
2. Since the encoder is shared for appearance and motion, how to ensure these two could be well disentangled? A deeper analysis of the proposed method would have been nice. 
 
3. The main contribution of the proposed method is to use a learned linear motion dictionary, and can improve the generated image quality. On the other hand, would the linear assumption limit to generate large and complex motion?

summary_of_the_review: On balance I'm somewhat positive about the paper.  However, I have found some considerable issues as described above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a novel method for self-supervised image animation using a source image for the identity of the subject to animate (e.g., a face) and a driving video, involving another subject, showing the motion to replicate (e.g., another face speaking). 
The main novelty of the work is to avoid explicit representation for the subject motion. The previous works decompose the subject into known parts (e.g., keypoints or regions) and defy local trajectories for them. Instead, the authors rely on an implicit representation of the motion as a trajectory in a learned latent space. The proposed architecture, given two frames, a source and a target one, generates an implicit transformation that brings the source one in a learned canonical reference pose and another transformation to go from canonical pose to target one. The two transformations combined can be decoded into the desired animation. This formulation greatly simplifies the network architecture and training regime w.r.t. previous works,without sacrificing the reconstruction quality. The proposed system achieves SOTA results in commonly used benchmarks for this field.  

main_review: Overall I found the paper quite interesting and the results are quite impressive. Some parts of the presentation could be improved. I’ll list below some of the strengths and weaknesses of the paper in the current form. 

STRENGTHS

+ The proposed method is significantly more streamlined w.r.t. to the main competitor that relies on breaking the motion to animate into smaller local motions. The result is that the model itself as well as the training regime is significantly simpler to follow and (probably) to reimplement based on the information in the paper.

+ The quality of the results shown is impressive and the quantitative analysis confirms the improvement w.r.t. previously published methods.

WEAKNESSES

1. The proposed technique does not explicitly disentangle the motion of the foreground object from the motion of the background, while one of the contributions of the main competitor (i.e., MRAA) was handling exactly this. This can generate unwanted artifacts where the background seems to stick to the motion of the foreground subjects. This effect is clearly visible in the  `realworld_vox.mp4` video in the supplementary material.

2. The method presentation in section 3 could be improved. 

     **a.** While reading, for me, it was not clear what $z_{s \leftarrow r}$ was supposed to represent till I read section 4.4. I would suggest moving part of this explanation directly in Section 3 in the form of intuitions about what the network it is supposed to model, and leave the experimental verification to Section 4.4. 

    **b.** It’s not clear why the two images $x_s$ and $x_d$ should be processed by the same encoder with shared parameters. Per my understanding, the outcome of $E(x_d)$ would be a latent transformation $z_{d \rightarrow r}$ which the following MLP needs to “invert” and express as a combination of the learned basis vectors. Have the author considered using a different encoder for directly predicting $z_{r \rightarrow d}$?

3. The motion dictionary expressed as an orthogonal basis in the latent space is a pretty cool idea, but it could have been evaluated more deeply in the ablation study. In particular:

    **a.** Table 5 seems to suggest an increase in performance connected with an increase of dimensions in the learned dictionary, however the evaluation stops at 20. Is there anything preventing the authors from trying more than 20 dimensions?

    **b.** Tab 4 shows the impact of using a dictionary to model the motion, however there isn’t an ablation study testing whether the orthogonality constraint is needed and/or is beneficial to performance. I wonder what would happen if you express the motion as a linear combination of motions without constraints.

4. The fact that nothing is specifically designed in the network to force $z_{s \rightarrow r}$ to represent a transformation that brings the source image in a learned reference pose is both a strength and a weakness of the method as it may expose it to unexpected failures when applying it to a new set of data where such canonical pose cannot be clearly learned from the data. 

5. Failure cases of the model are not discussed neither in the main paper nor in the supplementary material. This should be a good practice for any research work, but in particular for this model as moving from a more structured intermediate representation of the motion, to a completely implicit one could expose the model to very unexpected and catastrophic failures. I would encourage the authors to discuss some of these in the paper.

TYPOS & ERRORS

* Page 5 after equation 2: “Semantically each $d_i$ should represent an basic” -> “a basic”
* Beginning of Section 3.4.: “In reference” -> “At inference” ??
* First paragraph of section 4: (c) is mentioned two times, the first one might be a typo
* Is the driving image in the second row of Fig. 4 left correct? The poses of the generated images seem completely off.
* Tab 6 of the supplementary, please highlight with bold even numbers from the competitor.

summary_of_the_review: This work is quite interesting as it shows how it is possible to achieve compelling results in image animation without requiring any explicit intermediate structured representation for the motion of the main subject. The work builds on recent findings on the expressive power of the latent spaces of deep generative models for image manipulation and repurpose these for the task of animating images.
As a side effect of this drastically different approach, the method simplifies quite significantly the network architecture and the training regime while achieving equal or better results on standard benchmarks and user validation wrt the sota. The method presentation could be improved but the paper does not present major flaws in terms of methodology and experimental validation. For this reason I’m currently suggesting acceptance and rate  this paper at 8.
review_title: null

Meta-review: This paper proposes a self-supervised auto-encoder latent image animator that animates images via latent space navigation. The task of transferring motion from driving videos to source images is formulated as learning linear transformations in the latent space. Experiments conducted on real-world videos demonstrate that the proposed framework can successfully animate still images. The proposed framework is novel, the experimental results are supportive and promising. However, some related works are still missing and might need to be added to the current paper for discussion and comparison. 

The rebuttal has addressed all major concerns raised by all 5 reviewers. The revised paper also included some feedback from the reviewers, except those discussions and comparisons with some missing related works pointed out by reviewers. After the rebuttal, all reviewers tend to accept the paper. AC agrees with the reviewers and recommends accepting the paper as a poster. Lastly, AC urges the authors to further improve their paper by incorporating the discussion on other missing related works suggested by the reviewers.
Decision: Accept (Poster)