Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an extension of first-order motion model for image animation with a driving video by incorporating the U-net based discriminator from Schonfeld et al. The main contribution of this paper is their training strategy of generative network that mixes synthetic pixels with real pixels around the occlusion area where warping artifacts are more likely to appear. Results and experiments are reported on three diverse datasets: VoxCeleb, BAIR, and Tai-Chi-HD with comparisons to closely related works. 

+This paper is well written and easy to read. Most of sections are clearly presented with sufficient details. 
+Both qualitative and quantitative experiments are reported on three diverse datasets with solid baselines including the base methods: two variants of first-order motion models. 

-The main concern is its novelty. The paper is mostly built upon the first-order motion model with the adoption of previously published U-net discriminator. The proposed pixel mixing method for generative network training is interesting, but it doesn't show significant improvement over the baseline. One way to further validate the proposed idea could be doing general image inpainting problems.
-It'd be better to include video results for evaluation. 
review_title: An extension of first-order motion model for image animation with marginal improvement on visual quality

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes PriorityCut, a data augmentation technique for self-supervised image animation. It uses the top-k percent occluded pixels of the foreground for consistency regularization. By mixing real and fake images according to the proposed method, the generator and discriminator are encouraged to focus on difficult parts, thus improving results. Experiments show that the proposed method could provide mild improvements in some cases. 

**Strengths**
+ The proposed PriorityCut is simple to implement and could help in some cases. 

**Weaknesses**
- The idea is not significant. The overall framework is based on FOMM (Siarohin et al. 2019). It merely proposes a simple way to identify difficult pixels and have the generation module focus on the more difficult parts. The mixing operation is similar to CutMix (Yun et al. 2019). Different from CutMix that is demonstrated useful for many important computer vision tasks, the proposed data augmentation technique is designed and demonstrated only for a specific application, image animation. 

- The improvements are very subtle. Although the tables show mild quantitative improvements, the gains are not significant. The visual enhancements compared to the baseline method are not substantial, either. For most examples in the supplementary document, the improvements against the baseline are very subtle. Also, the paper motivates the approach by occlusions, but the places where the proposed method helps to improve do not always exhibit occlusions.

- The paper is not well-written. For example, since both the binary background mask and the occlusion map are binary masks, I assume that the mask $\hat{\mathcal{O}}_{fg}$ is binary. It is not clear how to obtain the PriorityCut mask by taking the top k percent occluded pixels. It would be better to use a symbol for the PriorityCut mask, rather than the confusing operation $\min_k \hat{\mathcal{O}}_{fg}$.


**Minor issues**
- The paper mentions that the red and green colors indicate the worse and better result than the baseline. However, in Tables 1, 2, and 3, most of the numbers are not colored. 

- The topic of the paper is about animating images. As the abstract states, the result is a video of a source image following a motion of a driving video. For video, in addition to image quality, it is essential to investigate its temporal coherence quality. Without seeing the videos, it is difficult to see how temporally coherent the resultant videos are.

**Post-rebuttal**

After reading the rebuttal and the other reviews, my rating remains the same, although the rebuttal addresses some issues. The added videos show good temporal coherence as previous methods, and several writing issues have been resolved. However, I still feel that the contribution of the proposed method is not significant enough. It would be more convincing if the paper can show that the proposed augmentation technique is effective for more applications. Also, I still think that the visual improvement is very subtle. For the six added videos, only fig_a2_label.mp4 shows clear improvement visually. 

review_title: Both the contribution and the improvement are not significant

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes PRIORITYCUT, where the authors create an occlusion mask based on predicted background map and occlusion map. The occlusion mask indicates which pixels have heavy motions and most of these pixels are on the edges.  Next, the authors utilize this map to create the reconstructed driving image. The benefits are that the discriminator could totally focus on the occluded regions and improve the generating effects. 

Comments:
The paper mainly combines the CutMix and First Order Motion Model, which is pretty interesting. However, I have several concerns :

Novelty:  The paper seems to combine two methods together. The network comes from First Order Motion Model while the PRIORITYCUT is built on top of CutMix. After reading the paper, my feeling is that the author slightly changes the formula in CutMix and use this trick to get better reconstruction results. However, I don’t think this paper brings too much new knowledge.

Experiments: in equation 2, there is the top-k number. Which k do you choose in your experiments?  I do see an ablation study in supp but it seems to be an evaluation of masked PSNR and SSIM, not the choice of k. I think it would be helpful to add this. To be specific, does top-k generalize CutMix? For example, if K = 0, it is CutMix?

The good point is that the authors outperform the state of the art. This is a very good point since while the proposed priority cut is simple, it works.


Conclusion: Overall, I think this paper proposes an interesting idea and shows good results. However, due to lack of creativity and inadequate ablation study, I rate it below the acceptance bar.

review_title: review

final_score: rating: 2: Strong rejection
review_content: **1. Presentation and clarity**

I believe the paper is very poorly structured, does not introduce related work properly, contains many unclear points in the presentation, making it almost impossible for the reader to grasp key ideas without reading at least three related works on which this paper is heavily based. Until one consults (Schonfeld et al., 2020) it is not clear what the per-pixel feedback means, it's also not clear what is inpainting regularizations. If we look at section 3.3 we might get confused what is the disciminator encoder and decoder, and how the losses are used to train them. These details are fleshed out in the appendix and in the prior work. 

I believe the flaw in presentation is due to weak originality of the paper. It borrows heavily from (Schonfeld et al., 2020) and (Siarohin et al., 2019b) and hence does not have much of original content. 

**2. Originality**

The paper replaces the discriminator of (Siarohin et al., 2019b) with the one presented in (Schonfeld et al., 2020)  supervising it with the images generated with the priority-cut scheme. The latter uses the occlusion mask of (Siarohin et al., 2019b) to generate a new image by combining the generated image with the driving image, attempting to show the discriminator which pixels require further attention. 

In my opinion, these ideas are quite marginal and do not contain any interest for community. The paper re-uses already existing and well working techniques, such as the whole framework of (Siarohin et al., 2019b).

**3. Results**

By looking at the results I cannot see significant differences compared with (Siarohin et al., 2019b). The artifacts present in the first order motion model are present here, so no noticeable improvement overall. The reader will be able to notice something only if they zoom into the figures and compare very small details. To facilitate such behavior, the authors increased the regions in which such details might be noticeable. Even with such visualization, I have to admit, the differences are insignificant. This observation is supported by the numerical results as well. The improvement over FOMM is 0.0012 in terms of L1 on VoxCeleb and 0.002 on Tai-Chi-HD and 0.0016 on BAIR. 

**4. Rating**

I believe, poor presentation alone is a sufficient reason to recommend rejection. If we set aside it for a moment, we notice that the presented ideas are simple adaptations from previously published papers and results do not show necessary improvement.
review_title: The paper discusses a method to slightly improve existing animation techniques using a U-net discriminator and a mask for inpainting regularization

Meta-review: All the reviewers agreed that the paper lacks novelty. The overall framework is based on FOMM (Siarohin et al. 2019); the mixing operation is similar to CutMix (Yun et al. 2019). The improvements over the prior work are very subtle. R1 and R3 mentioned the paper is not well-written. The rebuttal didn’t change the reviewers’ mind. In particular, the reviewers pointed out that only one out of six videos showed clear visual improvement in the added video results. After reading the paper, reviewer’s comments, the rebuttal with added results, the AC agrees with the reviewers that the paper is not ready for publication.
Decision: Reject