PriorityCut: Occlusion-aware Regularization for Image Animation
Authors: Wai Ting Cheung
Gyeongsu Chae
Venue:  (2021)
Keywords: image animation
occlusion
inpainting
gan
augmentation
regularization

Abstract: Image animation generates a video of a source image following the motion of a driving video. Self-supervised image animation approaches do not require explicit pose references as inputs, thus offering large flexibility in learning. State-of-the-art self-supervised image animation approaches mostly warp the source image according to the motion of the driving video, and recover the warping artifacts by inpainting. When the source and the driving images have large pose differences, heavy inpainting is necessary. Without guidance, heavily inpainted regions usually suffer from loss of details. While previous data augmentation techniques such as CutMix are effective in regularizing non-warp-based image generation, directly applying them to image animation ignores the difficulty of inpainting on the warped image. We propose PriorityCut, a novel augmentation approach that uses the top-$k$ percent occluded pixels of the foreground to regularize image animation. By taking into account the difficulty of inpainting, PriorityCut preserves better identity than vanilla CutMix and outperforms state-of-the-art image animation models in terms of the pixel-wise difference, low-level similarity, keypoint distance, and feature embedding distance.

Under review as a conference paper at ICLR 2021
PRIORITYCUT: OCCLUSION-AWARE REGULARIZATION FOR IMAGE ANIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Image animation generates a video of a source image following the motion of driving video. Self-supervised image animation approaches do not require explicit pose references as inputs, thus offering large flexibility in learning. State-of-the-art self-supervised image animation approaches mostly warp the source image according to the motion of the driving video, and recover the warping artifacts by inpainting. When the source and the driving images have large pose differences, heavy inpainting is necessary. Without guidance, heavily inpainted regions usually suffer from loss of details. While previous data augmentation techniques such CutMix are effective in regularizing non-wan see-waar-bbasse image generation, directly applying them to image animation ignores the difficulty inpainting the warped image. We propose PriorityCut, novel augmentation approach that uses the top-k percent occluded pixels of the foreground to regularize image animation. By taking into account the difficulty of inpainting, PriorityCur preserves better identity than vanilla CutMix and outperforms state-of-the-an image animation models in terms of the pixel-wise difference, low-level similarity, keypoint distance, and feature embedding distance.
Source Image
Driving Image Generated Image Occlusion Mask PriorityCut Mask CutMix Image
Warp-based image animation warps the source image based on the motion of the driving image and recovers the warping artifacts by inpainting. PriorityCut utilizes the occlusion information in image animation indicating the locations of warping artifacts to regularize discriminator predictions on inpainting. The augmented image has smooth transitions without loss or mixture of context.
Figure 1:
INTRODUCTION
Image animation takes an image and driving video as inputs and generates a video of the input image that follows the motion of the driving video. Traditional image animation requires a reference pose of the animated object such as facial keypoints or edge maps (Fu et al. 2019 Ha et al. 2019
Qian al., 2019 Zhang al. 20196 Otberdout al. 2020). Self-supervised image animation does not require explicit keypoint labels on the objects (Wiles et al. 2018 Kim et al. 2019 Siarohin et al.
2019a b). Without explicit labeling, these approaches often struggle to produce realistic images when
the poses between the source and the driving images differ significantly.
To understand this problem, we first look at the typical process of self-supervised image animation approaches. These approaches can be generalized into the following pipeline: (1) keypoint detection, (2) motion prediction, and (3) image generation. Keypoint detection identifies important points in the source image for movement. Motion prediction estimates the motion of the source image based on the driving image. Based on the results of keypoint detection and motion prediction, it warps the source image obtain intermediate image that closely resembles the motion of the driving image.

---

Under review as a conference paper at ICLR 2021
Image generation then recovers the warping artifacts by inpainting. Existing approaches mostly provide limited to no guidance on inpainting. The generator has to rely on the learned statistics to recover the warping artifacts. For instance. First Order Motion Model (Starohin et al.) (019b) predicts an occlusion mask that indicates where and how much the generator should inpaint. While it has shown significant improvements over previous approaches such as X2Face (Wiles al. (018) and Monkey-Net (Siarohin et al. 2019a), it struggles to inpaint realistic details around heavily occluded areas. The occlusion mask does not provide information on how well the generator inpaints.
We propose PriorityCut, a novel augmentation approach that uses the top-k percent occluded pixels of the foreground for consistency regularization. PriorityCut derives new mask from the occlusion mask and the background mask. Using the PriorityCut mask, we apply CutMix operation (Yun et al., 2019), a data augmentation that cuts and mixes patches of different images, to regularize discriminator predictions. Compared to the vanilla rectangular CutMix mask, PriorityCut mask is flexible in both shape and locations. Also, PriorityCut prevents unrealistic patterns and information loss unlike previous approaches (revs & Taylor 2017 Yun et al. 2019 Zhang et al. 2017). The subtle differences in our CutMix image allow the generator to take small steps in learning, thus refining the details necessary for realistic inpainting. We built PriorityCut on top of First Order Motion Model and experimented on the VoxCeleb (Nagrani et al.) 2017). BAIR (Ebert et al., (017), and Tai-Chi-HD (Siarohin al., (019b) datasets. Our experimental results show that PriorityCut outperforms state-of-the-art image animation approaches in pixel-wise difference, low-level similarity, keypoint distance, and feature embedding distance.
2 RELATED WORK
Data augmentation Our work is closely related to patch-based augmentation techniques. Cutout and its variants drop random patches of an image (DeVries & Taylor 2017, Singh al., 2018 Chen 2020). Mixup blends two images to generate a new sample (Zhang et al. (2017). 7). CutMix and its variants cut and mix patches of random regions between images (Takahashi al. 2019 Yun et al. 2019; Yoo et al., (020).. Yoo et al. (2020) observed that existing patch-basee data augmentation techniques either drop the relationship of pixels, induce mixed image contents within an image, or
cause a sharp transition in an image. In contrast, we design our augmentation to avoid these issues.
Image animation Traditional image animation requires reference pose of the animated object such as facial keypoints or edge maps (Fu et al.) 2019: Ha et al.) 2019) QQia et al. 2019 Zhang et al.) 2019b] Otberdout et al. (2020). Self-supervised image animation does not require explicit labels the objects. X2Face (Wiles et al. 2018) uses an embedding network and driving network to generate images. Kim et al. (2019) used keypoint detector and motion generator to predict videos of an action class based on single image. Monkey-Net (Siarohin et al. 2019a) generates images based on a source image, relative keypoint movements, and dense motion. First Order Motion Model (Siarohin et al.. 2019b) extended Monkey-Net by predicting Jacobians in keypoint detection and an occlusion mask. Burkov et al. (2020) achieved pose-identity disentanglement using a big identity encoder and a small pose encoder. Yao et al.] (2020) generated images based on optical flow predicted on 3D meshes. These approaches mostly provide limited to no guidance on inpainting. In contrast, our approach utilizes the occlusion information to guide inpainting.
Advancements in generative adversarial networks Researchers have proposed different solutions to address the challenges of GANs (Bissoto et al., 2019). Our work is closely related to architectural methods, constraint techniques, and souuge-t--imaee translation. [Chen et al. (2018) modulated the intermediate layers of a generator by the input noise vector using conditional batch normalization. Kurach et al.) (2019) conducted large-scale study on different regularization and normalization techniques. Some researchers applied consistency regularization on real images (Zhang ct al.) 2019a). and additionally on generated images and latent variables (Zhao et al.) 2020). Researchers also provided local discriminator feedback on patches (Isola et al.] 2017) and individual pixels with CutMix regularization (Schonfeld et al., 2020). Our work differs from Schonfeld et al. (2020) in the application domain, mask shape, and mask locations. First, their experiments non-warp- based image generation, but we experimented with image animation. Also, their CutMix mask is rectangular and is applied at arbitrary locations. In contrast, mask shape is irregular and
2

---

Under review as a conference paper at ICLR 2021
is applied to heavily occluded areas. In Section 3.1 we discuss the implications of directly applying the vanilla CutMix to image animation.
METHODOLOGY
The core of our methodology is to guide the model to gradually learn the crucial parts in inpainting. We first summarize the base architecture we built upon. Then, we introduce per-pixel discriminator feedback and its importance image animation. After that, we discuss the limitations of directly applying existing patch-bases data augmentation on image animation. Lastly, illustrate how the limitations of existing data augmentation techniques inspired the design of our approach.
3.1 BACKGROUND
First Order Motion Model We built our architecture on top of First Order Motion Model (iarohhin al. 2019b), a state-of-the-art model on image animation. First Order Motion Model consists of motion estimation module and an image generation module. The motion estimation module takes as inputs source image S and driving image D, and predicts dense motion field Ts- D and an occlusion mask ÖS.-D. The image generation module warps the source image based on the dense motion field Ts-DD and recovers warping artifacts by inpainting the occluded parts of the source image. For details of individual modules, see Section of the appendix.
Per-pixel discriminator feedback In image recovery, the generator needs to maintain both the global and local realism. Existing image animation techniques either provide no clues (Wiles) et al.] 2018 Siarohin et al., (2019a) limited clues like occlusion map (Siarohin et al.) 2019b Yao et al. 2020) to guide inpainting.
reenacted image can share similar pose as the driving image (global realism), but the subtle texture geometry differences can affect the perspective of identity (local realism). To address this issue, we adapted the U-Net discriminator architecture (Schonfeld al. 2020) to provide both global and per-pixel discriminator feedback. A U-Net discriminator D consists of an encoder D and a decoder Di The encoder serves the same purpose as an ordinary discriminator, which predicts if an image is real or fake (global realism). The decoder predicts if individual pixels are real or fake (local realism). The U-Net discriminator uses skip connections to feed information between matching resolutions of the encoder and the decoder. Per-pixel discriminator feedback is especially important in warping- image animation techniques. After warping, some regions the warped image resemble the source image less than the other regions, and recovering the artifacts around those regions requires relatively more inpainting effort. Per-pixel feedback helps the generator learn precisely where and how much to improve during inpainting. For details of our architectures, Section B of the appendix.
Consistency regularization While per-pixel discriminator feedback provides fine-grained feedback, there is guarantee consistent predictions. Researchers have demonstrated the effectiveness of CutMix (Yun et al. 2019) in regularizing the U-Net decoder (Schonfeld et al. 2020). However, directly applying vanilla CutMix to image animation has few limitations. First, unlike Schonfeld et al. 2020) that experimented on non-warp-base based image generation, image warping creates gradient of inpainting difficulty on a single image. Applying CutMix at arbitrary locations makes difficult for the generator to focus on improving the heavily occluded areas. Also, there is mismatch between the mask shape and the task nature. In image animation, there can be multiple occluded regions of irregular shapes. Simply using single rectangular mask for CutMix like [Schonfeld al. (2020) does not reflect the reality of the task. Finally, regularizing the discriminator with vanilla CutMix can provide only partial per-pixel feedback on image restoration. Yoo et al. (2020) suggested that good augmentation techniques should not have sharp transitions like CutMix (Yun) al. 2019), mixed image contents within image patch like Mixup (Zhang et al., 2017). losing the relationships of pixels like Cutout (Dees & Taylor 2017). In the vanilla CutMix image, part of the image context is replaced by that of another image. Mixing per-pixel feedback may confuse the generator on restoring the artifacts. To fully utilize per-pixel discriminator feedback, an augmentation mask should closely reflect the tasks model is trying to learn.
3

---

Under review as a conference paper at ICLR 2021
Occluded Pixels
Source Image
Driving Image
Generated Image
Occlusion Mask
Background Mask
PriorityCut Mask
CutMix Image
10%
30%
50%
Figure 2: Illustration of deriving Priority Cut masks from occlusion and background masks.
Vanilla CutMix Reconstruction
Vanilla CutMix Feedback
PriorityCut Reconstruction
PriorityCut Feedback
Source
Driving
Figure 3: Comparison of per-pixel discriminator feedback between vanilla CutMix and PriorityCur
3.2 PRIORITYCUT
Our approach is based on two key observations. One observation that occlusion in warping-base based image animation reflects the intensity of artifacts that need be recovered. Another observation is that heavy occlusion can happen on both the foreground and the background. To recover the artifacts effectively, the generator should focus its learning on heavily occluded areas and the main object.
Based on the above observations, we propose PriorityCut, a novel augmentation that uses the top-k percent occluded pixels of the foreground the CutMix mask. Figureejillusraate the derivation of PriorityCut masks from occlusion and background masks. Suppose Ma, is an alpha background mask predicted by the dense motion network, ranging between and 1. We first suppress the uncertain pixels of the alpha background mask Mby to obtain binary background mask May- Mag corresponds to the background mask predicted by the dense motion network with high confidence. The occlusion map ÖS [0, H| is an alpha mask, with 0 being fully occluded and I being not occluded. Equation uutilizes Mag to compute the occlusion map of the foreground Ô
Ö, Mbn Mbg) ÔS
(1)
where @ denotes the Hadamard product. retains only the foreground portions the occlusion masks shown in Figure which are also alpha masks. Given percentile k. denote the PriorityCut mask Mpe as the top-k percent occluded pixels of the foreground Ö Following Yun et al. (201)), we randomize the values of A: in our experiments. Equation Jutilizes the PriorityCut mask perform CutMix between the real images x and the generated images x'. To avoid sharp transitions, PriorityCut performs CutMix on the driving image and reconstruction D.
mix(æ, x', M Myc 3 r + - MM0 O x'
(2)
In Figure 2 the CutMix images look almost identical to the driving the generated images with only subtle differences in fine details. PriorityCut always assigns the fake pixels to locations where there large changes in motion, creating incentives for the generator to improve. For example, borders,
4

---

Under review as a conference paper at ICLR 2021
edges, in-between regions of distinct objects (e.g. face, mic, wall), or parts of objects (e.g. hair, eyes, nose, mouth). The design philosophy of PriorityCut follows that of CutBlur (Yoo et al. 2020). The augmented images have sharp transitions, mixed image contents, pixels. PriorityCur also adds another degree of flexibility to the mask shapes. The discriminator can longer rely a rectangular area like the vanilla CutMix to predict where the real and fake pixels concentrate at. This encourages the discriminator to learn properly the locations of the real and fake pixels. Figure 3 compares the per-pixel discriminator feedback between PriorityCut and vanilla CutMix. PriorityCut helps the discriminator learn clear distinctions between real and fake pixels around locations with large changes in motion. In contrast, vanilla CutMix helps the discriminator learn only vague estimations. In Section 4.3 we compare PriorityCut with applying vanilla CutMix at arbitrary locations.
loss of the relationships of
3.3 TRAINING LOSSES
We followed previous works (Siarohin et al. 2019b Schonfeld et al. 2020) to use a combination of losses. The U-Net discriminator loss LD consists of the adversarial losses of the U-Net encoder D and the U-Net decoder L and the consistency regularization loss L The
consistency regularization loss regularizes the U-Net decoder output on the CutMix image and the CutMix between the U-Net decoder outputs on real and fake images. The generator loss LG consists of the reconstruction loss Lree based on activations of the pre-trained VGG-19 network (Simonyan & Zisserman 2014), the equivariance loss Lequie on local motion approximation to encourage consistent keypoint predictions, the adversarial loss Lodvs and the feature matching loss fe similar to that of pix2pixHI (Wang et al. 2018). For more details, refer to Section B in the appendix.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We followed Siarohin et al. (2019b) to preprocess high-quality videos on the following datasets and resized them to 256 x 256 resolution: the VoxCeleb dataset (Nagrani al., 2017) (18.398 training and 512 testing videos after preprocessing); the Tai-Chi-HD dataset (Siarohin et al. 2019b (2, .99 training and 285 testing video chunks after preprocessing); the BAIR robot pushing dataset Ebert et al. 2017) (42,880 training and 128 testing videos).
Evaluation protocol We (Sarooin et al. 2019h) to quantitatively and qualitatively evaluate video reconstruction. For video reconstruction, used the first frame of the input video as the source image and each frame as the driving image. We evaluated the reconstructed videos against the ground truth videos on the following metrics: pixel-wise differences (Lı); PSNR, SSIM, and their masked versions (M-SNNR, M-SSIM); average keypoint distance (AKD). missing keypoint rate (MKR), and average Euclidean distance (AED) of feature embeddings detected by third -patty tools.
For details on dataset preprocessing and metric computation, refer to Section in the appendix.
4.2 COMPARISON WITH STATE-OF-TIE--ARY
We quantitatively and qualitatively compared PriorityCut with state-of-th tfht---rrttt-tttt self-supervised image animation methods with publicly available implementations.
X2Face. The reenactment system with an embedding and a driving network (Wiles et al. 2018).
Monkey-Net. The motion transfer framework based on keypoint detector, dense motion network, and motion transfer generator (Siarohin et al. 2019a).
First Order Motion Model. The motion transfer network that extends Monkey-Net by estimating affine transformations for the keypoints and predicting occlusion for inpainting (Siarohin et al. 2019b). We compared two versions of First Order Motion Model. The baseline model (FOMM) corresponds to the one in their published paper. The adversarial model (FOMM+) is concurrent
5

---

Under review as a conference paper at ICLR 2021
Model
L1 +
PSNR f
SSIM f
AKD
AED
All
Salient
Salient
All
Salient
Salient
X2Face
0.0739-2e-4
1901
20.04 30.6510..4
0.62516e-4
0.681
0.944
6.847.4433 0.366412e-
Monkey-Net
0.0477=le-4
22.47:1002
23. 2002
34.43.1004
0.730155-4
0.76914e4
0.962
1.892:144- 0.1967=5e-
FOMM
0.0413=9-5
24.28
20
36. 1920.0
0.791
1:4e4
0.825
0.969
1.290 06-3
FOMM+
0.0409 9e-5
24.26:200
25. 2002
36.2622004
0.790
0.822
0.970
1.305
0.13399
Ours
0.0401=9-5 24.45:000
36.45 25
20.04 0.793
2. 0.970
1.286
Table Comparison with otate-of-the-art for approaches for video reconstruction on VoxCeleb.
Model
PSNR f
SSIM
AKD 4
MKR 4
AED +
All
Salient
Salient
All
Salient
Salient
X2Face
0.0729:3e4
18.16
21.08=0.02
22.24
0.580=1e3 0.858
0.734
14.89
0.175=te3
0.2441
Monkey-Net 0.0691
18.89
22.02.000
22.70g004
0.599.36-
0.867
0.742
11.40atea 0.060
0.2319a7e4
FOMM
0.056922-4
21.29
24.65
20.0s
25.18
2801
0.651
0.891
0.771
6.87
0.038
0.1657.2te4
FOMM+
0.0555
21.35=a.00
=0.05
=:04
0.654=223
0,893
0.772
6.73
0.032=4x4
0, 0.1647
Ours
0.0549:2-4
21.54:a.0
24.98
25.33
0.653+22-3 0.896
0.768
6.78+6e2
0.030=444
0.1629:ae44
Table 2: Comparison with state-of-the-art for approaches for video reconstruction Tai-Chi-HD.
work with an adversarial discriminator. Since its authors have released both both models, we evaluated the baseline model and additionally the adversarial model.
Ours. Our extension of First Order Motion Model with U-Net discriminator to provide per-pixel discriminator feedback and PriorityCut to regularize inpainting.
Quantitative comparison Table and 3 show the quantitative comparison results of video reconstruction on the VoxCeleb. BATR. and Tai-Chi-HD datasets, respectively. For all tables, the down arrows indicate that lower values mean better results, and vice versa. We show the 95% confidence intervals, highlight the best results in bold and underline the second-best For variants of the baseline model that do not produce the hest or the second best results, the red and green texts indicate worse and better results than the baseline, respectively. This serves similar purpose as the ablation study, indicating the effectiveness of certain components in improving the baseline. PriorityCut outperforms state-of-the-arr models in every single metric for VoxCeleb and BAIR, and in most of the metrics for Tai-Chi-HD. Note that adversarial training alone (FOMM+) does not always guarantee improvements, as highlighted in red for VoxCeleb.
Qualitative comparison Figure shows the qualitative comparison for the VoxCeleb and BAIR datasets. The color boxes highlight the noticeable differences between the results of different models.
Model
L1 4
PSNR
SSIM
X2Face
0.04194564
21.3#0011
0.831+26-3
For the VoxCeleb dataset, X2Face produces slight to heavy distortions on the face, depending on the pose angles. Monkey-Net either fails to follow the pose angles or struggles to preserve the identity of the source image. FOMM follows closely the pose angles, but it struggles to inpaint the subtle details. For instance, the corner of the right eye extends all the way to the hair (frame 1; frame below), the hair on the background (frame 2), and the left eye in a polygon shape (frame 3 top). FOMM+ either amplifies the artifacts (frames and 3) or is uncertain about the texture (frame 2). In contrast, PriorityCut maintains clear distinction between the right eye and the hair (frame 1; frame 3 below), inpaints the left eye in the ellipse shape (frame 3 top), and has high confidence in the texture (frame 2).
Monkey-Net
4.0340.:4-44
23.1::0.1
0.86712e-3
FOMM
0.0292446.4
24.8+00.1
1.889+
Ours
0.0276111e4
25.3..0.1
1.
Table 3: Comparison with sfate-of-the-art for approaches for video reconstruction on BAIR.

---

Under review as a conference paper at ICLR 2021
Source
Driving
Driving
Driving
Source
Driving
Driving
Driving 3
et
X2Face
X2Face
Nen
Monkey-Ned
First Order
First Order
Motion
Motion Model
Model
First Order
Mation
Ours
Model+
Ours
(a) VoxCeleb
(b) BAIR
Figure 4: Qualitative comparison of state-of-the-art approaches for image animation.
For the BAIR dataset, X2Face produces trivial warping artifacts. Monkey-Net either erases the object (frame 1 left) introduces extra artifacts (frame 2 right). FOMM is uncertain about the texture (frame left; frames and 3) or the geometry (frame right). In contrast, PriorityCut inpaints both realistic texture and geometry. Note that the blue and yellow object in the third frame is stretched due to image warping. PriorityCur recovers the texture while FOMM splits the object into two parts.
For additional qualitative comparison, refer to Section of the appendix.
4.3 ABLATION STUDY
To validate the effects of each proposed component, evaluated the following variants of model on video reconstruction. Baseline: the published First Order Motion Model used in their paper, Adv: the concurrent work of First Order Motion Model with global discriminator; U-Ner: the architecture the global discriminator extended to the U-Net architecture; PriorityCut: our proposed approach that the top-k percent occluded pixels of the foreground the CutMix mask.
Quantitative ablation study Table 4 guaatittatively compares the results of video reconstruction on the VoxCeleb dataset (Nagrani et al. 2017). First, adversarial training improves only the L, distance and the non-salient parts, but worsens other metrics. U-Net discriminator improves L1 by margin with better AKD as positive side bonus, at the cost of further degraded AED. We experimented vanilla CutMix on top of the U-Net architecture. After adding
with adding either PriorityCut PuiorityCut, the full model outperforms the baseline model in every single metric. In particular, the improvement of AED shows the effectiveness of Priority Cut in guiding the model inpaint realistic facial features. However, vanilla CutMix pushes the generator to optimize only the pixel values, at the cost of significant degradation in keypoint distance (AKD) and identity preservation (AED).
7

---

Under review as a conference paper at ICLR 2021
Architecture
PSNR t
SSIM
AKD
AED
All
Salient
Salient
All
Salient
Salient
Baseline
0.041319e5
24.281002
25.1910.8 36.19_006
0.791 0.825 4e4
0.969_2e4
1.290
0.1324
+ Adv
0.0409
24.26=au2
25. 05.17=0002
36.26=a04
0.790=4e-4
0.822+2-4
0.970=1e-4
1.305=2e-5
0.1339=ae-
+ U-Net
0,0401 24.34=0u2
25.29
36.31=a04
0.791
0.824
0.969=2x-4
1.278 0.1347=de-4
+ PriorityCut 0.0401 199-5
24.45
25.35
36.45 3.04
0.7931
0.826
4e-4
0.970
1.286:12e3
0.1303 6:6e-
CutMix
0.039419e-5
24.51 25.45:10.00
36.42-002
0.792
0.826
0.969-24
1.295
2.1365
Table 4: Quantitative ablation study for video reconstruction on VoxCeleb.
Source
Driving
Driving
Driving
Driving
0.8
Baseline
Adv
0.6
+ U-Net
0.4
Ours
0.2
CuMMix
Figure 5: Qualitative ablation study for video reconstruction on VoxCeleb.
Figure F qualitatively compares video reconstruction on the VoxCeleb dataset (Nagrani et al. 2017). The first row is the ground truth. The heatmaps illustrate the differences between the ground truth and the reconstructed frames. The color boxes highlight the noticeable differences between architectures. First, adversarial training amplifies the texture (right eyes of frames 2 and 4; artifacts in the heatmaps of frames I and 2). Per-pixel discriminator feedback produces more precise texture than the adversarial model (heatmaps in frames I and 3; right eye of frame 4). Similar to quantitative ablation study, we qualitatively compared the effects of adding either PriorityCut or vanilla CutMix on top the U-Net architecture. PriorityCut is sensitive to areas with large changes in motion. Among different architectures, PriorityCut is the only one that maintains the mic shape (frame 1) and the distance between the mic and the mouth (frame 4). Also, the heatmaps of frames and for PriorityCuu resemble the ground truth the most. For vanilla CutMix, the mic shape problem persists (frame 1), the right eye shows ambiguous texture (frame above) and the gap between the mic and the mouth is closed (frame 4 below). Most importantly, all heatmaps of vanilla CutMix show trivial differences texture around the edge of the right face. These suggest that vanilla CutMix struggles in distinguishing between foreground and background around locations with large motions, since its augmentation mask was not designed to handle motions. Overall, qualitative ablation study results show the effectiveness of PriorityCut in capturing the subtle shape and texture.
Qualitative ablation study
8

---

Under review as a conference paper at ICLR 2021
5 DISCUSSION
This section summarizes the key observations and findings of our work.
Limitations of warp-based image animation Existing warp-based image animation techniques recover the warping artifacts by inpainting. We observed that large pose differences in image animation often critically influence identity preservation. Without proper guidance, the generator usually struggles at recovering the warping artifacts at locations with large motions. To address this challenge, proposed PriorityCut to regularize image animation based inpainting difficulty, capturing the aspects related to large changes in motion. Our experimental results with solid baselines and diverse datasets show that PriorityCut outperforms state-of-the-ar models in identity preservation.
Limitations of regularizing image animation While Schonfeld et al. (2020) demonstrated the effectiveness of vanilla CutMix on non-warp-asse image generation, we observed that directly applying vanilla CutMix to image animation ignores the inpainting difficultly with augmentation masks irrelevant to motions, and provides only partial per-pixel feedback on image restoration (Section 3.T). Our comparisons with vanilla CutMix image animation (Section (4.) support our observations and reveal contradictory findings to that of Scchonfeld et al. (2020): directly applying vanilla CutMix to image animation compromises crucial image animation properties such as pose and identity to pursue pixel realism. Adversarial training, being unsupervised approach as vanilla CutMix, faces similar trade-offs when applied to image animation. To address this challenge, PriorityCut takes motion into account and redefines the CutMix augmentation mask to supervise training on the difficult parts of inpainting. The all-round realism of PriorityCut is attributable to the tight coupling between its novel design and the nature of image animation. Our findings substantiate those of Yoo et al. ) an augmentation mask closely related to the task nature plays significant role effective learning.
Potential applications of PriorityCut One limitation of PriorityCut the dependency on an occlusion mask and background mask. Only state-of-the-arr image animation approaches use these masks (Kim al. 2019 [iarohin et al.] 20196; [urkovv et al., 20200, Yao al.) (020). However, we anticipate any warp- based image animation approaches can adopt PriorityCut with proper modifications. In addition to image animation, we expect PriorityCut to widely applicable to any research areas involve image warping, occlusion, motion or optical flow estimation such as facial expression and body pose manipulation, image inpainting, and video frame interpolation.
6 CONCLUSION
We proposed PriorityCut, novel augmentation approach that captures the crucial aspects related to large changes in motion to address the identify preservation problem in image animation. PriorityCut outperforms state-of-the-art image animation models in terms of the pixel-wise difference, low-level similarity, keypoint distance, and feature embedding distance. Our experimental results demonstrated the effectiveness of PriorityCut in achieving all-round realism and confirmed the significance of augmentation mask in balanced learning.
REFERENCES
Alceu Bissoto, Eduardo Valle, and Sandra Avila. The six fronts of the generative adversarial networks. arXiv preprint arXiv: 1900:19100 2019.
Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 13786-13795, 2020.
Pengguang Chen. Gridmask data augmentation. arXiv preprint arXiv:2001.04086 2020.
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. arXiv preprint 1rxvv:1800.01336 2018.

---

Under review as a conference paper at ICLR 2021
Terrance VeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv:1710.05268, 2017.
Chaoyou Fu, Yibo Hu, Xiang Wu, Guoli Wang, Qian Zhang, and Ran He. High fidelity face 12003, 2019.
manipulation with extreme pose and expression. arXiv preprint
Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. arXiv preprint arXiv:1911.08133 2019.
Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexei A Efros. amage-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1 112--1134, 2017.
Yunji Kim, Seonghyeon Nam, Cho, and Seon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. In Advances in Neural Information Processing Systems, pp. 3814-3824, 2019.
Karol Kurach, Mario Lučié, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly, A large-scale study on regularization and normalization in gans. In International Conference on Machine Learning, pp. 3581-35901 2019.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.0888 08612. 2017.
Naima Otberdout, Mohammed Daoudi, Anis Kacem, Lahoucine Ballihi, and Stefano Berretti. Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative adversarial nets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang. Fumin Shen, Chen Qian, and Ran He. Make a face: Towards arbitrary high fidelity face manipulation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10033-10044 2019.
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8207-8216, 2020.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2377-2386, 2019a.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in Neural Information Processing Systems, pp. 7135-7145, 2019b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint UrXiv:11099 1556, 2014.
Krishna Kumar Singh, Hao Yu, Aron Sarmasi, Gautam Pradeep, and Yong Jae Lee. Hide-and-seek: A data augmentation technique for weakly-supervised localization and beyond. arXiv preprint arXiv:1811.02545 2018.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions on Circuits and Systems for Video Technology, 2019.
Ting- Ting-Chun Wang, Ming- Yu Liu, Jun Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
10

---

Under review as a conference paper at ICLR 2021
Olivia Wiles, A Sophia Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV). pp. 670-686, 2018.
Guangming Yao, Yi Yuan, Tianjia Shao, and Kun Zhou. Mesh guided one-shot face reenactment using graph convolutional networks. arXiv preprini arXiv:2008.07758 2020.
Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn. Rethinking data augmentation for image super-resolution: comprehensive analysis and a new strategy. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8375-8384, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision, pp. 5023-6032, 2019.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lec. Consistency regularization for generative adversarial networks. arXiv preprint arXiv: 12027, 2019a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv: 2017.
Jiangning Zhang. Xianfang Zeng, Yusu Pan, Yong Liu, Yu Ding, and Changjie Fan. Faceswapnet: Landmark guided many- monnny face reenactment. arXiv preprint 1781v 2, 2019b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans, arXiv preprint arXiv:2002.047224,220 2020.


Reviews:
[{"review_id": "lCuMmBB7JXA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes PriorityCut, a data augmentation technique for self-supervised image animation. It uses the top-k percent occluded pixels of the foreground for consistency regularization. By mixing real and fake images according to the proposed method, the generator and discriminator are encouraged to focus on difficult parts, thus improving results. Experiments show that the proposed method could provide mild improvements in some cases. \n\n**Strengths**\n+ The proposed PriorityCut is simple to implement and could help in some cases. \n\n**Weaknesses**\n- The idea is not significant. The overall framework is based on FOMM (Siarohin et al. 2019). It merely proposes a simple way to identify difficult pixels and have the generation module focus on the more difficult parts. The mixing operation is similar to CutMix (Yun et al. 2019). Different from CutMix that is demonstrated useful for many important computer vision tasks, the proposed data augmentation technique is designed and demonstrated only for a specific application, image animation. \n\n- The improvements are very subtle. Although the tables show mild quantitative improvements, the gains are not significant. The visual enhancements compared to the baseline method are not substantial, either. For most examples in the supplementary document, the improvements against the baseline are very subtle. Also, the paper motivates the approach by occlusions, but the places where the proposed method helps to improve do not always exhibit occlusions.\n\n- The paper is not well-written. For example, since both the binary background mask and the occlusion map are binary masks, I assume that the mask $\\hat{\\mathcal{O}}_{fg}$ is binary. It is not clear how to obtain the PriorityCut mask by taking the top k percent occluded pixels. It would be better to use a symbol for the PriorityCut mask, rather than the confusing operation $\\min_k \\hat{\\mathcal{O}}_{fg}$.\n\n\n**Minor issues**\n- The paper mentions that the red and green colors indicate the worse and better result than the baseline. However, in Tables 1, 2, and 3, most of the numbers are not colored. \n\n- The topic of the paper is about animating images. As the abstract states, the result is a video of a source image following a motion of a driving video. For video, in addition to image quality, it is essential to investigate its temporal coherence quality. Without seeing the videos, it is difficult to see how temporally coherent the resultant videos are.\n\n**Post-rebuttal**\n\nAfter reading the rebuttal and the other reviews, my rating remains the same, although the rebuttal addresses some issues. The added videos show good temporal coherence as previous methods, and several writing issues have been resolved. However, I still feel that the contribution of the proposed method is not significant enough. It would be more convincing if the paper can show that the proposed augmentation technique is effective for more applications. Also, I still think that the visual improvement is very subtle. For the six added videos, only fig_a2_label.mp4 shows clear improvement visually. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OKbq-lc1iPF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes PRIORITYCUT, where the authors create an occlusion mask based on predicted background map and occlusion map. The occlusion mask indicates which pixels have heavy motions and most of these pixels are on the edges.  Next, the authors utilize this map to create the reconstructed driving image. The benefits are that the discriminator could totally focus on the occluded regions and improve the generating effects. \n\nComments:\nThe paper mainly combines the CutMix and First Order Motion Model, which is pretty interesting. However, I have several concerns :\n\nNovelty:  The paper seems to combine two methods together. The network comes from First Order Motion Model while the PRIORITYCUT is built on top of CutMix. After reading the paper, my feeling is that the author slightly changes the formula in CutMix and use this trick to get better reconstruction results. However, I don’t think this paper brings too much new knowledge.\n\nExperiments: in equation 2, there is the top-k number. Which k do you choose in your experiments?  I do see an ablation study in supp but it seems to be an evaluation of masked PSNR and SSIM, not the choice of k. I think it would be helpful to add this. To be specific, does top-k generalize CutMix? For example, if K = 0, it is CutMix?\n\nThe good point is that the authors outperform the state of the art. This is a very good point since while the proposed priority cut is simple, it works.\n\n\nConclusion: Overall, I think this paper proposes an interesting idea and shows good results. However, due to lack of creativity and inadequate ablation study, I rate it below the acceptance bar.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ka8dgAhR3eM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**1. Presentation and clarity**\n\nI believe the paper is very poorly structured, does not introduce related work properly, contains many unclear points in the presentation, making it almost impossible for the reader to grasp key ideas without reading at least three related works on which this paper is heavily based. Until one consults (Schonfeld et al., 2020) it is not clear what the per-pixel feedback means, it's also not clear what is inpainting regularizations. If we look at section 3.3 we might get confused what is the disciminator encoder and decoder, and how the losses are used to train them. These details are fleshed out in the appendix and in the prior work. \n\nI believe the flaw in presentation is due to weak originality of the paper. It borrows heavily from (Schonfeld et al., 2020) and (Siarohin et al., 2019b) and hence does not have much of original content. \n\n**2. Originality**\n\nThe paper replaces the discriminator of (Siarohin et al., 2019b) with the one presented in (Schonfeld et al., 2020)  supervising it with the images generated with the priority-cut scheme. The latter uses the occlusion mask of (Siarohin et al., 2019b) to generate a new image by combining the generated image with the driving image, attempting to show the discriminator which pixels require further attention. \n\nIn my opinion, these ideas are quite marginal and do not contain any interest for community. The paper re-uses already existing and well working techniques, such as the whole framework of (Siarohin et al., 2019b).\n\n**3. Results**\n\nBy looking at the results I cannot see significant differences compared with (Siarohin et al., 2019b). The artifacts present in the first order motion model are present here, so no noticeable improvement overall. The reader will be able to notice something only if they zoom into the figures and compare very small details. To facilitate such behavior, the authors increased the regions in which such details might be noticeable. Even with such visualization, I have to admit, the differences are insignificant. This observation is supported by the numerical results as well. The improvement over FOMM is 0.0012 in terms of L1 on VoxCeleb and 0.002 on Tai-Chi-HD and 0.0016 on BAIR. \n\n**4. Rating**\n\nI believe, poor presentation alone is a sufficient reason to recommend rejection. If we set aside it for a moment, we notice that the presented ideas are simple adaptations from previously published papers and results do not show necessary improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNMEGn7_QO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an extension of first-order motion model for image animation with a driving video by incorporating the U-net based discriminator from Schonfeld et al. The main contribution of this paper is their training strategy of generative network that mixes synthetic pixels with real pixels around the occlusion area where warping artifacts are more likely to appear. Results and experiments are reported on three diverse datasets: VoxCeleb, BAIR, and Tai-Chi-HD with comparisons to closely related works. \n\n+This paper is well written and easy to read. Most of sections are clearly presented with sufficient details. \n+Both qualitative and quantitative experiments are reported on three diverse datasets with solid baselines including the base methods: two variants of first-order motion models. \n\n-The main concern is its novelty. The paper is mostly built upon the first-order motion model with the adoption of previously published U-net discriminator. The proposed pixel mixing method for generative network training is interesting, but it doesn't show significant improvement over the baseline. One way to further validate the proposed idea could be doing general image inpainting problems.\n-It'd be better to include video results for evaluation. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agreed that the paper lacks novelty. The overall framework is based on FOMM (Siarohin et al. 2019); the mixing operation is similar to CutMix (Yun et al. 2019). The improvements over the prior work are very subtle. R1 and R3 mentioned the paper is not well-written. The rebuttal didn’t change the reviewers’ mind. In particular, the reviewers pointed out that only one out of six videos showed clear visual improvement in the added video results. After reading the paper, reviewer’s comments, the rebuttal with added results, the AC agrees with the reviewers that the paper is not ready for publication.
Decision: Reject