Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos
Authors: Aysegul Dundar
Kevin J Shih
Animesh Garg
Robert Pottorf
Andrew Tao
Bryan Catanzaro
Venue:  (2020)
Keywords: unsupervised landmark discovery

Abstract: Unsupervised landmark learning is the task of learning semantic keypoint-like
representations without the use of expensive keypoint-level annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions, conditioning only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest. Furthermore, the rendered background quality is also improved, as the background rendering pipeline no longer requires the ill-suited landmarks to model its pose and appearance. We demonstrate this improvement in the context of the video-prediction.

Under review as a conference paper at ICLR 2020
UNSUPERVISED DISENTANGLEMENT OF POSE, APPEARANCE AND BACKGROUND FROM IMAGES AND VIDEOS
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations. popular approach is factorize an image into pose and appearance data stream, then to reconstruct the image from the factorized components. The pose represen- tation should capture a of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks focus on the foreground object of interest. However, the reconstruc- tion task of the entire image forces the model to allocate landmarks to model the background. This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions, conditioning only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest. Furthermore, the rendered background qual- the background rendering pipeline no longer requires the
ity is also improved, ill-suited landmarks to model its pose and appearance. We demonstrate this im- provement in the context of the video-prediction task.
INTRODUCTION
Pose prediction a classical computer vision task that involves inferring the location and configu- ration of deformable objects within an image. It has applications in human activity classification, finding semantic correspondences across multiple object instances, and robot planning to name a few. One of the caveats of this task is that annotation is very expensive. Individual object "parts" need to be carefully and consistently annotated with pixel-level precision. Our work focuses on the task of unsupervised landmark learning, which aims find unsupervised pose representations from image data without the need for direct pose-level annotation.
A good visual landmark should be tightly localized, consistent across multiple object instances, and grounded on the foreground object of interest. Tight localization is important because many objects (such persons) are highly deformable. A landmark localized to smaller, rigid area of the object will offer more precise pose information in the event of object motion. Consistency across multiple object instances also important, as we wish for landmarks apply all instances within a visual category. Finally, and most relevant to our proposed method, we want our landmarks to focus on the foreground objects. A landmark that fires on the background is wasted landmark, as the background is constantly changing, and yields little information regarding the pose of our foreground object of interest.
Many unsupervised landmark learning methods perturb an input training image with various trans- formations, then require the model to learn semantic correspondences across the transformed vari- ants to piece together the unaltered input image. The primary issue with this approach is it penalizes the entire image reconstruction when we care only about the foreground, resulting in landmarks being allocated to the background. This poses number of issues, including increased memory requirements (more landmarks required to capture the foreground) and lower landmark reliability (landmarks assigned to background unstable). Our proposed method aims to reduce the likeli-

---

Under review as a conference paper at ICLR 2020
hood of landmarks being allocated to the background, thereby improving overall landmark quality and reducing the number of landmarks required to achieve state-of-the-ar performance.
Our work builds upon existing methods in sange-rrccostruuction -uiided landmark learning tech- niques (Jakab et al., 2018 Lorenz et al. 2019). We explicitly encourage our model to factorize the reconstruction task into separate foreground and background reconstructions, where only the foreground reconstruction is conditioned on learned landmarks. Our contributions are as follows:
1. We propose improvement to reconstruction-guin lows the landmarks to better focus on the foreground.
unsupervised landmark learning that al-
2. We demonstrate through empirical analysis that our proposed factorization allows for state-of- the-art landmark results with fewer learned landmarks, and that fewer landmarks are allocated to modeling background content.
We demonstrate that the overall quality of the reconstructed frame is improved via the factorized rendering, and include an application to the video-prediction task.
2 RELATED WORKS
Our work builds upon prior methods unsupervised discovery of image correspondences (Thewlis)
et al. 2017b] Zhang et al., 2018b Suwajanakorn et al. [0188] Thewlis et al., 2017a Kanazawa ctt al. 2016 akab et al. 2018: Lorenz et al. (019). Most relevant here are Jakab et al. (2018) and Lorenz et al. (2119). which learn the latent landmark activation maps via image factorization and reconstruction pipeline. Each image is factored into pose and appearance representations and decoder is trained to reconstruct the image from these latent factors. The loss is designed such that accurate image reconstruction can only be achieved when the landmarks activate at consistent locations between an image its war-we variant. [orenz et al.] (2019) specifically improves upon the method proposed by Jakab et al. (2018) such that instead of representing the appearance information as single vector for the entire image, there is a separate appearance encoding for each landmark in the pose representation. One limitation of these works is that the appearance and pose vectors also need to encode background information in order to reconstruct the entire image. Our work attempts to resolve this limitation by introducing unsupervised foreground backuuruudd separation into the pipeline, using the pose and appearance vectors for only the foreground rendering.
There are few other works that propose separate foreground and background in image rendering tasks. [alakrishnan et al. (2018) separates foreground and background for image synthesis in an unseen pose, but their method relies supervised 2D keypoints. Rhodin et al. (2018) and Rhodin cct al.] (2019) separate background from foreground for single and multi-person pose-cstimation. In both works, the background images computed by taking the median pixel value across all frames, and therefore require video sequence data with perfectly static backgrounds. Instead, our approach trains network to synthesize clean background from any input frame. It is therefore more forgiving with respect to background variation, and can even handle thit-plate-spline warped backgrounds after overfitting the training data. This allows us use our method on non-video datasets such as CelebA faces (Liu et al. 2015).
3
METHOD
Our method extends the pipeline proposed by Lorenz al. (2019) At high-level, it reconstructs an image from two perturbed variants: one where the appearance (color, lighting, texture) information perturbed, and one where the pose (position, orientation) of the object perturbed. The model must learn to extract the pose information from the appearance-perturee information from the pose-perturbed image. The model will learn a set of landmarks in the process as means to spatially- the information extracted from the two sources in order to reconstruct the original image.
image, and appearance
Our extension to Lorenz et al. (2019) factorizes the final reconstruction into separate foreground and background renders, where only the foreground is rendered conditioned on the landmark posi- tions. The background will be inferred directly from the pose-perturbed input image with simple UNet (Ronneberger et al., 2015). We want our UNet to have a limited capacity for handling com- plex changes in pose. The remaining complex pose changes (e.g. limb motion, object rotations) will
2

---

Under review as a conference paper at ICLR 2020
Apply littering (x)
Composite
My G
Mg) @
Gauasian
pose
MMj
*The Pose Encoder run twice
depp
Sample future frame Tiama
1st Layer Feature
maps
Appearance Ercoder
Foreground Decoder
5/9
Pose Enceder
Ri
pase temp
MaskNet
Misong
Mej
Manu
whadamard
M
BGkltt
Figure 1: This figure depicts an overview of our training pipeline on video data. Given an image frame z, we produce Tey(2) (Ir) and Teemp()), which are appearance and pose perturbed variants of respectively. The model learns combine the appearance information from Tempp (I), and combine with pose from the Tey(x) (x) in order to reconstruct foreground object from the foreground decoder. Foreground masks predicted part of the pipeline separate the foreground rendering from the background rendering. Specifically, the background is rendered from a UNet that learns extract clean backgrounds from Tremp(2). This allows the learned pose representation focus on the more dynamic foreground object. The pose encoder and MaskNet are each depicted twice as they are applied twice during the forward pass.
then be captured by the more flexible landmark representations. During training, this factorization guided by translating foreground/staric background assumption, though we demonstrate that landmark quality is improved even when this assumption is held weakly.
3.1 MODEL COMPONENTS
Our full pipeline comprises five components: the pose and appearance encoders, foreground de- coder, background reconstruction subnet, and foreground mask subnet.
The goal of the pose encoder aross Enee (z) is to take input image and output set of unsupervised part activation maps. Critically, we want these part activation maps be invariant to changes in local appearance, as well as be consistent across deformations. A heatmap that acti- vates on person's 's right hand should be invariant across varying skin tones and lighting conditions, well track the right hand's 's location across varying deformations and translations.
The appearance encoder &aPP = Enc (x; Ens extracts local appearance information, conditioned the cose-encoderr' activation maps. Given input image x, the pose encoder will first provide K x x W part activation maps pposs To extract local appearance vectors, the appearance encoder projects the image C H x W appearance feature map Mopp, We compute the appearance vector for the kth pose activation map as:
H
W
app
ΣD
poose
caj for co 11.C.
giving us K C-dimensional appearance vectors. Here, each activation map in ppose is softmax- normalized.
The method pipeline attempts to reconstruct the original imput image by combining the pose infor- mation from the K activation maps with the pooled appearance vectors for each of the K parts. As in Lorenz et al. (2019). we fit 2D Gaussian to each activation of the K activation maps by estimat- ing their respective means and either estimating using a pre-determined covariance. Each part is represented by &poose (mg, E.). where b4 R and Σ, x The 2D Gaussian approxima- tion forces each part activation map into a unimodal representation with a simple parameterization, thereby enforcing that each landmark appears in most one location per image.
3

---

Under review as a conference paper at ICLR 2020
The foreground decoder (FGDec) and background reconstruction subnet (BGNet) networks that attempt to reconstruct the foreground and background respectively. Our foreground decoder is based the architecture proposed in SPADE (Park et al. 2019). In SPADE, semantic maps used to predict spatially-aware affine transformation parameters for normalization schemes such as InstanceNorm. Herein, we project the 2D Gaussian parameters from Grose to a heatmap of the target output width and height use as semantic maps in the SPADE architecture. Following [orenz al. 2019), we use the formula:
s (k, I+ =z (1-
(2)
where sk, () is the heatmap value for part map k at coordinate location 1. In addition to feeding s(k, () as semantic map to SPADE, individual appearence vectors are also projected onto their respective heatmap to create localized appearance encoding to be fed into the decoder. Please see section 3.4 of Lorenz al. (2019) for details on this projection.
Unlike the foreground decoder, which is conditioned on bottlenecked pose-appearance represen- tation, the BGNet given direct access to image data, albeit the pose-perturbed variant of the input. Given static background video sequence, we assume it is easier for the BGNet to learn to directly copy background pixels (and remove the foreground when necessary) than is for the pose-appearance factorization to learn model the background. In the absence of BGNet-likk module, several landmarks will be allocated to capture the "pose" the background, despite being ill-suited for such task.
The final module is the foreground mask subnet (MaskNet), which infers the blending mask to com- posite the foreground and background renders. It can be interpreted foreground segmentation mask and conditioned on 4rose.
3.2 TRAINING PIPELINE
All network modules are jointly trained in a fully self superviperi fashion, using the final image reconstruction task as guidance. We follow the training method as detailed in Lorenz et al. (2019), with the addition of our proposed factorized rendering pipeline in the reconstruction phase. An illustration of this pipeline is depicted in Fig. !
Training involves reconstructing an image from its appearance and pose perturbed variants, learning to extract the un-perturbed element from each variant. As with Lorenz et al. (2019), we use color jittering to construct the ereearance-perturbbe temporally sample a frame to 60 timesteps apart from the same scene to attain the pose-perturbed variant T (hemp(2) (I). However, in the absence of video data, use thin-plate-spline warping to perturb pose Txps(()). In general, our method is able to work with both Tzemp(x) and Teps (z). though TPS- Iarrwirping has the downside of also warping the background pixels, making the task of BGNet more difficult. Let Epose be the moussian-heatnan fitted to the raw activation map ppose and let O represent element-wise multiplication. Our training procedure can be expressed as follows:
variant Tzy(z). When training from video data,
grose Eose (Te((z)) and poose Pose (Tremp (x:)
(3)
cj
happ- EPC (IPP(TTeem) (Ttemp(z); Ense (Tremp(z)))
(4)
Mej = MaskNet (Ppooe) and Mtemp = MaskNet ( temp)
(5)
5/9 9 = FGDec pose TapP) and 1b = BGNet((1 - -Mtemp) O Ttemp (x)),
(6)
ã Moj O 8/s + May) о bbg
where the goal is to minimize the reconstruction loss between the original input æ and the recon- struction F. As can be seen, neither the shape encoder nor the appearance encoder are ever given direct access to the original image x. The pose information feeding into the foreground decoder FGDec(.,) -) is based on the color-jittered input image, where only the local appearance information is perturbed. The appearance information is captured from lzemp(z) (or (x)), where the pose information is perturbed. Notice the shape encoder is also executed on both the pose-perturbed and color- jittere input images. This is necessary to map the localized appearance information for a particular landmark from its location in the pose perturbed image to its unaltered position in (z))
4

---

Under review as a conference paper at ICLR 2020
Table 1: Evaluation of landmark accuracy on Human3. 6M la and BBC Pose is normalized by image dimensions. For BBC Pose, we report the percentage of annotated keypoints predicted within a 6-pixel radius of the ground truth.
Human3.60 3.6M error
Human3.61
Error
BBC Pose
Acc.
supervised
Newell et al.
(2016)
2.16
supervised
Charles et al.
(2013)
79.9%
unsup.
Thewlis
(2017b)
7.51
Plister et al.
2015)
88.0%
Zhang et al.
20188)
4.91
unsup.
Jakab ct al.
(0118)
68.4%
Lorenz et (0199)
2.79
Lorenz et al.
(2019)
74.5%
Baseline (temp)
3.07
Baseline (temp)
73.3%
Baseline (temp.tps)
2.86
Baseline (temp. tps)
73.4%
Ours
2.73
Ours
78.8%
(a)
(b)
Finally, the predicted foreground- masks are computed for both the appearance and pose perturbed variants: Maj and Mtemp respectively. May should have a foreground mask correspond- ing to the original foreground's pose, and is used to blend the foreground and background renders in the final step. Maemp is the foreground mask for the pose-perturhed input image, and assists the BGNet in removing foreground information from its background render. Refer to Appendix A for architecture, loss, and training parameters.
EXPERIMENTS
Here, we analyze the effect of introducing foreground-backgrouns separation into an unsupervised- landmark pipeline. Through empirical analysis, we demonstrate that the learned landmarks less used for capturing background information, thereby improving overall landmark quality. Landmark quality is evaluated by using linear regression to map the unsupervised landmarks to annotated key- points, with the assumption that well-placed, spatially consistent landmarks lead low regression error. Finally, we include an additional application of our method in the video prediction task, demonstrating how the factorized rendering pipeline improves the overall rendered result.
4.1 DATASETS
We evaluate our method on Human3.66 6.6 (Tonescu et al.) 2013). BBC Pose (Charles et al.) 2013), 2004). Human3.66 6. is video dataset that
CelebA (Liu al., 2015), and KTH (Schuldt features human activities recorded with stationary cameras from multiple viewpoints. BBC Pose dataset contains video sequences featuring 9 unique sign language interpreters. Individual frames are annotated with keypoint annotations for the signer. While most of the motion is from the hand gestures of the signers, the background features a constantly changing display that makes clean back- ground separation more difficult. CelebA is an image-only dataset that features deypoint-annotatee celebrity faces. As with prior works, we separate out the smaller MAFL subset of the dataset, train our landmark representation on the remaining CelebA training set, and perform the annotated re- gression task on the MAFL subset. The KTH dataset comprises videos people performing one of six actions (walking, running, jogging, boxing, handwaving, hand-clapping). We use KTH for our video prediction application. Additional preprocessing details are given Appendix A
4.2 UNSUPERVISED LANDMARK EVALUATION
As with prior works JJkab et al. 2018 Thewlis et al., 2017a). we fit linear regressor (without intercept) to our learned landmark locations from our pose representation to supervised keypoint coordinates. Following Jakab et al. (2018). we create loose crop around the foreground object using the provided keypoint annotations, and evaluate our landmark learning method within said crop. Importantly, most prior methods have not released their evaluation code for all datasets, thus we were not able to control for cropping parameters and coordinate space. The former affects the relative size and aspect ratio of the foreground object to the input frame, whereas the latter affects the regression results in the absence of a bias term. As such, external comparisons on this task should be interpreted rough comparison best, and that the reader focus the comparison against our internal bascline, which is our rough implementation of Lorenz et al. (2019). We include our cropping details in Appendix
5

---

Under review as a conference paper at ICLR 2020
86
w/in foreground
10a
84
Accuracy
82
10
80
0urs16
Baseline 16
78
10a
Ours
76
Ours
Baseline 12
74
Baseline (temp)
10-1
Ours8
Baseline (temp.tps)
Baseline8
10
20
30
40
2
4
12
14
16
Number of learned keypoints
landmark id
(a)
(b)
Figure 2: Landmark analysis experiments. 2a plots the BBC validation dataset keypoint accuracy versus number of Iearned keypoints. By factorizing out the background rendering. we are able to achieve better todmmar--t--nnotatten mappings with fewer landmarks than the baseline. 2b plots the percentage of the per-landmark normalized activation maps contained within the provided foreground segmentation masks on Human3.61 sorted in ascending order. We compare model against our baseline at 8, 12, and 16 learned landmarks. We see that the least-contained landmarks in the proposed approach are significantly more contained than those of the baseline.
We report our regression accuracies on Human3.6n 6M, BBC, and CelebA/MAFL, with the first two being video-based datasets and the last being image only. Results are shown in Tables Ta and 2 respectively. For the video datasets, we found it best to use only Teemp(a) (x)) to sample perturbed poses from future frames during training. Only Tips(z) was possible for CelebA/MAFL. Our pri- mary baseline is our model without the explicit goeeroond-baackroow line, we report results using -ney (Baseline (temp)) as well as both Tremp and Tips (Baseline (temp.tps)). In all cases, demonstrate that including factorized foreground foreground-buckkrround render- ing improves landmark quality compared to the controlled baseline model. We also believe our performance is competitive if not state-of-the-at based
separation. For this base-
best-attempt at matching cropping and regression protocols for external comparisons. The results on CelebA demonstrate that our method works even given very weak static background assumptions. This is because Tips (æ) in- discriminately warps the entire image, creating a pose-perturbed variant with a heavily deformed background. Further discussion in Appendix B
Next, we analyze how factorizing out the background ren- dering influences landmark quality. In Fig. 2a we present an ablation study where we measure the regression-to- annotation accuracy against the number of learned land- marks. Compared to our baseline models, we can see that the lakkgrooudd-fctooizzaatii
Table 2:
Landmark evaluation on MAFL using 10 landmarks. Predic- tion eиror is scaled by inter-ocular dis- tance. While we can only use Tips to sample pose perturbations video dataset, we still see strong im- provements over the baseline. We also ablate the use of the masks M in Ours (No Mask), where no masks are pre- dicted and the predicted z and 4h are directly elementwise-added.
allows us to achieve better accuracy with fewer landmarks, and that the degradation less steep. Further, in Table 2 we include a No Mask baseline which is proposed model but sans predicted blending masks. Here, we combine foreground and back- ground directly with: improves over the unfactorized baseline, though the full pipeline still performs best.
this non-
3 2b9 This variant also
MAFL
Error
One of our primary claims is that by factorizing fore- ground and background rendering in the training pipeline, we allow the landmarks to focus on modeling the pose and appearance of the foreground objects, leaving the background rendering task to a less expressive, but cas- ier to learn mechanism. We attempt to validate this claim on the Human3.66 6.MM dataset, they provide foreground- background segmentation masks. If the landmarks truly focus more modeling the foreground more, then underlying activation heatmaps for each unsu- pervised landmark should be more contained within the provided segmentation masks in the factor- ized case. In Fig. 2b] compare the percentage of the normalized activation maps contained within the provided segmentation masks against our baseline model for 8, 12, and 16 landmark models. For each learned landmark, first compute its average activation mass contained within the foreground
Thewlis et al. (2017b)
6.32
Zhang et al. (20T8a)
3.46
Lorenz et al. (2019
3.24
Jakab et al. (0118)
3.19
Baseline (ips)
4.34
Ours (No Mask)
2.88
Ours
2.76

---

Under review as a conference paper at ICLR 2020
Figure 3: Qualitative results of our landmark prediction pipeline. From top to bottom, we show our regressed annotated keypoint predictions, our predicted foreground mask, and the underlying land- mark activation heatmaps. Datasets are BBC Pose, Human3.6 3.6M, and CelebA/MAFL respectively.
segmentations. We then the landmarks in ascending order of containment (horizontal axis of Fig. 2b) and plot the models'
curve.
The results in Fig. demoonstte that the foreground- boreground-backgrount factorization noticeably improves the least containment of the least-contained landmarks. Note that the lowest containment percent- ages for the baseline are 0.05, 0.4, and 0.3, whereas the factorized containment percentages an order of magnitude larger at 8.2, 4.8, and 2.5 for 16, 12, and 8 landmarks respectively. It is safe say that the least-contained landmarks for the baseline model are nearly completely utilized for modeling the image background (99%+ of the activation mass is on the background). While the proposed factorization does not climinate the problem, we believe this difference is a contributing factor to the improvements over our baseline.
We show qualitative results of our regressed annotated keypoint predictions, well as landmark activation and foreground mask visualizations in Fig. 3 From top to bottom, we show our regressed annotated keypoint predictions, our predicted foreground mask, and the underlying landmark activa- tion heatmaps. Datasets are BBC Pose, Human3.66 6.6,, and CelebA/MAFL, respectively. Notice that the degree of binarization in the predicted mask is indicative of the strength of the static background assumption the data. ftuman3.66M features strongly static background, whereas BBC Pose has a constantly updating display on the left, and CelebA was trained with Tips which indiscriminately warps both foreground and background. Nevertheless, our method still shows improves over the baseline despite imperfectly binarized foreground-backgroundd separation.
4.3 APPLICATION TO VIDEO PREDICTION
Lorenz et al. 2019) applied their model video-to-video style transfer on videos of BBC signers, indicating that the rendered images from the landmark model are temporally stable. One of the issues with these renders. however, is that the landmarks suited for modeling the background, resulting in low-fidelity rendered backgrounds. We demonstrate that our factorized formulation resolves this issue
We evaluate our rendering on the video prediction task on the KTH dataset, and compare against external methods. The unsupervised landmark model factorizes image data into pose (landmarks parameterized as 2D Gaussians) and appearance information. We assume the appearance informa- tion remains constant throughout each video sequence, and use LSTM predict how the 2D Gaussians move through time conditioned on an initial set of seed-frames. Refer to Appendix D for implementation details. We show our qualitative and quantitative results Fig. and 5 and respectively. We report SSIM. PSNR, and the percoptual-featur based LPIPS (Zhang et al., 2018a) metric. Note that the background-factorrizel approach significantly outperforms the unfactorized baseline on all performance metrics, indicating better background reconstruction, as the foreground a comparatively smaller portion of the frame. Our method is also competitive with state-of-ther -ftthe-rttttt models such Lee et al. (2018). In Fig. 4 we show our rendered foreground, mask, rendered back- ground, and the corresponding composition. Our method assumes a fixed background for the entire sequence, but predicts& a foreground and blending mask for each extrapolated timestep. Both our baseline and proposed method maintain better structural integrity than other methods. However, due to the imperfect binarization of the predicted mask, the foreground in the composite image may
7

---

Under review as a conference paper at ICLR 2020
t=3
[=8
t=11
[= t=14
t= 17
t=20
[=23
[=26
t=29
[=32
[=35
t=38
f
1
1
A
DRNET
A
SAVP Det
SAVP
Baseline
X
Foreground
Mask
Ours
Figure 4: Qualitative results KTH action test dataset comparing method prior work. Our baseline produces a sharp foreground, but the background does not match that the initial frames. Our proposed factorized rendering significantly improves the background fidelity. The bottom three rows shows our factorized outputs. From top to bottom, we have the rendered foreground, the predicted blending mask, and the rendered background (first image on bottom row) followed by the composite output.
1-LPIPS KTH
SSIM KTH
PSNR KTH
0.95
0.95
Mean I-LPIPS
Mean SSIM
0.90
Mean PSNR
0.90
28
0.85
0.85
26
0.80
24
0.80
0.75
20
0.75
0.70
15
20
30
35
10
15
20
Frame Number
Frame
Number
Frame Number
Ours
Bascline
"StVPddctemiiiisticcc (Lec et al., 2018)
SAVP al., 2018)
DRNET DDentonnet-al,2220
Figure 5: We base our main evaluation to LPIPS score which closely correlates with human percep- tion. We also provide SSIM and PSNR metrics for completeness. Our method is competitive with sate-off-the-ar methods on KTH, and shows a large improvement over our controlled baseline.
appear somewhat faded compared that of other methods. Improved binarization of the predicted masks remains a topic of future work.
CONCLUSION
We propose and study the effects of explicitly factorized foreground and background rendering on eccnstruction-guidde unsupervised landmark learning. Our experiments demonstrate that by using UNet to learn a simpler copy mechanism to copy roughly static background pixels, the model do better job of allocating landmarks to the foreground objects of interest. As such, we are able to achieve more accurate regressions to annotated keypoints with fewer landmarks, thereby reducing memory requirements. We also demonstrate applications of our pipeline to unsupervised-landman based video manipulation tasks. For future work, we are interested in finding ways to improve binarization of the predicted foreground masks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag, Synthesizing images of humans in unseen poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2340-8348, 2018.
J. Charles, T. Pfister, D. Magee, D. Hogg, and A. Zisserman. Domain adaptation for upper body pose tracking in signed TV broadcasts. In British Machine Vision Conference, 2013.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In Proceedings of the 35th International Conference on Machine Learning. 2018.
Emily Denton et al. Unsupervised learning of disentangled representations from video. Ad- vances in neural information processing systems, pp. 4414-4423, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): 1325- 1139,, 2013.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in Neural Information Processing Systems, 2018.
Angjoo Kanazawa, David W. Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching for single- view reconstruction. In The IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2016.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 2018.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Dominik Lorenz, Leonard Bereska, Timo Milbich, and Björn Ommer. Unsupervised part-based disentangling of object shape and appearance. In CVPR, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference on computer vision, pp. 483-499, Springer, 2016.
Taesung Park, Ming-Yr Yu Liu, Ting-Chun Wang, and Jun-Ya Zhu. Semantic image synthesis with apatially-adaptive normalization. In CVPR, 2019.
Tomas Pfister, James Charles, and Andrew Zisserman. Flowing convnets for human pose estimation in videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1913- 2015.
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3d human pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 750-767, 2018.
Helge Rhodin, Victor Constantin, Isinsu Katircioglu, Mathieu Salzmann, and Pascal Fua. Neural scene decomposition for multi-person motion capture. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7703-7713, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015.

---

Under review as a conference paper at ICLR 2020
C Schuldt, Laptev, and B Caputo. Recognizing human actions: local sym approach. In Proceed- ings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pp. 32 36. IEEE, 2004.
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In Advances in Neural Information Processing Systems, pp. 2059-2070, 2018.
Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In International Conference Computer Vision (ICCV), 2017a.
James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling. In Advances in Neural Information Processing Systems, pp. 844-855, 2017b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018a.
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised discovery of object landmarks as structural representations. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
10

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
The overall architecture consists of sub-networks as in: pose and appearance encod- ing networks, foreground mask subnet, background reconstruction subnet, and a foreground image decoder. We use the U-net architectures (Ronneberger et al. 2015) for the pose encoder, appearance encoder, foreground mask subnet and background reconstruction subnet, complete with skip connec- tions. The pose encoder has 4 blocks of convolutional dowsampling modules. Each convolutional downsampling module has a convolution
Architecture:
Insr--nstance Normalization- ReNLU and a downsampling layer. At each block, the number of filters doubles, starting from 64. The upsampling portion of the pose encoder has blocks of convolutional upsampling modules, and the number of channels is halved at every block starting from 512. The appearance encoder network has one convolutional downsampling module and one convolutional upsampling module. The foreground mask subnet has 3 blocks of convolutional dowsampling module and 3 blocks of upsampling module, and the number of channels is 32 at each module. Similarly, the background reconstruction subnet has 3 blocks of convolutional dowsampling module and blocks of upsampling module. At each block, the number of filters doubles starting from 32.
The image decoder has 4 convolution- Meetluttoonn--U---L ReLU-upsample modules. We first downsample the appear- ance featuremap by a factor of 8 in each spatial dimension. Number of output channels of each convolution-Red Seriuutioonne--uuuppe R-ppaaapagg module in the image decoder are 256, 256, 128, 64, and 3 respec- tively. We apply spectral normalization (Miyato et al. (018)) to each convolutional layer.
Loss Function and Optimization Parameters: network with VGG Perceptual loss which uses the pre-trained VGG19 model provided by the Py- Torch library. We apply the MSE loss on outputs of layers relu relu12. _2,, re re_2, re relu3_2, and relu4_2, weighted by
We train our the image faccorization-reconssttiiioooooi
32 16 g ,,and 1 respectively. We use Adam optimizer, learning rate of 1e and weight decay of 5e The network is trained on 8 GPUs with batch size of 16 images per GPU.
For BBC Pose, we first roughly crop around each signer by using the given keypoints. Specifically,
Dataset Preprocessing:
find the center of the keypoints and crop 300x300 around the center and resize the crops to 128x 128. For the Human3.6M dataset, we follow the procedure defined by IZhang al. (2018b) for training/validation splits. We find the center of the keypoints and crop 300x300 around the center and again resize the crops to 128 x 128. For the CelebA/MAFL. dataset, we follow Jakab al. (2018) by resizing the images 160x 160, and center crop by 128x 128.
Color jittering (Toz (2)) performed with torchvisionn t transformss co (0.4, 0.4, 0.4, 0)3) the input image
The thin plate spline transformation Tips(1)) allows us to perform a non-rigid warping of the image content based on applying perturbations a grid of control points in the image's coordinate space. This was implemented with cv2. SreateThinng more expressive than a standard affine transformation on the image, allowing us to deform the image content in more interesting ways, and thus a reasonable drop-in replacement for Ttemp(z) when temporal data is not available.
former () It is
Regression coordinates For BBC Pose and Human3.6M, we define the origin coordinate as the center of the image before regression. is unclear what other methods used for these datasets. For CelebA/MAFL, we follow Jakab et al. (2018) and set the origin at the top left corner.
Dataset-Speciffe Model parameters We use 30 landmarks of fitted covariances for the BBC Pose dataset, meaning we estimate the covariance from the part activation maps when fitting the Gaussians compute &pose For Human3.61 3.6M and CelehA, fixed diagonal covariance of 0.08. In general, fixed diagonal covariances lead to better performance on the landmark regression task than fitted covariance, though fitted covariances lead better image generation results. As such, we use fitted covariances for the video prediction task.
16 and 10 landmarks respectively with
B STATIC IMAGE REFACTORING DURING TRAINING
In Fig. 6 we show our model outputs from training CelebA. We that the pipeline has de- termined the center of the face to be foreground, with everything else as background. Importantly, the fourth column from the left shows that the BGNet is capable of memorizing how to rectify
11

---

Under review as a conference paper at ICLR 2020
BBC Pose
Accuracy
Center
Top left
Baseline temp
73.30
73.33
Baseline (temp, tps)
73.40
72.73
Ours
78.78
79.16
Table 3: BBC Pose evaluation with the origin of the coordinate space set at different locations. We reported center origin in the main paper. Note that the results differ despite coming from the same model output. Nevertheless, the improvement from our proposed method is still clear and the fluctuation in accuracy remains small.
chin-plate-splingg warped image. While this is case of overfitting, it is arguably the reason our ap- proach is able to perform weak. A fiitennni based approach for background subtraction would be feasible only on video sequence data with perfectly still backgrounds.
separation when the static background assumption
Figure 6: From left to right input image, color jittered image, thin-plate-spline warped image, constructed background, predicted foreground, mask, and reconstructed output.
C ADDITIONAL QUALITATIVE LANDMARK PREDICTION RESULTS
Here, we show additional results for the pose-regression task on various datasets. The regression quality is generally very accurate, though notice that, as with other unsupervised landmark ap- proaches, we cannot model keypoint visibility easily, nor can we distinguish between front and back facing subjects on the Human3. 6M dataset.
Figure 7: Additional qualitative results for keypoint predictions for BBC Pose, Human3.6M, 3.6M, and CelebA/MAFL respectively.
12

---

Under review as a conference paper at ICLR 2020
D VIDEO PREDICTION WITH UNSUPERVISED LANDMARKS
As can be seen in Fig. [ the rendering component of our pipeline is conditioned directly on an appearance encoding 4APP and pose encoding $rose, Conveniently, the pose representation is stored of 2D Gaussian distributions. which can can easily manipulate spatially by translating their means and updating their covariances. As such, we apply our model to the video prediction task by assuming hopp to constant throughout the prediction sequence, and using an LSTM to update Grose, conditioned on the first frames. We extract papp from the N frame in the input sequence.
Care must be taken to maintain positive definite covariance matri- during prediction. Thus decomposition of the covariance matrices as the prediction targets. In practice, the LSTM may produce an estimate for L which is not a valid Cholesky factor, but even this case will still produce a valid covariance matrix when LL" is computed.
the parameters of the Cholesky
(Aps.Al))
LSTM
An illustration of our LSTM setup is depicted in Fig g In addition the Cholesky parameterization. we also found that predicting the residual during extrapolation, rather than the state directly, was im- portant robust long term predictions. In practice. this helped im- prove both training performance and generalization. At each time- step, the LSTM takes a concatenation the previous state and state residual to predict the next state residual.
0..00.5.00
(
Σ-LL
dft Agt + AB
(u.. >
Figure The LSTM predicts perturbations to the Gaussian means and Cholesky parame- ters of the covariances.
Our LSTM comprises LSTM layers and a final linear layer. Each LSTM layer has 256 channels. For the KTH results, we trained our landmark model with 40 landmarks. We also use GAN loss term improve image quality. Due to the KTH dataset being grayscale, we had to predict coding all the foreground details. The LSTM was trained predict 10 future frames from an input sequence of 10.
masks at half resolution to prevent the masks from en-
13

---

Under review as a conference paper at ICLR 2020
E ADDITIONAL VIDEO PREDICTION RESULTS
t=3
t=11
t=
14
t=17
t=20
1=23
t=26
[=29
t=32
t=35
t=38
5
I
DRNET
SAVP Det
SAVP
R
Baseline
Foreground
A
A
Mask
Ours
Figure 9: Qualitative results on KTH action test dataset comparing our method to prior work. SAVP and -Ave-deteriinistic methods produce blurry foreground images. On the other hand, the base- line method produces sharp foreground but the background does not match the initial frames. Our method maintains sharpness and high fidelity to the background. We show the foreground image constructions and masks that are used to produce output images. the last row, first column shows the reconstructed background image.
F ADDITIONAL VIDEO PREDICTION EXPERIMENT ON BAIR PUSH DATASET
We additionally video prediction experiments the BAIR action-free dataset Finn et al. 2016). This dataset consists of videos with robot arms moving randomly with diverse set of objects on a table. The videos have spatial resolution of 64. For this dataset, trained with 10 input future setup (never conditioning on its own output during training). We show our qualitative and quantitative results the BAIR dataset in Fig. 10 11 and respectively.
video prediction LSTM
14

---

Under review as a conference paper at ICLR 2020
1=4
1=8
t=12
t=16
t=20
1=24
t=28
t=30
GT
SVGLP
SAVP Det
SAVP
Baseline
Foreground
Mask
Ours
Figure 10: Qualitative results the BAIR dataset comparing our method to prior work. Methods are conditioned on initial frames to predict the next 28. SVGLP, SAVP and SAVP-deterministic methods produce blurry outputs in the previously occluded regions. Our method maintains sharpness and high fidelity to the background. We show the foreground image reconstructions and masks that are used to produce output images. In the last row, first column shows the reconstructed background image.
1-LPIPS on BAIR
SSIM on BAIR
PSNR on BAIR
Mean I-LPIPS
0.96
0.90
24
Mean SSIM
Mean PSNR
42
0.94
0.85
200
0.92
0.80
0.90
0.75
10
1's
20
25
15
Frame Num ber
Framd Number
Frame Number
Ours
-Baseline
P. P-determimistic (Lee al., 2018)
SAVP (Lee et al., 2018)
SVGLP (Denton & Fergus, 2018)
We base main evaluation to LPIPS score Zhang et al. 2018a) which closely cor- relates with human perception. We also provide SSIM and PSNR metrics for completeness. Our implementation achieves better LPIPS score than the competing methods. Importantly, the factor- ized method significantly outperforms our baseline by large margin on all metrics.
Figure 11:
15


Reviews:
[{"review_id": "SkeSv4c_FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors propose to design an unsupervised learning framework, which can capture pose representation by reconstructing images or videos. \n\nThe paper is not written well. Too much components in the design make this work hard to follow. The novelty is also relatively limited, since a large part of this work has been done in Lorenz et al. (2019). Moreover, the only supervision is the reconstruction loss. I am just curious how the neural network can learn the semantics representation (such as foreground and background), without any guidance in the corresponding modules? \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJxhp7OpKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an unsupervised method to get disentanglement of pose, appearance, background from both images domain and video domain. 5 sub-network are used to model pose, appearance, foreground, background, and decoders.  Their methods let the network focus more on the foreground to regress the landmark and improve state-of-the-art performance on landmark regression (unsupervised.), video prediction and image reconstruction. \nHowever, there are still lots of details missing for the training of the whole network even with the supplementary.  \n1. what are the details of the color jitter process? how do you know it is foreground and only colorizing this part?\n2. why the video prediction only on KTH. H36M is also a video-based dataset.\n3. what is the thin-plate-spline warped image?\n4. how do you generate T_temp(x) for image-based dataset?\n5. are the learned landmark all unimodal? as for 2d pose estimation, even we give them unimodal gt, sometimes the prediction is bimodal.\n6. how do you make the covariance as learned?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlyQwWpYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an unsupervised approach for learning landmarks in images or videos with single objects by separating the representation of the image into foreground and background and factorizing the representation of the foreground into pose and appearance. It builds upon previous work [Jakab 2018, Lorenz 2019] who proposed to train by reconstructing the original image from one version of the image with perturbed appearance and another with perturbed pose. It extends this approach by introducing an additional separation of foreground and background in the image. \n\nStrengths:\n+ Nicely motivates the approach of separating foreground and background\n+ Fewer landmarks are needed than in previous work\n+ Approach seems beneficial for video prediction\n+ Clear and well written\n+ Detailed description of architecture and training\n \nWeaknesses:\n- The changes and improvements feel somewhat incremental\n- Some uncertainty about the solidity of the evaluation/comparability with baselines\nResults on CelebA somewhat weak\n\nOverall the paper is well written, easy to follow, presents a straightforward extension of previous work and appears to show an improvement. I’m thus generally supportive of the paper.\n\nOne question I’d like to see addressed in the response is about the evaluation: as you state, the details about cropping are not known for previous work, introducing some uncertainty into the external comparisons. It would be great if you could provide some more details about the steps you took to verify that your internal baseline is indeed comparable to previous work (e.g. Lorenz 2019). For instance, on CelebA your baseline seems to fall short of Lorenz 2019, which may suggest that your substantial looking improvement on BBC Pose is indeed due to more favorable cropping.\n\nThe second concern is about the results on CelebA presented in Fig. 6 in the Appendix: It looks like the background net reconstructs almost the entire image. I would have expected that hair style, shape of ears or existence of a beard would equally warrant landmarks and appearance. It seems a bit odd that the foreground is so focused on the central part of the face. Do you have an explanation for that?\n\nMinor comments:\n- Test in Fig. 1 too small and not readable when printed\n- Fig. 1 seems to be missing an arrow pointing from image to appearance encoder\n- Why are there more than 8 points in Fig. 2 for “Ours8” (and more than 12 for “Ours12”)?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for unsupervised learning of keypoint landmarks from images and videos by decomposing them into the foreground and static background. The technical approach builds upon related prior works such as Lorenz et al. 2019 and Jakab et al. 2018 by extending them with foreground/background separation. The proposed method works well for static background achieving strong pose prediction results. The weaknesses of the paper are that (1) the proposed method is a fairly reasonable but incremental extension of existing techniques; (2) it relies on a strong assumption on the property of static backgrounds; (3) video prediction results are of limited significance and scope. In particular, the proposed method may work for simple data like KTH but is very limited for modeling videos as it is not well-suited to handle moving backgrounds, interactions between objects (e.g., robot arm in the foreground and objects in the background), and stochasticity. 
Decision: Reject