Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents an unsupervised method to get disentanglement of pose, appearance, background from both images domain and video domain. 5 sub-network are used to model pose, appearance, foreground, background, and decoders.  Their methods let the network focus more on the foreground to regress the landmark and improve state-of-the-art performance on landmark regression (unsupervised.), video prediction and image reconstruction. 
However, there are still lots of details missing for the training of the whole network even with the supplementary.  
1. what are the details of the color jitter process? how do you know it is foreground and only colorizing this part?
2. why the video prediction only on KTH. H36M is also a video-based dataset.
3. what is the thin-plate-spline warped image?
4. how do you generate T_temp(x) for image-based dataset?
5. are the learned landmark all unimodal? as for 2d pose estimation, even we give them unimodal gt, sometimes the prediction is bimodal.
6. how do you make the covariance as learned?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper presents an unsupervised approach for learning landmarks in images or videos with single objects by separating the representation of the image into foreground and background and factorizing the representation of the foreground into pose and appearance. It builds upon previous work [Jakab 2018, Lorenz 2019] who proposed to train by reconstructing the original image from one version of the image with perturbed appearance and another with perturbed pose. It extends this approach by introducing an additional separation of foreground and background in the image. 

Strengths:
+ Nicely motivates the approach of separating foreground and background
+ Fewer landmarks are needed than in previous work
+ Approach seems beneficial for video prediction
+ Clear and well written
+ Detailed description of architecture and training
 
Weaknesses:
- The changes and improvements feel somewhat incremental
- Some uncertainty about the solidity of the evaluation/comparability with baselines
Results on CelebA somewhat weak

Overall the paper is well written, easy to follow, presents a straightforward extension of previous work and appears to show an improvement. I’m thus generally supportive of the paper.

One question I’d like to see addressed in the response is about the evaluation: as you state, the details about cropping are not known for previous work, introducing some uncertainty into the external comparisons. It would be great if you could provide some more details about the steps you took to verify that your internal baseline is indeed comparable to previous work (e.g. Lorenz 2019). For instance, on CelebA your baseline seems to fall short of Lorenz 2019, which may suggest that your substantial looking improvement on BBC Pose is indeed due to more favorable cropping.

The second concern is about the results on CelebA presented in Fig. 6 in the Appendix: It looks like the background net reconstructs almost the entire image. I would have expected that hair style, shape of ears or existence of a beard would equally warrant landmarks and appearance. It seems a bit odd that the foreground is so focused on the central part of the face. Do you have an explanation for that?

Minor comments:
- Test in Fig. 1 too small and not readable when printed
- Fig. 1 seems to be missing an arrow pointing from image to appearance encoder
- Why are there more than 8 points in Fig. 2 for “Ours8” (and more than 12 for “Ours12”)?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, authors propose to design an unsupervised learning framework, which can capture pose representation by reconstructing images or videos. 

The paper is not written well. Too much components in the design make this work hard to follow. The novelty is also relatively limited, since a large part of this work has been done in Lorenz et al. (2019). Moreover, the only supervision is the reconstruction loss. I am just curious how the neural network can learn the semantics representation (such as foreground and background), without any guidance in the corresponding modules? 

review_title: Official Blind Review #1

Meta-review: The paper proposes an approach for unsupervised learning of keypoint landmarks from images and videos by decomposing them into the foreground and static background. The technical approach builds upon related prior works such as Lorenz et al. 2019 and Jakab et al. 2018 by extending them with foreground/background separation. The proposed method works well for static background achieving strong pose prediction results. The weaknesses of the paper are that (1) the proposed method is a fairly reasonable but incremental extension of existing techniques; (2) it relies on a strong assumption on the property of static backgrounds; (3) video prediction results are of limited significance and scope. In particular, the proposed method may work for simple data like KTH but is very limited for modeling videos as it is not well-suited to handle moving backgrounds, interactions between objects (e.g., robot arm in the foreground and objects in the background), and stochasticity. 
Decision: Reject