Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a simple yet effective framework for self-supervised learning of human skeletons and object outlines. An encoder is firstly used to regress the key points in a heatmap manner. Then a shared edge weight graph is generated by the keypoint predictions. An edge map is drawn based on the keypoint locations and edge weights. To get rid of the structure information in the image, the authors also randomly masked the original image as an appearance material to recover the input from edge maps. The model is then trained by the perceptual reconstruction loss. Experiments on human skeletons and other object outline datasets show promising results (robust key points and edge predictions).
strengths_and_weaknesses: Strengths:
1) This paper is well-written. The idea of this paper is novel and interesting.
2) This paper successfully built a simple yet effective self-supervised learning framework.  Different from previous self-supervised landmark prediction networks which normally enforce the key points to follow the same transformation as images, this paper constructed a shared graph and individual edge weights to reconstruct the image. The differentiable weights were inspired by the previous work on drawing and sketching, and it's very interesting to introduce them into this area for skeleton predictions.
3) To avoid the graph bottleneck from modelling appearance, this paper additionally feeds the encoder with a marked input to enrich the required appearance for reconstructing images.
4) The experimental results of this paper are impressive. By using linear regression to regress predicted facial key points to ground truths, the model achieved a very promising NRMSE result. On other additional datasets, the model is also able to predict very satisfying skeletons and key points, which cater to human perceptions. In the provided videos, the predictions are also stable in continuous frames. The predicted edge weight, which is shared in a specific task, can also well model the dependency of different key points.
5) Interesting ablation study on the number of key points and edge thickness was given.

Weaknesses:

1) The training process of this paper should be described in more detail in the main paper. For example, do you use a concentration constraint as in [115] to help learn heatmaps? Otherwise, the keypoint detector seems to be slow to converge.
2) Some figures in the paper are blurry when zoomed in (especially for the skeletons), which should be improved.
3) The authors argued that texture doesn't matter as predictions for zebra are not affected by the stripes. However, this can be because the zebra is jointly trained with horses. What if the model is only trained with objects with significant textures?
questions: As in the Weaknesses.
limitations: Yes. The method has limitations when the background has structures or the object is occluded.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a new self-supervised landmark estimation technique from image collections without any GT annotations. In contrast the existing works the discovers spatially independent landmarks, this work proposes to reason about connections/edges between the landmarks (skeletons). The core of the technique is auto-encoder with the estimated edge maps and masked images as bottleneck layers forcing the network to disentangle structure information into the learned keypoints and edges. Experiments on humans and face datasets demonstrate better results compared to existing landmark and part discovery works. Visual results demonstrate generalization of the technique to other object types (although the quality of results is not clear from the visuals on these datasets).
strengths_and_weaknesses: Strengths:
- A simple yet effective technique for landmark discovery.
- The use of simple masking strategy to bottleneck the structure information in an image instead of learning a separate appearance encoder.
- The use differentiable edge estimation to represent structure with keypoint edges instead of just keypoints.
- State-of-the-art results on face and human datasets.

Weaknesses:
- Much of the results are shown on 2D-like object images with relatively constrained viewpoint variations. There are no quantitative results on more 3D like object datasets such as Pascal-VOC where there are 3D objects such as cars, animals from different viewpoints. For instance, existing works (e.g. [33]) report results on pascal-voc parts dataset. Does the proposed approach fail on such datasets with more 3D viewpoint variations? Paper claims the technique works well on all types of objects, but metrics are limited to only certain type of 2D objects.
- Quantitative metrics are only shown on humans and face datasets. It is difficult to assess the quality of the discovered skeletons on other object datasets (such as zebras, hands) just from visuals. Also, the visuals in Figure-1 only shows horses and zebras from very similar viewpoints.

questions: Experiments to further strengthen the paper:
- Did authors try using unsupervised saliency to make the network focus more on foreground regions? Does the improve the results?
- If we assume a generic object skeleton is given (i.e., edge weights are given), can the proposed approach learn to attach the given skeleton to the right locations in the images?

Suggestions:
- It is better to give an intuitive explanation of edge computation (eq.3) so that authors need to look into [66] to understand this part o the paper.
- Font sizes is too small in figures-4 and 5. Increase it for better readability.

Minor corrections:
- Eq.1: Missing opening bracket in the denominator.
- Eq.3: Missing 'equal' sign.
- line 264: 'a masked images' -> 'masked images'.
limitations: Several limitations are discussed in the paper. It is better to also add a discussion on how much camera viewpoint variations does the proposed approach can handle across the image collection.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to learn 2D keypoints from images, where the keypoints are linked by straight edges, from images of the same object class. The method is trained by reconstructing the original input image using masked input images and the learned edge map. The edge weights are learned and are shared across all instances in the dataset. Quantitative evaluations focus on datasets with human poses such as Human3.6M and CelebA, and qualitative results are on datasets such as Zebra, Horses, and Flowers. Results show that the proposed edge modeling is important for learning keypoints across datasets.
strengths_and_weaknesses: Strengths
- To the best of my knowledge, the proposed method is new for learning keypoints in a self-supervised way with learned edge weights across keypoints.

- The evaluation is on a range of datasets, with fairly extensive ablation studies of the method with a range of hyperparameters. 

- The presentation of the paper is clear, with lots of visualizations of the method, including discussions of failure cases and limitations.

Weaknesses
- The paper mentions that the training required a collection of images of the same object class (+ the graph edge weights are shared). Could the authors further discuss this requirement (perhaps in limitations in the main paper, if the class requirement is strict)? How similar do the objects need to be - would this method work on image datasets like the CUB dataset? 

- The keypoints visually seem to cover object edges a lot of the time - I'm not sure if the keypoints are covering meaningful locations of the image (ex: in the provided supplementary videos & especially for flowers in Figure 6 of the appendix). Maybe this is intentional by the authors - could this be clarified? Could the authors also discuss relation of this work to works on unsupervised foreground segmentation, since it looks like the edges are covering a lot of the foreground (ex: [A])?

- The authors mention that their approach is simpler than alternatives (lines 13, 265) - however, there are simpler approaches for keypoint discovery that perform comparably on Human3.6M (ex: [39] [B] - both have fewer components than this model and doesn't need edges - they both need video though). Does the author mean that their method is simpler for image-based keypoint discovery? Also I think that Table 2 could be updated with [B] which is a self-supervised method that achieves 2.53 $\pm$ 0.056 on Human3.6M. 

[A] Savarese et al., Information-Theoretic Segmentation by Inpainting Error Maximization, CVPR 2021

[B] Sun et al., Self-Supervised Keypoint Discovery in Behavioral Videos, CVPR 2022
questions: I listed some questions above in weaknesses.

Some more questions:
- Could the authors further clarify "structured background"? Is it when the dataset have all the same background, or is it when the background have straight / patterned edges?

- What is the effect of using the masked image for appearance vs. extracting an appearance feature from the image as in [38]?
limitations: Yes the authors included limitations and potential negative societal impacts.
review_title: null

Meta-review: metareview: Building from works on unsupervised keypoint discovery for a domain of 2D images, this work proposes to jointly learn a skeletal structure that links discovered keypoints, and further proposes a novel image masking strategy for extracting limited background information, to force the keypoints to capture maximum information about the scene. The evaluations span a variety of datasets, with quantitative numbers on human face and body pose datasets, and show improvements from the proposed approach. A novel idea, executed well, and of interest to many at NeurIPS. Congratulations to the authors, and please fix visualization issues etc. before camera-ready / next revision. 

confidence: Certain
Decision: Accept