Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The following work proposes a formulation of the unsupervised latent-landmark learning autoencoder where the landmark bottleneck makes no assumptions about the number of instances of a keypoint to appear in each image -- an important distinction as prior approaches assume each landmark to appear exactly once per image. As with prior methods, two encoders are used, one for pose-invariant feature information, and the other for feature-invariant pose information. Output from the two encoders is combined and jointly decoded to reconstruct the original input. Pose invariance is achieved by feeding one encoder a thin-plate-spline warped image. Landmark activation locations are represented with predicted heatmaps. Unlike previous methods, these heatmaps are not assumed to have a single peak. Rather, the top-K activations are identified via NMS and retained for reconstruction. An online K-means loss is used to cluster feature representations at landmark locations where the resultant centroids are the resultant landmark descriptor.
strengths_and_weaknesses: Strengths
I think this work attempts to tackle an important issue, which is the applicability of unsupervised landmark methods. Due to the 1-instance-per-image assumption, these methods tend to be impractical when there are unknown number of instances or out-of-plane rotations.

Weaknesses
- Unless I've misunderstood a key component, the proposed method appears to be closely related to the autoencoders used in quantization literature, most notably VQVAE (van den Oord, 2017) and gumbel-softmax approaches. There's already a lot of extant literature based on top of that, learning discrete visual tokenizations of images that are critically neglected from this study. At the very least, these should serve as baselines for the clustering approach used in this work.
- There are some validity issues regarding the landmark-to-keypoint evaluation. 
    - The first issue is directed towards the validity of the metric itself, and not specific to this work. Prior methods using this evaluation perform a linear regression without an intercept to map landmarks to keypoints. The lack of the intercept makes it such that the regression result is highly sensitive to the positioning of the origin in the coordinate space. On centered objects, placing the origin in the middle of the image will often result in better linear mappings than should the origin be placed in any of the 4 corners of the image.
    - The linear mapping formulation isn't really applicable to the landmark formulation in this method, as the authors noted in the text. It's not clear whether or not the linear layer used by this work includes the bias term. Furthermore, I'm not sure whether the tensor representation implies an ordering to the multiple instances of the same landmark, or this has been handled by the authors in their specific formulation. 
   - In general, I don't think we can conclude anything from the comparison in table 3, though I understand the necessity of attempting the comparison regardless. Assuming my understanding is correct, this should probably be noted in the paper.
questions: - It would be great if the authors could clarify for me what is meant by the intuition behind the Wasserstein loss (that the distributions of features and prototypes should match).
- Please address my concerns regarding the potential relationship to vector quantization literature.
limitations: limitations appropriately addressed
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a new method for learning task-agnostic keypoints in an unsupervised way. The proposed method encodes semantics into the keypoints by reconstructing the original image from sparse keypoints descriptors. A group of prototypes is learned during training and the descriptors are constrained to be around the prototypes. The proposed method can predict keypoints for multiple instances and achieves comparable results with the state of the art. 
strengths_and_weaknesses: Strengths:
1. The task of unsupervised keypoint localization for multiple instances is interesting and may facilitate future research.
2. The proposed method is technically sound and achieves comparable results with the state of the art. 
3. The visualization of the prototype on the MNIST-Hard dataset is interesting. It is better to show the visualization of face/body keypoints too. 

Weakness:
1. Multiple instances supporting is an important part of the proposed method, however, only a single instance quantitative evaluation is shown for some complex tasks (face/ body keypoint localization). The effectiveness of the proposed method needs to be further proved in such kinds of tasks. 
2. The reconstruction loss L_recon seems to be mandatory in the proposed method and it has been performed for all the experiments.  But this loss may affect the robustness of the method on datasets with complex backgrounds which is more common in the multi-instance dataset. The authors may need to add more experiments on such kinds of datasets, e.g. the COCO dataset, The WIDER FACE dataset. 
3. The details of how to learn the prototype is not clear in the submission. It is better to show how to update the prototype based on the sliced Wasserstein loss. 
 4. The different settings of prototype number, i.e. M, need to be carefully analyzed. 
questions: 1. In Table 4, what will happen if the L_recon or L_sw is removed from the entire loss?
2. In Table4, the localization results seem to change slightly with different settings of losses. Is it because the localization is too easy on the MNIST-Hard dataset. 
3. Some typos need to be refined, e.g. 
L. 67, 'priori'->'prior'
In the supplementary L. 532, P_i represents both point position and prototype. 
In the supplementary L. 537, should 'descriptors' be 'prototype'?
limitations: The authors have addressed the limitations in the submission.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper describes a method for learning keypoint detectors in an unsupervised manner. The key extension over related work is that the proposed method does not build on the assumption that only one target keypoint is present in an image. Technically this is achieved by having one keypoint heatmap/response map in the architecture that is shared among all keypoints, whereas related work estimates one heatmap per keypoint. Other technical strategies that are used to encourage equivariance and to clustering of the feature descriptors have also been used in prior work, but are combined in this work to enable fully unsupervised learning of keypoints detectors. The experimental evaluation shows that the proposed method performs comparably to related work on rather artificial datasets and simple datasets of faces and humans, while not building on the single object assumption.
strengths_and_weaknesses: Strengths:
+ The paper addresses an important problem, in that it aims to resolve the major issue that so far methods for unsupervised keypoint discovery assume that only one object is present in an image. While this is a valid assumption when learning from video streams (where objects can be segmented fairly well based on their movement), it is a very limiting factor when learning from unordered sets of images. I have myself been concerned about this issue for some time, and I appreciate that this paper makes progress towards resolving this limitation.
+ The paper is very well written and describes the key concepts very intuitively. I have enjoyed reading the paper and could understand the key concepts already after the first reading pass.

Weaknesses:
- Key concepts presented and combined in this paper are very well known. While using a single heatmap to represent the activation of all keypoints to resolve the assumption about keypoint numbers is relevant, the contrastive learning of feature representations, clustering of feature representations, and equivariance losses are well known and widely applied.
- The experimental evaluation is limited as it is conducted on very simple datasets. I understand that this is meant to be a proof-of-concept evaluation, but given the limited variability of the object shapes, appearance and background it remains unclear if much more simple method  simple clustering of features [C,E] would suffice. I highly recommend testing the proposed method on datasets of real-world images with more complex variations in shape and appearance  such as PASCAL3D+ or VehiclePart [D] or even KITTI.
- Several related works that for unsupervised learning of landmark and part representations are not mentioned [A-C] and should be compared to.
- The claim that the proposed method can model occlusion (e.g. l330) is not quantified and only shown qualitatively on images with very minor occlusion. I would recommend to tone down this claim as it seems unjustified.

[A] Unsupervised learning of object landmarks by factorized spatial embeddings.

[B] Unsupervised learning of object frames by dense equivariant image labelling.

[C] Visual concepts and compositional voting

[D] Detecting semantic parts on partially occluded objects

[E] Learning deep parsimonious representations

=====================POST REBUTTAL=========================
After reading the other reviews and the rebuttal I vote for accepting the paper. My concerns have been addressed by the additional results and clarifications provided in the rebuttal. I also think that the concerns of the other reviewers were addressed sufficiently. Therefore I raise my initial score to 7.
questions: I would like the authors to comment on the weaknesses that I described. I will carefully reconsider my rating depending the authors response especially concerning the lack of novelty and simplicity of the datasets in the experiments, as well as the discussion with the other reviewers.
limitations: The limitation section addresses the core limitations adequately and concisely.
review_title: null

Meta-review: metareview: The meta reviewer has carefully read the paper, reviews, rebuttals, and discussions. The authors did a good job in rebuttal. The additional results and clarifications addressed the reviewers' concerns.  The manuscript crosses the acceptance bar. The authors are still suggested to revise the paper considering the reviewers' comments.
confidence: Certain
Decision: Accept