Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ** Paper Summary **

This paper presents semi-supervised keypoint localization networks and loss functions to overcome the need for the labeled keypoint data for that task. It simultaneously generates keypoint heatmaps and pose invariant keypoint representations, where these representations were separately used to enforce translation equivariance, and translation invariance, and semantic consistency, respectively. The proposed method attains the improvement on several benchmarks for human and animal body landmark localization.

** Paper Strength **

+ Learning keypoints with a minimal supervision is an essential step for numerous applications, thus solving such a problem is important.
+ Using two kinds of representations, i.e., heatmap and keypoint feature, makes sense, enabling to apply different loss functions that achieve different aspects, e.g., translation equivariance and translation invariance, and semantic consistency.
+ Compared to other previous methods to train the keypoint estimation networks in a self-supervised manner, e.g., Thewlis et al., 2019, the proposed methods incorporates the semantic consistency loss that makes the networks achieve a semantic awareness. 
+ But, results on several benchmarks were clearly state-of-the-arts. 

** Paper Weakness **

- Even though this paper was tailored to semi-supervision learning, unsupervised losses themself contribute the performance gains solely. So, I'm really interested in what happens the networks are trained only with unsupervised losses, i.e., SC and TC. It would be interesting because some datasets might not have any ground truth keypoints. In Table 2 and 3, the authors tried to provide an ablation study relevant to this, but the case of percentage of labeled images as 0 would be interesting. 
- Conceptually, the transformation equivariance loss in (2) is similar to equivariant landmark transformation loss proposed by Honari et al. (2018) and recent other variants. The authors argue the proposed loss in (2) are different with them in that the previous methods leverage an inverse transformation. But, the methodological improvement is marginal and incremental.
- Transformation invariance loss in (4) is also interesting, but the loss itself might not contribute the performance gain. For example, if z and z' converge to 0, the loss is going to be minimum, but the networks are not trained well. 
- In addition, because the paper solves the semi-supervised learning, the weights of loss components are important hyper-parameters, but there lack the ablation study for those.
review_title: Review for "semi-supervised keypoint localization"

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**

The paper presents an approach to keypoint localization (to retrieve people/animals pose) combining labeled and unlabeled data. Features are extracted and concatenated into a single descriptor per keypoints, by multiplying feature maps and heatmaps and max-pooling over the spatial domain, and used for semantic classification. Images are transformed with simple perspective augmentations. The non-supervised part comes in enforcing that keypoint representations for unlabeled images remain close.

**Pros**

* Very simple formulation.
* Thorough evaluation on four datasets with strong results.
* Informative ablation tests.

**Cons**

* Relatively light on contributions, as the paper is quite simple.

**Details**

Please cite Adam (mentioned but not cited).

The baselines/datasets seem sufficient, but I might have missed relevant works (I do not work in this field).

"Dialated": dilated

---

P.S. I received an automated email complaining that my review was too short. The paper is nice, straightforward, and easy to follow. The evaluation seems thorough. I did not find any mistakes. I don't have too much to say about it. My review is positive and, I hope, constructive.
review_title: SEMI-SUPERVISED KEYPOINT LOCALIZATION

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper presents an interesting method for semi-supervised keypoint localization, that jointly learns the keypoint heatmaps and pose-invariant keypoint representations. The model is trained semi-supervised by applying transformation consistency and semantic consistency constraints. The proposed method is evaluated on several benchmarks and it significantly outperforms other semi- & un- supervised methods.

##################################################################

Reasons for score: 

The paper is mostly clear and well-motivated. The authors develop a novel approach to tackle the problem of semi-supervised keypoint localization. The reviewer appreciates the novelty and ingenuity of this approach. It achieves the state-of-the-art performance, compared to other semi-supervised methods. The proposed method is easy-to-implement and can be added to any existing keypoint localization networks. Also the extra keypoint classification branch is only used for training and can be discarded during inference. 

There is still room for greatly improving the experiment section. Hopefully the authors can address the concerns in the rebuttal period. One major problem is the missing comparisons with pseudo-labeling baselines (Dong & Yang, 2019; Radosavovic et al., 2018). Another limitation of this work is the relatively narrow scope. The paper only focuses on one application of semi-supervised keypoint localization. However, I believe such techniques may also applicable to other tasks. 

##################################################################

Pros:

1.The proposed method is very interesting and novel. The authors propose to add a keypoint classification branch, and design several well-motivated consistency losses. 
2.The paper is clear and well-written.
3.The proposed method is easily added to any existing keypoint localization networks, which means many previous works can be jointly trained in a much larger unlabeled dataset and it may enhance the robustness and performance of the models.

##################################################################

Cons:

1. One major problem is the missing comparisons with pseudo-labeling baselines (Dong & Yang, 2019; Radosavovic et al., 2018). The reviewer believes that the comparisons are critical in showing the effectiveness of the proposed method. 
2. The authors said “where there is a high risk of transferring inaccurate pseudo-labeled examples to the retraining stage that is harmful for the model.”However, no evidence is provided in the experiment sections.
3. Although the proposed method provides several ablation studies, the reviewer suggests to consider adding the following experiments to enhance the quality of the paper:

(1)	When percentage of labeled images is 100%, which is totally supervised, the proposed method already outperforms the baseline. Is this method also applicable to improving the performance of fully-supervised cases? Please analyze why. 

(2) Ablation study of only adding the keypoint classification branch (without the consistency losses).

(3) Lacking of ablation study about TC in Table 2. The effectivess of transformation equivariance and transformation invariance should be considered separately.


##################################################################

Questions during rebuttal period:

1.In Table3, it reads“Our method outperforms the supervised score only by a small margin with 10% of unlabelled data.” It would be more convenient, if the supervised scores are also listed in the Table. 

Some typos:

(1) 4.1 ATRW: where l is is the head → where l is the head
(2) Table 2： Usupervised losses → Unsupervised losses

review_title: Official Blind Review1

final_score: rating: 6: Marginally above acceptance threshold
review_content: It can be applied to point heatmaps based network by adding a semantic representation learn by a three loss terms: one supervised and two semi-supervised. The proposed architecture combines a Keypoint Localization Network with a Keypoint Classification Network. Experiments are achieved on four public datasets. 

The main contribution of the paper is the model and losses proposed to train, in a semi-supervised way, the network. 

Contributions are clearly stated and validated. 
The idea of using intermediate features to produce a map that will select keypoint features with an element-wise product with the heatmap is good. Its looks like a kind of attention module. Additional information should be provide to explain the differences: this is mandatory
The transformation consistency constraints are also good ideas and the modified transformation equivariance is a smart trick. 
Experiments have been achieved in order to compare the proposed semi-supervised model with other semi-supervised models (2 are selected). Results are good. 
Moreover, an ablation study is proposed. It reports the effect of each unsupervised loss. It should be interesting to add the score of the model without unsupervised loss. It will, for example show if for 100% sample trained, adding unsupervised loss improves the method. A visualisation ok keypoint embedding using the tSNE technic is also proposed. I'am not sure that this figure gives usefull information for the study if not compared to the one obtained by only supervised training. 

The implementation details points that "unsupervised loss may hurt the learning at the beginning" and proposes to use ground truth heatmaps. This is not really an implementation detail to my point of view are more information and experiments about this trick should be given. 

review_title: This paper proposes a model for landmark/keypoint localization trained in a semi-supervised way

Meta-review: This paper received three borderline reviews (2+ / 1-) and one positive review.  Having read through the reviews and author responses, the AC recommends the paper to be accepted.  The method, while simple, is proven experimentally to be effectively and will add to the body of work on key-point localization.   The authors are requested to add their additional baselines in the response text to the revision of their paper if it has not already been done.

Decision: Accept (Poster)