Semi-supervised Keypoint Localization
Authors: Olga Moskvyak
Frederic Maire
Feras Dayoub
Mahsa Baktashmotlagh
Venue: ICLR 2021 Poster (2021)
Keywords: semi-supervised learning
keypoint localization
limited data
unsupervised loss

Abstract: Knowledge about the locations of keypoints of an object in an image can assist in fine-grained classification and identification tasks, particularly for the case of objects that exhibit large variations in poses that greatly influence their visual appearance, such as wild animals. However, supervised training of a keypoint detection network requires annotating a large image dataset for each animal species, which is a labor-intensive task. To reduce the need for labeled data, we propose to learn simultaneously keypoint heatmaps and pose invariant keypoint representations in a semi-supervised manner using a small set of labeled images along with a larger set of unlabeled images. Keypoint representations are learnt with a semantic keypoint consistency constraint that forces the keypoint detection network to learn similar features for the same keypoint across the dataset. Pose invariance is achieved by making keypoint representations for the image and its augmented copies closer together in feature space. Our semi-supervised approach significantly outperforms previous methods on several benchmarks for human and animal body landmark localization.

Published as a conference paper at ICLR 2021
SEMI-SUPERVISE KEYPOINT LOCALIZATION
Olga Moskvyak, Frederic Maire, Feras Dayoub
School of Electrical Engineering and Robotics Queensland University of Technology, Australia
{olga moskvyak, f. mairee, feras dayoub edu. edu
Mahsa Baktashmotlagh
School of Information Technology and Electrical Engineering The University of Queensland, Australia www.akttaske wwwaaaasstttaaaiaaaaaa Laghéuqq edu. au
ABSTRACT
Knowledge about the locations of keypoints of object in image assist in fine- eme-ggained classification and identification tasks, particularly for the case of ob- jects that exhibit large variations in poses that greatly influence their visual appear- ance, such as wild animals. However, supervised training of keypoint detection network requires annotating a large image dataset for each animal species, which is labor-intensive task. To reduce the need for labeled data, we propose to learn simultaneously keypoint heatmaps and pose invariant keypoint representations a semi-supervised manner using small set of labeled images along with larger set of unlabeled images. Keypoint representations are learnt with a semantic keypoint consistency constraint that forces the keypoint detection network to learn similar features for the same keypoint across the dataset. Pose invariance is achieved by making keypoint representations for the image and its augmented copies closer to- gether in feature space. Our semi-supervised approach significantly outperforms previous methods on several benchmarks for human and animal body landmark localization.
INTRODUCTION
Detecting keypoints helps with fine-grained classification (Guo & Farrell 2019) and re- identificaiion (Zhu al. 2020 Sarfraz et 2018). In the domain of wild animals (Mathis al. 2018 Moskvyak
ata.] Liu et al. 201 2019ap by. annotating data is especially challenging due to large pose variations and the need for domain experts to annotate. Moreover, there less commercial interest in keypoint
estimation for animals compared to humans, and little effort is invested in collecting and annotating public datasets.
Unsupervised detection of landmarks Jakab et al. 2018 Thewlis et al. 2017 2019) can extract useful features, but are not able to detect perceptible landmarks without supervision. On the other hand, supervised learning has the risk of overfitting if trained only a limited number of labeled examples. Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data during training. It is mostly studied for classification task (van Engelen & Hoos) 2019) but it is also important for keypoint localization problem because amnotating multiple key- points per image a time-consuming manual work, for which precision is the most important factor. Pseudo-labeling (Lee) 2013) is common semi-supervised approach where unlabeled examples are assigned labels (called pseudo-labels) predicted by model trained on labeled subset. A heuristic unsupervised criterion is adopted to select the pseudo-labeled data for a retraining procedure. More recently, the works of (Dong & Yang 2019 Radosavovic et al. 2018) apply variations to selection criteria in pseudo-labeling for semi-supervised facial landmark detection. However, there are less variations in facial landmark positions than in human or animal body joints, where there is a high
We use terms keypoints or landmarks interchangeably in our work. These terms are more generic than hody joints (used in human pose estimation) hecause our method applicable variety of categories.

---

Published as a conference paper at ICLR 2021
Kcypoint localization network
Losses
Ground truth heatmaps for subset of the dataset
Image dataset
Supervised
Trans. equivariance
Trans. invariance
Semantic consistency
Figure Our semi-supervised keypoint localization system learns list of heatmaps and list of semantic keypoint representations for each image. In addition to supervised loss optimized on the labeled subset of the data, we propose several unsupervised constraints of transformation equivariance, transformation invariance, and semantic consistency.
risk of transferring inaccurate pseudo-labeled examples the retraining stage that is harmful for the model.
Previous work of (Honari al. 2018 in semi-supervised landmark detection utilizes additional class attributes and test only on datasets that provide these attribute annotations. Our work focuses on keypoint localization task in a common real-world scenario where annotations are provided for a small subset of data from a large unlabeled dataset. More specifically, we propose method for semi-supervised keypoint localization that learns list of heatmaps and list of semantic keypoint representations for each image (Figure [T. A semantic keypoint representation is a vector of real numbers in a low-dimensional space relative the image size, and the same keypoints in differ- ent images have similar representations. We leverage properties that are specific to the landmark localization problem to design constraints for jointly optimizing both representations.
We extend transformation consistency constraint of (Honari et al. 2018) to be able to apply it on each representation differently (i.c. transformation equivariant constraint for heatmaps and trans- formation invariant constraint for semantic representations). Moreover, consistency constraint that encourages detecting similar features across images for the same land- mark independent of the pose of the object (e.g. eye in all images should look similar). Learning both representations simultaneously allows us to use the power of both supervised and unsupervised learning.
formulate semantic
Our work is motivated by data scarcity in the domain of wild animals, but not limited to animals, and well, it is applicable to human body landmarks detection. The contribution of work is three-fold:
We propose a technique for semi-supervised keypoint localization that jointly learns key- point heatmaps and semantic representations optimised with supervised and unsupervised constraints;
Our method can be easily added any existing keypoint localization networks with no structural and with minimal computational overhead;
We evaluate the proposed method on annotated image datasets for both humans and an- imals. As demonstrated by our results, our method significantly outperforms previously proposed supervised and unsupervised methods on several benchmarks, using only limited labeled data.
The paper is organised follows. Related work semi-supervised learning and keypoint localiza- tion is reviewed Section[] Our proposed method is described in Section Experimental settings, datasets and results are discussed in Section!! 4
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Keypoint localization. Supervised keypoint localization research is driven by few large datasets with labeled keypoints that span across several common research domains including human pose es- timation (Andriluka et al. al.] (2144) and facial keypoints (Sagonas al. 2016). Challenges in obtaining keypoint annotations have led to the rise in unsupervised landmark localization research. Several unsupervised methods leverage the concept of equivariance which means that landmark coordinates stay consistent after synthetic transformations or in subsequent video frames. [hewlis et al. (2017) propose to learn viewpoint-indeeendent representations that are equivariant to different transforma- tions and Dong et al. 2018) exploit the coherence of optical flow source of supervision, Zhang ett al. (2018) learn landmark encodings by enforcing constraints that reflect the necessary properties for landmarks such separability and concentration. Dakab et al. (2018) propose generative ap- proach where the predicted heatmaps are used to reconstruct the input image from a transformed copy. Recent work (Thewlis et al. 2019) enforce the consistency between instances of the same ob- ject by exchanging descriptor vectors. These methods are mostly evaluated on faces of people that have less degrees of freedom during movements and transformations than human or animal body joints. We compare our method to the combination of supervised and aforementioned unsupervised methods in Section
Semi-supervised learning is the most studied for the classification task. Pseudo-labeling (Lee) 2013) is method that uses the model's class predictions as artificial labels for unlabeled examples and then trains the model to predict these labels. Another technique is a consistency regularization which states that realistic perturbations of input examples from unlabeled dataset should not signifi- cantly change the output of a neural network. Consistency regularization is used in 11-model (Laine) & Aila 2017) and further improved by Temporal Ensembling (Laine & Aila 2017) which maintains an exponential moving average prediction for each training example and Mean Teacher (Tarvainen) & Valpola 2017) that averages model weights instead of model predictions. Recent methods UDA
(Xie et al. 2019). ReMixMatch (Berthelot et al. (020) and FixMatch (Sohn et al. (2020) use com- bination of consistency loss, pseudo-labeling and advanced augmentation techniques in addition to color perturbations and spatial transformations. In this work, we investigate adjustments required to apply consistency loss to keypoint localization which we discuss in Section 3.2
Semi-supervised learning for keypoint localization. To the best of our knowledge, there are a few works in semi-supervised keypoint localization. Dong & Yang (2019) build on the pseudo-labeling technique and propose a teacher model and two students to generate more reliable pseudo- for unlabeled images. However, the method is evaluated on face landmarks and in cases with high variations of poses, there a high possibility of inaccurate aseudo-labels that cannot be filtered out and be harmful during the retraining stage. |Honari et al. (2018): : Ukita & Uematsu (2018) learn keypoints in semi-supervised manner but utilise extra annotations to guide landmark learning such as action labels (running, jumping) for juman joints emotion labels (smiling, yawning) for facial keypoint localization. Different from previous work our approach does not use any class labels and learns directly from unlabeled data with high pose variations.
SEMI-SUPERVISED LEARNING FOR KEYPOINT LOCALIZATION
In this work, we propose semi-supervised technique for keypoint localization that learns from an image set where ground truth annotations are provided only for a small subset of the dataset. The overall architecture consists of two components: a keypoint localization network (KIN) that out- puts keypoint heatmaps of the image, and keypoint classification network (KCN) that classifies keypoints given semantic keypoint representation as input. Our method does not pose any con- straints on the architecture of the KLN and it can be added to any existing keypoint localization network with minimal modifications.
We optimize heatmaps with the supervised loss and the transformation equivariance constraint. Si- multaneously, keypoint representations are optimized with transformation invariance and semantic consistency constraints (Figure [T]. We discuss each constraint and related components of the archi- tecture in the next sections.
3

---

Published as a conference paper at ICLR 2021
keypoint
Keypoint rxpresentation
KLN
h
Semantic keypoint class scores
GMP
KCN
Intermediate Returres
Ixl Conv
Figure 2:
Semantic consistency criteria. Keypoint representation is defined for each keypoint by multiplying a corresponding predicted heatmap h; with intermediate features F. Keypoint repre- sentations are classified with shared network Φ and the feedback is added to the total loss.
3.1 SEMANTIC KEYPOINT REPRESENTATIONS
Keypoint heatmaps are optimized to estimate locations of keypoints in the image. However, heatmaps do carry any information about semantic type of the keypoint (e.g, beak eye for a bird). In semi-supervised regime, the feedback provided by unlabeled examples are not as effective the ones coming from labeled examples. To extract useful information from unlaheled images, we propose learning semantic keypoint representation. In particular, keypoint localization network is encouraged to detect similar features for the same semantic keypoint across the dataset by incorporating the feedback from keypoint representation classifier in the objective function.
Motivation for our approach is that the same keypoints should activate the same feature maps. Let us consider KLN as a function f(x; 0) with an input image æ and trainable parameters O that outputs heatmaps = (2;; 0). We collect intermediate feature maps from KI.N, upscale them the spatial dimension of output heatmaps, concatenate by channels, and pass through convolutional layer with filters of size one (Figure 2). The resulting feature map F has the shape (C, H, W). Then, feature maps F' are element-win mask out activations corresponding to the detected keypoint. The output of this operation is K feature maps of size (C, H. W). Global Max Pooling (GMP) is applied over feature maps to keep the highest value for each channel. We call the produced vector 2; = GMP(F @ h2) for each keypoint i (1, K} semantic keypoint representation.
multiplied with each keypoint heatmap h, E {1, ..., K} seperately to
Finally, we pass keypoint representations to a simple KCN (d) which a fully connected network with input and output layer for classification with cross-entropy loss. The feedback from the cross-entropy loss makes up semantic consistency (SC) loss:
1
(x)
&i log(ó(z;)
where ÿ a vector of ground truth semantic labels for keypoints because the order keypoints in heatmap is fixed.
One advantage of our method is its efficiency as only adds a small number parameters to the network to address the task of keypoint representation classification. Specifically, KCN is small fully connected network shared between keypoints and it has less than a thousand of parameters depending the number of keypoints. Our approach is related attention modules (Vaswani al.) 2017 Hu et al. 2020) as our network has the ability to focus on subset of features using element- wise multiplication with heatmaps. However, our model this attention-based mechanism to leam additional keypoint representations from unlabeled data by optimizing set of unsupervised losses.
3.2 TRANSFORMATION CONSISTENCY CONSTRAINT
The difference between keypoint heatmaps and semantic keypoint representations is that the former transformation equivariant and the latter is transformation invariant. In other words, the output

---

Published as a conference paper at ICLR 2021
heatmaps should be consistent with viewpoint variations of the image while keypoint representa- tions should be preserved for all different transformations of the image. We call this property a transformation consistency constraint.
Transformation equivariance (TE) enforces a commutative property on the landmark localization and augmentation operations that include spatial transformation (e.g, rotations and translations), meaning that the order of applying these two operations does not matter. Let g(,, 8) be an augmen- tation function with augmentation parameters which are not trainable and sampled randomly each time. Transformation equivariance constraint is formulated as: f g(#) = 9 f(x). We measure transformation equivariance loss Lu over predicted heatmaps by squared Euclidean distance:
Le x 0) Eu (1) s); gf 0), s)]{
Note that, after applying transformation, some landmarks may go outside of the image boundary, and cause the visibility issue. This problem is alleviated in our formulation by applying the same transformation
image. This is different from equivariant landmark transformation (ELTT loss proposed by Honari et al. (2018) which computes an inverse transformation instead. In essence, inverse transformation cannot bring these landmarks back meaning that inverse transformation does not output the original image. Our approach avoids this issue.
Transformation invariance (TI) of keypoint representations enforced by pulling correspond- ing vectors for the image and its augmented copy closer together. First, we concatenate keypoint representations in one vector get a holistic representation & of the image æ:
z |1, 22, mx}
We apply a random spatial transformation to the input image to get image x', compute keypoint representations z 2/, z'x- and concatenate them get vector z'. Finally, we enforce pose invariance by penalizing distance between representations of original and transformed images and formulate transformation invariance loss Lii
Ca, x²) = Em,ar [| z²11²]
(4)
The overall objective is the weighted sum of losses:
L Cuup + XzCae + dbCLe + Xeha
(5)
where Loup is a supervised mean squared error between predicted and ground truth heatmaps for the labeled subset. Parameters A are defined experimentally.
EXPERIMENTS
4.1
DATASETS
We evaluate our method on two datasets with annotated human body joints and two datasets of wild animals.
MPII Human Pose dataset (Andriluka et al. 2014 is a collection of images showing people doing real-world activities with annotations for the full body. Due to the fact that test annotations are not released publicly, we use training and validation splits of MPII in our experiments. We use 10,000 images for training to speed up experiments as we run multiple training runs for each subset of labeled examples. Our validation and test sets consist of 3,311 and 2,958 images respectively. Annotations contain coordinates for 16 body joints with a visibility flag.
LSP (Leeds Sports Pose) (Johnson & Everingham 2010 2011) dataset is collection of annotated images with people doing sports such as athletics, badminton or soccer. Each image has been an- notated with 14 joint locations. We use 10,000 images from extended (Johnson & Everingham, 2011) version for training and 2,000 images from original (Johnson & Everingham 2010) dataset for testing and validation.
CUB-200-2011 (Welinder et al. 2010) is a dataset of 200 fine-grained classes of bird species. We split dataset into training, validation and testing with disjoint classes test classes does not appear
5

---

Published as a conference paper at ICLR 2021
during training. First 100 classes are used for training (5,864 images), 50 classes for validation (2,958 images) and the last 50 classes (2,966 images) for testing. Each image is annotated with 15 body keypoints such as beak, left eye and throat. We use class label only for splitting the dataset and do not use it anywhere in out method.
ATRW (Li et al., 2019) is a dataset of Amur tigers images captured in multiple wild zoos in un- constrained settings. Professionals annotated 15 skeleton keypoints for each tiger. We use 3,610 images for training, 516 for validation and 1,033 for testing with annotations provided by authors. This dataset is more challenging than birds as four-legged animals exhibit more pose variations.
Training set for each dataset is split into labeled and unlabeled subsets by randomly picking 5%, 10%. 20% or 50% of the training examples and discarding the labels for the rest of the data. The procedure is repeated three times so all experiments are run three times to obtain the mean and standard deviation of the results. Validation and test sets are fixed for all experiments. Validation set is used to tune hyperparameters and test set is used to report the final results. The order of the keypoints is explicitly defined in annotations and is fixed for the training and inference.
The evaluation metric is PCK (probability of correct keypoint) from (Yang & Ramanan 2013) where keypoint is considered correctly localized if it falls within al pixels of the ground truth position (ar a constant and the maximum side of the bounding box). The PCK (a = 0.1) score is reported for LSP, CUB-200-2011 200-2011 and ATRW datasets. For MPII we an adaptation (Andriluka cet al., 2014) which is PCKh (head-normalized probability of correct keypoint) where / is the head size that corresponds to 60% of the diagonal length of the ground truth head bounding box (provided in the MPII annotations).
4.2 IMPLEMENTATION DETAILS
Images are resized to the input size 256 x 256 and heatmaps are predicted at size 64 x 64, We adapt HRNet-32 (Sun et al.) 2019) architecture KLN because it is originally designed for key- point localization and retains features at high spatial dimension (c.g. 64 x 64 for the input of size 256 x 256). We collect intermediate features at the output of each multi-scale subnetwork, after catenation we get 352 channels and then apply 64 convolutional filters of size one. GMP results in representations of length 64 for each keypoint. We also experimented with collecting more features from different layers but it did not improve the performance. KCN is a fully connected network that accepts keypoint representation of size 64 and classifies keypoints based their semantic labels (from 10 to 17 depending on the dataset).
We use perspective transformations as an augmentation function g where parameters s of the trans- formation are sampled randomly using a method from (Moskvyak & Maire, (017)) to avoid extreme warping. We also experimented with simple affine transformations but perspective gave better re- sults most likely due to higher variability of transformations.
Unsupervised losses may hurt the learning at the beginning because output heatmaps and interme- diate feature maps are random during first epochs. A possible solution is to vary the contribution of unsupervised losses according to predefined strategy. To avoid tuning many hyperparameters, semi-supervised approach uses ground truth heatmaps in unsupervised losses for the labeled samples in a batch. This approach has only one hyperparameter. percentage of the labeled samples in batch. We found that there is enough feedback from labeled examples when the batch has 50% of labeled and 50% of unlabeled examples.
We adopt Adam Kingma& & Ba 2015) optimizer with learning rate 10-4 for all experiments. Models are trained until the accuracy on the validation set has stopped improving. The weights of loss components were determined experimentally (A1, A2, d3, ds) (10%, 0.5, 102, 102). We provide the sensitivity analysis in Section
4.3 RESULTS
Comparison with the supervised baseline. We train HRNet-32 (Sun et al. 2019) with the super- vised loss as a baseline from the official implementation the labeled subsets with 5%, 10%, 20%, 50% and 100% of the dataset. The baseline performance decreases significantly when the amount of training data is reduced on human poses and tigers datasets (Table T). On birds dataset, we observe

---

Published as a conference paper at ICLR 2021
Table I: PCK @0.1 for keypoint localization with different percentage of labeled images. We report mean and standard deviation from three runs for different randomly sampled labeled subsets. Pseudo-labeled (PL) baseline evaluated for 100% of labeled data because there is unlabeled data to generate pseudo-labels for.
Percentage of labeled images
Method
5%
10%
20%
50%
100%
Dataset 1: MР1I
HRNet (Sun et al. 2019
66.22.11.60
1.00
71883
75.73:10.35
81.11
PL Radosavvvic al. 2018
1.74
64.78=1.44
69.35=1.11
77.43=0.48
BLT (Hoan et al.
2018
68.27=0.64
71.03=0.46
72.37=0.58
77.75=00.1
81.01=0.15
Gen (Jakab et al. 2018)
71.59+1.12
0..6
74.95+0.32
79.86+0011
80.92+0.32
Ours
0.83
76.56
7..46
80.75+
82.12-0.14
Dataset 2: LSP
HR (Sun et al. 2019
40.19
1..1
55.22+
6..61
02.12
PL (Radosavovic 2018
2018
37.36 1.89
42.05+1.66
48.86 1.23
0.94
ELT Honari et al.
41.77+1.55
47.22+00.91
57.34+0.94
66.81+0.62
7..2
Gen (Jkab et al.
2018
1.40
67.75+1.00
68.80=0.91
69.7010.77
72.25
Ours
0.99
69.56-1.66
71.85
12.5910.56
74.29
Dataset 3: CUB-200-2011
HRNet (Sun et al. 2019
85.77=0.38
88.62=0014
90.18=0.22
92.60-10.28
93.62+0.13
PL Radosavovic al.
2018
86.31=0.45
89.51#0.32
90.88=0.28
92.78+0.27
ELT Honari et al.
2016 8
0.35
89.48+00225
90.86=10.13
92.26++0.06
93.77+0.18
Gen (Jakab et 2018)
88.37=0.40
90.38++.22
91.31+0.21
92.79++0.14
93.62+0.25
Ours
91.11
91.4710.36
92.36-10.00
92.80-10.24
93.8110.13
Dataset 4: ATRW
HRNet (Sun et al. 2019
0.87
77.55+00.84
86.41:0.45
92.17+0.18
94.44.0.100
PI.
Radosavovic al.
2018
67.97+1.07
7.72
84.69+0.57
92.15+0.24
ELT Honari al. 2018
1.25
80.3510.96
87.98=0.47
92.8010.21
94.7510.14
Gen @Jkab ci al.
2018
89.54
90.4811.49
91.1620.13
92.27=0.24
94.80+0.13
Ours
9.657
94.29+0.66
94.49-0.36
94.63-10.18
95.31=0.12
only small decrease in the baseline score (Table We explain it by the fact that there are more variations in poses of four-legged animals and human body joints than of birds. Supervised results on MPII are lower than the official ones because the training set is smaller and we do not include ad- ditional tricks during training (e.g. half body transforms) and testing (post-processing and averaging flipped images).
Our method significantly improves the baseline on all datasets (Table Our proposed unsupervised constraints are the most beneficial for low data regimes with 5%, 10% and 20% labeled images. For example, our method increases the score from 40% to 66% on LSP dataset with 5% of labeled data. On the challenging tigers dataset, our approach reaches the score of 92% trained with only 5% labeled examples when the supervised model shows the score 69% while trained on the same labeled data. Experiments show that the influence of additional unsupervised losses decreases when more labeled examples are added to the training. Experiments show that our method on 100% labeled data outperforms the supervised baseline by a small margin because by learning supplementary semantic keypoint representations with unsupervised losses the model learns to generalize better.
Comparison with the pseudo-labeled baseline. We apply pseudo-labeled (PL) method from Ra- dosavovic al. (2018) on our datasets (Table D We use the same model HRNet-32 as in all our experiments for a fair comparison. Overall, the pseudo-labeled baseline is inferior to our method on all datasets used in study. We explain it by the fact that Radosavovic al. (2018) trained on datasets that are by order magnitude larger than our data so models pretrained on the labeled subset are already good enough to generate reliable pseudo-labels.

---

Published as a conference paper at ICLR 2021
Table 2: Ablation study LSP. We isolate gains from each unsupervised loss evaluate the tribution of each constraint. SC semantic consistency TE- transformation equivariance and TI transformation invariance. TE is not evaluated separately because it does not optimize both repre- sentations. Results are reported on one run.
Percentage of labeled images
Unsupervised losses
5%
10%
20%
50%
100%
TE + + SC
66.32
69.09
71.62
72.19
74.44
TE+TI
46.76
55.18
64.01
67.54
72.11
SC TI+SC
64.74
67.43
69.65
70.61
72.85
65.23
68.11
70.12
71.28
73,56
TE + SC TI
65.78
68.51
70.56
71.77
73.89
43.62
53.74
61.12
65.32
71.80
Supervised baseline
39.16
44.36
54.23
61.73
71.91
Table 3: Influence of the amount of unlabeled data. We train our model with the fixed 5% of labeled data and vary the amount of unlabeled data. Results reported compared with the supervised baseline (SB) trained with the same labeled data.
run. The results
Percentage of unlabeled images
Dataset
10%
20%
50%
100%
SB
CUB-200-2011
87.01
88.33
89.44
91.34
85.33
ATRW
72.04
76.65
86.56
93.02
69.84
Comparison with related methods. We compare approach with previously proposed semi-
supervised and unsupervised methods for landmark detection (Table D The equivariant landmark transformation (ELT) loss from (Honari et al., 2018) forces model to predict equivariant landmarks with respect to transformations applied to an image. ELT loss gives a small improvement over the baseline model and is inferior our method all datasets. Jakab et al.] (2018) learn keypoints without supervision by encouraging the keypoints to capture the geometry of the object by learning generate the input image given its predicted keypoints and an augmented copy. For a fair compar- ison we inject the models from Jakab et al. (2018) into our training pipeline and add the supervised loss for the labeled examples in each batch. All other parameters are kept the same including aug- mentation, subsets of data and training schedule. We observe that the generation approach improves over ELT loss and the baseline however it is inferior to our method. The generation approach also introduces more parameters (in the reconstruction part of the network) than our approach that adds only a small keypoint classifier network.
Ablation study. We investigate the influence of different loss components of our methods on LSP dataset (Table 2). At first, we remove semantic consistency loss component (Eq, [ and observe the significant drop in the score especially in low labeled data regime. For example, with 5% of labeled data the score drops from 66% when trained with the combination TE + TI + SC to 46% for the combination TE + TI. When we return semantic consistency and remove transformation 4 the results are reduced slightly. The results of ablation study shows
consistency losses (Eq. 2 that the semantic consistency loss component is more influential than the transformation consistency. Both TE and TI losses contribute to the performance gain and their combination achieves better results than each loss separately. We argue that our TE loss is an improvement over ELT loss (Honari et al. 2018). We replaced our TE loss with an inverse transformation loss of Honari et al. (018) in our framework, and applied it on ATRW and CUB-200-201 1 datasets with 20% of labeled data. We observed that the score decreased by 1% on both datasets.
We also analyse the influence of the amount of unlabeled data in our method (Table 3}. We conduct experiments where the amount of labeled examples is fixed at 5% and the number of unlabeled examples is reduced to 50%, 20% and 10% of the number of original unlabeled samples. We observe that the goes down as the amount of unlabeled data is reduced. Our method outperforms the supervised score only by a small margin with 10% of unlabeled data. We conclude that the number of unlabeled examples plays important role in training with unsupervised losses.
We conduct an ablation study to get an insight on using ground truth heatmaps in unsupervised losses. Experiments the LSP dataset show decrease of 1-2% in the score for all cases when ground truth heatmaps are not used (Table The results prove the benefit of using the signal from available ground truth heatmaps.
8

---

Published as a conference paper at ICLR 2021
Table 4:
An ablation study of using ground truth heatmaps in unsupervised losses LSP dataset. Results are reported on one run.
Percentage of labeled images
Method
5%
10%
20%
50%
100%
With g/t heatmaps
66.32
69.09
71.62
72.19
74.44
Without g/t heatmaps
64.75
67.27
69.91
70.55
73.65
Sensitivity analysis of weight loss components. We fixed the weight A = 10 and tested weights:
A2 2 (0.1, 0.5, 1.). A3 (10*, 102, 10a) and X4 (10*, 100, 109). The ranges of weight values different due to differences in scales for mean squared error and cross-entropy loss. Experiments on LSP dataset show that our method is sensitive to variations of TE (Aş) and TI (A1) losses (Figure 3) The most notable drop in accuracy is observed when the weight of SC loss (A2) is reduced to 0.1 and the accuracy is the same level when X2 equals 0.5 and 1.0. We select the combination of (A2, λ3, A4) (0.5, 102, 10²) that achieves the highest score.
weight of SC 0.1
weight of SC 0.5
Accuracy, %
weight of SC 1.0
72
70
(10%, 1044)
(10%,, 1002)
(10%, 1013)
(1022, 1000)
(1922 1022)
(103, 10"1) 10"3)
(1013, 10
(1005, 10
(1922
weights of (TE, Ti)
Figure 3: Sensitivity analysis of the weights of loss components.
Analysis of keypoint representations. We analyze the learned keypoint representation with SNE (van der Maaten & Hinton 2008). The t-SNE algorithm maps high dimensional space (64 dimensions in our case) into a two dimensional while preserving the similarity between points. The t-SNE plot for the keypoint representations of L.SP test set (Figure shows that representations for the same keypoints are clustered together.
Right snide
Right elbow
Right
Right
Right hip
Left shoulder
Left knce
Left writ
Right wrist
Head top
Figure 4: iSNE visualization of keypoint embeddings for human body landmarks on LSP test set.
CONCLUSION
We presented new method for semi-supervised keypoint localization. We show that reliable key- points can be obtained with a limited number of labeled examples. This is achieved by learning semantic keypoint representations simultaneously with keypoint heatmaps using a set of unsuper- vised constraints tailored for the keypoint localization task. We applied our method to predict human body joints and animal body keypoints and demonstrated that it outperforms current supervised and unsupervised methods. Morcover, it reaches the same performance as the model trained the whole labeled dataset with only 10% of labeled images on tigers ATRW dataset and with 50% la- beled images on challenging human poses LSP dataset. We plan to investigate the applicability of our method to domain adaptation for keypoint localization the future work.

---

Published as a conference paper at ICLR 2021
REFERENCES
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state of the art analysis. In Proc. CVPR, 2014.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: supersupe learning with distribution matching and augmenta- tion anchoring. In Proc. ICLR, 2020.
Xuanyi Dong and Yezhou Yang. Teacher supervises students how to learn from partially labeled images for facial landmark detection. In Proc. ICCV, 2019.
Xuanyi Dong, Shoou-I Yu, Xinshuo Weng. Shih-En Wei, Yi Yang, and Yaser Sheikh. Supervision- by-reiistration: An unsupervised approach to improve the precision of facial landmark detectors. In Proc. CVPR, 2018.
Pei Guo and Ryan Farrell. Aligned to the object, not to the image: A unified pose-aligned represen- tation for fine- theeggraded recognition. In Proc. WACV, 2019.
Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Joseph Pal, and Jan Kautz. Improving landmark localization with semi-supervised learning. Proc. CVPR, 2018.
Jie Hu, L. Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and--ecctatton networks. 2020.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. Proc. NeurlPS, 2018.
Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In Proc. BMVC, 2010.
Sam Johnson and Mark Everingham. Learning effective human pose estimation from inaccurate annotation. Proc. CVPR, 2011.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.
S. Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proc. ICLR, 2017.
D. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In Proc. ICML Workshops, 2013.
Shuyuan Li, Jianguo Li, Weiyao Lin, and Hanlin Tang. Amur tiger re-identification in the wild. In Proc. ICCV Workshops, 2019.
Cen Liu, R. Zhang, and Lijun Guo. Part-pose guided amur tiger re-identification. In Proc. ICCV Workshops, 2019a
Nannan Liu, Q Zhao, Nan Zhang, Xinhua Cheng, and Jianing Zhu. Pose-guided complementary features learning for amur tiger re-identification. In Proc. ICCV Workshops, 2019b.
Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, V. Murthy, M. Mathis, and M. Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21:1281-1289, 2018.
O. Moskvyak and F. Maire. Learning geometric equivalence between patterns using embedding neural networks. In Proc. DICTA, 2017.
Olga Moskvyak, F. Maire, Feras Dayoub, and Mahsa Baktashmotlagh. Keypoint-aligned embed- dings for image retrieval and re-identification. arXiv preprint, arXiv:2008. 1 1368, 2020.
Ilija Radosavovic, P. Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation:
Towards omni-supervised learning. In Proc. CVPR, 2018
10

---

Published as a conference paper at ICLR 2021
Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: database and results. Image and Vision Computing, 47:3-18, 2016.
M. S. Sarfraz, Schumann, Eberle, and Stieflhhagen. pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking. In Proc. CVPR, 2018.
Kihyuk Sohn, David Berthelot, C. Li, Zizhao Zhang, N. Carlini, E. D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint, arXiv:2001.0777 100.62000.07685, 2020.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proc. CVPR, 2019.
Antti Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proc. NIPS, 2017.
James Thewlis, Hakan Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factor- ized spatial embeddings. In Proc. 2017.
James Thewlis, Samuel Albanie, Hakan Bilen, and A. Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In Proc. ICCV, 2019.
N. Ukita and Yusuke Uematsu. Semi- and weakly-supervised human pose estimation. Computer Vision Image Understanding, 170:67-78, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2603 2008.
Jesper E. van Engelen and H. Hoos. A survey on semi-supervised learning. Machine Learning, 109: 373 373-440, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NIPS, 2017.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report 200-00-2000-00,, California Institute of Technology, 2010.
Qizhe Xie, Zihang Dai, E. Hovy, Minh- Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmen- tation for consistency training. arXiv preprint, arXiv:1904.1 12848, 2019.
Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. IEEE Transactions Pattern Analysis and Machine Intelligence, 35:2878-2894 2013.
Y. Zhang. Yijie Guo, Jin, Yijun Luo, Zhiyuan He, and H. Lee. Unsupervised discovery of object landmarks as structural representations. In Proc. CVPR, 2018.
Zhihui Zhu, X. Jiang, Feng Zheng, Xiao wei Guo, Feiyue Huang, W. Zheng, and Xing Sun. Viewpoint-aware loss with angular regularization for person re-identification. In Proc. AAAI, 2020.
11


Reviews:
[{"review_id": "TYQ8n1FzYdT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe paper presents an approach to keypoint localization (to retrieve people/animals pose) combining labeled and unlabeled data. Features are extracted and concatenated into a single descriptor per keypoints, by multiplying feature maps and heatmaps and max-pooling over the spatial domain, and used for semantic classification. Images are transformed with simple perspective augmentations. The non-supervised part comes in enforcing that keypoint representations for unlabeled images remain close.\n\n**Pros**\n\n* Very simple formulation.\n* Thorough evaluation on four datasets with strong results.\n* Informative ablation tests.\n\n**Cons**\n\n* Relatively light on contributions, as the paper is quite simple.\n\n**Details**\n\nPlease cite Adam (mentioned but not cited).\n\nThe baselines/datasets seem sufficient, but I might have missed relevant works (I do not work in this field).\n\n\"Dialated\": dilated\n\n---\n\nP.S. I received an automated email complaining that my review was too short. The paper is nice, straightforward, and easy to follow. The evaluation seems thorough. I did not find any mistakes. I don't have too much to say about it. My review is positive and, I hope, constructive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LVzk2pcF13K", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It can be applied to point heatmaps based network by adding a semantic representation learn by a three loss terms: one supervised and two semi-supervised. The proposed architecture combines a Keypoint Localization Network with a Keypoint Classification Network. Experiments are achieved on four public datasets. \n\nThe main contribution of the paper is the model and losses proposed to train, in a semi-supervised way, the network. \n\nContributions are clearly stated and validated. \nThe idea of using intermediate features to produce a map that will select keypoint features with an element-wise product with the heatmap is good. Its looks like a kind of attention module. Additional information should be provide to explain the differences: this is mandatory\nThe transformation consistency constraints are also good ideas and the modified transformation equivariance is a smart trick. \nExperiments have been achieved in order to compare the proposed semi-supervised model with other semi-supervised models (2 are selected). Results are good. \nMoreover, an ablation study is proposed. It reports the effect of each unsupervised loss. It should be interesting to add the score of the model without unsupervised loss. It will, for example show if for 100% sample trained, adding unsupervised loss improves the method. A visualisation ok keypoint embedding using the tSNE technic is also proposed. I'am not sure that this figure gives usefull information for the study if not compared to the one obtained by only supervised training. \n\nThe implementation details points that \"unsupervised loss may hurt the learning at the beginning\" and proposes to use ground truth heatmaps. This is not really an implementation detail to my point of view are more information and experiments about this trick should be given. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4etN2AwuCbf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper presents an interesting method for semi-supervised keypoint localization, that jointly learns the keypoint heatmaps and pose-invariant keypoint representations. The model is trained semi-supervised by applying transformation consistency and semantic consistency constraints. The proposed method is evaluated on several benchmarks and it significantly outperforms other semi- & un- supervised methods.\n\n##################################################################\n\nReasons for score: \n\nThe paper is mostly clear and well-motivated. The authors develop a novel approach to tackle the problem of semi-supervised keypoint localization. The reviewer appreciates the novelty and ingenuity of this approach. It achieves the state-of-the-art performance, compared to other semi-supervised methods. The proposed method is easy-to-implement and can be added to any existing keypoint localization networks. Also the extra keypoint classification branch is only used for training and can be discarded during inference. \n\nThere is still room for greatly improving the experiment section. Hopefully the authors can address the concerns in the rebuttal period. One major problem is the missing comparisons with pseudo-labeling baselines (Dong & Yang, 2019; Radosavovic et al., 2018). Another limitation of this work is the relatively narrow scope. The paper only focuses on one application of semi-supervised keypoint localization. However, I believe such techniques may also applicable to other tasks. \n\n##################################################################\n\nPros:\n\n1.The proposed method is very interesting and novel. The authors propose to add a keypoint classification branch, and design several well-motivated consistency losses. \n2.The paper is clear and well-written.\n3.The proposed method is easily added to any existing keypoint localization networks, which means many previous works can be jointly trained in a much larger unlabeled dataset and it may enhance the robustness and performance of the models.\n\n##################################################################\n\nCons:\n\n1. One major problem is the missing comparisons with pseudo-labeling baselines (Dong & Yang, 2019; Radosavovic et al., 2018). The reviewer believes that the comparisons are critical in showing the effectiveness of the proposed method. \n2. The authors said “where there is a high risk of transferring inaccurate pseudo-labeled examples to the retraining stage that is harmful for the model.”However, no evidence is provided in the experiment sections.\n3. Although the proposed method provides several ablation studies, the reviewer suggests to consider adding the following experiments to enhance the quality of the paper:\n\n(1)\tWhen percentage of labeled images is 100%, which is totally supervised, the proposed method already outperforms the baseline. Is this method also applicable to improving the performance of fully-supervised cases? Please analyze why. \n\n(2) Ablation study of only adding the keypoint classification branch (without the consistency losses).\n\n(3) Lacking of ablation study about TC in Table 2. The effectivess of transformation equivariance and transformation invariance should be considered separately.\n\n\n##################################################################\n\nQuestions during rebuttal period:\n\n1.In Table3, it reads“Our method outperforms the supervised score only by a small margin with 10% of unlabelled data.” It would be more convenient, if the supervised scores are also listed in the Table. \n\nSome typos:\n\n(1) 4.1 ATRW: where l is is the head → where l is the head\n(2) Table 2： Usupervised losses → Unsupervised losses\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2uMZ4MvuUCd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Paper Summary **\n\nThis paper presents semi-supervised keypoint localization networks and loss functions to overcome the need for the labeled keypoint data for that task. It simultaneously generates keypoint heatmaps and pose invariant keypoint representations, where these representations were separately used to enforce translation equivariance, and translation invariance, and semantic consistency, respectively. The proposed method attains the improvement on several benchmarks for human and animal body landmark localization.\n\n** Paper Strength **\n\n+ Learning keypoints with a minimal supervision is an essential step for numerous applications, thus solving such a problem is important.\n+ Using two kinds of representations, i.e., heatmap and keypoint feature, makes sense, enabling to apply different loss functions that achieve different aspects, e.g., translation equivariance and translation invariance, and semantic consistency.\n+ Compared to other previous methods to train the keypoint estimation networks in a self-supervised manner, e.g., Thewlis et al., 2019, the proposed methods incorporates the semantic consistency loss that makes the networks achieve a semantic awareness. \n+ But, results on several benchmarks were clearly state-of-the-arts. \n\n** Paper Weakness **\n\n- Even though this paper was tailored to semi-supervision learning, unsupervised losses themself contribute the performance gains solely. So, I'm really interested in what happens the networks are trained only with unsupervised losses, i.e., SC and TC. It would be interesting because some datasets might not have any ground truth keypoints. In Table 2 and 3, the authors tried to provide an ablation study relevant to this, but the case of percentage of labeled images as 0 would be interesting. \n- Conceptually, the transformation equivariance loss in (2) is similar to equivariant landmark transformation loss proposed by Honari et al. (2018) and recent other variants. The authors argue the proposed loss in (2) are different with them in that the previous methods leverage an inverse transformation. But, the methodological improvement is marginal and incremental.\n- Transformation invariance loss in (4) is also interesting, but the loss itself might not contribute the performance gain. For example, if z and z' converge to 0, the loss is going to be minimum, but the networks are not trained well. \n- In addition, because the paper solves the semi-supervised learning, the weights of loss components are important hyper-parameters, but there lack the ablation study for those.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received three borderline reviews (2+ / 1-) and one positive review.  Having read through the reviews and author responses, the AC recommends the paper to be accepted.  The method, while simple, is proven experimentally to be effectively and will add to the body of work on key-point localization.   The authors are requested to add their additional baselines in the response text to the revision of their paper if it has not already been done.

Decision: Accept (Poster)