Semi-supervised Keypoint Localization

1 INTRODUCTION . Detecting keypoints helps with fine-grained classification ( Guo & Farrell , 2019 ) and re-identification ( Zhu et al. , 2020 ; Sarfraz et al. , 2018 ) . In the domain of wild animals ( Mathis et al. , 2018 ; Moskvyak et al. , 2020 ; Liu et al. , 2019a ; b ) , annotating data is especially challenging due to large pose variations and the need for domain experts to annotate . Moreover , there is less commercial interest in keypoint estimation for animals compared to humans , and little effort is invested in collecting and annotating public datasets . Unsupervised detection of landmarks1 ( Jakab et al. , 2018 ; Thewlis et al. , 2017 ; 2019 ) can extract useful features , but are not able to detect perceptible landmarks without supervision . On the other hand , supervised learning has the risk of overfitting if trained only on a limited number of labeled examples . Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data during training . It is mostly studied for classification task ( van Engelen & Hoos , 2019 ) but it is also important for keypoint localization problem because annotating multiple keypoints per image is a time-consuming manual work , for which precision is the most important factor . Pseudo-labeling ( Lee , 2013 ) is a common semi-supervised approach where unlabeled examples are assigned labels ( called pseudo-labels ) predicted by a model trained on a labeled subset . A heuristic unsupervised criterion is adopted to select the pseudo-labeled data for a retraining procedure . More recently , the works of ( Dong & Yang , 2019 ; Radosavovic et al. , 2018 ) apply variations to selection criteria in pseudo-labeling for semi-supervised facial landmark detection . However , there are less variations in facial landmark positions than in human or animal body joints , where there is a high 1We use terms keypoints or landmarks interchangeably in our work . These terms are more generic than body joints ( used in human pose estimation ) because our method is applicable to a variety of categories . risk of transferring inaccurate pseudo-labeled examples to the retraining stage that is harmful for the model . Previous work of ( Honari et al. , 2018 ) in semi-supervised landmark detection utilizes additional class attributes and test only on datasets that provide these attribute annotations . Our work focuses on keypoint localization task in a common real-world scenario where annotations are provided for a small subset of data from a large unlabeled dataset . More specifically , we propose a method for semi-supervised keypoint localization that learns a list of heatmaps and a list of semantic keypoint representations for each image ( Figure 1 ) . A semantic keypoint representation is a vector of real numbers in a low-dimensional space relative to the image size , and the same keypoints in different images have similar representations . We leverage properties that are specific to the landmark localization problem to design constraints for jointly optimizing both representations . We extend a transformation consistency constraint of ( Honari et al. , 2018 ) to be able to apply it on each representation differently ( i.e . transformation equivariant constraint for heatmaps and transformation invariant constraint for semantic representations ) . Moreover , we formulate a semantic consistency constraint that encourages detecting similar features across images for the same landmark independent of the pose of the object ( e.g . an eye in all images should look similar ) . Learning both representations simultaneously allows us to use the power of both supervised and unsupervised learning . Our work is motivated by data scarcity in the domain of wild animals , but is not limited to animals , and as well , it is applicable to human body landmarks detection . The contribution of our work is three-fold : • We propose a technique for semi-supervised keypoint localization that jointly learns keypoint heatmaps and semantic representations optimised with supervised and unsupervised constraints ; • Our method can be easily added to any existing keypoint localization networks with no structural and with minimal computational overhead ; • We evaluate the proposed method on annotated image datasets for both humans and animals . As demonstrated by our results , our method significantly outperforms previously proposed supervised and unsupervised methods on several benchmarks , using only limited labeled data . The paper is organised as follows . Related work on semi-supervised learning and keypoint localization is reviewed in Section 2 . Our proposed method is described in Section 3 . Experimental settings , datasets and results are discussed in Section 4 . 2 RELATED WORK . Keypoint localization . Supervised keypoint localization research is driven by a few large datasets with labeled keypoints that span across several common research domains including human pose estimation ( Andriluka et al. , 2014 ) and facial keypoints ( Sagonas et al. , 2016 ) . Challenges in obtaining keypoint annotations have led to the rise in unsupervised landmark localization research . Several unsupervised methods leverage the concept of equivariance which means that landmark coordinates stay consistent after synthetic transformations or in subsequent video frames . Thewlis et al . ( 2017 ) propose to learn viewpoint-independent representations that are equivariant to different transformations and Dong et al . ( 2018 ) exploit the coherence of optical flow as a source of supervision . Zhang et al . ( 2018 ) learn landmark encodings by enforcing constraints that reflect the necessary properties for landmarks such as separability and concentration . Jakab et al . ( 2018 ) propose a generative approach where the predicted heatmaps are used to reconstruct the input image from a transformed copy . Recent work ( Thewlis et al. , 2019 ) enforce the consistency between instances of the same object by exchanging descriptor vectors . These methods are mostly evaluated on faces of people that have less degrees of freedom during movements and transformations than human or animal body joints . We compare our method to the combination of supervised and aforementioned unsupervised methods in Section 4 . Semi-supervised learning is the most studied for the classification task . Pseudo-labeling ( Lee , 2013 ) is a method that uses the model ’ s class predictions as artificial labels for unlabeled examples and then trains the model to predict these labels . Another technique is a consistency regularization which states that realistic perturbations of input examples from unlabeled dataset should not significantly change the output of a neural network . Consistency regularization is used in Π-model ( Laine & Aila , 2017 ) and further improved by Temporal Ensembling ( Laine & Aila , 2017 ) which maintains an exponential moving average prediction for each training example and Mean Teacher ( Tarvainen & Valpola , 2017 ) that averages model weights instead of model predictions . Recent methods UDA ( Xie et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) and FixMatch ( Sohn et al. , 2020 ) use a combination of consistency loss , pseudo-labeling and advanced augmentation techniques in addition to color perturbations and spatial transformations . In this work , we investigate adjustments required to apply consistency loss to keypoint localization which we discuss in Section 3.2 . Semi-supervised learning for keypoint localization . To the best of our knowledge , there are a few works in semi-supervised keypoint localization . Dong & Yang ( 2019 ) build on the pseudo-labeling technique and propose a teacher model and two students to generate more reliable pseudo-labels for unlabeled images . However , the method is evaluated on face landmarks and in cases with high variations of poses , there is a high possibility of inaccurate pseudo-labels that can not be filtered out and be harmful during the retraining stage . Honari et al . ( 2018 ) ; Ukita & Uematsu ( 2018 ) learn keypoints in a semi-supervised manner but utilise extra annotations to guide landmark learning such as action labels ( running , jumping ) for juman joints or emotion labels ( smiling , yawning ) for facial keypoint localization . Different from previous work our approach does not use any class labels and learns directly from unlabeled data with high pose variations . 3 SEMI-SUPERVISED LEARNING FOR KEYPOINT LOCALIZATION . In this work , we propose a semi-supervised technique for keypoint localization that learns from an image set where ground truth annotations are provided only for a small subset of the dataset . The overall architecture consists of two components : a keypoint localization network ( KLN ) that outputs keypoint heatmaps of the image , and a keypoint classification network ( KCN ) that classifies keypoints given a semantic keypoint representation as input . Our method does not pose any constraints on the architecture of the KLN and it can be added to any existing keypoint localization network with minimal modifications . We optimize heatmaps with the supervised loss and the transformation equivariance constraint . Simultaneously , keypoint representations are optimized with transformation invariance and semantic consistency constraints ( Figure 1 ) . We discuss each constraint and related components of the architecture in the next sections . 3.1 SEMANTIC KEYPOINT REPRESENTATIONS . Keypoint heatmaps are optimized to estimate locations of keypoints in the image . However , heatmaps do not carry any information about a semantic type of the keypoint ( e.g , a beak or an eye for a bird ) . In semi-supervised regime , the feedback provided by unlabeled examples are not as effective as the ones coming from labeled examples . To extract useful information from unlabeled images , we propose learning a semantic keypoint representation . In particular , keypoint localization network is encouraged to detect similar features for the same semantic keypoint across the dataset by incorporating the feedback from a keypoint representation classifier in the objective function . Motivation for our approach is that the same keypoints should activate the same feature maps . Let us consider KLN as a function f ( x ; θ ) with an input image x and trainable parameters θ that outputs heatmaps h = f ( x ; θ ) . We collect intermediate feature maps from KLN , upscale them to the spatial dimension of output heatmaps , concatenate by channels , and pass through a convolutional layer with C filters of size one ( Figure 2 ) . The resulting feature map F has the shape ( C , H , W ) . Then , feature maps F are element-wise multiplied with each keypoint heatmap hi , i ∈ { 1 , ... , K } seperately to mask out activations corresponding to the detected keypoint . The output of this operation is K feature maps of size ( C , H , W ) . Global Max Pooling ( GMP ) is applied over feature maps to keep the highest value for each channel . We call the produced vector zi = GMP ( F hi ) for each keypoint i ∈ { 1 , ... , K } a semantic keypoint representation . Finally , we pass keypoint representations to a simple KCN ( φ ) which is a fully connected network with an input and an output layer for classification with cross-entropy loss . The feedback from the cross-entropy loss makes up a semantic consistency ( SC ) loss : Lsc ( x ) = − 1 K K∑ i=1 ŷi log ( φ ( zi ) ) ( 1 ) where ŷ is a vector of ground truth semantic labels for keypoints because the order of keypoints in a heatmap is fixed . One advantage of our method is its efficiency as it only adds a small number of parameters to the network to address the task of keypoint representation classification . Specifically , KCN is a small fully connected network shared between keypoints and it has less than a thousand of parameters depending on the number of keypoints . Our approach is related to attention modules ( Vaswani et al. , 2017 ; Hu et al. , 2020 ) as our network has the ability to focus on a subset of features using elementwise multiplication with heatmaps . However , our model uses this attention-based mechanism to learn additional keypoint representations from unlabeled data by optimizing a set of unsupervised losses .

Summary: This paper presents semi-supervised keypoint localization networks and loss functions to overcome the need for the labeled keypoint data for that task. It simultaneously generates keypoint heatmaps and pose invariant keypoint representations, where these representations were separately used to enforce translation equivariance, and translation invariance, and semantic consistency, respectively. The proposed method attains the improvement on several benchmarks for human and animal body landmark localization.