Semi-supervised Keypoint Localization

1 INTRODUCTION . Detecting keypoints helps with fine-grained classification ( Guo & Farrell , 2019 ) and re-identification ( Zhu et al. , 2020 ; Sarfraz et al. , 2018 ) . In the domain of wild animals ( Mathis et al. , 2018 ; Moskvyak et al. , 2020 ; Liu et al. , 2019a ; b ) , annotating data is especially challenging due to large pose variations and the need for domain experts to annotate . Moreover , there is less commercial interest in keypoint estimation for animals compared to humans , and little effort is invested in collecting and annotating public datasets . Unsupervised detection of landmarks1 ( Jakab et al. , 2018 ; Thewlis et al. , 2017 ; 2019 ) can extract useful features , but are not able to detect perceptible landmarks without supervision . On the other hand , supervised learning has the risk of overfitting if trained only on a limited number of labeled examples . Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data during training . It is mostly studied for classification task ( van Engelen & Hoos , 2019 ) but it is also important for keypoint localization problem because annotating multiple keypoints per image is a time-consuming manual work , for which precision is the most important factor . Pseudo-labeling ( Lee , 2013 ) is a common semi-supervised approach where unlabeled examples are assigned labels ( called pseudo-labels ) predicted by a model trained on a labeled subset . A heuristic unsupervised criterion is adopted to select the pseudo-labeled data for a retraining procedure . More recently , the works of ( Dong & Yang , 2019 ; Radosavovic et al. , 2018 ) apply variations to selection criteria in pseudo-labeling for semi-supervised facial landmark detection . However , there are less variations in facial landmark positions than in human or animal body joints , where there is a high 1We use terms keypoints or landmarks interchangeably in our work . These terms are more generic than body joints ( used in human pose estimation ) because our method is applicable to a variety of categories . risk of transferring inaccurate pseudo-labeled examples to the retraining stage that is harmful for the model . Previous work of ( Honari et al. , 2018 ) in semi-supervised landmark detection utilizes additional class attributes and test only on datasets that provide these attribute annotations . Our work focuses on keypoint localization task in a common real-world scenario where annotations are provided for a small subset of data from a large unlabeled dataset . More specifically , we propose a method for semi-supervised keypoint localization that learns a list of heatmaps and a list of semantic keypoint representations for each image ( Figure 1 ) . A semantic keypoint representation is a vector of real numbers in a low-dimensional space relative to the image size , and the same keypoints in different images have similar representations . We leverage properties that are specific to the landmark localization problem to design constraints for jointly optimizing both representations . We extend a transformation consistency constraint of ( Honari et al. , 2018 ) to be able to apply it on each representation differently ( i.e . transformation equivariant constraint for heatmaps and transformation invariant constraint for semantic representations ) . Moreover , we formulate a semantic consistency constraint that encourages detecting similar features across images for the same landmark independent of the pose of the object ( e.g . an eye in all images should look similar ) . Learning both representations simultaneously allows us to use the power of both supervised and unsupervised learning . Our work is motivated by data scarcity in the domain of wild animals , but is not limited to animals , and as well , it is applicable to human body landmarks detection . The contribution of our work is three-fold : • We propose a technique for semi-supervised keypoint localization that jointly learns keypoint heatmaps and semantic representations optimised with supervised and unsupervised constraints ; • Our method can be easily added to any existing keypoint localization networks with no structural and with minimal computational overhead ; • We evaluate the proposed method on annotated image datasets for both humans and animals . As demonstrated by our results , our method significantly outperforms previously proposed supervised and unsupervised methods on several benchmarks , using only limited labeled data . The paper is organised as follows . Related work on semi-supervised learning and keypoint localization is reviewed in Section 2 . Our proposed method is described in Section 3 . Experimental settings , datasets and results are discussed in Section 4 . 2 RELATED WORK . Keypoint localization . Supervised keypoint localization research is driven by a few large datasets with labeled keypoints that span across several common research domains including human pose estimation ( Andriluka et al. , 2014 ) and facial keypoints ( Sagonas et al. , 2016 ) . Challenges in obtaining keypoint annotations have led to the rise in unsupervised landmark localization research . Several unsupervised methods leverage the concept of equivariance which means that landmark coordinates stay consistent after synthetic transformations or in subsequent video frames . Thewlis et al . ( 2017 ) propose to learn viewpoint-independent representations that are equivariant to different transformations and Dong et al . ( 2018 ) exploit the coherence of optical flow as a source of supervision . Zhang et al . ( 2018 ) learn landmark encodings by enforcing constraints that reflect the necessary properties for landmarks such as separability and concentration . Jakab et al . ( 2018 ) propose a generative approach where the predicted heatmaps are used to reconstruct the input image from a transformed copy . Recent work ( Thewlis et al. , 2019 ) enforce the consistency between instances of the same object by exchanging descriptor vectors . These methods are mostly evaluated on faces of people that have less degrees of freedom during movements and transformations than human or animal body joints . We compare our method to the combination of supervised and aforementioned unsupervised methods in Section 4 . Semi-supervised learning is the most studied for the classification task . Pseudo-labeling ( Lee , 2013 ) is a method that uses the model ’ s class predictions as artificial labels for unlabeled examples and then trains the model to predict these labels . Another technique is a consistency regularization which states that realistic perturbations of input examples from unlabeled dataset should not significantly change the output of a neural network . Consistency regularization is used in Π-model ( Laine & Aila , 2017 ) and further improved by Temporal Ensembling ( Laine & Aila , 2017 ) which maintains an exponential moving average prediction for each training example and Mean Teacher ( Tarvainen & Valpola , 2017 ) that averages model weights instead of model predictions . Recent methods UDA ( Xie et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) and FixMatch ( Sohn et al. , 2020 ) use a combination of consistency loss , pseudo-labeling and advanced augmentation techniques in addition to color perturbations and spatial transformations . In this work , we investigate adjustments required to apply consistency loss to keypoint localization which we discuss in Section 3.2 . Semi-supervised learning for keypoint localization . To the best of our knowledge , there are a few works in semi-supervised keypoint localization . Dong & Yang ( 2019 ) build on the pseudo-labeling technique and propose a teacher model and two students to generate more reliable pseudo-labels for unlabeled images . However , the method is evaluated on face landmarks and in cases with high variations of poses , there is a high possibility of inaccurate pseudo-labels that can not be filtered out and be harmful during the retraining stage . Honari et al . ( 2018 ) ; Ukita & Uematsu ( 2018 ) learn keypoints in a semi-supervised manner but utilise extra annotations to guide landmark learning such as action labels ( running , jumping ) for juman joints or emotion labels ( smiling , yawning ) for facial keypoint localization . Different from previous work our approach does not use any class labels and learns directly from unlabeled data with high pose variations . 3 SEMI-SUPERVISED LEARNING FOR KEYPOINT LOCALIZATION . In this work , we propose a semi-supervised technique for keypoint localization that learns from an image set where ground truth annotations are provided only for a small subset of the dataset . The overall architecture consists of two components : a keypoint localization network ( KLN ) that outputs keypoint heatmaps of the image , and a keypoint classification network ( KCN ) that classifies keypoints given a semantic keypoint representation as input . Our method does not pose any constraints on the architecture of the KLN and it can be added to any existing keypoint localization network with minimal modifications . We optimize heatmaps with the supervised loss and the transformation equivariance constraint . Simultaneously , keypoint representations are optimized with transformation invariance and semantic consistency constraints ( Figure 1 ) . We discuss each constraint and related components of the architecture in the next sections . 3.1 SEMANTIC KEYPOINT REPRESENTATIONS . Keypoint heatmaps are optimized to estimate locations of keypoints in the image . However , heatmaps do not carry any information about a semantic type of the keypoint ( e.g , a beak or an eye for a bird ) . In semi-supervised regime , the feedback provided by unlabeled examples are not as effective as the ones coming from labeled examples . To extract useful information from unlabeled images , we propose learning a semantic keypoint representation . In particular , keypoint localization network is encouraged to detect similar features for the same semantic keypoint across the dataset by incorporating the feedback from a keypoint representation classifier in the objective function . Motivation for our approach is that the same keypoints should activate the same feature maps . Let us consider KLN as a function f ( x ; θ ) with an input image x and trainable parameters θ that outputs heatmaps h = f ( x ; θ ) . We collect intermediate feature maps from KLN , upscale them to the spatial dimension of output heatmaps , concatenate by channels , and pass through a convolutional layer with C filters of size one ( Figure 2 ) . The resulting feature map F has the shape ( C , H , W ) . Then , feature maps F are element-wise multiplied with each keypoint heatmap hi , i ∈ { 1 , ... , K } seperately to mask out activations corresponding to the detected keypoint . The output of this operation is K feature maps of size ( C , H , W ) . Global Max Pooling ( GMP ) is applied over feature maps to keep the highest value for each channel . We call the produced vector zi = GMP ( F hi ) for each keypoint i ∈ { 1 , ... , K } a semantic keypoint representation . Finally , we pass keypoint representations to a simple KCN ( φ ) which is a fully connected network with an input and an output layer for classification with cross-entropy loss . The feedback from the cross-entropy loss makes up a semantic consistency ( SC ) loss : Lsc ( x ) = − 1 K K∑ i=1 ŷi log ( φ ( zi ) ) ( 1 ) where ŷ is a vector of ground truth semantic labels for keypoints because the order of keypoints in a heatmap is fixed . One advantage of our method is its efficiency as it only adds a small number of parameters to the network to address the task of keypoint representation classification . Specifically , KCN is a small fully connected network shared between keypoints and it has less than a thousand of parameters depending on the number of keypoints . Our approach is related to attention modules ( Vaswani et al. , 2017 ; Hu et al. , 2020 ) as our network has the ability to focus on a subset of features using elementwise multiplication with heatmaps . However , our model uses this attention-based mechanism to learn additional keypoint representations from unlabeled data by optimizing a set of unsupervised losses .

Summary: The paper presents an approach to keypoint localization (to retrieve people/animals pose) combining labeled and unlabeled data. Features are extracted and concatenated into a single descriptor per keypoints, by multiplying feature maps and heatmaps and max-pooling over the spatial domain, and used for semantic classification. Images are transformed with simple perspective augmentations. The non-supervised part comes in enforcing that keypoint representations for unlabeled images remain close.