Pseudo-Labeled Auto-Curriculum Learning for Semi-Supervised Keypoint Localization

1 INTRODUCTION . Keypoints ( also termed as landmarks ) are a popular representation of objects that precisely represent locations of object parts and contain concise information about shapes and poses . Example keypoints are ” right shoulder ” on a human body or the ” tail tip ” of a cat . Keypoint localization is the basis of many visual tasks , including action recognition ( Yan et al. , 2018 ) , face recognition ( Bulat & Tzimiropoulos , 2017 ) , fine-grained classification ( Gavves et al. , 2013 ; 2015 ) and reidentification ( Zhao et al. , 2017 ; Miao et al. , 2019 ) . Keypoint localization has achieved great success with the advent of deep learning in recent years . However , the success of deep networks relies on vast amounts of labeled data , which is often expensive and time-consuming to collect . Semi-supervised learning ( SSL ) is one of the most important approaches for solving this problem . It leverages extensive amounts of unlabeled data in addition to sparsely labeled data to obtain gains in performance . Pseudo-labeling ( PL ) has become one of the most popular SSL approaches due to its simplicity . PL-based methods iteratively add unlabeled samples into the training data by pseudo-labeling them with a model trained on a combination of labeled and pseudo-labeled samples . PL-based methods commonly require a predefined handpicked threshold ( Lee et al. , 2013 ; Oliver et al. , 2018 ) , to filter out low-confidence noisy predictions . However , a single fixed threshold does not take into account the dynamic capacity of the current model for handling noisy pseudo-labels , leading to sub-optimal performance . In this work , we borrow ideas from Curriculum Learning ( CL ) ( Bengio et al. , 2009 ) and design our curriculum as a series of thresholds for PL , which is tuned according to the feedback from the model . CL is a widely used strategy to control the model training pace by selecting from easier to harder samples . With a carefully designed curriculum , noticeable improvement is obtained . However , traditional CL methods suffer from hand-designed curricula , which heavily rely on expertise and detailed analysis for specific domains . Manual curriculum design based on handcrafted criteria is always tedious and sub-optimal . Moreover , curriculum design ( or threshold setting ) is complicated . High-confidence pseudo-labels typically correspond to easier samples with clean labels , while low-confidence pseudo-labels correspond to harder samples with noisy labels . How to design a curriculum to balance the correctness , representativeness , and difficulty of pseudo-labeled data is an open problem . This paper is devoted to tackling the aforementioned problem , i.e . how to automatically learn an optimal learning curriculum for pseudo-labeling in a data-driven way . To this end , we propose a novel method , called PseudoLabeled Auto-Curriculum Learning ( PLACL ) . PLACL formulates the curriculum design problem as a decision-making problem and leverages the reinforcement learning ( RL ) framework to solve it . Additionally , PL-based methods suffer from confirmation bias ( Tarvainen & Valpola , 2017 ) , also known as noise accumulation ( Zhang et al. , 2016 ) , and concept drift ( Cascante-Bonilla et al. , 2021 ) . This long-standing issue stems from the use of noisy or incorrect pseudo-labels in subsequent training stages . As a consequence , the noise accumulates and the performance degrades as the learning process evolves over time . To mitigate this problem , we propose the cross-training strategy which alternatively performs pseudo-label prediction and model training on separate sub-datasets . We benchmark PLACL on six keypoint localization datasets , including LSPET ( Johnson & Everingham , 2011 ) , MPII ( Andriluka et al. , 2014 ) , CUB-200-2011 ( Welinder et al. , 2010 ) , ATRW ( Li et al. , 2019b ) , MS-COCO Lin et al . ( 2014 ) , and AnimalPose ( Cao et al. , 2019 ) . We empirically show that PLACL is general and can be applied to various keypoint localization tasks ( human and animal pose estimation ) and different keypoint localization networks . With a simple yet effective search paradigm , our method significantly boosts the keypoint estimation performance and achieves superior performance to other SSL methods . We hope our method will inspire the community to rethink the potential of PL-based methods for semi-supervised keypoint localization . Our main contributions can be summarized as follows : • We propose Pseudo-Labeled Auto-Curriculum Learning ( PLACL ) . It is an an automatic pseudo-labeled data selection method , which learns a series of dynamic thresholds ( or curriculum ) via reinforcement learning . To the best of our knowledge , this is the first work that explores automatic curriculum learning for semi-supervised keypoint localization . • We propose the cross-training strategy for pseudo-labeling to mitigate the long-standing problem of confirmation bias . • Extensive experiments on a wide range of popular datasets demonstrate the superiority of PLACL over the previous state-of-the-art SSL approaches . In addition , PLACL is modelagnostic and can be easily applied to different keypoint localization networks . 2 RELATED WORKS . 2.1 SEMI-SUPERVISED KEYPOINT LOCALIZATION . Keypoint localization focuses on predicting the keypoints of detected objects , e.g . human body parts ( Lin et al. , 2014 ; Jin et al. , 2020 ) , facial landmarks ( Bulat & Tzimiropoulos , 2017 ) , hand keypoints ( Zimmermann & Brox , 2017 ) and animal poses ( Li & Lee , 2021 ; Cao et al. , 2019 ) . However , training a keypoint localization model often requires a large amount of data , which is expensive and time-consuming to collect . Semi-supervised keypoint localization is one of the most promising ways to solve this problem . Semi-supervised keypoint localization can be categorized into consistency regularization based methods and pseudo-labeling based methods . Consistency regularization methods ( Honari et al. , 2018 ; Moskvyak et al. , 2020 ) assume that the output of the model should not be invariant to realistic perturbations . These approaches typically rely on modality-specific augmentation techniques for regularization . Pseudo-labeling methods ( Ukita & Uematsu , 2018 ; Dong & Yang , 2019 ; Cao et al. , 2019 ; Li & Lee , 2021 ) use labeled data to predict the labels of the unlabeled data , and then train the model in a supervised way with a combination of labeled and selected pseudo-labeled data . Our approach also builds upon pseudo-labeling methods . In contrast to previous works , we propose to learn pseudo-labeled data selection via reinforcement learning . 2.2 CURRICULUM LEARNING . Curriculum learning is firstly introduced by Bengio et al . ( 2009 ) . It is a training strategy that trains machine learning models from easy to complex samples , imitating human education . The curriculum is often pre-determined by heuristics ( Khan et al. , 2011 ; Bengio et al. , 2009 ; Spitkovsky et al. , 2009 ) . However , it requires expert domain knowledge and exhaustive trials to find a good curriculum suitable for a specific task and its dataset . Recently , automatic curriculum learning methods are introduced to break through these limits . Popular ones include self-paced learning methods ( Kumar et al. , 2010 ; Jiang et al. , 2014 ; Zhao et al. , 2015 ) and reinforcement learning ( RL ) based methods ( Graves et al. , 2017 ; Matiisen et al. , 2019 ; Fan et al. , 2018 ) . Our approach can be categorized as RL-based methods . Unlike previous works that mainly focus on supervised learning , our approach is specially designed for the SSL paradigm . Our work is mostly related to Curriculum Labeling ( Cascante-Bonilla et al. , 2021 ) . It adopts a hand-crafted curriculum based on Extreme Value Theory ( EVT ) to facilitate model training . In contrast , we propose an automatic curriculum learning approach by searching for dynamic thresholds for pseudo-labeling . In addition , the curriculum of ( Cascante-Bonilla et al. , 2021 ) is coarse-grained on the round level , while our curriculum is fine-grained on the epoch level . 2.3 REINFORCEMENT LEARNING FOR AUTOML . Reinforcement learning ( RL ) has shown impressive results in a range of applications . Well-known examples include game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ; 2017 ) and robotics control ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ) . Recent works have employed RL to the AutoML , automating the design of a machine learning system , e.g . searching for neural architectures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Baker et al. , 2017 ; Pham et al. , 2018 ) , augmentation policies ( Cubuk et al. , 2019 ) , activation functions ( Ramachandran et al. , 2017 ) , loss functions ( Li et al. , 2019a ; 2021 ) , and training hyperparameters ( Dong et al. , 2020 ) . In contrast to these works , we apply RL to the automatic selection of pseudo-labeled data in the context of pseudo-labeling . 3 PSEUDO-LABELED AUTO-CURRICULUM LEARNING ( PLACL ) . 3.1 OVERVIEW . Our PLACL algorithm is illustrated in Fig . 1 . The training process consists ofR self-training rounds and each round consists of N training epochs . ( 0 ) In the initial round ( r = 0 ) , we pre-train a keypoint localization network Θ0ω on the labeled data , where ω denotes the weights of the network . And for the r-th round , ( 1 ) The trained network Θrω is used to predict pseudo-labels for unlabeled data . ( 2 ) We adopt reinforcement learning ( RL ) to automatically generate the learning curriculum . Specifically , our curriculum ( Γr ) consists of a series of thresholds for pseudo-labeled data selection . Γr = [ γr1 , . . . , γ r N ] , where γ r i ∈ [ 0 , 1 ] is the threshold for each epoch i . ( 3 ) We then select reliable pseudo-labeled data by the searched curriculum . ( 4 ) We retrain a new model ( Θr+1ω ) using both the labeled samples and selected pseudo-labeled samples . ( 5 ) This process is repeated for R rounds . 3.2 PSEUDO-LABEL SELECTION FOR SEMI-SUPERVISED KEYPOINT LOCALIZATION . We denote the labeled dataset with Nl samples as Dl = { ( I li , Y l i ) ∣∣Nl i=1 } , where Ii and Yi denote the i-th training image and its keypoint annotations ( the x-y coordinates of K keypoints ) . The Nu unlabeled images are denoted as Du = { ( Iui ) | Nu i=1 } , which are not associated with any ground-truth keypoint labels . Generally , we have |Nl| |Nu| . Pseudo-labeling based method builds upon the general idea of self-training ( McLachlan , 1975 ) , where the keypoint localization network Θω goes through multiple rounds of training . In the initialization round , the model is first trained with the small labeled training set Dtrain = Dl in a usual supervised manner . In subsequent rounds , the trained model is used to estimate labels for the unla- beled data D̃u = { ( Iui , Ỹ u i ) ∣∣∣Nu i=1 } . Here , we omit the superscript r for simplicity . Specifically , given an unlabeled image Iui , the trained keypoint localization network Θω predicts K heatmaps . Each heatmap is a 2D Gaussian centered on the joint location , which represents the confidence of the k-th keypoint . The output pseudo-labeled keypoint location ( Ỹ ui ) is the highest response in the heatmap space . And the confidence score C ( Θω ( Iui ) ) is the response value at the keypoint location . Then , pseudo-label selection process is adopted . Let g = [ g1 , ... , gNu ] ⊆ { 0 , 1 } Nu be a binary vector representing the selection of pseudo-labels , where gi denotes whether the keypoint prediction on Iui is selected . gi = { 1 , if C ( Θω ( Iui ) ) > γ 0 , otherwise ( 1 ) where γ ∈ ( 0 , 1 ) is the confidence threshold . Pseudo-labeled samples with higher confidence are added to the training set . Dtrain = { ( I li , Y l i ) ∣∣Nl i=1 } ∪ { ( Iui , Ỹ u i ) ∣∣∣Nu i=1 where gi = 1 } . ( 2 ) Then the keypoint localization network is retrained with a combination of labeled and pseudolabeled training data Dtrain .

Summary: A semi-supervised learning method (PLACL) is proposed. This method employs a Pseudo-Labeling (PL) approach. It consists in iteratively (these iterations are called rounds):  1. predicting pseudo-labels to unlabeled data using the current model, 2. training a series of models from scratch using the labeled data and selections of pseudo-labeled data. The pseudo-label selection is performed using a series of thresholds (called curriculum) over the scores output by the model.  The authors propose an Auto-Curriculum Learning (ACL) strategy to automatically update the curriculum using a reinforcement learning approach (PPO2). The fact that the curriculum is updated at each round is called "Curriculum Residual Learning". They also employ a cross-training strategy to prevent the issue of confirmation bias.  The performances of PLACL are evaluated on a keypoint localization application on 5 different datasets. When the percentage of labeled data is very low (5% or 10%), PLACL outperforms the state of the art semi-supervised learning competitors.