Pseudo-Labeled Auto-Curriculum Learning for Semi-Supervised Keypoint Localization

1 INTRODUCTION . Keypoints ( also termed as landmarks ) are a popular representation of objects that precisely represent locations of object parts and contain concise information about shapes and poses . Example keypoints are ” right shoulder ” on a human body or the ” tail tip ” of a cat . Keypoint localization is the basis of many visual tasks , including action recognition ( Yan et al. , 2018 ) , face recognition ( Bulat & Tzimiropoulos , 2017 ) , fine-grained classification ( Gavves et al. , 2013 ; 2015 ) and reidentification ( Zhao et al. , 2017 ; Miao et al. , 2019 ) . Keypoint localization has achieved great success with the advent of deep learning in recent years . However , the success of deep networks relies on vast amounts of labeled data , which is often expensive and time-consuming to collect . Semi-supervised learning ( SSL ) is one of the most important approaches for solving this problem . It leverages extensive amounts of unlabeled data in addition to sparsely labeled data to obtain gains in performance . Pseudo-labeling ( PL ) has become one of the most popular SSL approaches due to its simplicity . PL-based methods iteratively add unlabeled samples into the training data by pseudo-labeling them with a model trained on a combination of labeled and pseudo-labeled samples . PL-based methods commonly require a predefined handpicked threshold ( Lee et al. , 2013 ; Oliver et al. , 2018 ) , to filter out low-confidence noisy predictions . However , a single fixed threshold does not take into account the dynamic capacity of the current model for handling noisy pseudo-labels , leading to sub-optimal performance . In this work , we borrow ideas from Curriculum Learning ( CL ) ( Bengio et al. , 2009 ) and design our curriculum as a series of thresholds for PL , which is tuned according to the feedback from the model . CL is a widely used strategy to control the model training pace by selecting from easier to harder samples . With a carefully designed curriculum , noticeable improvement is obtained . However , traditional CL methods suffer from hand-designed curricula , which heavily rely on expertise and detailed analysis for specific domains . Manual curriculum design based on handcrafted criteria is always tedious and sub-optimal . Moreover , curriculum design ( or threshold setting ) is complicated . High-confidence pseudo-labels typically correspond to easier samples with clean labels , while low-confidence pseudo-labels correspond to harder samples with noisy labels . How to design a curriculum to balance the correctness , representativeness , and difficulty of pseudo-labeled data is an open problem . This paper is devoted to tackling the aforementioned problem , i.e . how to automatically learn an optimal learning curriculum for pseudo-labeling in a data-driven way . To this end , we propose a novel method , called PseudoLabeled Auto-Curriculum Learning ( PLACL ) . PLACL formulates the curriculum design problem as a decision-making problem and leverages the reinforcement learning ( RL ) framework to solve it . Additionally , PL-based methods suffer from confirmation bias ( Tarvainen & Valpola , 2017 ) , also known as noise accumulation ( Zhang et al. , 2016 ) , and concept drift ( Cascante-Bonilla et al. , 2021 ) . This long-standing issue stems from the use of noisy or incorrect pseudo-labels in subsequent training stages . As a consequence , the noise accumulates and the performance degrades as the learning process evolves over time . To mitigate this problem , we propose the cross-training strategy which alternatively performs pseudo-label prediction and model training on separate sub-datasets . We benchmark PLACL on six keypoint localization datasets , including LSPET ( Johnson & Everingham , 2011 ) , MPII ( Andriluka et al. , 2014 ) , CUB-200-2011 ( Welinder et al. , 2010 ) , ATRW ( Li et al. , 2019b ) , MS-COCO Lin et al . ( 2014 ) , and AnimalPose ( Cao et al. , 2019 ) . We empirically show that PLACL is general and can be applied to various keypoint localization tasks ( human and animal pose estimation ) and different keypoint localization networks . With a simple yet effective search paradigm , our method significantly boosts the keypoint estimation performance and achieves superior performance to other SSL methods . We hope our method will inspire the community to rethink the potential of PL-based methods for semi-supervised keypoint localization . Our main contributions can be summarized as follows : • We propose Pseudo-Labeled Auto-Curriculum Learning ( PLACL ) . It is an an automatic pseudo-labeled data selection method , which learns a series of dynamic thresholds ( or curriculum ) via reinforcement learning . To the best of our knowledge , this is the first work that explores automatic curriculum learning for semi-supervised keypoint localization . • We propose the cross-training strategy for pseudo-labeling to mitigate the long-standing problem of confirmation bias . • Extensive experiments on a wide range of popular datasets demonstrate the superiority of PLACL over the previous state-of-the-art SSL approaches . In addition , PLACL is modelagnostic and can be easily applied to different keypoint localization networks . 2 RELATED WORKS . 2.1 SEMI-SUPERVISED KEYPOINT LOCALIZATION . Keypoint localization focuses on predicting the keypoints of detected objects , e.g . human body parts ( Lin et al. , 2014 ; Jin et al. , 2020 ) , facial landmarks ( Bulat & Tzimiropoulos , 2017 ) , hand keypoints ( Zimmermann & Brox , 2017 ) and animal poses ( Li & Lee , 2021 ; Cao et al. , 2019 ) . However , training a keypoint localization model often requires a large amount of data , which is expensive and time-consuming to collect . Semi-supervised keypoint localization is one of the most promising ways to solve this problem . Semi-supervised keypoint localization can be categorized into consistency regularization based methods and pseudo-labeling based methods . Consistency regularization methods ( Honari et al. , 2018 ; Moskvyak et al. , 2020 ) assume that the output of the model should not be invariant to realistic perturbations . These approaches typically rely on modality-specific augmentation techniques for regularization . Pseudo-labeling methods ( Ukita & Uematsu , 2018 ; Dong & Yang , 2019 ; Cao et al. , 2019 ; Li & Lee , 2021 ) use labeled data to predict the labels of the unlabeled data , and then train the model in a supervised way with a combination of labeled and selected pseudo-labeled data . Our approach also builds upon pseudo-labeling methods . In contrast to previous works , we propose to learn pseudo-labeled data selection via reinforcement learning . 2.2 CURRICULUM LEARNING . Curriculum learning is firstly introduced by Bengio et al . ( 2009 ) . It is a training strategy that trains machine learning models from easy to complex samples , imitating human education . The curriculum is often pre-determined by heuristics ( Khan et al. , 2011 ; Bengio et al. , 2009 ; Spitkovsky et al. , 2009 ) . However , it requires expert domain knowledge and exhaustive trials to find a good curriculum suitable for a specific task and its dataset . Recently , automatic curriculum learning methods are introduced to break through these limits . Popular ones include self-paced learning methods ( Kumar et al. , 2010 ; Jiang et al. , 2014 ; Zhao et al. , 2015 ) and reinforcement learning ( RL ) based methods ( Graves et al. , 2017 ; Matiisen et al. , 2019 ; Fan et al. , 2018 ) . Our approach can be categorized as RL-based methods . Unlike previous works that mainly focus on supervised learning , our approach is specially designed for the SSL paradigm . Our work is mostly related to Curriculum Labeling ( Cascante-Bonilla et al. , 2021 ) . It adopts a hand-crafted curriculum based on Extreme Value Theory ( EVT ) to facilitate model training . In contrast , we propose an automatic curriculum learning approach by searching for dynamic thresholds for pseudo-labeling . In addition , the curriculum of ( Cascante-Bonilla et al. , 2021 ) is coarse-grained on the round level , while our curriculum is fine-grained on the epoch level . 2.3 REINFORCEMENT LEARNING FOR AUTOML . Reinforcement learning ( RL ) has shown impressive results in a range of applications . Well-known examples include game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ; 2017 ) and robotics control ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ) . Recent works have employed RL to the AutoML , automating the design of a machine learning system , e.g . searching for neural architectures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Baker et al. , 2017 ; Pham et al. , 2018 ) , augmentation policies ( Cubuk et al. , 2019 ) , activation functions ( Ramachandran et al. , 2017 ) , loss functions ( Li et al. , 2019a ; 2021 ) , and training hyperparameters ( Dong et al. , 2020 ) . In contrast to these works , we apply RL to the automatic selection of pseudo-labeled data in the context of pseudo-labeling . 3 PSEUDO-LABELED AUTO-CURRICULUM LEARNING ( PLACL ) . 3.1 OVERVIEW . Our PLACL algorithm is illustrated in Fig . 1 . The training process consists ofR self-training rounds and each round consists of N training epochs . ( 0 ) In the initial round ( r = 0 ) , we pre-train a keypoint localization network Θ0ω on the labeled data , where ω denotes the weights of the network . And for the r-th round , ( 1 ) The trained network Θrω is used to predict pseudo-labels for unlabeled data . ( 2 ) We adopt reinforcement learning ( RL ) to automatically generate the learning curriculum . Specifically , our curriculum ( Γr ) consists of a series of thresholds for pseudo-labeled data selection . Γr = [ γr1 , . . . , γ r N ] , where γ r i ∈ [ 0 , 1 ] is the threshold for each epoch i . ( 3 ) We then select reliable pseudo-labeled data by the searched curriculum . ( 4 ) We retrain a new model ( Θr+1ω ) using both the labeled samples and selected pseudo-labeled samples . ( 5 ) This process is repeated for R rounds . 3.2 PSEUDO-LABEL SELECTION FOR SEMI-SUPERVISED KEYPOINT LOCALIZATION . We denote the labeled dataset with Nl samples as Dl = { ( I li , Y l i ) ∣∣Nl i=1 } , where Ii and Yi denote the i-th training image and its keypoint annotations ( the x-y coordinates of K keypoints ) . The Nu unlabeled images are denoted as Du = { ( Iui ) | Nu i=1 } , which are not associated with any ground-truth keypoint labels . Generally , we have |Nl| |Nu| . Pseudo-labeling based method builds upon the general idea of self-training ( McLachlan , 1975 ) , where the keypoint localization network Θω goes through multiple rounds of training . In the initialization round , the model is first trained with the small labeled training set Dtrain = Dl in a usual supervised manner . In subsequent rounds , the trained model is used to estimate labels for the unla- beled data D̃u = { ( Iui , Ỹ u i ) ∣∣∣Nu i=1 } . Here , we omit the superscript r for simplicity . Specifically , given an unlabeled image Iui , the trained keypoint localization network Θω predicts K heatmaps . Each heatmap is a 2D Gaussian centered on the joint location , which represents the confidence of the k-th keypoint . The output pseudo-labeled keypoint location ( Ỹ ui ) is the highest response in the heatmap space . And the confidence score C ( Θω ( Iui ) ) is the response value at the keypoint location . Then , pseudo-label selection process is adopted . Let g = [ g1 , ... , gNu ] ⊆ { 0 , 1 } Nu be a binary vector representing the selection of pseudo-labels , where gi denotes whether the keypoint prediction on Iui is selected . gi = { 1 , if C ( Θω ( Iui ) ) > γ 0 , otherwise ( 1 ) where γ ∈ ( 0 , 1 ) is the confidence threshold . Pseudo-labeled samples with higher confidence are added to the training set . Dtrain = { ( I li , Y l i ) ∣∣Nl i=1 } ∪ { ( Iui , Ỹ u i ) ∣∣∣Nu i=1 where gi = 1 } . ( 2 ) Then the keypoint localization network is retrained with a combination of labeled and pseudolabeled training data Dtrain .

Summary: The paper introduces a method for semi-supervised keypoint localization based on pseudo-labeling with auto-curriculum learning. The Auto-curriculum learning approach learns a series of dynamic thresholds for automatic selection of high-quality pseudo-labeled examples for model retraining. The reinforcement learning (RL) framework, more specifically, the proximal policy optimization algorithm, is used to search for the optimal curriculum. The method is evaluated on four benchmarks in keypoint localization.