Year: 2021

Abstract:
Localizing keypoints of an object is a basic visual problem. However, supervised learning of a keypoint localization network often requires a large amount of data, which is expensive and time-consuming to obtain. To remedy this, there is an ever-growing interest in semi-supervised learning (SSL), which leverages a small set of labeled data along with a large set of unlabeled data. Among these SSL approaches, pseudo-labeling (PL) is one of the most popular. PL approaches apply pseudo-labels to unlabeled data, and then train the model with a combination of the labeled and pseudo-labeled data iteratively. The key to the success of PL is the selection of high-quality pseudo-labeled samples. Previous works mostly select training samples by manually setting a single confidence threshold. We propose to automatically select reliable pseudo-labeled samples with a series of dynamic thresholds, which constitutes a learning curriculum. Extensive experiments on six keypoint localization benchmark datasets demonstrate that the proposed approach significantly outperforms the previous state-of-the-art SSL approaches.

heading: 1 INTRODUCTION
text: Keypoints (also termed as landmarks) are a popular representation of objects that precisely represent locations of object parts and contain concise information about shapes and poses. Example keypoints are ”right shoulder” on a human body or the ”tail tip” of a cat. Keypoint localization is the basis of many visual tasks, including action recognition (Yan et al., 2018), face recognition (Bulat & Tzimiropoulos, 2017), fine-grained classification (Gavves et al., 2013; 2015) and reidentification (Zhao et al., 2017; Miao et al., 2019).
Keypoint localization has achieved great success with the advent of deep learning in recent years. However, the success of deep networks relies on vast amounts of labeled data, which is often expensive and time-consuming to collect. Semi-supervised learning (SSL) is one of the most important approaches for solving this problem. It leverages extensive amounts of unlabeled data in addition to sparsely labeled data to obtain gains in performance. Pseudo-labeling (PL) has become one of the most popular SSL approaches due to its simplicity. PL-based methods iteratively add unlabeled samples into the training data by pseudo-labeling them with a model trained on a combination of labeled and pseudo-labeled samples.
PL-based methods commonly require a predefined handpicked threshold (Lee et al., 2013; Oliver et al., 2018), to filter out low-confidence noisy predictions. However, a single fixed threshold does not take into account the dynamic capacity of the current model for handling noisy pseudo-labels, leading to sub-optimal performance. In this work, we borrow ideas from Curriculum Learning (CL) (Bengio et al., 2009) and design our curriculum as a series of thresholds for PL, which is tuned according to the feedback from the model. CL is a widely used strategy to control the model training pace by selecting from easier to harder samples. With a carefully designed curriculum, noticeable improvement is obtained. However, traditional CL methods suffer from hand-designed curricula, which heavily rely on expertise and detailed analysis for specific domains. Manual curriculum design based on handcrafted criteria is always tedious and sub-optimal. Moreover, curriculum design (or threshold setting) is complicated. High-confidence pseudo-labels typically correspond to easier samples with clean labels, while low-confidence pseudo-labels correspond to harder samples with noisy labels. How to design a curriculum to balance the correctness, representativeness, and difficulty of pseudo-labeled data is an open problem. This paper is devoted to tackling
the aforementioned problem, i.e. how to automatically learn an optimal learning curriculum for pseudo-labeling in a data-driven way. To this end, we propose a novel method, called PseudoLabeled Auto-Curriculum Learning (PLACL). PLACL formulates the curriculum design problem as a decision-making problem and leverages the reinforcement learning (RL) framework to solve it.
Additionally, PL-based methods suffer from confirmation bias (Tarvainen & Valpola, 2017), also known as noise accumulation (Zhang et al., 2016), and concept drift (Cascante-Bonilla et al., 2021). This long-standing issue stems from the use of noisy or incorrect pseudo-labels in subsequent training stages. As a consequence, the noise accumulates and the performance degrades as the learning process evolves over time. To mitigate this problem, we propose the cross-training strategy which alternatively performs pseudo-label prediction and model training on separate sub-datasets.
We benchmark PLACL on six keypoint localization datasets, including LSPET (Johnson & Everingham, 2011), MPII (Andriluka et al., 2014), CUB-200-2011 (Welinder et al., 2010), ATRW (Li et al., 2019b), MS-COCO Lin et al. (2014), and AnimalPose (Cao et al., 2019). We empirically show that PLACL is general and can be applied to various keypoint localization tasks (human and animal pose estimation) and different keypoint localization networks. With a simple yet effective search paradigm, our method significantly boosts the keypoint estimation performance and achieves superior performance to other SSL methods. We hope our method will inspire the community to rethink the potential of PL-based methods for semi-supervised keypoint localization.
Our main contributions can be summarized as follows:
• We propose Pseudo-Labeled Auto-Curriculum Learning (PLACL). It is an an automatic pseudo-labeled data selection method, which learns a series of dynamic thresholds (or curriculum) via reinforcement learning. To the best of our knowledge, this is the first work that explores automatic curriculum learning for semi-supervised keypoint localization.
• We propose the cross-training strategy for pseudo-labeling to mitigate the long-standing problem of confirmation bias.
• Extensive experiments on a wide range of popular datasets demonstrate the superiority of PLACL over the previous state-of-the-art SSL approaches. In addition, PLACL is modelagnostic and can be easily applied to different keypoint localization networks.

heading: 2 RELATED WORKS

heading: 2.1 SEMI-SUPERVISED KEYPOINT LOCALIZATION
text: Keypoint localization focuses on predicting the keypoints of detected objects, e.g. human body parts (Lin et al., 2014; Jin et al., 2020), facial landmarks (Bulat & Tzimiropoulos, 2017), hand keypoints (Zimmermann & Brox, 2017) and animal poses (Li & Lee, 2021; Cao et al., 2019). However, training a keypoint localization model often requires a large amount of data, which is expensive and time-consuming to collect. Semi-supervised keypoint localization is one of the most promising ways to solve this problem. Semi-supervised keypoint localization can be categorized into consistency regularization based methods and pseudo-labeling based methods. Consistency regularization methods (Honari et al., 2018; Moskvyak et al., 2020) assume that the output of the model should not be invariant to realistic perturbations. These approaches typically rely on modality-specific augmentation techniques for regularization. Pseudo-labeling methods (Ukita & Uematsu, 2018; Dong & Yang, 2019; Cao et al., 2019; Li & Lee, 2021) use labeled data to predict the labels of the unlabeled data, and then train the model in a supervised way with a combination of labeled and selected pseudo-labeled data. Our approach also builds upon pseudo-labeling methods. In contrast to previous works, we propose to learn pseudo-labeled data selection via reinforcement learning.

heading: 2.2 CURRICULUM LEARNING
text: Curriculum learning is firstly introduced by Bengio et al. (2009). It is a training strategy that trains machine learning models from easy to complex samples, imitating human education. The curriculum is often pre-determined by heuristics (Khan et al., 2011; Bengio et al., 2009; Spitkovsky et al., 2009). However, it requires expert domain knowledge and exhaustive trials to find a good curriculum suitable for a specific task and its dataset. Recently, automatic curriculum learning methods are
introduced to break through these limits. Popular ones include self-paced learning methods (Kumar et al., 2010; Jiang et al., 2014; Zhao et al., 2015) and reinforcement learning (RL) based methods (Graves et al., 2017; Matiisen et al., 2019; Fan et al., 2018). Our approach can be categorized as RL-based methods. Unlike previous works that mainly focus on supervised learning, our approach is specially designed for the SSL paradigm. Our work is mostly related to Curriculum Labeling (Cascante-Bonilla et al., 2021). It adopts a hand-crafted curriculum based on Extreme Value Theory (EVT) to facilitate model training. In contrast, we propose an automatic curriculum learning approach by searching for dynamic thresholds for pseudo-labeling. In addition, the curriculum of (Cascante-Bonilla et al., 2021) is coarse-grained on the round level, while our curriculum is fine-grained on the epoch level.

heading: 2.3 REINFORCEMENT LEARNING FOR AUTOML
text: Reinforcement learning (RL) has shown impressive results in a range of applications. Well-known examples include game playing (Mnih et al., 2015; Silver et al., 2016; 2017) and robotics control(Schulman et al., 2015; Lillicrap et al., 2016). Recent works have employed RL to the AutoML, automating the design of a machine learning system, e.g. searching for neural architectures (Zoph & Le, 2017; Zoph et al., 2018; Baker et al., 2017; Pham et al., 2018), augmentation policies (Cubuk et al., 2019), activation functions (Ramachandran et al., 2017), loss functions (Li et al., 2019a; 2021), and training hyperparameters (Dong et al., 2020). In contrast to these works, we apply RL to the automatic selection of pseudo-labeled data in the context of pseudo-labeling.

heading: 3 PSEUDO-LABELED AUTO-CURRICULUM LEARNING (PLACL)

heading: 3.1 OVERVIEW
text: Our PLACL algorithm is illustrated in Fig. 1. The training process consists ofR self-training rounds and each round consists of N training epochs. (0) In the initial round (r = 0), we pre-train a keypoint localization network Θ0ω on the labeled data, where ω denotes the weights of the network. And for the r-th round, (1) The trained network Θrω is used to predict pseudo-labels for unlabeled data. (2) We adopt reinforcement learning (RL) to automatically generate the learning curriculum. Specifically, our curriculum (Γr) consists of a series of thresholds for pseudo-labeled data selection. Γr = [γr1 , . . . , γ r N ], where γ r i ∈ [0, 1] is the threshold for each epoch i. (3) We then select reliable pseudo-labeled data by the searched curriculum. (4) We retrain a new model (Θr+1ω ) using both the labeled samples and selected pseudo-labeled samples. (5) This process is repeated for R rounds.

heading: 3.2 PSEUDO-LABEL SELECTION FOR SEMI-SUPERVISED KEYPOINT LOCALIZATION
text: We denote the labeled dataset with Nl samples as Dl = {( I li ,Y l i )∣∣Nl i=1 } ,where Ii and Yi denote the i-th training image and its keypoint annotations (the x-y coordinates of K keypoints). The Nu unlabeled images are denoted as Du = { (Iui )| Nu i=1 } , which are not associated with any ground-truth keypoint labels. Generally, we have |Nl| |Nu|. Pseudo-labeling based method builds upon the general idea of self-training (McLachlan, 1975), where the keypoint localization network Θω goes through multiple rounds of training. In the initialization round, the model is first trained with the small labeled training set Dtrain = Dl in a usual supervised manner. In subsequent rounds, the trained model is used to estimate labels for the unla-
beled data D̃u = {( Iui , Ỹ u i )∣∣∣Nu i=1 } . Here, we omit the superscript r for simplicity. Specifically, given an unlabeled image Iui , the trained keypoint localization network Θω predicts K heatmaps. Each heatmap is a 2D Gaussian centered on the joint location, which represents the confidence of the k-th keypoint. The output pseudo-labeled keypoint location (Ỹ ui ) is the highest response in the heatmap space. And the confidence score C (Θω(Iui )) is the response value at the keypoint location.
Then, pseudo-label selection process is adopted. Let g = [g1, ..., gNu ] ⊆ {0, 1}Nu be a binary vector representing the selection of pseudo-labels, where gi denotes whether the keypoint prediction on Iui is selected.
gi = { 1, if C (Θω(Iui )) > γ 0, otherwise (1)
where γ ∈ (0, 1) is the confidence threshold. Pseudo-labeled samples with higher confidence are added to the training set.
Dtrain = {( I li ,Y l i )∣∣Nl i=1 } ∪ {( Iui , Ỹ u i )∣∣∣Nu i=1 where gi = 1 } . (2)
Then the keypoint localization network is retrained with a combination of labeled and pseudolabeled training data Dtrain.

heading: 3.3 CROSS-TRAINING STRATEGY
text: In this section, we introduce the cross-training strategy in our curriculum learning framework. In a typical self-training round, the model predicts noisy pseudo-labels which are used in subsequent training stages. Since the pseudo-label prediction is performed on a dataset of known data (on which training is performed), the noise accumulates in a positive feedback loop. This causes the long-standing issue of confirmation bias (Tarvainen & Valpola, 2017). To mitigate this problem, we propose the cross-training strategy. Specifically, we randomly partition the unlabeled data Du into two complementary subsets D(1)u and D(2)u . The partition is fixed across all the training rounds. The self-training conducts alternatively for D(1)u and D(2)u . For r = {1, . . . , R}, when r is odd, we use a combination of D̃(1)u and Dl to train the model, and the trained model predicts pseudo-labels for the next round on D(2)u . When r is even, we use a combination of D̃(2)u and Dl to train the model, and perform pseudo-label prediction on D(1)u . Before each round, the model parameters are re-initialized with random weights following Cascante-Bonilla et al. (2021) to avoid noise accumulation.

heading: 3.4 CURRICULUM RESIDUAL LEARNING
text: Directly learning for R rounds of curriculum parameters separately can be inefficient. Considering that the model is reinitialized and retrained in each self-training round, we assume that the optimal curricula for different rounds should have some similar patterns. We propose a greedy multi-step searching algorithm. For round r, we use the searched curriculum in the previous round r − 1 as the base curriculum to guide the searching of current curricula Γr. Inspired by ResNet (He et al.,
2016), we propose the curriculum residual learning strategy. Formally, we learn a bias term ∆Γr around the base curriculum, i.e. Γr = (Γr−1)∗ + ∆Γr, where (Γr−1)∗ means the searched optimal curriculum for round r − 1, and (Γ0)∗ is initialized with all zeros. We empirically find that this strategy accelerates the model convergence speed and achieves marginally better performance. To further reduce the search space, we choose every G epochs as an epoch group, which shares the same threshold parameters for pseudo-labeled data selection. In total, we have NG epoch groups, and the size of each epoch group is G. Therefore, the search space is reduced by a factor of G, where G = 10 in our implementation.

heading: 3.5 CURRICULUM SEARCH VIA REINFORCEMENT LEARNING
text: Our PLACL can be formulated as an optimization problem shown in Eq. 3. In the inner-loop, we optimize the weights ω of the keypoint localization network Θω to minimize the training loss L (see Alg. 1). In the outer-loop, we apply proximal policy optimization (PPO2) algorithm (Schulman et al., 2017) to search for the curriculum Γ that maximize the evaluation metric ξ (e.g. PCK) on the validation set Dval (see Alg. 2).
max Γ ξ(Γ) = ξ(Θω∗(Γ);Dval), s.t. ω∗(Γ) = arg min ω L(Θω;Dtrain,Γ). (3)
The training consists of R rounds. In each round r, the PPO2 search process consists of T sampling steps. In each step, M sets of parameters are sampled independently from a truncated normal distribution (Nakano et al., 2012; Fujita & Maeda, 2018), ∆Γr ∼ Ntrunc[0,1] ( µrt , σ 2I ) , where µrt and σ2I are the mean and covariance (σ is fixed to 0.2 in practice). These sampled parameters are used to construct M different training curricula for training M keypoint localization networks separately. Then the mean of the distribution is updated by PPO2 algorithm according to the evaluation score of the M networks.
The objective function of PPO2 is formulated in Eq. 4.
J (µr) = Eπ [ min ( πµr ( ∆Γrj ;µ r, σ2I )
πµrt ( ∆Γrj ;µ r t , σ 2I ) ξ̃ (Γrj) , CLIP
( πµr ( ∆Γrj ;µ r, σ2I )
πµrt ( ∆Γrj ;µ r t , σ 2I ) , 1− , 1 + ) ξ̃ (Γrj) )] ,
(4)
where the function CLIP(x, 1− , 1+ ) clips x to be no more than 1+ and no less than 1− . Following the common practice (Li et al., 2021), the mean reward is subtracted for better convergence. ξ̃ ( Γrj ) = ξ ( Γrj ) − 1M ∑M j=1 ξ ( Γrj )
and the policy πµr is defined as the probability density function (PDF) of the truncated normal distribution. PPO2 enforces the probability ratio between old and new policies πµr ( ∆Γrj ;µ r, σ2I ) /πµrt ( ∆Γrj ;µ r t , σ 2I )
to stay within a small interval to control the size of each policy update. We then compute the gradients and update the parameters by µrt+1 ← µrt + α∇µrJ (µr) with a learning rate of α > 0. After T sampling steps, we choose µrt with the highest average evaluation score as (Γr)∗. And (Γr)∗ is used as the base curriculum for the next round. And after R rounds, our final optimal curriculum is obtained, Γ∗ = [(Γ1)∗, . . . , (ΓR)∗].
Training details In the training phase, the keypoint localization network and the curriculum search policy are simultaneously optimized. For the outer-loop, the PPO2 (Schulman et al., 2017) search procedure is conducted for T = 16 sampling steps, and in each step M = 8 sets of parameters (curriculum) are sampled. The clipping threshold is = 0.2, and µrt+1 is updated with the learning rate of α = 0.2. We empirically use R = 6 self-training rounds, and group size G = 10 for curriculum search. For the inner-loop, we follow the common practice (Sun et al., 2019) to train the keypoint localization network with Mean-Squared Error (MSE) loss for N = 210 epochs per round. Adam (Kingma & Ba, 2015) with a learning rate of 0.001 is adopted. We reduce the learning rate by a factor of 10 at the 170-th and 200-th epochs. Although the RL search process increases the training complexity, the total training cost is not too high (only 1.5 days with 32 NVIDIA Tesla V100 GPUs). More detailed training settings for each task are provided in §4.2 and §4.3.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS AND EVALUATION METRICS
text: Datasets: To show the versatility of PLACL, we conduct experiments on 5 diverse datasets. LSPET (Leeds Sports Pose Extended Dataset) (Johnson & Everingham, 2010; 2011) consists of images of people doing sports activities. We use 10,000 images from (Johnson & Everingham, 2011) for training and 2,000 images from (Johnson & Everingham, 2010) for validation and testing. MPII Human Pose dataset (Andriluka et al., 2014) is a well-known benchmark for human pose estimation. The images are collected from YouTube videos, showing people doing daily human activities. We follow (Moskvyak et al., 2020) to use 10,000 random images from MPII train for training, 3,311 images from MPII train for validation and MPII val for evaluation. CUB-200-2011 (CaltechUCSD Birds-200-2011) (Welinder et al., 2010) dataset is a well-known dataset for SSL. It consists of 200 fine-grained bird species with 15 keypoint annotations. We follow (Moskvyak et al., 2020) to split dataset into training (100 categories with 5,864 images), validation (50 categories with 2,958 images) and testing (50 categories with 2,966 images). ATRW (Li et al., 2019b) dataset contains
images of 92 Amur tigers captured from multiple wild zoos in challenging and unconstrained conditions. For each tiger, 15 body keypoints are annotated. The dataset consists of 3,610 images for training, 516 for validation, and 1,033 for testing. MS-COCO’2017 (Lin et al., 2014) is a popular large-scale benchmark for human pose estimation, which contains over 150,000 annotated people. We randomly select 500 images from COCO train for validation, the remaining training set (115k images) for training, and COCO val (5k images) for evaluation. We use this dataset to validate the applicability of our approach on large-scale data. AnimalPose (Cao et al., 2019) dataset contains 5,517 instances of five animal categories: dog, cat, horse, sheep, and cow. It consists of 2,798 images for training, and the 810 images for validation and 1,000 images for testing. We use it to test the generalization and domain-transfer capacity of our proposed method.
Evaluation Metrics: PCK (Probability of Correct Keypoint): A detected keypoint is considered correct if the distance between the predicted and true keypoint is within a certain threshold (αl), where α is a constant and l is the longest side of the bounding box. We adopt PCK@0.1 (α = 0.1) for LSPET (Johnson & Everingham, 2011), CUB-200-2011 (Welinder et al., 2010), and ATRW (Li et al., 2019b) datasets. PCKh is adapted from PCK, where l is the head size that corresponds to 60% of the diagonal length of the ground-truth head box. We adopt PCKh@0.5 (α = 0.5) for MPII (Andriluka et al., 2014) dataset. Standard AP (Average Precision) is another commonly used evaluation metric. It is based on object keypoint similarity (OKS), which measures the distance between predicted keypoints and ground-truth keypoints normalized by the scale of the object. We use mAP for AnimalPose (Cao et al., 2019) datasets.

heading: 4.2 COMPARISONS WITH THE STATE-OF-THE-ART SSL APPROACHES
text: In Table 1, we compare with the supervised baseline (HRNet (Sun et al., 2019)) and other stateof-the-art SSL approaches. We experiment with different percentages of labeled images (5%, 10%, 20%, 50%, and 100%). For fair comparisons, all results are obtained using HRNet-w32 backbone with the input size of 256× 256. We follow (Moskvyak et al., 2020) to prepare datasets and exclude half body transforms, and testing tricks (post-processing, and flip testing).
Comparisons with consistency regularization methods. ELT (Honari et al., 2018) (equivariant landmark transformation) loss encourages the model to output keypoints that are equivariant to input transformations. Gen (Jakab et al., 2018) learns to extract geometry-related features through conditional image generation. SSKL (Moskvyak et al., 2020) learns the pose invariant keypoint representations with semantic keypoint consistency constraints. These consistency regularization methods have shown superior results over the supervised baseline, however, they are inferior to our PLACL method on all datasets and different percentages of labeled samples. Especially we show that PLACL is mostly effective for low data regimes. For example, in CUB-200-2011 dataset, PLACL with only 5% labeled data achieves better performance (93.01 vs 92.80) than SSKL with 50% labeled data. And in LSPET dataset, we show that PLACL improves the performance of baseline by a large margin from 40.19 to 70.76 with 5% labeled images.
Comparisons with pseudo-labeling method. We also compare with a pseudo-labeling (PL) baseline (Radosavovic et al., 2018). Overall our proposed PLACL significantly outperforms the pseudolabeled (PL) baseline on all datasets. As pointed out by Moskvyak et al. (2020), the vanilla pseudolabeling approach does not perform well for the keypoint localization task with a low data regime, due to the lack of an effective pseudo-label selection scheme. Previous PL approaches generally require a relatively large (> 50%) labeled dataset to generate reliable pseudo-labels. Instead, PLACL is able to automatically select high-quality pseudo-labeled samples, which is the key to the success of pseudo-labeling based methods.
Comparisons with curriculum-learning method. Curriculum Labeling (CL) (Cascante-Bonilla et al., 2021) is a recently proposed approach that applies a hand-crafted curriculum to facilitate training of SSL. We observe that our proposed PLACL significantly outperforms CL, which validates the effectiveness of our proposed automatic curriculum learning.
Experiments on large-scale datasets. Inspired by Zhou et al. (2020), in order to decrease the RL curriculum search cost for the large-scale MS-COCO (Lin et al., 2014) and full MPII (Andriluka et al., 2014) datasets, we use a light proxy task with reduced number of training samples (5k) for RL curriculum search. After the search procedure, we re-train the keypoint networks with the searched
curriculum on the full training set and evaluate them on the test set. Please refer to A.4 for more analysis about proxy tasks and A.5 for experiments on the full MPII (Andriluka et al., 2014) dataset.

heading: 4.3 EVALUATION OF GENERALIZATION CAPACITY
text: Generalization to different domains. We investigate the generalization ability of our proposed model on domain transfer. To this end, we conduct experiments on AnimalPose (Cao et al., 2019) dataset in Table 2. Specifically, we follow (Cao et al., 2019) to choose one animal class (e.g. cat) as the target domain and the remaining four classes for the source domain. The images of the source domain are fully annotated, while the images of the target domain are unlabeled. For fair comparisons, we adopt the AlphaPose model (Fang et al., 2017), which uses ResNet-101 (Xiao et al., 2018) as the backbone. All models are trained with the pose-labeled human dataset involved. The AlphaPose baseline model is pre-trained on the human dataset and fine-tuned on the labeled source animal data. For pseudo-labeling based approaches (Inoue et al., 2018; Cao et al., 2019), the unlabeled target animal data is used for pseudo-labeling. For domain adaptation approaches (Tzeng et al., 2015; Long et al., 2016), the unlabeled target data is used for domain transfer. Please refer to Cao et al. (2019) for details about the compared approaches. We observe that our proposed approach consistently outperforms the previous state-of-the-art methods on cross-domain semi-supervised learning.

heading: 4.4 ANALYSIS
text: Number of self-training rounds. Along with the increasing of self-training rounds (R), the quality of the pseudo-labels gradually improves (see Fig. 3) and the test accuracy increases (see Fig. 2a) until saturation. The experiments are conducted on multiple datasets with 5% labeled data. Interest-
ingly, different datasets require a different number of rounds to achieve optimal, because four-legged animals (ATRW (Li et al., 2019b)) have more pose variations than birds (CUB-200-2011 (Welinder et al., 2010)). We use R = 6, because further increasing R does not bring significant gains.
Comparisons with pseudo-labeling with different fixed thresholds. As shown in Fig. 2b, we compare our PLACL with 10 static thresholds from 0.0 to 0.9. We observe that PLACL clearly outperforms all these fixed threshold alternatives. Moreover, we find that the accuracy will be significantly affected by different thresholds (61.01% PCK for γ = 0.0 vs 67.35% PCK for γ = 0.7).
Choice of epoch group size. In Fig. 2b, we also compare the performance of different epoch group sizes (G). We empirically find that smaller G will produce better performance, but at the cost of increased search space. We choose G = 10 to trade-off between accuracy and efficiency.

heading: 4.5 ABLATION STUDIES
text: In Table 3, we present ablation studies to measure the contribution of each component. All ablative experiments are conducted on LSPET (Johnson & Everingham, 2011) dataset with 5% labeled data. PCK@0.1 is adopted as the evaluation metric.
Effect of cross-training strategy. We find that without cross-training strategy the accuracy significantly drops from 70.76 to 67.13, due to noise accumulation over time. Especially, we find that there is little to no improvement after the first self-training round.
Effect of curriculum learning. We compare PLACL with the alternative that only searches for a fixed threshold via RL. We find that using dynamic thresholds improves upon the fixed-threshold alternative by a large margin, which validates the effectiveness of curriculum learning.
Effect of parameter search. We compare PPO2 (Schulman et al., 2017) search with Random Search (w/o PPO2 search). We randomly sampled T ×M curricula and pick out the best one for comparisons. PPO2 search obtains much better performance (70.76 vs 65.42). This indicates that the searching problem is non-trivial and that our searching algorithm is very effective.
We also compare with manually designed curricula whose thresholds are gradually decreasing on the epoch level. We tried five curricula with different decrease slopes and reported the best one as Manually Design. We observe that PLACL significantly outperforms the manually-designed curriculum baseline which validates the effectiveness of automatic curriculum search.

heading: 5 CONCLUSIONS
text: In this paper, we propose a novel Pseudo-labeled Auto-Curriculum Learning (PLACL) for the task of semi-supervised keypoint localization. We propose to learn a curriculum to automatically select reliable pseudo-labels and propose cross-training strategy to mitigate the confirmation bias problem. Extensive experiments on 6 diverse datasets validate the effectiveness and versatility of the proposed method. The ideas of pseudo-labeling and consistency regularization are complementary and we will explore the combination of these two techniques in the future. We believe that our proposed approach is generic and we plan to investigate the applicability of PLACL on other visual tasks, such as object detection and semantic segmentation.

heading: A APPENDIX
text: A.1 PSEUDO-CODE FOR PLACL ALGORITHM
Here we present the pseudo-code for the proposed Pseudo-Labeled Auto-Curriculum Learning (PLACL) algorithm. In PLACL, the keypoint localization network and the curriculum policy are jointly optimized. In the inner-loop, we optimize the keypoint localization network as shown in Algorithm 1. In the outer-loop, the curriculum policy is updated according to the performance of the keypoint localization network, as shown in Algorithm 2.
Algorithm 1: Inner-loop network training Input:
1. Labeled data Dl, pseudo-labeled data D̃u. 2. Number of training groups per round NG; 3. Curriculum Γ;
Output: Obtained optimal network weights ω∗(Γ). Random initialization of network weights ; for g = 1 to NG do
Update current threshold γg = Γ[g] ; Compute data selection vector g (Eq. 1) ; Construct the training set Dtrain (Eq. 2) ; Update network weights ω via back-propagation;
return ω∗(Γ) ;
Algorithm 2: Pseudo-Labeled Auto-Curriculum Learning (PLACL) Input:
1. Labeled data Dl, unlabeled data Du. 2. Initialized distribution µ and σ2. 3. Number of rounds R. 3. Searching steps T ; Sampling number M . 4. Evaluation metric (e.g. PCK@0.1) ξ.
Output: Obtained the optimal curriculum Γ∗ and the final network Θω∗ . Initialize (Γ0)∗ with all zeros. Pre-train keypoint localization network Θ0ω using labeled data Dl. for r = 1 to R do
if r%2 == 1 then Predict keypoint pseudo-labels D̃(1)u with Θr−1ω∗ ; D̃u = D̃(1)u else Predict keypoint pseudo-labels D̃(2)u with Θr−1ω∗ ; D̃u = D̃(2)u for t = 1 to T do for j = 1 to M do
Sample parameter ∆Γrj,t ∼ Ntrunc [0,1] ( µrt , σ 2I )
; Γrj,t = ∆Γ r j,t + (Γ
r−1)∗ ; Get Θrω∗,j,t via inner-loop network training using Γ r j,t (Alg. 1) ;
Compute the evaluation metric ξ ( Γrj,t )
= ξ(Θrω∗,j,t;Dval) ; Compute the objective function J (µ) (Eq. 4); Update µrt+1 ← µrt + α∇µrJ (µr) ;
(Γr)∗ = arg maxµrt ∑M j=1 ξ ( Γrj,t ) ,∀t = 1, . . . , T ; Get Θrω∗ via network training using (Γ r)∗ (Alg. 1) ;
return Γ∗ = [(Γ1)∗, . . . , (ΓR)∗] and ΘRω∗ ;
A.2 VISUALIZATION OF PSEUDO-LABELED SAMPLES
In order to provide a better illustration of how pseudo-labels evolve in self-training rounds, we visualize some pseudo-labeled samples for different datasets. We observe that the quality of pseudolabels gradually improves with the increase of the self-training rounds.
Round 0 Round 1 Round 2 Round 3 Round 4
A.3 GENERALIZATION TO DIFFERENT KEYPOINT LOCALIZATION MODELS.
Table 4 shows the improvement when PLACL is applied to the recent state-of-the-art keypoint localization models which vary in model architectures and training/testing techniques. The experiments are conducted on LSPET dataset with 5% labeled images and 95% unlabeled images. We show that PLACL consistently improves the performance of the state-of-the-art approaches by a large margin. PLACL does not require any knowledge of the keypoint localization models, making it easy to use in practice.
A.4 ANALYSIS OF PROXY TASKS
There are a lot of methods that target at improving the searching efficiency in literature, e.g. using reduced-training proxy tasks (input size, model size, training samples, and training epochs can be reduced (Zhou et al., 2020)). With these techniques, we are able to get orders of magnitude less computation cost, but can still match the performance. Therefore, we believe that the scalability is not a problem. For example, in order to further reduce the complexity, we use a light proxy task (with reduced training samples) for the RL search process. Specifically, we randomly select a small proportion of the training data (e.g. 5k images) for efficient curriculum search. After the search procedure, we re-train the keypoint networks with the searched curriculum on the full training set and evaluate them on the test set.
As shown in Table 5, we randomly select different number of training images (5k and 10k) for RL curriculum search. We find that reducing the number of training images by half (from 10k to 5k) does not decrease the final performance much, which validates the effect of using proxy tasks. Such a strategy enables us to apply the proposed PLACL approach to large-scale datasets, such as MS-COCO (Lin et al., 2014) and the full MPII (Andriluka et al., 2014) datasets.
A.5 EXPERIMENTS ON THE FULL MPII DATASET
As shown in Table 6, we provide the results on the full MPII (Andriluka et al., 2014) dataset. Since the codes of SSKL (Moskvyak et al., 2020) are not publicly available, we only compare with CL (Cascante-Bonilla et al., 2021) in the experiments. We see that our proposed PLACL consistently outperforms CL, especially for low labeled data regime (5% and 10%). Note that our results on the full MPII are obtained using reduced-training proxy tasks, i.e. we use 5K images for RL curriculum search, and re-train the model with the obtained curriculum on the full training set.

Review summaries:
1. This paper introduces Curriculum Learning to semi-supervised keypoint localization, which is an automatic pseudo-labeled data selection method. The method uses reinforcement learning to learns a series of dynamic thresholds.   2. Besides, this paper proposes the cross-training strategy for pseudo-labeling to alleviate confirmation bias.   3. The experiments shows that the proposed method can effectively improve the performance in different dataset and surpass other semi-supervised methods. 

A semi-supervised learning method (PLACL) is proposed. This method employs a Pseudo-Labeling (PL) approach. It consists in iteratively (these iterations are called rounds):  1. predicting pseudo-labels to unlabeled data using the current model, 2. training a series of models from scratch using the labeled data and selections of pseudo-labeled data. The pseudo-label selection is performed using a series of thresholds (called curriculum) over the scores output by the model.  The authors propose an Auto-Curriculum Learning (ACL) strategy to automatically update the curriculum using a reinforcement learning approach (PPO2). The fact that the curriculum is updated at each round is called "Curriculum Residual Learning". They also employ a cross-training strategy to prevent the issue of confirmation bias.  The performances of PLACL are evaluated on a keypoint localization application on 5 different datasets. When the percentage of labeled data is very low (5% or 10%), PLACL outperforms the state of the art semi-supervised learning competitors. 

The paper introduces a method for semi-supervised keypoint localization based on pseudo-labeling with auto-curriculum learning. The Auto-curriculum learning approach learns a series of dynamic thresholds for automatic selection of high-quality pseudo-labeled examples for model retraining. The reinforcement learning (RL) framework, more specifically, the proximal policy optimization algorithm, is used to search for the optimal curriculum. The method is evaluated on four benchmarks in keypoint localization.