Modeling the Uncertainty with Maximum Discrepant Students for Semi-supervised 2D Pose Estimation
Authors: Jiaqi Wu
Junbiao Pang
Qingming Huang
Venue: Submitted to ICLR 2023 (2023)
Keywords: semi-supervised learning
pose estimation
uncertainty
convolutional neural networks

Abstract: Semi-supervised pose estimation is a practically challenging task for computer vision. Although numerous excellent semi-supervised classification methods have emerged, these methods typically use confidence to evaluate the quality of pseudo-labels, which is difficult to achieve in pose estimation tasks. For example, in pose estimation, confidence represents only the possibility that a position of the heatmap is a keypoint, not the quality of that prediction. In this paper, we propose a simple yet efficient framework to estimate the quality of pseudo-labels in semi-supervised pose estimation tasks from the perspective of modeling the uncertainty of the pseudo-labels. Concretely, under the dual mean-teacher framework, we construct the two maximum discrepant students (MDSs) to effectively push two teachers to generate different decision boundaries for the same sample. Moreover, we create multiple uncertainties to assess the quality of the pseudo-labels. Experimental results demonstrate that our method improves the performance of semi-supervised pose estimation on three datasets.

Under review as a conference paper at ICLR 2023
MODELING THE UNCERTAINTY WITH MAXIMUM DIS- CREPANT STUDENTS FOR SEMI-SUPERVISED 2D POSE ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-supervised pose estimation is practically challenging task for computer vi- sion. Although numerous excellent semi-supervised classification methods have emerged, these methods typically use confidence to evaluate the quality of pseudo- labels, which is difficult to achieve in pose estimation tasks. For example, in pose estimation, confidence represents only the possibility that a position the heatmap is keypoint, not the quality of that prediction. In this paper, pro- pose a simple yet efficient framework to estimate the quality of pseudo-labels semi-supervised pose estimation tasks from the perspective of modeling the certainty of the pseudo-labels. Concretely, under the dual mean-teacher frame- work, construct the two maximum discrepant students (MDSs) to effectively push two teachers to generate different decision boundaries for the same sample. Moreover, we create multiple uncertainties to assess the quality of the pseudo- labels. Experimental results demonstrate that our method improves the perfor- mance semi-supervised pose estimation on three datasets. The project code and our dataset are publicly available on https: //::///11 thuc. com/ Q1201 MDSS/trres/master
INTRODUCTION
As an essential task in computer vision, pose estimation has been widely used in various fields such as action recognition, security monitoring, animal experimentation. Most pose estimation methods use MSE loss to transform keypoint location predictions into Gaussian heatmap regression. State-
arrthc-art methods (Zhang et all (20211) Zhang et al, (20020))) Huang et al. (2I))) Lictal (2019))
Cheng et al. (2019). Chen et all (2018), Xiao et all (2018). Tang et all (2018), Newell et all (2017))
Cao et all 72017)) Newell et al (2016) Wei el all (2(T6) require a large amount labeled data.
However, labeling pose keypoints is costly and inme-intensive labor for many practical tasks. How to effectively train model without sufficient labeled data has attracted attention in the field of pose estimation.
simple idea is to utilize semi-supervised classification methods for the pose estimation task. Nu- merous outstanding research have emerged in the field of semi-supervised classification, which can be categorized into methods based on consistency constraints and pseudo-labels. For example, Mry- atl (2018), Sajadi al, (2016) mainly employ rational stochastic transformations and pertur- bations to improve the gain, which are derived from consistency constraints. Meanwhile, Iaavamen Valpola (2017), Laine & AiR (20T) utilize high-quality pseudo-label filters from the predictions of unlabeled samples. In addition, the two methods are combined for better performance, such as Zhang et al] (20211) Chen et all (2020), Berthelot et all (2019).
However, semi-supervised classification methods commonly evaluate the quality of generated pseudo-labels based on confidence, such as Ke all (2019) which cannot be applied to semi- supervised pose estimation. In the pose estimation task, confidence can only be used to obtain the keypoint locations of the heatmap predicted by the model, not to evaluate the quality of the predic- tion. As shown in Fig. IL with the increase of epochs, the model's prediction confidence of this key point gradually increases, but the error of this prediction does not effectively decrease. This phenomenon indicates that confidence in pose estimation is not directly related to the quality of pre-

---

Under review as a conference paper at ICLR 2023
confidence -12
Con confocance
Commoon
Emo
-14
-84
02
280
403
Egoch
Figure 1: Error and confidence curves for two keypoint pseudo-labels, 1106_5 and 2077_4 It suggests that confidence is not directly related to the accuracy the prediction itself. Therefore, confidence does not effectively assess the quality of pseudo-labels.
dictions from the model, and thus cannot be used to assess the quality of pseudo-labels, With these noisy and even erroneous pseudo-labels, the performance naturally deteriorates. That is, the perfor- mance of semi-supervised learning is lower than that of supervised learning with the same amount of labeled data. Therefore, effectively selecting high-quality pseudo-labels remi-supervised pose estimation is an essential and challenging task.
Therefore, we construct dual mean-teat mean-teacher structure evaluate the quality the pseudo-label by computing the distance between the predicted keypoint positions of multiple pose estimators to measure the discriminative power of the model on the samples. Furthermore, high-quality pseudo- labels are added to the training dataset to gradually train the model. To avoid coupling between multiple pose estimators. we introduce a simple and effective method to build the maximum differ- regrestors (MDRs) by adding parameter adversarial mechanism between two students. When the parameters each regressor are different, is possible to ensure that their decision boundaries do not overlap, thus helping ensure the accuracy of the evaluated uncertainty. Meanwhile, the parametric adversarial mechanism can balance the performance the two MDSs to guarantee the quality of pseudo-labels. The key contributions of this work are:
We propose semi-supervised pose estimation method based on the Dual Mean-Teacher framework. We construct multiple uncertainties to evaluate the quality of pscudo-labels generated by the model and efficiently extract high-quality pseudo-labels for further model training.
We propose moodel-parameee adversarial mechanism effectively increase diversity and avoid the degradation of uncertainty assessment quality caused by the coupling of multiple pose regressors.
Our experiments show that MDSs effectively select high-quality pseudo-labels without using confidence as an evaluation criterion and achieve better performance on the three datasets.
RELATIVE WORK
2.1 2D POSE ESTIMATION
In terms of organization of supervised information, 2D pose estimation methods can be divided into keypoint regression- based methods and heatmap-based methods. For example, [ooshv Szegedy (2014) directly regress the keypoint positions using the coordinate values of the keypoints as su- pervision information. Moreover, Huang et al. (2019), Newell all (2(16) and Wel WeteLal (2016) are heatmap based keypoint estimation methods. The keypoint coordinate values are converted into heatmap as supervised information to train the model, and then the appropriate position is selected from the heat output by the model the position prediction of the keypoints according confi- dence. We choose ceatmap-based methods because direct coordinate regression is more difficult and less favorable for semi-supervised pose estimation.
2

---

Under review as a conference paper at ICLR 2023
2.2 SEMI-SUPERVISED LEARNING
Many excellent semi-supervised methods have emerged in the field of classification. For example, Mıyato et (2018), Sajjadi et all (201)) consistency constraint based methods. Crucial this approach is how to efficiently improve the effect of consistency constraints through reasonable data augmentation and perturbation, thus obtaining more supervised information from unlabeled data to improve model performance. While Tarvanen & Valpola (20T7)) Laine & Aila (20T6), based on the idea of self-training generate pseudo-labels from the model to supervise their training. So this ap- proach relies on how to generate and select high-quality pseudo-labell In addition, the two methods are combined for better performance, such as Zhang el al. (202T), Chen al (M12)) Berthelor all (2019). The core idea of Ge all (2020) is the composition of two mean-teachers providing each other with pseudo-labels, Our work differs from it in that we maximize the difference in parameters between the two students to improve the discrepancy between the two models. We combine the two using the Teacher. Our model not only obtains supervised information based on consis- tency constraints from unlaheled data, but also generates high-quality pseudo-labels through mutual generalization of teacher and student models.
2.3 QUALITY ASSESSMENT OF PSEUDO-LABELS
superviper classification methods normally filter sulf-super information by combining uncertainty and confidence, such as Ke et all (2019), Sohn all (2022). This is because uncertainty commonly quantified by evaluating the consistency of multiple predictions for the same sample, which requires the results of each prediction to be plausible. However, unlike the classification task, confidence in pose estimation can only be used to obtain keypoint locations from heatmaps, not to evaluate the quality of model predictions.
Yang et all (2021)) calculates the uncertainty the pseudo-labels a predictive way. contrast, our work proposes that confidence should no longer be used an evaluation standard when evaluating the quality of pseudo-labels, and we use the tri-uncertainty to replace confidence.
Therefore, method evaluates the quality of pseudo-labels through uncertainty without confi- dence. To ensure uncertainty accuracy, we construct an uncertainty evaluation network based on the dual Teachre structure and add model parameter adversarial mechanism that the decision surfaces between multiple pose estimators do not overlap.
3 PROPOSED METHOD
3.1 SEMI-SUPERVISED POSE ESTIMATION
The supervised pose estimation is to achieve accurate predictions of the position of K keypoints. Most state-of-the-art methods convert the position of keypoints as heatmap H. H a distribution, encoding the possibility that each pixel in image I is the keypoint. Denote the labeled training dataset as L = {(I', H') " As a result, the prediction of keypoints' locations converted as the Gaussian heatmap regression by minimize the MSE loss between the prediction and its ground-truth.
The pose loss for labeled data is as follows:
L/ = EIe ( - 2|
where f(10)) is the model.
Comparing with supervised learning, the semi-supervised pose estimation solves the problem of less amount of labeled data. Denote the unlabeled training dataset 24 -I"" For unla- beled data, the ground-truth is the pseudo-label generated the semi-supervised learning. The pose loss for the unlabeled data is as follows:
D Eieu [[ H
We use Mean Teacher (MT) [arvamen & Valpola (2017) in semi supi-vupee pose estimation. In MT, the teacher generates the pseudo-labels for unlabeled data to train the student, the parameters of
3

---

Under review as a conference paper at ICLR 2023
MDS1
Teacher
Consistency Loss
heatmap
Uncortainty Estimator
Student 1
Pase Loss
Selected Psaudo-labels
Input
Adversarial
healmap
Student
EMA
Consistency Loss
Teacher 2
Forward
Rank
EMA
MDS2
Backward
Coord
Adversarial
Parameters
Figure 2: Overview of the proposed maximum discrepant students (MDSs) framework.
teacher obtained from student by the Exponential Moving Average (EMA) strategy. The EMA as follows:
0 = =f a
(3)
where 0z. O' are the teacher's and student's parameters in / epoch, respectively, a is a smoothing coefficient cryper-parameter Taking advantage of EMA, teacher can generate more accurate pseudo- labels.
The quality of the pseudo labels is related to the predictive power of the generator at each epoch. However, models in semi-supervised learning typically use single regressor to generate pseudo- labels, such as the teacher MT. A single regressor obtains barely additional information. With small amount of labeled data, the model tends to under- or over-fit, fit, and thus generate noisy pseudo- labels. Therefore, an effective quality evaluation method for pseudo-labels is needed to pick up high-quality pseudo-labels to train the model.
3.2 MAXIMUM DISCREPANT STUDENTS
The MDSs framework is illustrated in Fig. M It is composed of two MDS modules, and uncer- tainty estimator E. Both students and teachers use the same pose estimation model. each MDS, the teacher predicts the pscudo-label to train the student and uses the Exponential Moving Average (EMA) weights of the student. The uncertainty estimator evaluates the uncertainty the pseudo- label by calculating the distance between the discrepancy of two students S and S2 by the parameters adversarial loss, as equation F
predictions from teachers Ti and Tg. We maximize
Each epoch, teachers generate the pseudo-labels of unlabeled data. The uncertainty estimator E estimates the uncertainty of these pseudo-labels, and evaluates the quality of these pseudo-labels by the uncertainty. The top k high-quality pscudo-labels are added into labeled dataset. Then two students are trained by this expanded labeled dataset.
3.3 OBJECTIVE FUNCTION
In supervised pose estimation, the MSE loss is used to train the model. In MDSs, we directly use the ground-truth to calculate pose loss for labeled data. For unlabeled data, we use the pseudo-labels generated by teacher 72 to calculate the pose loss of student Si, which is follows:
Lpe, Erec ((I)ss)) HIl2 + Ereu [f[ 12.
where 8s,, 0T, are the parameter of the student S₁ and the teacher T2. respectively. Hy, is the pseudo-label generated by the teacher Tz.
4

---

Under review as a conference paper at ICLR 2023
Correspondingly, the pose loss of student S2 is calculated with the pseudo labels generated by teacher T1, which is as follows
Lps, Blec f(1 Hil? + Ereu [f.( Hy, 11?
(5)
where 02. Or, are the parameter of the student S2 and the teacher Ti, respectively. Hn is the pseudo-label generated by the teacher Tj.
In real life, when two experts in the same field have different answers to the same question, it is most likely that the question itself is very uncertain. Inspired by this, the MDSs framework is based on two constraints.
On one hand, two teachers are used evaluate the uncertainty of the same sample. achieve this goal, the first constraint is that two teachers should have balanced performance, as equation ! and equation
On the other hand, two students should keep divergences. So, another constraint is the parameler udversarial loss, through which we ensure that the two experts are not homogenized. To maximize the discrepancy between two MDSs, minimize the cosine distance of students S₁ and S2 by the parameters adversarial loss:
La = 0s. 092
(6)
10s. |00011
Otherwise, following the semi-supervised approach, the consistency loss to updating S and S2 is as follows:
L. Ereccal - ((1 + - s(T(0T,)13
We summarize the objective function as follows:
\Lps + Lps,) ) daLa + AcLes
(8)
where Ap. Aa and d₂ are the weights to balance all losses.
3.4 QUALITY PSEUDO-LABELS
Crucial in the semi-supervised pose estimation task is how to judge the quality of pseudo-label.
First, we take the prediction f[I of the teacher Or on the raw sample as the pseudo-truth value of the pseudo-label. This is because, in pose estimation tasks, the prediction accuracy of raw samples is much higher than that of augmented samples, which are generated by data augmentation perturbation.
Second, to evaluate the quality of pseudo-labels, we consider the following two points:
For high-quality pseudo-labels, the model should have predictive consistency for different augmented samples of the raw sample. A model has a high epistemic capacity on raw samples if its prediction uncertainty on augmented samples is low. In turn, high uncertainty implies low epistemic capacity of the model for the raw sample. Therefore, we obtain M randomly augmented samples of this raw sample and calculate the distance between the predictions of these augmented samples to quantify the uncertainty of the corresponding pseudo-label. The lower the uncertainty, the higher quality of the pseudo-labels.
For high-quality pseudo-labels, the model should have predictive stability for the raw sam- ple and its augmented samples. Therefore, we evaluate the quality of the pseudo-labels by computing their uncertainties via moving weighted average.
3.5 THE QUANTIFICATION OF UNCERTAINTY
To better assess the quality of pseudo-labels, we construct triplet uncertainties, which are augmented internal uncertainty unc und augmented external uncertainty unc and raw internal uncertainty ex

---

Under review as a conference paper at ICLR 2023
The augmented internal uncertainty is the average distance between the predictions (P) {f
((("")0))]))) (r))m=) generated by the teacher on the M augmented samples.
unc
M(D(C((D)
sug
where M(-) is to calculate the mean value from a distances set, D is function to calculate the Euclidean distance between two coordinates, is a combinatorial function, selecting prediction of all, non-repeatedly.
The augmented external uncertainty is the distance between the mean prediction pør, of one teacher Tí and the mean prediction Por, of another teacher T2 on the same set of augmented samples.
unc
D(Por, Por, )
(10)
aug
The raw external uncertainty is the distance between the prediction por, of one teacher and the prediction por, of another teacher on the same raw sample.
unc D(por, P077))
(11)
When selecting high-quality pseudo-labels we require that all three uncertainty values should be smaller than the threshold = 3.0.
4 EXPERIMENTAL RESULTS
4.1 DATASET
The FLIC Mathis et (2018) dataset consists of 5003 images taken from movies, including 3987 training data and 1016 test data. Most images have single instance and few have multiple in- stances. Each instance contains 11 keypoints.
our experiments, we use single instance samples, of which 1000 are used the training set and 500 are used the testing set. Fifty percent of the samples in the training set are labeled. We selected keypoints on the left shoulder and left waist to participate in the experiment. When computing the PCK, shoulder and the left waist as the scale factor. The metric of PCK @0.2 is reported.
the distance between the left
The Pranay Sapp & Taskar (2013) dataset is public mouse dataset consisting of 1000 images with one instance per image. Each mouse instance has four keypoints such as nose, left ear, right ear, and tail root. In our experiments, we use 100 samples the training set and 500 samples as the test set. Thirty percent of the samples in the training set are labeled. When computing the PCK, we use the distance between the left ear and the right ear as the scale factor. The metric of PO. is reported.
The Mouse dataset was captured by ourselves from real experimental environments. The dataset contains 1000 RGB images with 256 x 256 pixels. The pose annotation of the image consists of nine keypoints such as nose, left eye, right eye, left ear, right ear, head, tail root, tail and tip. In our experiments, we use 100 samples as the training set and 500 samples as the test set. Thirty percent of the samples the training set are labeled. When computing the PCK, we use the distance between the left ear and the right ear as the scale factor. The metric of PCK @0.2 0.2 is reported.
4.2 IMPLEMENTATION DETAILS
We experimented with the Stacked Hourglass (HG), Mean Teacher (MT), Easy-Hard Augmentation (EHA) and our MDSs on the FLIC, Pranav and Mouse datasets, respectively. Among the methods, HG is supervised learning and the remaining three methods are semi-supervised learning.
Stacked Hourglass (HG). Newell al (016 the HG, the number of stacks is 3, and other set- tings are the same as these in Newell et all (20G).
This datasct is available at this project address.

---

Under review as a conference paper at ICLR 2023
62-
ACU/TC
BM
MU55
Sporh
D1A
HCS
M055
12-
DIAA
MDD5
Epech
Epechs
Figure 3: The error and PCK @0.2 of different methods three datasets, respectively. HG Newell cL (2016) is supervised learning. MT Tarvanen & Valpola (2MT7). Easy-Hard Augmentation (EHA) Xie
(2022)) and MDSs are semi-supervised learning. Left Colomn: Result FLIC dataset, 50% of the 1000 samples are labeled. Middle Colomn: Result on Pranav dataset, 30% of the 100 samples are labeled. Right Colomn: Result on Mouse dataset, 30% of the 100 samples are labeled.
Mean Teacher (MT). Tarvainen Valpola (2017) In the MT, we change the classification model to Stacked Hourglass Newell all (2016). Moreover, the number of stacks is 3. The initial weight of the consistency loss between teacher and student is 0.0, and after 50 epochs, it increases from 0.0 to 20.0. The initial value of pose loss is the constant 10.
Easy-Hard Augmentation (EHA). etal (2021) EHA based on the idea of using predictions of masy-to-augment samples to supervise the predictions of hard-to-augment samples. Thus, as mentioned cf (2021D, we constructed easy-hard augmentation pair with random rotations of 30F and 60F. The initial weight of the consistency loss between Easy and Hard models is 0.0, which increased from 0.0 to 20.0 after 50 epochs. The weight of pose loss is constant 10.
MDSs. We use Stacked Hourglass the pose estimation model, the number of stacks is 3. In the pseudo-label generation process, we generate different data-augmented samples for each sam- ple to calculate its uncertainty. Moreover, when selecting high-quality pseudo-labels, the triple- uncertainty's 's common threshold is 3.0. The value of Xp is the constant 10. The initial value of X 0.005 and after 500 epochs decreases 0.0. The initial value of A is 0.0, and after 50 epochs, increases from 0.0 to 20.0.
In all of the above experiments, all input images are resized 256 256 pixels. Moreover, use the data augmentation that includes random rotation (+/ 30 degrees) (except EHA, which has been indicated in its paragraph), and random scaling (0.75 1.25) and random horizontal flip. The training batch size is two and the total number of training epochs is 500.
4.3 COMPARISONS WITH OTHER METHODS
shown in Table MDSs achieve the best results, both in terms of enror and PCK @0.2, on both Mouse and Pranav datasets. MDSs achieve better results on the FLIC dataset in terms of error, with at least a 3-points improvement over the other methods. The error and PCK @0.2 curves for different models on three datasets shown in Fig. & It can be seen that, on Pranav and Mouse datasets, MDSs perform considerably better than the other methods in terms of both PCK @0.2 and Error curves throughout the training period. On the FLIC dataset, the lack of labeled data makes it difficult to generate pscudo-labels with high accuracy, as the model's 's knowledge the samples is not sufficient. However, then, the curve of the MDSs still better than others.

---

Under review as a conference paper at ICLR 2023
Accuracy
Accuracy pseudo-labels filtered by uncertainty and confidence
0.2
Accuracy of pseudo-labels filtered by uncertainty
Accuracy of pseudo-labels filtered confidence
Accuracy of model on test dataset
0.0
Epoch
Error of pseudo-labels filtered by uncertainty and confidence
of pseudo-labels filtered uncertainty
Error of pseudo-labels filtered by confidence
Error of model on test dataset
Error
100
200
500
Epoch
Figure 4: The PCK @02 (Top) and error (Bottom) of different methods on the Mouse datasets.
Table 1: MSE and PCK@0) of different methods FLIC, Pranav, Mouse Dataset. "1000*0.5"
represents the represents 1000 samples as the training set, 50% which are labeled.
Method
Training set
MSE
PCK@0.2
supervised Newell all (2016)
29.70
0.477
Mean-Teacher Tarvainen & Valpola (2017))
FLIC 1000+0.5
33.68
0.469
EHA Xie eL (2(21)
32.67
0.428
Ours
27.47
0.420
supervised Mean-Teacher
9.39
0.485
Pranav 100*0.3
6.54
0.554
EHA
7.18
0.502
Ours
5.66
0.660
supervised
5.29
0.583
Mean-Teacher
Mouse 100*0.3
4.75
0.627
EHA
5.18
0.482
Ours
4.10
0.687
8

---

Under review as a conference paper at ICLR 2023
4.4 THE EFFECT OF UNCERTAINTY
To further illustrate the effectiveness of our method, we select small amount of labeled data as unlabeled data generate and select high-quality pseudo-labels. Moreover, calculating the error and PCK @0.2 these pseudo-labels to observe the effect of uncertainty on the quality evaluation of pseudo labels. We performed the following experiments:
In MDSs, triplet uncertainty and confidence are used to select high-quality pseudo-labed
MDSs, triplet uncertainties are used to select high-quality pseudo-labels.
In MDSs, confidence is used to select high-quality pseudo-labels.
The experimental results are shown Figs. BL On the one hand, the pseudo-label quality of the methods using triplet uncertainty is much higher than the prediction performance the current model on the test set. These results suggest that triplet uncertainty can effectively select high-quality pseudo-labels. On the other hand, the pseudo-label quality is similar for both methods using triplet uncertainty, while the method using confidence alone has the worst pseudo-label quality.
4.5 ABLATION STUDY
Table 2:
Evaluation result of MDSs under different
settings.
PAL is the
parometer adversarial loss of MDSs, as equation B
Index
PAL
PAL Weight
PCK 0.2
1
0.675
2
0.000005
0.687
3
V
0.005
0.693
In order to illustrate the role of model parameter adversarial mechanism in MDSs, we ran experi- ments with different parameter adversarial loss weights (PAL Weight), and the experimental results are shown in Table 2 Use the parameter adversarial mechanism, the error and PCK @0.2 of the model are better than those without parameter adversarial mechanism.
CONCLUSION
In this paper, Maximum Difference Student (MDSs) framework is proposed to effectively improve performance of semi-supervised pose estimation tasks. In, evaluation metrics based on triplet uncertainty can effectively select high-quality pseudo-labels and completely replace confidence. The model parameter adversarial mechanism can further improve the consistency and stability of the predictions of the two teachers and effectively boost the performance of the model. Our method achieves better results on three datasets: FLIC, Pranav, and Mouse.
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint arXiv: 19119999199999 2019.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime aulti-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017.
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistcaally-inoomed interpolation of hidden space for semi-super text classification. arXiv preprint arXiv:22044 12239, 2020.
Yilun Chen. Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 7103-7112, 2018

---

Under review as a conference paper at ICLR 2023
Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas Huang, and Lei Zhang. Bottom-up higher-resolution networks for multi-person pose estimation. arXiv preprint arXiv:1908.10333 7, 2019.
Yixiao Ge. Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation person re-identification. arXiv preprint urXiv:200..0526, 2020.
Junjie Huang, Zheng Zhu, and Guan Huang. Multi-stage hrnet: multiple stage high-resolution network for human pose estimation. arXiv preprint arXiv:1910.05901, 2019.
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson WH Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 6728 6736, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1eXiv.:610.02200 2016.
Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng. Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. Rethinking multi-stage networks for human pose estimation. arXiv preprint Xrviv: 1901.0014, 2019.
Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Macken- zie Weygandt Mathis, and Matthias Bethge. Deeplabeut: markerless pose estimation user- defined body parts with deep learning, Nature neuroscience, 21(9):1211-128, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(::11979 1993, 2018.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference on computer vision, pp. 483-499, 2016.
Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for
joint detection and grouping. Advances in neural information processing systems, 30, 2017.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
Ben Sapp and Ben Taskar. Modec: Multimodal decomposable models for human pose estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3674-3681, 2013.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Wei Tang, Pei Yu, and Ying Wu. Deeply learned compositional models for human pose estimation. In Proceedings of the European conference on computer vision (ECCV), pp. 190-206, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1653-1660, 2014.
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, al. Convolutional pose machines. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7724-4732, 2016.
10

---

Under review as a conference paper at ICLR 2023
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pp. 1666481, 2018.
Rongchang Xie, Chunyu Wang, Wenjun Zeng, and Yizhou Wang. An empirical study of the col- lapsing problem in semi-supervised 2d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11240-1 1249, 2021.
Wenfei Yang, Tianzhu Zhang, Xiaoyuan Yu, Tian Qi, Yongdong Zhang, and Feng Wu. Uncertainty guided collaborative training for weakly supervised temporal action detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 53-63, 2021.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-super learning with curriculum pseudo la- beling. Advances in Neural Information Processing Systems, 34, 2021.
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-awee aware coordinate repre- sentation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7093-7102, 2020a.
Zhe Zhang, Chunyu Wang, Wenhu Qin, and Wenjun Zeng. Fusing wearable imus with multi-view images for human pose estimation: A geometric approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2200-2209, 2020b.


Reviews:
[{"review_id": "zoGxrin0iWi", "reviewer": "Reviewer_WbjR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a semi-supervised pose estimation method based on the Dual Mean-Teacher framework. It aims to efficiently select high quality pseudo-labels. The framework, as the name suggests, is decomposed of two Mean-Teacher modules where teacher generates pseudo-labels by taking the parameters from student with Exponential Moving Average strategy to train the student. MSE loss along with parameter adversarial loss to keep the divergence and consistency loss is used to train the model.\n", "summary_of_the_review": "Even though the quantitative results are good; I am not very convinced with the experimental settings yet: The method is working well for small-scale data; while it is showing the inferior results for large-scale data. Also, measures are inconsistent from dataset to dataset. I’d like to request authors’ thought on these points. Currently, I’m marginally below the threshold for this paper.\n", "strength_and_weaknesses": "The proposed method outperforms the other methods in terms of MSE score and the stability i.e., much smaller fluctuations on the error curve and unlike the other methods, it does not use the confidence for evaluation of quality of pseudo-labels.\n\nThe authors use 3 datasets in which two of them are much smaller dataset (Mouse, Pranav) than the other one (FLIC). In small datasets in terms of PCK@0.2 their approach outperforms the other methods whereas in the large dataset it has the lowest PCK@0.2 among all which raises the question of generalizability of this method to larger datasets. Besides, the error curve on only one dataset (Mouse) is reported.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tOkllq0V_cT", "reviewer": "Reviewer_2eQb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the semi supervised pose estimation problem with the mean-teacher framework.\nSpecifically, the key insight is to maximum the discrepancy between two student models and use the corresponging teacher models to generate pseudo labels  to supervise each other. \nBesides, a discrepancy loss is used to maximum the discrepancy between the weights of two student models, and an uncertainty estimation method is used to evaluate the uncertainty of pseudo labels", "summary_of_the_review": "My main concern is the technique contribution and experiment evaluation.", "strength_and_weaknesses": "\nStrengths:\n1. Well written and easy to read.\n2. Theperformance seem to be good.\nWeakness:\n1. Key references are missing. Using two student and teacher models to supervised each other have been explored in previous works.\nSpecifically, the motivation of [1] is to utilize the discrepancy between two models to generate soft pseudo labels, [2] models the uncertainty of the pseudo labels in a predictive way.\n[1] Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification \n[2] Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localization\n2. Technique contributions are not clear. This paper seems to be the use of existing  techniques on a new task, although this is reasonable, it lacks task specific technique contributions.\n3. Experimental evaluations are not convincing. (1) For semi supervise learning, a key metrich is the performance with different labeled data ratios, but this is missing in this paper.  (2) There are many ways for uncertainty estimation, the advantage of the way in this paper is not clear.\n4. There are some typos in this paper, such are \"the prediction of the hard-agumented sample [Paper]\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bCdaVTablZ", "reviewer": "Reviewer_rtkR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a semi-supervised pose estimation approach based \nthe dual mean-teacher framework. To evaluate the quality of the generated pseudo-labels, the approach makes use of the teacher networks' predicted uncertainty, where the uncertainty aims to select high-quality pseudo-labels, to be added to the unlabeled data and gradually train the model. Next, the paper presents an adversarial mechanism to build the maximum discrepancy within the student networks. Finally, the proposed approach is evaluated on three datasets (FLIC, Pranav, and Mouse), where it shows solid performance.", "summary_of_the_review": "The paper presents an interesting idea but the novelty is limited. Moreover, the evaluation would be more beneficial with a more decent comparison with prior works on at least an additional standard dataset. Finally, the paper writing needs further work.", "strength_and_weaknesses": "Strength: \n- The paper studies an open problem, namely semi-supervised 2D pose estimations.\n- The proposed method is simple, straightforward and easy to develop. \n- The ablation studies are thorough. \n\nWeaknesses:\n- The results shown do not clearly show a significant improvement compared to other methods in most of the examples, e.g. Xie et al. Table1).\n- The presented dual teacher framework, mean teacher, and the uncertainty estimation protocol are known. Thus, the paper lacks a major contribution. \n- Evaluation on more human pose data, e.g. MPII or COCO, would  be helpful for comparing with a wider range of methods.\n- The paper writing has space for improvement. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RfHGrXf0XN", "reviewer": "Reviewer_nhdF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript recommends an ensemble-like approach for semi-supervised pose estimation. A teacher-student setting where self-supervised pseudo-labels are created by the teacher for unlabeled data is extended to use two teacher-student pairs. The discrepancy between the two teachers is used to assess the quality of the pseudo-labels and to select pseudo-labels that are more likely to be correct. The discrepancy between the students is used in a loss function. The method is evaluated on three datasets, showing improved results.", "summary_of_the_review": "While the manuscript explores an interesting, general idea, it lacks important details and clarity and does not sufficiently demonstrate that the architectural improvements lead to improved results.", "strength_and_weaknesses": "\n**Novelty**: The idea of using an ensemble of self-supervised / semi-supervised networks is well thought of and an interesting direction. The method itself could generalize to other settings as well.\n\n**Presentation**: The manuscript is in parts unclear regarding notation (see below). Some parts of the manuscript appear unfinished (invalid references, see below).\n\n**Reproducibility**: I believe it is unlikely that the work can be reproduced. Details on the network architecture, training setup and hyperparameters are missing (see below).\n\n**Experimental Setup**: The method is evaluated on three smaller datasets (which are not cited) and compared with mostly older methods. It remains unclear how it would perform on larger datasets against state of the art. Additionally, the experiments do not show if the improvement over the baseline method is due to the proposed architectural improvements or simply due to the doubeling of the network's capacity.\n\n**Results**: The results are good and show a significant improvement over the compared-with methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject