Boosting Semi-Supervised 2D Human Pose Estimation by Revisiting Data Augmentation and Consistency Training

## Abstract

Abstract The 2D human pose estimation (HPE) is a basic visual problem. However, its supervised learning requires massive keypoint labels, which is labor-intensive to collect. Thus, we aim at boosting a pose estimator by excavating extra unlabeled data with semi-supervised learning (SSL). Most previous SSHPE methods are consistency-based and strive to maintain consistent outputs for differently augmented inputs. Under this genre, we find that SSHPE can be boosted from two cores: advanced data augmentations and concise consistency training ways. Specifically, for the first core, we discover the synergistic effects of existing augmentations, and reveal novel paradigms for conveniently producing new superior HPE-oriented augmentations which can more effectively add noise on unlabeled samples. We can therefore establish paired easy-hard augmentations with larger difficulty gaps. For the second core, we propose to repeatedly augment unlabeled images with diverse hard augmentations, and generate multi-path predictions sequentially for optimizing multi-losses in a single network. This simple and compact design is interpretable, and easily benefits from newly found augmentations. Comparing to SOTA approaches, our method brings substantial improvements on public datasets. Code is in https://github.com/hnuzhy/MultiAugs

###### Keywords:

## 1 Introduction

2D human pose estimation (HPE) aims to detect and represent human parts as sparse 2D keypoint locations in RGB images. It is the basis of many visual tasks such as action recognition [, ], person re-identification [, ] and 3D human shape regression [, ]. Modern data-driven HPE has been substantially improved by generous deep supervised learning approaches [, , , , , , ]. This greatly benefits from the collection and annotation of many large-scale public HPE datasets [, , , ]. However, compared to image classification and detection tasks requiring plain labels, obtaining accurate 2D keypoints from massive images is laborious and time-consuming. To this end, some researches [, , , ] try to alleviate this problem by introducing the semi-supervised 2D human pose estimation (SSHPE), which can subtly leverage extensive easier obtainable yet unlabeled 2D human images in addition to partial labeled data to improve performance. Although methods [, ] have improved the accuracy of SSHPE task, they overlooked two fundamental questions:

Figure: (a) Single Network
Refer to caption: extracted/5456722/figures/frameworkOldA.png

Q1: How to judge the discrepancy of unsupervised data augmentations with different difficulty levels?
In [], as shown in Fig. 0(a), for a batch of unlabeled images $\mathbf{I}$, its easy augmentation $\mathbf{I}_{e}$ and hard augmentation $\mathbf{I}_{h}$ are generated separately. Then, predicted heatmap $\mathbf{H}_{e}$ of $\mathbf{I}_{e}$ is used as a pseudo label to teach the network to learn the harder counterpart $\mathbf{I}_{h}$ with its yielded heatmap $\mathbf{H}_{h}$. [] finds that the large gap between two augmentations $(\mathbf{I}_{e},\mathbf{I}_{h})$ matters. Essentially, this is a pursuit for more advanced data augmentations in SSHPE. To rank augmentations of different difficulty levels, [] observes precision degradation of a pretrained model by testing it on a dataset after corresponding augmentation. However, we declare that this manner is not rigorous. An obvious counter-case is that over-augmented samples approaching noise will get worst evaluation results, but such hard augmentations are meaningless. Contrastingly, we deem that a persuasive ranking requires independent training for each augmentation. We answer this question in detail in Sec. 3.1.

Q2: How to generate multiple unsupervised signals for consistency training efficiently and concisely?
Previous work [] proposes to use a Single-Network to perform the unsupervised consistency training on the easy-hard pair $(\mathbf{I}_{e},\mathbf{I}_{h})$. It also gives a more complicated Dual-Network as in Fig. 0(b) for cross-training of two easy-hard pairs. SSPCM [] even constructs a Triple-Network as in Fig. 0(c) for interactive training of three easy-hard pairs. This pattern of adding more networks with the increase of unsupervised signal pairs can certainly bring gains. But this is cumbersome and will decelerate the training speed proportionally. In practice, considering that augmentations are always performed on the same input, we can repeatedly augment $\mathbf{I}$ multiple times with $n$ diverse hard augmentations, and generate multi-path easy-hard pairs $\{(\mathbf{I}_{e},\mathbf{I}_{h_{1}}),(\mathbf{I}_{e},\mathbf{I}_{h_{2}}),...,
(\mathbf{I}_{e},\mathbf{I}_{h_{n}})\}$. In this way, we can use only one single network (refer Fig. 6(a)) to optimize $n$ pairs of losses. This is also applicable to dual networks (refer Fig. 6(b)). We discuss this question in detail in Sec. 3.2.

In this paper, we mainly revisit Q1 and Q2 to boost SSHPE. For Q1, after properly ranking existing basic augmentations [, , , , ], we naturally try to extend it to discover new strong augmentations through reasonable sequential combinations inspired by the AutoAug families [, , , ]. Rather than trivial enumeration, we notice the synergistic effects between different augmentations, and reveal novel paradigms for easily generating superior combinations. Our paradigms for the SSHPE task contain three principles: (P1) Do not combine MixUp-related augmentations. (P2) Try to utilize the synergistic effects. (P3) Do not over-combine too many augmentations. These principles have favorable interpretability, and bypass painstaking designs in other advanced augmentations [, , , , , ]. For Q2, we quantitatively validated the superiority of multi-path design over commonly used heatmaps fusion [] and confidence masking [, ]. Combining it with newly found advanced augmentations, our Single-Network based approach can surpass the original Dual-Network [] and come close to SSPCM using a Triple-Network. Our contributions are as follows:

- •
We comprehensively evaluated the difficulty levels of existing data augmentations suitable for the SSHPE task, validated their synergistic effects by properly combining different basic augmentations, and presented novel combination paradigms which are intuitively interpretable.
- •
We proposed to generate multi-path predictions of separately strongly augmented samples for training only one single model, rather than adding auxiliary networks. Thus, we can optimize multiple unsupervised losses efficiently and concisely, and benefit from distinct superior augmentations.
- •
We achieved new SOTA results on public SSHPE benchmarks with less training time and parameters under same settings of previous methods.

## 2 Related Work

Semi-Supervised Learning (SSL).
SSL is invented for improving the classification task by exploiting a small set of labeled data and a large set of unlabeled data. It can be categorized into pseudo-label (PL) based [, , , , , ] and consistency-based [, , , , , , ]. PL-based methods iteratively add unlabeled images into the training data by pseudo-labeling them with a pretrained or gradually enhanced model. It needs to find suitable thresholds to mask out uncertain samples with low-confidence, which is a crucial yet tricky issue. Consistency-based methods enforce model outputs to be consistent when its input is randomly perturbed. They have shown to work well on many tasks. For example, MixMatch [] combines the consistency regularization with the entropy minimization to obtain confident predictions. FixMatch [] utilizes a weak-to-strong consistency regularization and integrates the pseudo-labeling to leverage unlabeled data. FlexMatch [] and FreeMatch [] adopt the curriculum learning and adaptive thresholding based on FixMatch, respectively. CRMatch [] and SAA [] try to design strategies and augmentations to enhance the consistency training. These SSL methods give us primitive inspirations.

Semi-Supervised Human Pose Estimation (SSHPE).
The SSHPE task is relatively less-studied comparing to other visual tasks such as classification and object detection. A few representative methods are based on pseudo labeling [, ] or consistency training [, , , ]. SSKL [] designs a semantic keypoint consistency constraint to learn invariant representations of same keypoints. It has been evaluated on small-scale HPE benchmarks MPII [] and LSP [], instead of the larger COCO []. Following it, PLACL [] introduces the curriculum learning by auto-selecting dynamic thresholds for producing pseudo-labels via reinforcement learning. Inspired by co-training [] and dual-student [], Dual-Network [] points out the typical collapsing problem in SSHPE, and proposes the easy-hard augmentation pair on the same input to imitate teacher-student signals without relying on Mean Teacher []. SSPCM [] extends the Dual into Triple by adding an auxiliary teacher for interactive training in multi-steps. It designs a handcrafted pseudo-label correction based on the predicted position inconsistency of two teachers, and has achieved SOTA performances. We revisit the less efficient-consistency training way in [, ], and propose to upgrade the Single-Network furtherly by multi-path predictions.

Unsupervised Data Augmentations.
The UDA [] has emphasized and verified the key role of high-quality noise injection (*e.g*., data augmentations) in improving unsupervised consistency training. It utilizes advanced augmentations [, ] to promote the SSL classification. Then, [] transfers the positive correlation between strong augmentations and SSL performance to the HPE field. It introduces a more advanced augmentation called Joint Cutout inspired by Cutout []. Similarly, SSPCM [] provides a harder keypoints-sensitive augmentation Cut-Occlude inherited from CutMix []. In this paper, we thoroughly revisit existing data augmentations suitable to SSHPE, give a rank of their difficulty levels by controlled trainings, and produce simple paradigms for getting novel superior joint-related augmentations. We also compare them with other well-designed counterparts [, , ] to reveal our advantages.

## 3 Empirical Studies

Problem Definition: The task of 2D HPE is to detect $k$ body joints in an image $\mathbf{I}\in\mathbb{R}^{h\times w\times 3}$. The state-of-the-art methods tend to estimate $k$ Gaussian heatmaps $\mathbf{H}\in\mathbb{R}^{\frac{h}{s}\times\frac{w}{s}\times k}$ downsampled $s$ times. For inference, each keypoint is located by finding the pixel with largest value in its predicted heatmap. We denote the labeled and unlabeled training sets as $\mathcal{D}^{l}\!=\!\{(\mathbf{I}^{l}_{i},\mathbf{H}^{l}_{i})\}|^{N}_{i=1}$ and $\mathcal{D}^{u}\!=\!\{\mathbf{I}^{u}_{i}\}|^{M}_{i=1}$, respectively. The $\mathbf{H}^{l}$ are ground-truth heatmaps generated using 2D keypoints. For supervised training of the network $f$, we calculate the MSE loss:

$$ $\small\mathcal{L}_{s}=\mathbb{E}_{\mathbf{I}\in\mathcal{D}^{l}}||f(T_{e}( \mathbf{I}))-T_{e}(\mathbf{H})||^{2}~{}$ (1) $$

where $T_{e}$ represents an easy affine augmentation including a random rotation angle from $[-30^{\circ},30^{\circ}]$ and scale factor from $[0.75,1.25]$ (denoted as $T_{A30}$). For unlabeled images, we calculate the unsupervised consistency loss:

$$ $\small\mathcal{L}_{u}=\mathbb{E}_{\mathbf{I}\in\mathcal{D}^{u}}||T_{e \rightarrow h}(f(T_{e}(\mathbf{I})))-f(T_{h}(\mathbf{I}))||^{2}~{}$ (2) $$

where $T_{h}$ is a harder augmentation with strong perturbations than affine-based $T_{e}$. The $T_{e\rightarrow h}$ means a known affine transformation on heatmaps if $T_{h}$ contains additional rotation and scaling operations. In this way, we can obtain a paired easy-hard augmentations $(\mathbf{I}_{e},\mathbf{I}_{h})\!=\!(T_{e}(\mathbf{I}),T_{h}(\mathbf{I}))$ for generating corresponding teacher signals and student signals. During training, we should stop gradients propagation of teacher signals to avoid collapsing.

Next, we answer two questions Q1 and Q2 by extensive empirical studies in Sec. 3.1 and Sec. 3.2, respectively. After that, we provide a perspective for understanding the pursuit of strong augmentations in Sec. 3.3.

### 3.1 Paradigms of Generating Superior Augmentations

Figure: Figure 2: Comparison of applying different easy-hard pairs for training a Single-Network model as in Fig. 0(a). We can sort these six augmentations indisputably based on either best mAP results or distinct convergence curves.
Refer to caption: extracted/5456722/figures/AugsRankingV2.png

Ranking of Basic Augmentations.
The core of the easy-hard pair paradigm $(\mathbf{I}_{e},\mathbf{I}_{h})$ is a more advanced augmentation. For this reason, Dual-Network [] and SSPCM [] propose pseudo keypoint-based augmentations Joint Cutout ($T_{JC}$) and Joint Cut-Occlude ($T_{JO}$), respectively. They also reach a similar yet crude conclusion about difficulty levels of existing augmentations: $\{T_{JO},T_{JC}\}\!>\!\{T_{RA},T_{CM},T_{CO},T_{MU},T_{A60}\}$, where $T_{RA}$, $T_{CM}$, $T_{CO}$ and $T_{MU}$ are RandAugment [], CutMix [], Cutout [] and Mixup [], respectively. The $T_{A60}$ consists of two $T_{A30}$. We give them a new ranking by conducting more rigorous trainings one-by-one. The $T_{RA}$ is removed for it contains repetitions with $T_{CO}$ and $T_{A60}$. As shown in Fig. 3, we divide the rest by their distinguishable gaps into four levels: $\{T_{JC},T_{JO}\}\!>\!\{T_{CO},T_{CM}\}\!>\!\{T_{MU}\}\!>\!\{T_{A60}\}$.

Synergy between Augmentations.
Then, instead of laboriously designing stronger augmentations, we consider to conduct two or more augmentations in sequence to obtain superior combinations conveniently. This idea is feasible because it essentially belongs to the AutoAug families [, , , ]. Instead of auto-searching, we expect to find some heuristic strategies for the HPE task. In fact, after performing joint-related $T_{JO}$ or $T_{JC}$ on one image, we can continue to perform some joint-unrelated augmentations such as $T_{CM}$, $T_{CO}$ and $T_{MU}$ on random areas. As shown in Fig. 3, applying $T_{JOCO}$ (a $T_{CO}$ after $T_{JO}$) or $T_{JCCM}$ (a $T_{CM}$ after $T_{JC}$) will bring harder samples for generating more effective student signals, but not destroy the semantic information visually. We call this discovery the synergistic effect between different augmentations. The $T_{A60}$ can server as an essential factor in any $T_{h}$ for keeping the geometric diversity.

Selection of Augmentations Combination.
Now, if selecting from the rest five basic augmentations, there are up to 26 choices $(2^{5}-\binom{5}{0}-\binom{5}{1})$. The optimal combinations are still time-consuming to acquire. Fortunately, not arbitrary number or kind of augmentations are collaborative. We intuitively summarize three simplistic principles. (P1) A global $T_{MU}$ does not make sense for the HPE task. (P2) Stacking augmentations with the similar perturbation type (e.g., $T_{JO}\!\sim\!T_{CM}$ and $T_{JC}\!\sim\!T_{CO}$) or difficulty level (e.g., $T_{JO}\!\sim\!T_{JC}$ and $T_{CM}\!\sim\!T_{CO}$) may not bring significant gain. (P3) Adding too many augmentations (e.g., three or four) will be profitless or even harmful for seriously polluting the image. We thus nominate the most likely superior combinations: $T_{JOCO}$ and $T_{JCCM}$. A case of setting $T_{e}$ as $T_{A30}$ and $T_{h}$ as $T_{JOCO}$ for getting corresponding easy teacher signals $\mathbf{H}_{e}$ and hard student signals $\mathbf{H}_{h}$ is shown as below:

$$ $\displaystyle\mathbf{H}_{e}=T_{A30\rightarrow A60}(f(\mathbf{I}_{e})),\quad \mathbf{I}_{e}=T_{A30}(\mathbf{I})$ (3) $\displaystyle\mathbf{H}_{h}=f(\mathbf{I}_{h}),\quad\mathbf{I}_{h}=T_{A30}(T_{ CO}(T_{JO}(T_{A30}(\mathbf{I}))))~{}$ $$

where $T_{A60}$ is default to $T_{h}$, and divided into two separate $T_{A30}$ for being compatible with $T_{e}$. To further verify the above intuitive principles, we follow the augmentations ranking way and conduct empirical studies on the performance of up to 13 selected representative combinations out of 26 choices.

| Id | Combination | mAP |
| --- | --- | --- |
| – | $T_{JC}$ | 42.4 |
| – | $T_{JO}$ | 41.9 |
| $c_{1}$ | $T_{JC,CM}$ | 42.7 |
| $c_{2}$ | $T_{JO,CO}$ | 43.7 |
| $c_{3}$ | $T_{JC,MU}$ | 41.8 |
| $c_{4}$ | $T_{JO,MU}$ | 42.1 |
| $c_{5}$ | $T_{JC,CO}$ | 42.1 |
| $c_{6}$ | $T_{JO,CM}$ | 42.5 |
| $c_{7}$ | $T_{JC,JO}$ | 42.0 |
| $c_{8}$ | $T_{JC,CM,MU}$ | 42.0 |
| $c_{9}$ | $T_{JO,CO,MU}$ | 42.8 |
| $c_{10}$ | $T_{JC,JO,CO}$ | 41.7 |
| $c_{11}$ | $T_{JC,JO,CM}$ | 42.3 |
| $c_{12}$ | $T_{JC,CO,CM}$ | 42.8 |
| $c_{13}$ | $T_{JO,CO,CM}$ | 42.8 |
|  |  |  |

| Id | MultiAugs | mAP |
| --- | --- | --- |
| $m_{1}$ | $T_{JC},T_{CM}$ | 43.0 |
| $m_{2}$ | $T_{JO},T_{CO}$ | 43.4 |
| $m_{3}$ | $T_{JC},T_{JO}$ | 43.1 |
| $m_{4}$ | $T_{JCCM},T_{CO}$ | 43.7 |
| $m_{5}$ | $T_{JOCO},T_{CM}$ | 43.1 |
| $m_{6}$ | $T_{JOCO},T_{JC}$ | 44.2 |
| $m_{7}$ | $T_{JCCM},T_{JO}$ | 42.9 |
| $m_{8}$ | $T_{JC},T_{JO},$<br>$T_{CO},T_{CM}$ | 43.6 |
| $m_{9}$ | $T_{JCCM}$,<br>$T_{JCCM}$ | 43.6 |
| $m_{10}$ | $T_{JOCO}$,<br>$T_{JOCO}$ | 44.0 |
| $m_{11}$ | $T_{JOCO},$<br>$T_{JCCM}$ | 44.9 |
| $m_{12}$ | $T_{JOCO},T_{JC},$<br>$T_{JCCM},T_{JO}$ | 45.5 |
|  |  |  |

As shown in Tab. 3.1 and Fig. 3.1, we can examine three principles one-by-one: (P1) $T_{MU}$ often causes adverse or inferior effects for each combination. Please refer paired combinations $c_{1}$-$c_{3}$, $c_{2}$-$c_{4}$, $c_{1}$-$c_{8}$ and $c_{2}$-$c_{9}$. Thus, we do not add it for finding superior combinations. (P2) Synergistic effects between augmentations do exist. Please refer paired combinations $c_{1}$-$c_{5}$, $c_{1}$-$c_{7}$, $c_{2}$-$c_{6}$ and $c_{2}$-$c_{7}$. Especially, the $T_{JCJO}$ with two most advanced augmentations performs the worst among combinations {$c_{1}$,$c_{2}$,$c_{5}$,$c_{6}$,$c_{7}$}, which roundly reveals the harm of violating the principle of synergy. (P3) Do not overly combine too many augmentations. Please refer paired combinations $c_{1}$-$c_{12}$, $c_{2}$-$c_{13}$, $c_{1}$-$c_{11}$ and $c_{2}$-$c_{10}$. Stacking more augmentations brings non-significant gains or even results in degradation. We attribute it to deviating from the rule of collaboration and possibly producing meaningless or difficult-to-recognize images. Based on these facts, we have sufficient reasons not to check the performance of rest combinations. And we recommend two new strongest combinations $T_{JOCO}$ and $T_{JCCM}$.

### 3.2 Multi-path Consistency Losses

To further amplify the advantage of easy-hard augmentation, [] adopts two independent networks containing two easy-hard pairs for producing two consistency losses. SSPCM [] continues this route by designing a Triple-Network with three easy-hard pairs. Meanwhile, some SSL methods construct multi-view inputs for unlabeled data without adding accompanying networks. For example, SwAV [] enforces the local-to-global consistency among a bag of views with different resolutions. ReMixMatch [] feeds multiple strongly augmented versions of an input into the model for training. Therefore, we wonder whether such a simple idea can also benefit the SSHPE task.

Specifically, rather than feeding a single hard augmentation $\mathbf{I}_{h}$ into the model, we independently yield $n$ strongly augmented inputs $\mathcal{I}^{n}\!=\!\{\mathbf{I}_{h_{1}},\mathbf{I}_{h_{2}},...,\mathbf{I}_{h_
{n}}\}$ from $\mathbf{I}$ by applying $n$ hard data augmentations $\mathcal{T}^{n}\!=\!\{T_{h_{1}},T_{h_{2}},...,T_{h_{n}}\}$ accordingly. The augmentation set $\mathcal{T}^{n}$ is de-emphasized in order and non-deterministic, and will generate distinct multi-path augmented inputs $\mathcal{I}^{n}$. Then, we can calculate $n$-stream heatmaps $\mathcal{H}^{n}\!=\!\{f(T_{h_{i}}(\mathbf{I}_{h_{i}}))|^{n}_{i=1}\}$. This multi-path augmentation framework is illustrated in Fig. 7. For regularizing $n$ easy-hard pairs, we obtain multi-path consistency losses using Eq. 2 in separate, and optimize them jointly by applying multi-loss learning:

$$ $\small\mathcal{L}^{*}_{u}=\mathbb{E}_{\mathbf{H}_{h_{i}}\in\mathcal{H}^{n}} \sum\nolimits^{n}_{i=1}||\mathbf{H}_{e}-\mathbf{H}_{h_{i}}||^{2}~{}$ (4) $$

where $\mathbf{H}_{e}$ and $\mathbf{H}_{h_{i}}$ are obtained as dissected in Eq. 3. The $\mathbf{H}_{e}$ keeps constant for each $\mathbf{H}_{h_{i}}$. Comparably, Data Distill [] applies a single model to multiple transformations of unlabeled data to train a student model. Then, it ensembles predictions to obtain keypoint locations, and re-generates a pseudo heatmap for supervision. Differently, we argue that conducting a fusion on predicted heatmaps in SSHPE is harmful. We consider that there are always differences in the estimation of keypoint positions for each $\mathbf{I}_{h_{i}}$. It is an ill-posed problem to heuristically evaluate the consistency regularization contribution of each heatmap in pixel during ensemble. We will explain this in ablation studies Sec. 5.4.

Despite the simplicity, such a minor modification brings consistent gains over the original Single-Network under same SSL settings. With our discovered augmentation combinations $T_{JOCO}$ and $T_{JCCM}$, the boosted Single-Network can surpass the original Dual-Network evidently. We validated in ablation studies that the performance gain is non-trivial. We conjecture that regularizing multiple hard augmentations with a shared easy augmentation can be regarded as enforcing consistency among advanced augmentations as well, which inherits the concept of training positive-negative paired samples in contrastive learning [, , ] and its SSL-related variations [, , ].

Effectiveness Verification. As shown in Tab. 3.1 and Fig. 3.1, we also experimentally verified the major advantage of the multi-path augmentations (dubbed as MultiAugs) strategy. Here, we have two variables of MultiAugs: the number of paths and the category of augmentations. For acquisition of an optimal augmentations set, similarly, we continue with the rules summarized in the previous section, and elect 12 different MultiAugs schemes for illustrating. We can witness the effectiveness of MultiAugs from two aspects: (1) It can inherit and even expand the synergistic effects between different augmentations. (2) It can alleviate the defects caused by excessive stacking of augmentations.

Specifically, by comparing $c_{1}$-$m_{1}$, $c_{2}$-$m_{2}$ and $c_{7}$-$m_{2}$, we find that MultiAugs provides comparable performance with the same augmentations. By comparing $c_{10}$-$m_{6}$, $c_{11}$-$m_{7}$, $c_{12}$-$m_{4}$ and $c_{13}$-$m_{5}$, we can observe the sustained large gains brought by MultiAugs. The scheme $m_{8}$ does not obtain a better result than $m_{11}$ showing that new augmentations $T_{JOCO}$ and $T_{JCCM}$ have their essential properties. Schemes $m_{9}$ and $m_{10}$ which utilize a single augmentation twice are also inferior to $m_{11}$ showing the cooperativity of using multi-path distinct augmentations. Moreover, the optimal scheme $m_{12}$ further unleashes capabilities of four advanced augmentations. In order to balance performance and time consumption, we do not add more augmentation paths.

### 3.3 Analysis of Superior Augmentations

Figure: (a) Top-50 singular values
Refer to caption: extracted/5456722/figures/plotSVD.png

In this part, we tentatively analyze why employing a superior augmentation to strongly augment the unlabeled data can improve model performance. Different from UDA [] using the improved connectivity of constructed graphs to explain, we start from the perspective of shaped feature space of unlabeled data based on the singular value spectrum, which is widely considered to be related to the model transferability and generalization [, ]. Specifically, we perform singular value decomposition (SVD) on features $\mathbf{F}\!\in\!\mathbb{R}^{N\times D}$(^1^11We denote $N$ as the number of samples and $D$ as feature dimensions (a.k.a, $D\leq N$).) extracted by various trained models with different strong augmentations on one dataset: $\mathbf{F}\!=\!\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{T}$, where $\mathbf{U}$ and $\mathbf{V}$ is the left and right singular vector matrices, respectively, and $\mathbf{\Sigma}$ denotes the diagonal singular value matrix $\{\sigma_{1},\sigma_{2},...,\sigma_{D}\}$. Then, we plot calculated singular values in Fig. 5(a). To further measure the flatness of the singular value distribution, we calculate the entropy of normalized singular values $\mathsf{H}_{nsv}$:

$$ $\small\mathsf{H}_{nsv}=-\sum\nolimits^{D}_{i=1}\frac{\sigma_{i}}{\sum\nolimits ^{D}_{j=1}\sigma_{j}}\log{\frac{\sigma_{i}}{\sum\nolimits^{D}_{j=1}\sigma_{j}} }~{}$ (5) $$

Usually, a larger $\mathsf{H}_{nsv}$ indicates that the feature space captures more structure in the data and thus spans more dimensions due to more discriminated representations learned. As shown in Fig. 5(b), the model performance is positively correlated with the $\mathsf{H}_{nsv}$ value. Therefore, a superior augmentation facilitates better model generalization to unseen test sets.

## 4 Our Overall Framework: MultiAugs

We leverage unlabeled images by applying multiple augmentations with integrating two key techniques introduced in Sec. 3.1 and Sec. 3.2. Firstly, assuming that we have obtained an optimal augmentation set $\mathcal{\widehat{T}}^{n}\!=\!\{\widehat{T}_{h_{i}}|^{n}_{i=1}\}$, where $\widehat{T}_{h_{i}}$ may be an old single augmentation or a novel combined one. Then, we present how to construct our overall training framework in Fig. 7 based on either the Single-Network or the Dual-Network.

Figure: (a) MultiAugs - Single Network
Refer to caption: extracted/5456722/figures/frameworkNewA.png

MultiAugs (Single-Network).
It is a consistency-based approach. We only need to maintain a single model as in Fig. 6(a) during training. For each input batch with equal number of labeled images $\mathbf{I}^{l}$ and unlabeled images $\mathbf{I}^{u}$, we calculate the supervised loss with the ground-truth heatmaps as in Eq. 1, and the multiple unsupervised losses as in Eq. 4, respectively. The final loss is obtained by adding the two loss functions $\mathcal{L}=\mathcal{L}_{s}+\lambda\mathcal{L}^{*}_{u}$ with $\lambda=1$. Note that we only pass the gradient back through $n$ hard augmentations $\mathcal{\widehat{I}}^{n}$ for generating teacher signals to avoid collapsing. Based on this boosted Single-Network, we complete all ablation experiments by changing the augmentation categories and quantities in $\mathcal{\widehat{T}}^{n}$ for controlling the unsupervised factor $\mathcal{L}^{*}_{u}$.

MultiAugs (Dual-Network).
As shown in Fig. 6(b), this framework learns two identical yet independent networks with each similar to the Single-Network. For one input batch in every step, each of the two networks serves as both a teacher and a student. They both are fed by easy and hard augmentations of unlabeled images $\mathbf{I}^{u}$ when they produce teacher signals and student signals, respectively. Assuming we have two networks $f_{A}$ and $f_{B}$, and also the augmented easy images $\mathbf{I}^{u}_{e}$ using $T_{e}$ and $n$-path hard images $\{\mathbf{I}^{u}_{h_{1}},\mathbf{I}^{u}_{h_{2}},...,\mathbf{I}^{u}_{h_{n}}\}$ using $\mathcal{\widehat{T}}^{n}$, we first predict the following four types of heatmaps:

$$ $\displaystyle\mathbf{H}_{e,A}=T_{A30\rightarrow A60}(f_{A}(\mathbf{I}^{u}_{e}) ),\quad\mathcal{H}_{A}=\{\mathbf{H}_{h_{i},A}|^{n}_{i=1},\mathbf{H}_{h_{i},A}= f_{A}(\mathbf{I}^{u}_{h_{i}})\}$ (6) $\displaystyle\mathbf{H}_{e,B}=T_{A30\rightarrow A60}(f_{B}(\mathbf{I}^{u}_{e}) ),\quad\mathcal{H}_{B}=\{\mathbf{H}_{h_{i},B}|^{n}_{i=1},\mathbf{H}_{h_{i},B}= f_{B}(\mathbf{I}^{u}_{h_{i}})\}$ $$

where $T_{A30\rightarrow A60}$ is a pre-generated affine transformation. Based on above heatmaps, we calculate two unsupervised losses for training two networks as follows:

$$ $\displaystyle\mathcal{L}^{*}_{u,A}=\mathbb{E}_{\mathbf{H}_{h_{i},A}\in\mathcal {H}_{A}}\sum\nolimits^{n}_{i=1}||\mathbf{H}_{e,B}-\mathbf{H}_{h_{i},A}||^{2}$ (7) $\displaystyle\mathcal{L}^{*}_{u,B}=\mathbb{E}_{\mathbf{H}_{h_{i},B}\in\mathcal {H}_{B}}\sum\nolimits^{n}_{i=1}||\mathbf{H}_{e,A}-\mathbf{H}_{h_{i},B}||^{2}~{}$ $$

where we swap positions of teacher signals $\mathbf{H}_{e,A}$ and $\mathbf{H}_{e,B}$ for realizing the cross training of networks $f_{B}$ and $f_{A}$. Following [], we report the average accuracy of the final two well-trained and performance-approached models. Besides, $f_{A}$ and $f_{B}$ can have different structures as in [, ], where the large one often helps to distill a better small model, but not vice versa. We do not intend to explore this consensus anymore in this paper.

## 5 Experiments

### 5.1 Datasets and Metrics

COCO.
The dataset COCO [] has 4 subsets: train-set (118K images), val-set (5K images), test-dev and test-challenge. It is a popular large-scale benchmark for human pose estimation, which contains over 150K annotated people. In addition, there are 123K wild unlabeled images (wild-set). We randomly selected 1K, 5K and 10K labeled data from train-set. In some experiments, unlabeled data came from the remaining images of train-set. In other experiments, we used the whole train-set as the labeled dataset and wild-set as the unlabeled dataset. The metric of mAP (Average AP over 10 OKS thresholds) is reported.

MPII and AI-Challenger.
The dataset MPII [] has 25K images and 40K person instances with 16 keypoints. The dataset AI-Challenger (AIC) [] train-set has 210K images and 370K person instances with 14 keypoints. We use MPII as the labeled set, AIC as the unlabeled set. The metric of PCKh0.5 is reported.

### 5.2 Implementation Details

For a fair comparison with prior works, we use SimpleBaseline [] to estimate heatmaps and ResNet [] and HRNet [] as backbones. The input image size is set to $256\times 192$. We adopt the PyTorch 1.30 and 4 A100 GPUs with each batch size as 32 for training. The initial learning rate is 1e-3. When training on COCO with 10K labeled data, it decreases to 1e-4 and 1e-5 at epochs 70 and 90, respectively, with a total of 100 epochs. When using 1K or 5K labeled data, total epochs are reduced to 30 or 70, respectively. When training on the complete COCO or MPII+AIC, it drops to 1e-4 and 1e-5 at epochs 220 and 260, respectively, with a total of 300 epochs. When testing, we do not flip horizontally.

For data augmentation settings, we keep the easy augmentation $T_{e}$ as $T_{A30}$ in all experiments. The hard augmentations set $\mathcal{\widehat{T}}^{4}\!=\!\{T_{JOCO},T_{JCCM},T_{JC},T_{JO}\}$ is chosen for the comparison C1. While, the less optimal augmentations set $\mathcal{\widehat{T}}^{2}\!=\!\{T_{JOCO},T_{JCCM}\}$ is chosen for comparisons C2, C3, C4 and C5 to balance the performance and training time. We present details of finding two novel superior hard augmentations $T_{JOCO}$ and $T_{JCCM}$, and selecting the optimal augmentation set $\mathcal{\widehat{T}}^{4}$ in ablation studies.

### 5.3 Performance Comparison

| Method | Net.<br>Num. | 1K | 5K | 10K | All |
| --- | --- | --- | --- | --- | --- |
| Supervised [] | 1 | 31.5 | 46.4 | 51.1 | 67.1 |
| PseudoPose [] | 2 | 37.2 | 50.9 | 56.0 | — |
| DataDistill [] | 2 | 37.6 | 51.6 | 56.6 | — |
| PoseCons [] | 1 | 42.1 | 52.3 | 57.3 | — |
| PoseDual [] | 2 | 44.6 | 55.6 | 59.6 | — |
| SSPCM [] | 3 | 46.9 | 57.5 | 60.7 | — |
| Ours (Single) | 1 | 45.5 | 56.2 | 59.9 | — |
| Ours (Dual) | 2 | 49.7 | 58.8 | 61.8 | — |
|  |  |  |  |  |  |

**Table 4: Results on the COCO val-set when labeled images from the train-set and unlabeled images from the wild-set.**
| Method | Backbone | Networks | AP | AP${}_{.5}$ | AR | AR${}_{.5}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Supervised [] | ResNet50 | Single | 70.9 | 91.4 | 74.2 | 92.3 |
| PoseDual [] | ResNet50 | Dual | 73.9 | 92.5 | 77.0 | 93.5 |
| SSPCM [] | ResNet50 | Triple | 74.2 | 92.7 | 77.2 | 93.8 |
| Ours (Single) | ResNet50 | Single | 74.0 | 92.5 | 77.1 | 93.9 |
| Ours (Dual) | ResNet50 | Dual | 74.6 | 93.5 | 77.6 | 94.0 |
| Supervised [] | ResNet101 | Single | 72.5 | 92.5 | 75.6 | 93.1 |
| PoseDual [] | ResNet101 | Dual | 75.3 | 93.6 | 78.2 | 94.1 |
| SSPCM [] | ResNet101 | Triple | 75.5 | 93.8 | 78.4 | 94.2 |
| Ours (Single) | ResNet101 | Single | 75.4 | 93.4 | 78.4 | 94.1 |
| Ours (Dual) | ResNet101 | Dual | 76.3 | 93.5 | 79.2 | 94.4 |
| Supervised [] | HRNet-w48 | Single | 77.2 | 93.5 | 79.9 | 94.1 |
| PoseDual [] | HRNet-w48 | Dual | 79.2 | 94.6 | 81.7 | 95.1 |
| SSPCM [] | HRNet-w48 | Triple | 79.4 | 94.8 | 81.9 | 95.2 |
| Ours (Single) | HRNet-w48 | Single | 79.3 | 94.6 | 81.8 | 95.1 |
| Ours (Dual) | HRNet-w48 | Dual | 79.5 | 94.6 | 82.1 | 95.2 |
|  |  |  |  |  |  |  |

**Table 5: Comparison to the SOTA methods on the COCO test-dev. The COCO train-set is the labeled set and COCO wild-set is the unlabeled set. The person detection results are provided by SimpleBaseline and flipping strategy is used.**
| Method | Backbone | Input Size | GFLOPS | #Params | AP | AR |
| --- | --- | --- | --- | --- | --- | --- |
| SimpleBaseline [] | ResNet50 | 256$\times$192 | 8.9 | 34.0 | 70.2 | 75.8 |
| HRNet [] | HRNet-w48 | 384$\times$288 | 32.9 | 63.6 | 75.5 | 80.5 |
| MSPN [] | ResNet50 | 384$\times$288 | 58.7 | 71.9 | 76.1 | 81.6 |
| DARK [] | HRNet-w48 | 384$\times$288 | 32.9 | 63.6 | 76.2 | 81.1 |
| UDP [] | HRNet-w48 | 384$\times$288 | 33.0 | 63.8 | 76.5 | 81.6 |
| TransPose-H-A6 [] | HRNet-w48 | 256$\times$192 | 21.8 | 17.5 | 75.0 | — |
| TokenPose-L/D24 [] | HRNet-w48 | 384$\times$288 | 22.1 | 29.8 | 75.9 | 80.8 |
| HRFormer [] | HRFormer-B | 384$\times$288 | 26.8 | 43.2 | 76.2 | 81.2 |
| ViTPose [] | ViT-Large | 256$\times$192 | 59.8 | 307.0 | 77.3 | — |
| DUAL (+HRNet) [] | HRNet-w48 | 384$\times$288 | 65.8 | 127.2 | 76.7 | 81.8 |
| DUAL (+DARK) [] | HRNet-w48 | 384$\times$288 | 65.8 | 127.2 | 77.2 | 82.2 |
| SSPCM (+DARK) [] | HRNet-w48 | 384$\times$288 | 98.7 | 190.8 | 77.5 | 82.4 |
| Ours (Dual) (+HRNet) | HRNet-w48 | 384$\times$288 | 65.8 | 127.2 | 76.8 | 81.8 |
| Ours (Dual) (+DARK) | HRNet-w48 | 384$\times$288 | 65.8 | 127.2 | 77.3 | 82.3 |
|  |  |  |  |  |  |  |

**Table 6: The results on the val-set of the MPII dataset. HRNet is trained only on the MPII train-set. The “*” means using extra labeled dataset AIC. The “+” means applying the model ensemble.**
| Method | Hea | Sho | Elb | Wri | Hip | Kne | Ank | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HRNet [] | 97.0 | 95.7 | 89.4 | 85.6 | 87.7 | 85.8 | 82.0 | 89.5 |
| HRNet* [] | 97.4 | 96.7 | 92.1 | 88.4 | 90.8 | 88.6 | 85.0 | 91.7 |
| PoseDual [] | 97.4 | 96.6 | 91.8 | 87.5 | 89.6 | 87.6 | 83.8 | 91.1 |
| Ours (Dual) | 97.3 | 96.8 | 91.7 | 87.5 | 90.3 | 88.6 | 84.6 | 91.4 |
| Ours+ (Dual) | 97.3 | 96.8 | 91.9 | 88.1 | 90.6 | 89.2 | 85.0 | 91.7 |
|  |  |  |  |  |  |  |  |  |

**Table 7: Comparisons on the MPII test-set (PCKh0.5). We use HRNetW32 as the backbone. Input size is 256$\times$256. The MPII and AIC (w/o labels) are used for training.**
| Method | Hea | Sho | Elb | Wri | Hip | Kne | Ank | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Newell *et al*. [] | 98.2 | 96.3 | 91.2 | 87.1 | 90.1 | 87.4 | 83.6 | 90.9 |
| Xiao *et al*. [] | 98.5 | 96.6 | 91.9 | 87.6 | 91.1 | 88.1 | 84.1 | 91.5 |
| Ke *et al*. [] | 98.5 | 96.8 | 92.7 | 88.4 | 90.6 | 89.4 | 86.3 | 92.1 |
| Sun *et al*. [] | 98.6 | 96.9 | 92.8 | 89.0 | 91.5 | 89.0 | 85.7 | 92.3 |
| Zhang *et al*. [] | 98.6 | 97.0 | 92.8 | 88.8 | 91.7 | 89.8 | 86.6 | 92.5 |
| Xie *et al*. [] | 98.7 | 97.3 | 93.7 | 90.2 | 92.0 | 90.3 | 86.5 | 93.0 |
| Huang *et al*. [] | 98.7 | 97.5 | 94.0 | 90.6 | 92.5 | 91.1 | 87.1 | 93.3 |
| Ours (Dual) | 98.8 | 97.6 | 94.1 | 90.3 | 92.4 | 91.1 | 87.2 | 93.4 |
|  |  |  |  |  |  |  |  |  |

We mainly compare our MultiAugs with representative SSHPE methods PoseDual [] and SSPCM [] under five conditions:

C1: Firstly, we conduct experiments on the COCO train-set with 1K, 5K and 10K labeled data, and evaluate on the val-set. As shown in Tab. 5.3 and Fig. 5.3, our method brings substantial improvements under the same setting. For example, when using a Single-Network, our method exceeds both PoseCons and PoseDual significantly, and is close to the SSPCM based on three networks. When using a Dual-Network, our method exceeds previous SOTA results by 2.8 mAP, 1.3 mAP, and 1.1 mAP under 1K, 5K and 10K settings, respectively. Note that our method can bring greater gains with less labeled data (*e.g*., 1K images), which further explains its efficiency and superiority.

C2: Then, we conduct larger scale SSHPE experiments on the complete COCO dataset by using train-set as the labeled dataset and wild-set as the unlabeled dataset. As shown in Tab. 4, regardless of using any backbone, our method can always improve all supervised baseline results, and bring more gains than two compared SSHPE methods [] and [] with using dual networks. When using a single network, our method is still superior to PoseDual [], and fairly close to SSPCM [] based on triple networks.

C3: Besides, we also report our results using HRNet-w48 on the COCO test-dev in Tab. 5. Our method can slightly outperform the PoseDual but fall behind the best SSHPE. We attribute it to our fewer training epochs and less parameters which may lead to weaker generalization. Besides, we can observe that MultiAugs outperforms some burdensome transformer-based methods [, , ], which reveals the significance of rational utilization of unlabeled data.

C4: Finally, we allocate the MPII train-set as a labeled set and the whole AIC as an unlabeled set. This is a more realistic setting where labeled and unlabeled images are from different datasets. Tab. 6 shows results on the MPII val-set. Our method outperforms both the supervised HRNet [] and semi-supervised PoseDual [] by a large margin under the same backbone. It is worth noting that our semi-supervised MultiAugs with applying the model ensemble can even approach the supervised HRNet with using extra labeled AIC.

C5: Then, as shown in Tab. 7, our results on the MPII test-set surpass all those of previous fully supervised methods and two semi-supervised counterparts PoseDual and SSPCM. This further validates the effectiveness and superiority of our proposed method.

### 5.4 Ablation Studies

To further study our proposed MultiAugs, we conduct ablation experiments under the setting of COCO 1K with a total of 30 epochs. The learning rate drops 10$\times$ twice separately at epochs 20 and 25. All studied models take ResNet18 as the backbone, and follow the Single-Network framework.

| Augmentations | Year | mAP |
| --- | --- | --- |
| $T_{RA}$ [] | CVPRW’2020 | 41.9 |
| $T_{TA}$ [] | ICCV’2021 | 40.2 |
| $T_{YOCO}$ [] ($T_{RA}$) | ICML’2022 | 42.5 |
| $T_{YOCO}$ [] ($T_{TA}$) | ICML’2022 | 41.6 |
| $T_{JC}$ [] | ICCV’2021 | 42.4 |
| $T_{JO}$ [] | CVPR’2023 | 41.9 |
| Ours ($T_{JCCM}$) | — | 42.7 |
| Ours ($T_{JOCO}$) | — | 43.7 |
|  |  |  |

#### 5.4.1 Comparing to Other Advanced Augmentations.

Three advanced augmentations are selected: RandAugment [], TrivialAugment [] and YOCO []. We refer to them as $T_{RA}$, $T_{TA}$ and $T_{YOCO}$. For $T_{YOCO}$, it may be based on $T_{RA}$ or $T_{TA}$. And we compare them with previous SOTA augmentations $T_{JO}$ and $T_{JC}$ for SSHPE, and our recommended $T_{JOCO}$ and $T_{JCCM}$. We did not compare to Mixup families [, , , ] or AutoAug families [, , , ]. Because Mixup is verified not to work with SSHPE, and AutoAug needs to search optimal parameters. Finally, as shown in Tab. 5.4 and Fig. 5.4, our optimal combinations are always better than $T_{RA}$-based or $T_{TA}$-based $T_{YOCO}$, which are meticulously designed and also composed of existing basic augmentations. These further prove the superiority and conciseness of our synergistic combinations.

#### 5.4.2 Training Techniques of Multiple Heatmaps.

In this part, we present additional techniques commonly used for unsupervised consistency training. Especially, for predicted multi-heatmaps, we propose to optimize them by applying the multi-loss learning (ML) as in Eq. 4. Other two alternative techniques are confidence masking (CM) and heatmaps fusion (HF). For CM, the consistency loss term in each mini-batch is computed only on keypoint channels whose maximum activation value is greater than a threshold $\tau$, which is set as 0.5. For HF, also termed as heatmaps ensemble, we sum and average multi-path heatmaps to obtain a fused heatmap for loss computing. Then, we compare MultiAugs (e.g., $m_{11}$ and $m_{12}$) using either of these three techniques ML, CM and HF.

As shown in Fig. 11, our ML is strictly superior than the other two techniques under either $m_{11}$ or $m_{12}$. For CM, we assume it may filter out some keypoint heatmaps with low confidence but high quality. This surely has a negative impact. For HF, although it is widely used in other SSL tasks for model ensemble, it may not necessarily be applicable to our intermediate keypoint heatmaps. We deem this is because each predicted heatmap is distinctive and meaningful (see Fig. 11). It is tricky to replace them equivalently with a fused heatmap. In comparison, our multi-loss learning is a simple yet effective choice.

Figure: Figure 10: The convergence curves and best mAP results of two MultiAugs schemes with using three different training techniques.
Refer to caption: extracted/5456722/figures/MultiHeatsASV2.png

## 6 Conclusions

In this paper, we aim to boost semi-supervised human pose estimation (SSHPE) from two perspectives: data augmentation and consistency training. Instead of inventing advanced augmentations in isolation, we attempt to synergistically utilize existing augmentations, and handily generate superior ones by novel combination paradigms. The discovered collaborative combinations have intuitive interpretability. We verified their advantages in solving the SSHPE problem. For consistency training, we abandon the convention of stacking networks to increase unsupervised losses, and train a single network by optimizing multi-path consistency losses for the same batch of unlabeled images. Combined with the optimal hard augmentations set, this plain and compact strategy is proven to be effective, and leads to better performance on public benchmarks.

Last but not least, we declare that the synergy effects of augmentations and multiple consistency losses are generic and generalizable for other SSL vision tasks such as image classification, object detection and semantic segmentation. We will explore them in the future.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See above
soundness: 2
strengths: 1. The paper is well-written and easy to follow
2. Personally, the idea of ranking basic augmentation looks interesting to me. From my own experience, heatmap-based prediction has different behavior from augmentations from traditional classifier-based models so this line of work looks interesting to me.
3. The experiment results show decent amount of improvement.
summary: This paper presents a method for SSL 2D Human Pose estimation that improves existing methods from two aspects: data augmentation and better consistency loss design.
weaknesses: 1. It seems that the dual network design is still a dominant approach based on experiment results. The single network with proposed components still cannot outperform dual network approaches.
2. How does the performance look like with ResNet-50? Some other work also evaluates with this model. Honestly, I don't really think ResNet-18 is that important these days.
3. The improvement on the coco training + coco wild seems to be marginal.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: - Based on the results in Table 3, the proposed method exceeds previous SOTA results.
- The rank and the combination of augmentations for semi supervised learning is important, and this work first try to dive deeper into this study.
summary: This paper aims to boost semi-supervised human pose estimation from two perspectives: data augmentation and consistency training. For data augmentation, this paper conducts empirical studies to get the easy-hard rank of augmentations (joint cutout, joint cut-occlude, randaugment, cutmix, etc.) for human pose and recommend two augmentation combinations (i.e., a cutout after joint cut-occlude and a cutmix after joint cutout). For consistency training, the paper proposes using one easy augmentation to supervise multiply hard augmentation. Experiments on COCO and MPII datasets show that the proposed method outperforms existing SOTA methods.
weaknesses: 1. L.95 introduces a principle "Do not combine MixUp-related augmentations" while L.224 P1 claims that "A global $T_{MU}$ does not make sense for the HPE task." It would be better to provide more details. What is the meaning of global? Moreover, it seems mixup is used in some heatmap-based works like [R1], and it works well.

- [R1] From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation. CVPR2021.

2. It would be better to include $T_{CM}$, $T_{CO}$ and $T_{MU}$  in Table 1.

3. Section 3.1 concludes that the rank and the combination of data augmentations are entirely based on empirical studies.  For example, it is easy to understand $T_{A60}$ is harder than $T_{A30}$, but it is not intuitive to understand $T_{JC}$ is harder than $T_{CO}$. I am wondering if any insights or theoretical analysis can be provided to help readers understand more intuitively. For example, L. 228 claims that "we thus nominate the most likely superior combinations: TJOCO and TJCCM". The two combinations are useful. However, can we have any insights or principles for other augmentations which not discussed in this paper. It would be meaningful if we can quickly rank any new augmentations. 

4.  Section 3.1 did not discusses the parameters of each augmentation. I am not sure the default parameters of each augmentation. Moreover, I am wondering if the parameters will have significant influences on the rank and the combination. For example, Dual-Network (Xie et al., 2021) prefers to use JC 5 and RA 20 to show the parameters directly.

5. Section 5.3 discusses more augmentations and training techniques. But they are not ablation study. It would be better to clearly show the baseline and the effect of each module/strategy.

6. It would be better to show some qualitative results in the main paper or in the supplementary.

7. I am not sure the comparison between the multi-path consistency loss and others is fair enough due to the difference of batch size. It would be better to compare the pair {$I_{e1}$,...,$I_{en}$} + {$I_{h1}$,...,$I_{hn}$} with {$I_{e}$} + {$I_{h1}$,...,$I_{hn}$}, and also discuss the influence of batch size of other SOTA methods.

8. Minor comments
- there is an incorrect usage of cite in L.47.
- there are incorrect punctuation in formulas like Eqs.3 and 4.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: It would be helpful if the authors could address key areas including Novelty and Differentiation, Comparison with Foundation Models, Justification of the SSHPE Setting, as well as provide insights on the suggested experimentation, ablation studies, and clarity improvements. Clarifying these aspects would significantly enhance the understanding of the paper’s contributions. Addressing these points would also help position the work within the broader landscape of current methods, highlighting any distinct impact and practical relevance.

# UPDATE (After Discussion Period)

I appreciate the author's response and thanks for sharing a detailed response and addressing concerns related to novelty, experimentation and ablation studies. The statement made by the authors, "Almost all of the above methods have not been or cannot be tested on the 2D HPE datasets used in this paper," appears unjustified and lacks merit. Several existing approaches, including foundational models like SAPIENS and BEDLAM, offer the capability to perform zero-shot evaluations on 2D HPE datasets without requiring additional training. Conducting such evaluations is straightforward and would provide a meaningful benchmark against proposed methods. This comparison is critical to establish the relevance and competitive performance of the proposed method in the broader context of foundation models.

Without such comparisons, the study's scope of semi-supervised learning remains speculative, as it lacks a clear understanding of how the proposed method measures up to existing works. Hence, **I would like to retain my current rating**, as the lack of rigorous comparative evaluation constitutes a significant shortcoming.
soundness: 2
strengths: 1.	Focus on a Practical Problem: The paper addresses a fundamental and labor-intensive challenge in 2D human pose estimation (HPE)—the need for extensive labeled data. By leveraging semi-supervised learning (SSL) to utilize unlabeled data, the approach targets a practical solution that could reduce the dependency on costly and time-consuming data annotation.
2.	Innovative Augmentation Strategies: The authors identify a unique contribution by combining existing data augmentations to create “easy-hard” augmentation pairs that introduce a wider difficulty spectrum. This approach leverages the synergistic effects of established augmentations to generate novel HPE-specific augmentations, potentially enhancing model robustness by training on more challenging, noise-introduced variations of data.
3.	Simplified and Effective Consistency Training: Instead of relying on complex multi-network architectures, the paper proposes a single-network design that optimizes multiple losses through sequential multi-path predictions on heavily augmented data. This simplified approach is both interpretable and efficient, making it accessible for broader implementation while maintaining strong performance gains.
4.	Commitment to Open Science: By releasing code for academic use, the authors support transparency and reproducibility, enabling other researchers to build upon this work and further validate the approach across diverse HPE tasks and datasets.
summary: The authors propose a method to enhance semi-supervised human pose estimation (SSHPE) through synergistic data augmentation and multi-path consistency training. Instead of creating isolated or complex augmentations, they combine existing augmentations in a complementary way that intuitively benefits SSHPE, achieving stronger results through these collaborative transformations. For consistency training, they forgo traditional multi-network stacking in favor of training a single network with multi-path consistency losses across multiple augmented views of the same unlabeled image batch, yielding both efficiency and accuracy gains.
weaknesses: ### Limited Novelty

The proposed method in this paper lacks sufficient novelty and fails to contribute meaningful advancements to the field of semi-supervised human pose estimation (SSHPE). Human pose estimation, as noted, is already a thoroughly researched area with many well-established methods for both data augmentation and consistency training. Specifically, the augmentations Joint Cutout (TJC) and Joint Cut-Occlude—are already known [2]. Similarly, the concept of consistency training has been explored extensively in prior work (e.g., Xie et al., 2021; Moskvyak et al., 2021; Li & Lee, 2023; Huang et al., 2023), reducing the originality of the current approach.

Additionally, this paper’s methodology appears closely aligned with SSPCM [2], sharing a similar approach to consistency training without offering meaningful differentiation both in terms of novelty and accuracy (Table 3,4). The methods do not extend beyond existing approaches like POST [1], which employs consistency training alongside augmentation but tackles a more complex problem involving significant distribution shifts—something that requires a more robust approach. Given that the paper neither introduces novel methodologies nor demonstrates improvements over current state-of-the-art techniques, its contributions are limited in both theoretical and practical impact. Consequently, the paper falls short in providing sufficient novelty or relevance to warrant further consideration.

The current approach to selecting augmentations is somewhat rudimentary; exploring more advanced augmentation generation paradigms, such as PoseAug [3], could substantially enhance the contribution by introducing more targeted, pose-specific transformations. To further strengthen the paper’s impact, the authors could also address more complex challenges within semi-supervised human pose estimation, such as managing significant distribution shifts or effectively handling occlusions, both of which would demonstrate the robustness and adaptability of the proposed method in diverse and realistic scenarios.

### Lack of experminets

The proposed method for semi-supervised human pose estimation (SSHPE) would benefit from a deeper comparison with existing foundation models, particularly in a zero-shot setting. For instance, models like Sapiens [4] have demonstrated strong generalization in zero-shot applications, and evaluating the proposed SSHPE approach against such foundation models would clarify its relative strengths and weaknesses. Additionally, the paper overlooks a critical experiment involving the use of high-accuracy 3D pose estimation models, such as those utilizing the SMPL model, which effectively parameterizes human poses, handles occlusions, and supports 3D-to-2D projection via camera transformations. Testing whether training a 2D pose estimation model under a semi-supervised setting indeed outperforms simply projecting a state-of-the-art 3D model (e.g., BEDLAM [5], which is trained solely on synthetic data) into 2D, would be crucial in establishing the necessity of the SSHPE approach. Given BEDLAM’s potential for zero-shot 2D projection, it is likely that both Sapiens and BEDLAM could outperform the proposed SSHPE model without additional training.

These omissions raise an essential question about the need for SSHPE approaches in the presence of robust, generalizable foundation models and accurate 3D pose estimators. To justify the SSHPE setting, it is recommended that the authors add a dedicated section explaining its relevance and necessity. Additionally, expanded experimental results are needed to establish a clear advantage of the proposed method over these established models, particularly to demonstrate that the proposed approach fills a specific gap that existing foundation models and 3D estimators do not.

### Results

Loss Function Effectiveness: Currently, the paper does not provide sufficient insight into the effectiveness of the proposed loss functions,  $\mathcal{L}_u$  and  $\mathcal{L}_S$. Including an ablation study to examine the individual impact of these losses on labeled and unlabeled datasets would provide a clearer understanding of their contributions. This would help substantiate the effectiveness of each component and offer more transparency on how these loss functions drive model performance.

Comparison with UDA Methods: The semi-supervised setting described shares similarities with unsupervised domain adaptation (UDA) approaches, suggesting a relevant opportunity for comparison. Including benchmark results against state-of-the-art UDA methods, such as UDAPE [6], which leverages consistency loss minimization, would highlight the unique value of the proposed method. Adding these comparisons in Table 4 would show how the approach fares relative to established UDA methods and underscore its novelty and advantage in this space.

### Lack of clarity
The paper lacks essential clarity, making it challenging for readers unfamiliar with the field to follow the methodology and results. Specifically:

Undefined Variables: Key variables such as  $u$, $l$, $N$, and $M$ in lines 181-182 are not clearly defined, which disrupts the reader’s ability to interpret the equations and understand their significance within the proposed method.

Unclear Justification for Downsampled Heatmaps: In line 161, there is a reference to predicting downsampled heatmaps without any clear explanation or citation to support this choice. Given that downsampling can impact resolution and accuracy, it’s essential to provide references to prior methods that utilize this approach and clarify why it’s suitable in this context.

### References

[1] Prior-guided Source-free Domain Adaptation for Human Pose Estimation, ICCV 2023  
[2] Semi-Supervised 2D Human Pose Estimation Driven by Position Inconsistency Pseudo Label Correction Module, CVPR 2023  
[3] PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation, CVPR 2021  
[4] Sapiens: Foundation for Human Vision Models, ECCV 2024  
[5] BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion, CVPR 2023  
[6] A Unified Framework for Domain Adaptive Pose Estimation, ECCV 2022

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors need to explain how they divide the different sections of Table 5. This is currently unclear from the table.
2. What is the effective training time of the proposed approach under different conditions/networks?
3. What is the significance of Fig.7 when Table 3 already presents the same results?
soundness: 3
strengths: 1. The authors propose a simple and seemingly effective way to improve 2d human pose estimation from RGB images using advanced data augmentation combinations and a multi-path augmentation framework for training a single network.
2. Table 3: the proposed approach outperforms other baselines showing its effectiveness on the COCO val set with different labeled train set sizes.
3. Table 4: the proposed approach outperforms other baselines when using the Resnet50 and ResNet101 architectures showing its effectiveness on the COCO val set when using the entire COCO labeled train set with unlabeled wild set for training.
4. Table 9: Results on WEPDTOF-Pose which is an indoor dataset indicate that the proposed approach may be generalizable across datasets.
summary: The authors propose a method for boosting 2D human pose estimation performance in a semi-supervised setting. Towards this, the authors propose two cores: advanced data augmentation and concise consistency training ways. They hypothesize that these two cores help the model’s interpretability, benefitting from the augmentations, and aid it in performing superior to SoTA semi-supervised approaches.
weaknesses: 1. While the overall approach seems to work somewhat effectively, the contribution in the novelty aspect is limited. The authors propose a method to combine the different augmentations to create sensible hard augmentations but the takeaways are generalized in nature and seem obvious. For example, selecting combining augmentations in Sec 3.1 does not seem to take out-of-the-box thinking and may be deduced from running a combination of experiments.
2. Table 4: while the gains are noticeable when using ResNet as the backbone, the same cannot be said for HRNet as the backbone. This raises concerns regarding the effectiveness of using the proposed augmentations and approach. Can the authors justify why choosing a different architecture may not result in a noticeable performance boost? The current results indicate the proposed approach may be architecture-dependent.
3. Sec 5.2 S3 → is there a reason the authors do not train for 400 epochs if it is the only other thing besides the number of networks presenting their method from performing not as well as SSPCM?
4. Table 5: the proposed approach performs similarly to DUAL on the test set. This raises concerns over the generalizability of the method.
5. Sec 5.2 S1 → Superior performance on 1 dataset for a small number of samples may not necessarily generalize to using small train sets from other datasets. Is it possible to show results on other datasets, e.g. MPII (which is already evaluated), H3.6, or LSP? Additionally, can the authors justify this statement? The authors state in Dataset Setup that the small subsets of 1K, 5K, and 10K were randomly selected. From the reported numbers, the corresponding baselines do not seem to be trained on the same selected labeled set. It may then be likely (not certainly) that the reported numbers for the proposed approach may be biased towards the evaluation set because of the selected subset for training.
6. S5 Table 8: While the overall performance is better by 0.1 (not significant), the method outperforms 4 of 7 joints (as opposed to claiming all).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see the weakness part.
soundness: 2
strengths: 1.This paper identifies that the existing SSHPE methods lack rigor in ranking the difficulty levels of applied data augmentations and discovers synergistic effects among different augmentations. It proposes a more rigorous difficulty ranking for data augmentations.
2.This paper provides a comprehensive evaluation of existing advanced data augmentation methods. Rather than designing new augmentation techniques, the paper employs rule-based constraints to combine existing augmentations, nominating the most likely superior combinations: TJOCO and TJCCM.
3.A novel multi-path approach that applies multi-path augmentations and multi-path loss training to a single network can surpass certain dual network.
summary: The paper proposes a data augmentation method with the consistency training strategy to improve the performance of semi-supervised 2D human pose estimation,
weaknesses: 1.The introduction of new data augmentation requires a re-evaluation of the selection of the optimal augmentation combination.
2.The combined data augmentation will increase the training time. Is this time consumption still lower than that of stacked networks?  
3.Different combinations may yield varying performance across datasets, so can the selected optimal combination consistently provide the best results?
4.The selection process for the optimal augmentation combination is time-intensive.

5. Although the paper provides a ranking of data augmentation techniques, the criteria for evaluating the difficulty levels of augmentations appear somewhat heuristic and may lack a strong quantitative foundation.

6. The paper lacks experiments using only the multi-path loss, making it difficult to determine the individual performance improvements contributed by the multi-path loss and the two new data augmentations.

7. The paper does not clearly explain the rationale for choosing the TJOCO and TJCCM data augmentations.

Meta-review: This paper received five reviews, and the authors submitted a response addressing the queries raised. While two reviewer shifted to a positive stance after the rebuttal, albeit with some reservations, the others retained their initial ratings, which were largely below the acceptance threshold. Key concerns included the paper's marginal improvements over existing work and the lack of essential experiments for fair comparisons.

The overall consensus is that the paper requires significant revisions to meet publication standards. After careful consideration, the Area Chair panel decided not to accept the paper in its current form. We encourage the authors to address the reviewers' feedback thoroughly and submit a stronger, more comprehensive version in the future.

Decision: Reject