Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a semi-supervised pose estimation approach based 
the dual mean-teacher framework. To evaluate the quality of the generated pseudo-labels, the approach makes use of the teacher networks' predicted uncertainty, where the uncertainty aims to select high-quality pseudo-labels, to be added to the unlabeled data and gradually train the model. Next, the paper presents an adversarial mechanism to build the maximum discrepancy within the student networks. Finally, the proposed approach is evaluated on three datasets (FLIC, Pranav, and Mouse), where it shows solid performance.
strength_and_weaknesses: Strength: 
- The paper studies an open problem, namely semi-supervised 2D pose estimations.
- The proposed method is simple, straightforward and easy to develop. 
- The ablation studies are thorough. 

Weaknesses:
- The results shown do not clearly show a significant improvement compared to other methods in most of the examples, e.g. Xie et al. Table1).
- The presented dual teacher framework, mean teacher, and the uncertainty estimation protocol are known. Thus, the paper lacks a major contribution. 
- Evaluation on more human pose data, e.g. MPII or COCO, would  be helpful for comparing with a wider range of methods.
- The paper writing has space for improvement. 
clarity,_quality,_novelty_and_reproducibility: The presented approach lacks implementation details for reproducibility. The clarity part of the paper needs further work, i.e. definitions and method explanation, as well as overall writing needs some polishing. Finally, the main novelty of the paper is to estimate the quality of the pseudo-labels using uncertainty measures instead of confidence scores. This is a valid point but it does not deliver a major novelty. 
summary_of_the_review: The paper presents an interesting idea but the novelty is limited. Moreover, the evaluation would be more beneficial with a more decent comparison with prior works on at least an additional standard dataset. Finally, the paper writing needs further work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a semi-supervised pose estimation method based on the Dual Mean-Teacher framework. It aims to efficiently select high quality pseudo-labels. The framework, as the name suggests, is decomposed of two Mean-Teacher modules where teacher generates pseudo-labels by taking the parameters from student with Exponential Moving Average strategy to train the student. MSE loss along with parameter adversarial loss to keep the divergence and consistency loss is used to train the model.

strength_and_weaknesses: The proposed method outperforms the other methods in terms of MSE score and the stability i.e., much smaller fluctuations on the error curve and unlike the other methods, it does not use the confidence for evaluation of quality of pseudo-labels.

The authors use 3 datasets in which two of them are much smaller dataset (Mouse, Pranav) than the other one (FLIC). In small datasets in terms of PCK@0.2 their approach outperforms the other methods whereas in the large dataset it has the lowest PCK@0.2 among all which raises the question of generalizability of this method to larger datasets. Besides, the error curve on only one dataset (Mouse) is reported.

clarity,_quality,_novelty_and_reproducibility: The idea of using mean-teacher networks looks novel and intersting. The paper is including details for reproducing their results.
summary_of_the_review: Even though the quantitative results are good; I am not very convinced with the experimental settings yet: The method is working well for small-scale data; while it is showing the inferior results for large-scale data. Also, measures are inconsistent from dataset to dataset. I’d like to request authors’ thought on these points. Currently, I’m marginally below the threshold for this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript recommends an ensemble-like approach for semi-supervised pose estimation. A teacher-student setting where self-supervised pseudo-labels are created by the teacher for unlabeled data is extended to use two teacher-student pairs. The discrepancy between the two teachers is used to assess the quality of the pseudo-labels and to select pseudo-labels that are more likely to be correct. The discrepancy between the students is used in a loss function. The method is evaluated on three datasets, showing improved results.
strength_and_weaknesses: 
**Novelty**: The idea of using an ensemble of self-supervised / semi-supervised networks is well thought of and an interesting direction. The method itself could generalize to other settings as well.

**Presentation**: The manuscript is in parts unclear regarding notation (see below). Some parts of the manuscript appear unfinished (invalid references, see below).

**Reproducibility**: I believe it is unlikely that the work can be reproduced. Details on the network architecture, training setup and hyperparameters are missing (see below).

**Experimental Setup**: The method is evaluated on three smaller datasets (which are not cited) and compared with mostly older methods. It remains unclear how it would perform on larger datasets against state of the art. Additionally, the experiments do not show if the improvement over the baseline method is due to the proposed architectural improvements or simply due to the doubeling of the network's capacity.

**Results**: The results are good and show a significant improvement over the compared-with methods.

clarity,_quality,_novelty_and_reproducibility: - p.3, chapter 3.1:  How is y used - its in {0,1}^K, but should it not represent a keypoint location? Why does for x^u_j j runs from N+1 to N+M, when x^u and x^l are already differentiated with the superscript?  Further confusing, why do later in Eq. (4), x^l_i and x^u_i exist at the same time (i.e. labeled and unlabeled x with the same index)?  
 x^l_i, y^l_i, and K are defined twice (once in the first, once in the second paragraph of Sec. 3.1).

- Eq.(8): L_p (the pose loss) is undefined (likely L_p_S1 + L_p_S2?)

- Eq.(9): I am unsure how the function C works. Does it select only two predictions, all (distinct) pairs of predictions, does it sample a set of pairs (and if yes, how many)?  
 Also, Eq. (1),(4),(5) indicate that the output of f(x|theta) is a heatmap. Yet in Eq.(9), it appears to be used as producing a coordinate.  
 x ~ ^m is undefined. What is the difference between {p ~ }_m=1^M in Eq.(9) and p~_\theta_T_1 in Eq.(10)? The notation appears to suddenly switch from f(x|theta) to p_\theta.

- p.5: The parameter adversarial loss (Eq. (6)) does not really enforce non-homogenous students, since there is no semantic 1-to-1-correspondence between their parameters. Two students can, for example, contain permutations of the same feature maps that have a small L_a, even though they compute the same function.  While the ablation in Table 2 shows a positive impact of this loss, I am unsure about its interpretation and motivation.

    Further, the evaluation in Table 2 falls short of optimizing the PAL weight. Larger weights than the presented optimum of 0.005 could lead to even better results.

    If the "PAL weight" is "\lambda_a", I'd recommend to re-use that symbol in Table 2.

- The method as presented is likely not reproducible. Details on the network architecture and training strategy and parameters are missing. Additionally, several hyperparameters are not given. Eq.(8): The values of \lambda_* are not given (except for \lambda_a in Table 2?). The threshold \epsilon from Sec. 3.5 is not given, neither is a way of deriving it (it probably depends on the dataset, since the unc-Values from Eq.(9),(10),(11) are not normalized).

- There appear to be no citiations for the datasets (FLIC, Pranav, Mouse).

- The compared-with methods appear rather "old" for deep-learning based methods (2016, 2017 and one from 2021). It is unclear why the results of other more recent work is not listed in the results of Table 1.

- While the experiments show an improvement over the baseline "Mean-Teacher", it is not demonstrated that this improvement is due to the modified training architecture, or simply because the proposed network has twice the capacity compared to Mean-Teacher.



Minor issues (typos etc.). No need to respond.

- Multiple references are made to "Eq. equation XX" - "Eq." and "equation" are redundant and one of them should be deleted.

- p.3, "is the corresponding *the* ground-truth" - delete second the

- p.4, invalid reference: "T_1 and T_2, as Eq. equation ??"

- p.7, missing \cite: "of the hard-augmented sample [Paper]"

summary_of_the_review: While the manuscript explores an interesting, general idea, it lacks important details and clarity and does not sufficiently demonstrate that the architectural improvements lead to improved results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the semi supervised pose estimation problem with the mean-teacher framework.
Specifically, the key insight is to maximum the discrepancy between two student models and use the corresponging teacher models to generate pseudo labels  to supervise each other. 
Besides, a discrepancy loss is used to maximum the discrepancy between the weights of two student models, and an uncertainty estimation method is used to evaluate the uncertainty of pseudo labels
strength_and_weaknesses: 
Strengths:
1. Well written and easy to read.
2. Theperformance seem to be good.
Weakness:
1. Key references are missing. Using two student and teacher models to supervised each other have been explored in previous works.
Specifically, the motivation of [1] is to utilize the discrepancy between two models to generate soft pseudo labels, [2] models the uncertainty of the pseudo labels in a predictive way.
[1] Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification 
[2] Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localization
2. Technique contributions are not clear. This paper seems to be the use of existing  techniques on a new task, although this is reasonable, it lacks task specific technique contributions.
3. Experimental evaluations are not convincing. (1) For semi supervise learning, a key metrich is the performance with different labeled data ratios, but this is missing in this paper.  (2) There are many ways for uncertainty estimation, the advantage of the way in this paper is not clear.
4. There are some typos in this paper, such are "the prediction of the hard-agumented sample [Paper]".
clarity,_quality,_novelty_and_reproducibility: See above Strength And Weaknesses
summary_of_the_review: My main concern is the technique contribution and experiment evaluation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper receives 3x marginally below the acceptance threshold and 1x reject, not good enough. The reviews are mostly negative: The results shown do not clearly show a significant improvement compared to other methods. The authors use 3 datasets in which two of them are much smaller dataset (Mouse, Pranav) than the other one (FLIC). This raises the question of generalizability of this method to larger datasets. Technical contributions are not clear.


justification_for_why_not_higher_score: This paper receives 3x marginally below the acceptance threshold and 1x reject, not good enough.
justification_for_why_not_lower_score: NA
Decision: Reject