Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve the well-known pseudo-labeling by Confident Sinkhorn Allocation (CSA) which confidently assigns the labels via optimal transport. The authors demonstrated CSA’s performance by comparing it with other SSL strategies.
strength_and_weaknesses: Strength:
1. CSA first conducts sample selection by removing low-confidence samples and then focuses on assigning labels. The overall framework is convincing and easy to follow. 
2. The variance of correct assignment, i.e., epistemic uncertainty, is considered for evaluating the uncertainty, which is an interesting idea.

Weaknesses:
1. The so-called domain-specific and pseudo-labeling has been attributed to two separate components which were thought based on the different designing principles. Please refer to [1].
2. I understand that authors would like to show correct assignments affect the optimal decision boundary by theorem 1. As it extends Yang&Xu’s theory, I am curious if it gives a better theoretical conclusion, e.g., closer to the decision boundary.
3. The discussion about the most provable class and second-most probable class is related to the theorem presented in the work [1], because both considered the relative relation between outputs.
4. T-test for sample selection is based on training M models, which incurs much computation cost. Note that the dropout used in Rizves et al is only involving inference, but CSA is with an ensemble style. An explanation about it should be added.
5. Another concern is that T-test is used for sample selection, but it has not used epistemic uncertainties, i.e., the variance of I. Notice that total variance is designed for multilabel classification, is it feasible for multiclass semi-supervised classification?
6. The contribution compared with Tai et al from the label assignment section is not very convincing to me. Does CSA outperform theirs by T-test and some additional constraints over class population size?

[1] Taming Overconfident Prediction on Unlabeled Data from Hindsight, https://arxiv.org/abs/2112.08200


clarity,_quality,_novelty_and_reproducibility: This paper is overall good and well presented. I personally would not encourage more research efforts on improving pseudo-labeling. The major reason is that studying this specific topic for SSL is narrow because according to the experimental results, such an elaborate design only slightly enhances the SSL performance. It is okay for the method such as FixMatch back to two years ago, but it is not very interesting today. 
summary_of_the_review: I have listed my concerns about this work above and I am expecting the authors' response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an improved pseudo-labeling method for semi-supervised learning methods. The paper first proposes a method for filtering out some unlabeled data that has high uncertainty based on  Welch’s T-test and  Total variance.  The paper then extends the methods of SLA [1] to assign labels for unlabeled data.



strength_and_weaknesses: **Strengths**:

* The paper proposes a method that does not need a predefined threshold to filter the unlabeled data


**Weakness**:

* The proposed method has limited novelty. First, considering uncertainty in pseudo-labeling is not new, e.g. [2] considered this scheme. Second, using optimal transport is not new as well. It is proposed in SLA [1].


* The proposed method does not need a predefined threshold to filter the unlabeled data, but the method needs additional hyper-parameters, such as fraction of assigned label $\rho$, which counteract the benefit of the proposed methods.


* The proposed methods need additional computation cost due to optimal transport calculation and multiple model training on the labeled data.  


References:

[1] Sinkhorn label allocation: Semi-supervised classification via annealed self-training. ICML.2021

[2] In defense of pseudo labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. ICLR 2021
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written.

The novelty is limited.

The results seem to be reproducible.
summary_of_the_review: This paper proposes an improved pseudo-labeling method for semi-supervised learning methods, but the novelty of the proposed methods is limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel problem for pseudo-labeling based semi-supervised learning. Especially, the authors proposed Confident Sinkhorn Allocation (CSA), which assigns labels to only examples with high confidence scores and learns the best label allocation via optimal transport.
strength_and_weaknesses: (+) The paper is generally well-written.

(+) The theoretical analyses of this paper are solid. 

(+) The experiments are sufficient.


(-) I feel that the proposed method is computational expensive. In each iteration, M models should be trained on labeled data, and an inner loop exists for Sinkhorn’s algorithm. Therefore, I want to see the total running time comparison of the proposed CSA and baseline methods. Besides, how about the scalability of the proposed method?

(-) Also regarding the model assembling, I doubt that the performance improvement of CSA over other comparators in Table 2 is due to this operation. Moreover, I found that the performance improvement on various dataset is quite marginal. Therefore, I want to know whether the comparison is fair, and whether the comparison demonstrates the effectiveness of the proposed way for calculating the uncertainty. Maybe, the authors should conduct statistical significance test to justify the superiority. 

clarity,_quality,_novelty_and_reproducibility: As mentioned previously, I think the paper is generally well written. The details of the proposed algorithm are clearly stated.
summary_of_the_review: I like the solid theoretical analyses of this paper, although the experiments are comparatively weak. I would temporarily give a positive mark for this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studied pseudo-labeling for semi-supervised learning and proposed a better uncertainty-aware method. The main issue is the limited novelty given the existence of related methods, for example, domain-specific and pseudo-labeling has been attributed to two separate components which were thought based on the different designing principles. Moreover, the significance may also be an issue: the paper focused on tabular data, the computation should be expensive, and the improvement looked marginal. The rebuttals by the authors didn't successfully resolve the concerns from the reviewers, especially the novelty issue. Consequently, we cannot accept it for publication. If the authors would like to revise the paper and try another machine learning conference, I suggest the authors to improve the quality of experiments in the revised manuscript (even better if the novelty can be further improved).
justification_for_why_not_higher_score: The main issue is the limited novelty given the existence of related methods.
Decision: Reject