Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles an imbalanced semi-supervised learning (SSL) problem by proposing a new way to construct the pseudo labels of unlabeled samples, coined INlier Pseudo-Labeling (INPL). Unlike the previous approaches, which use the softmax confidence from the training classifier, INPL instead uses the energy score calculated by logsumexp of logits. As INPL only changes the score function for unlabeled samples, it is compatible with the existing imbalance or semi-supervised learning methods. Also, it does not incur additional computational costs. Significant empirical improvements on widely used imbalanced SSL benchmarks (CIFAR-10-LT and CIFAR-100-LT) and more general scenarios (existence of OOD samples in unlabeled data or balanced class distribution) demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: **Pros.**

- **Simple and effective solution for an important problem**. Considering the class imbalance scenario is an essential step for applying SSL in a more realistic scenario but has yet to be explored. The proposed method can be used with a simple modification of the existing baseline, and it shows significant empirical gain; hence, it has the potential to be widely used to mitigate this problem without additional burdens.

**Cons.**

- **Omitted baselines in robust semi-supervised literature**. The proposed idea, which considers the imbalanced SSL under the OOD detection framework, is interesting and somewhat new. However, at the same time, there are concerns about its technical novelty since many relevant works in robust SSL literature have yet to be mentioned and compared [1,2,3,4]. Overall, these works detect the OOD samples in unlabeled data using their score functions; while there are no works that use energy score, but using the energy score to detect the OOD samples is not a new one [5]. Hence, in some sense, the proposed method could be viewed as a combination of existing techniques for the new problem. So, it seems to be necessary to compare with the previous robust SSL methods as well. 
- **Absence of large-scale experiments.** Previous imbalanced SSL works have proposed the experimental results on large datasets such as ImageNet-127 [6] or LSUN [7]; however, INPL has only been demonstrated under relatively small benchmarks (CIFAR-LT-10 and 100). For a real-world application, experiments on such large benchmarks are essential.

[1] Nair et al., RealMix: Towards Realistic Semi-Supervised Deep Learning Algorithms., arXiv:19.12

[2] Guo et al., Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data., ICML 20

[3] Saito et al., OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers., NeurIPS 21

[4] Park et al., OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data., ECCV 22 (w)

[5] Liu et al., Energy-based Out-of-distribution Detection., NeurIPS 20

[6] Wei et al., CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning., CVPR 21

[7] Lee et al., ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning., NeurIPS 21
clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality**. Overall, the writing is very clear and easy to follow. In addition, the organization of the main draft is well-established.

**Novelty**. While the proposed idea is a new thing in the imbalanced SSL, it has been investigated in the other relevant literature; hence, this work is limited in technical novelty.

**Reproducibility**. The authors provide a detailed presentation of the experimental setups, hence it seems to be easy to reproduce the results.


summary_of_the_review: As aforementioned, this paper is quite limited in the aspect of technical novelty. However, the empirical results are significant, and the presentation is clear. Hence, I believe that the acceptance of this work will be a useful addition to the ICLR community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a new perspective of pseudo-labeling for imbalanced SSL.
Without relying on model confidence, they propose to measure whether an unlabeled sample is likely to be “in-distribution,”; i.e., close to the current training data. 
To decide whether an unlabeled sample is “in-distribution” or “out-of-distribution,” the authors adopt the energy score from out-of-distribution detection literature. 
As training progresses and more unlabeled samples become in-distribution and contribute to training, the combined labeled and pseudo-labeled data can better approximate the accurate class distribution to improve the model. 

strength_and_weaknesses: Strength
- The authors propose a new perspective to tackle the imbalanced semi-supervised learning problem, and the proposed solution is original.
- The proposed method shows favorable performance compared to the recent methods. 

Weakness
- It would be better to improve the explanation of the motivation. For example, Fig. 3 says that the energy-based criterion selects fewer incorrect samples for the pseudo-labels. However, it is hard to see that in the figure. It would be better to provide a quantitative measure to show precisely how many incorrect samples are selected with which criterion.
- Also, instead of discarding the fixmatch-based criterion, how about using both criteria? In other words, use the pseudo-labels when the confidence is above a certain threshold, and the energy is below a certain threshold.
- More importantly, there is no analysis of the design choice for the proposed method. Instead of simply borrowing the popular energy function, it would be better to analyze how to modify the loss function or what is the best hyper-parameter (at least about the energy threshold) for the best performance.
- Finally, there is no evaluation result on realistic datasets such as iNaturalist or ImageNet datasets. Otherwise, it is hard to verify that the proposed method is helpful for realistic scenarios.

clarity,_quality,_novelty_and_reproducibility: Despite minor concerns, the proposed method is clear and novel. The quality of this paper looks favorable overall.
summary_of_the_review: Despite several concerns about the experiments, the observation in this work is interesting, and the proposed method is novel.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies imbalanced semi-supervised learning. SOTA imbalanced semi-supervised learning methods are often based on pseudo-labeling and consistency regularization, which still relies on confidence thresholding. In this paper, the authors formulate the pseudo-labeling problem as a classification problem, i.e., using the energy score to determine where an instance is in distribution or out of distribution. The authors claim that the proposed approach is more reliable compare to existing ones. Empirical results demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths

1.	The idea is simple and clear, and the paper is easy to understand and follow.

2.	This paper studies an important problem in imbalanced semi-supervised learning, i.e., the reliability of the pseudo labels.

Weaknesses

1.	The authors claim that the confidence-based methods are not reliable in imbalance semi-supervised learning, but it is not clear why the proposed method is reliable. The proposed method and the existing methods both depend on pre-defined thresholds.

2.	The motivation of the method is not very strong, and the contribution of this work is limited. The authors only change the criterion to pseudo-labeling an instance with energy score without proposing a new SSL approach.

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility of this work is good, but the novelty and contribution are limited to some extent.
summary_of_the_review: Generally speaking, this paper proposes a simple method for imbalance semi-supervised learning, but the proposed method is not new. The authors did not give a sufficient discussion about the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new pseudo-labeling method based on the energy instead of a confidence for its application to imbalanced semi-supervised learning. Experimental results show improvement over existing methods.
strength_and_weaknesses: * Strength
  - The paper is written clearly. 
  - The proposed method is very simple and shown to be effective. While simple, I tend to believe it is more than incremental, as it demonstrates consistent improvement across multiple SSL settings.

* Weakness
  - The comparison between the energy-based scoring and the confidence-based scoring should be more in-depth. Especially, the negative energy score $T \log \sum_{i=1}^{K} \exp(\frac{f_{i}(x)}{T})$ approximates a max logit $\max_{i=1}^{K} f_{i}(x)$ when there is a dominant one. Given such similarity, it is somewhat surprising that this simple change leads to such a huge difference.
    - The energy formulation is not invariant w.r.t the addition of values to logits, i.e., $E(f(x)) \neq E(f(x) + \mathrm{constant})$, while confidence is, i.e., $\mathrm{confidence}(f(x)) = \mathrm{confidence}(f(x) + \mathrm{constant})$. Would this difference be one of the reasons why energy formulation is better than the confidence, as the magnitude of the embedding is taken into account?
    - One interesting control experiment is to use a max-logit-based pseudo labeling, as it is similar to energy score in the sense that it is variant to the addition to logits.
clarity,_quality,_novelty_and_reproducibility: * The paper is clearly written.

* While simple, it shows significant empirical improvement.

* The paper contains sufficient detail for reproducibility.
summary_of_the_review: The paper presents a simple trick that works well across SSL benchmarks. The paper could be improved with more in-depth analysis on why it works better, but overall I think it is a simple method that would benefit the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an interesting approach, Inlier Pseudo-labeling (IPL), towards the challenging imbalanced semi-supervised learning (SSL) problem. A nontrivial insight is observed by the authors: To decide whether an unlabeled sample is "in-distribution'' or "out-of-distribution'' can be beneficial to imbalanced SSL. The proposed approach is based on the energy score from out-of-distribution detection literature (LeCun, 2006). While simple, the IPL approach achieves significantly better performance compared to the standard method, FixMatch. When combined with other more recent SoTA methods, the improvement is much smaller but still significant. A strength is that the IPL works well on the large-scale ImageNet dataset.

Paper received rather controversial comments at first. Authors actively prepared the rebuttal, providing with most new results and elaborations requested by the reviewers. After discussion, three reviewers are on the positive side, while only one reviewer is on the negative side. AC carefully read the paper, the reviews, the revised version, and the rebuttal, and acknowledge that the response has addressed the open concerns reasonably well. Considering that introducing the energy-based criterion to SSL is new, and the simple IPL works generally well for imbalanced SSL datasets, AC believes the paper is worth to be seen by the SSL community at the conference.

In addition to the current revised version, authors shall integrate all rebuttal material into the paper, especially the elaboration on the motivation of the energy-based score and why it is beneficial.
justification_for_why_not_higher_score: The simple approach, IPL, works well for several SSL datasets. Simple, neat, and solid approach that works is always a good addition to the conference. Nonetheless, presenting it as a poster is a proper decision, given that the energy-based criterion is leveraged from the well-established outlier detection field to the SSL field.
justification_for_why_not_lower_score: Reviewers are in general consensus of acceptance. While one reviewer has a lower score (3: Reject), the concerns were addressed reasonably well after the AC's investigation.
Decision: Accept: poster