Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper tackles the class-imbalanced problem with a semi-supervised learning scenario. Unlike the previous approaches, which require a complicated sampling strategy and multiple training pipelines, the authors provide a simple and unified framework, UDAL, by connecting the ideas of progressive distribution alignment (proposed for imbalanced semi-sup) to logits adjustment (proposed for imbalanced sup). This approach incurs no additional training time on top of the underlying semi-supervised learner. Significant empirical improvement on widely used benchmarks (CIFAR-10-LT, CIFAR-100-LT, and ImageNet-127) demonstrates the effectiveness of the proposed method.
main_review: **Pros.**

- **Clarity**. Overall, the writing is clear and easy to follow.
- **Important problem with a simple and effective solution**. Considering the class imbalance scenario is an essential step for applying SSL in a more realistic scenario but has been less explored. The proposed method can be used with a simple modification of the existing baseline and it shows significant empirical gain; hence, it has the potential to be widely used to mitigate this problem without additional burdens. Although the proposed method can be viewed as a simple combination of the existing methods, I believe that the simplicity and effectiveness of the proposed method will be interesting to the reader in ICLR.

**Cons.**

- **More general imbalanced semi-supervised learning scenarios.** Although I agree that a considered scenario (i.e., distributions of both labeled and unlabeled are same) is most natural, I wonder that the proposed method can be applicable to more generic scenarios (i.e.,  distributions of both labeled and unlabeled are different). Is there any way to apply the proposed UDAL to such a challenging scenario?
- **Effect of the strategy in (Kang et al., 2020).** In Section 4.3., the authors present that they use a deferred re-sampling strategy which is introduced by Kang et al., 2020 ("... spend the very last stages of training aligning to a relatively balanced class distribution..") Can the authors give the details about this? Also, is this necessary for empirical improvement?

Kang et al., Decoupling representation and classifier for long-tailed recognition., In *ICLR*, 2020
summary_of_the_review: Although the proposed UDAL can be viewed as a simple combination of the existing methods (progressive distribution alignment and logits adjustment), I believe that the simplicity and empirical effectiveness of the proposed method will be interesting to the reader in ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This approach combines distribution alignment with logit adjustment, resulting in an efficient method for solving the aforementioned problem while improving performance in the test setting. Unlike existing state-of-the-art approaches such as CReST, the approach involves no sampling state, and imbalance mitigation is achieved just by modifying the loss functions of the model. Experiments are conducted over three benchmark vision datasets of varying complexity: long-tailed versions of CIFAR10, CIFAR100, and ImageNet-127. Experimental results are competitive with or exceed other methods on the tested datasets with 5x training speedup compared to CReST.
main_review: Strengths:
+ This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This is an underexplored problem that is relevant to the AI/ML community and has practical real-world impact.
+ In contrast to other papers addressing this problem, the proposed approach relies only on diistribution alignment without sampling-based strategies. This is done by moving the distribution into the cross-entropy loss computations of recent semi-supervised learning algorithms: FixMatch and MixMatch.
+ This work connects ideas of progressive distribution alignment with logit adjustment from the fully supervised, imbalanced learning setting.
+ By considering distribution alignment alone, training time is reduced (5x reduction from CReST).
+ The method achieves significantly better performance characteristics as more labeled data becomes available, and the approach scales to larger datasets, outperforming the best existing method on ImageNet-127 (in terms of accuracy).
+ The proposed method is well-framed in the existing literature.
+ Section 2 provides a nice overview of the prerequisities needed to understand the method.
+ Eqs. 8 and 9 are interesting insights.
+ The approach is simple, requiring only a few additional lines of code to implement with no alteration to the training scheme and no increased training time.
+ The approach only has two hyperparameters which need to be tuned.
+ Experiments are conducted over three benchmark vision datasets of varying complexity: long-tailed versions of CIFAR10, CIFAR100, and ImageNet-127.
+ Experimental setup seems reasonable, and follows a similar procedure to CReST.
+ Experimental results are competitive or exceeds other methods on the tested datasets.
+ Ablations are run to validate each component of the proposed approach.

Weaknesses:
- Imbalanced semi-supervised learning, while an important problem, is not a new problem, somewhat limiting novelty.
- The proposed approach is a combination of two existing approaches, somewhat limiting novelty.
- Experiments are conducted on only a single network architecture. It would be useful to show it works on a wide range of neural network architectures.
- It would be useful to see if the results are statistically significant.

Questions:
- What happens if the assumption that the unlabeled set of examples is imbalanced in a different way than the training examples?
- Please define "strong" and "weakly" augmented versions of the unlabeled example.
- Are there any significant drawbacks to this approach compared to others?
- What happens in the balanced setting; does it fall back to a standard semi-supervised learning approach?
summary_of_the_review: This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This is an underexplored problem that is relevant to the AI/ML community and has practical real-world impact. In contrast to other papers addressing this problem, the proposed approach relies only on diistribution alignment without sampling-based strategies. This is done by moving the distribution into the cross-entropy loss computations of recent semi-supervised learning algorithms: FixMatch and MixMatch. The paper is generally well-structured and clearly written. The experimental set up is sound, and experimental results show the promise of the model. The approach is well-grounded in the existing literature, and its motivation is clear. As it stands, I do not see any major flaws with the approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on class-imbalanced semi-supervised learning and proposes a new method that combines distribution alignment and logit adjustment techniques. Experiments show that the proposal can achieve performance improvement.
main_review: This paper focuses on class-imbalanced semi-supervised learning and proposes a new method that combines distribution alignment and logit adjustment techniques. The proposal is simple. However, the proposed method relies on the assumption that the imbalance distribution between labeled and unlabeled datasets is the same. This is a very strong prior knowledge and in reality, we can not obtain this knowledge.  Thus, the application of the proposal is limited. Moreover, the new proposal is mainly based on existing techniques, although combining these techniques can achieve a performance improvement, the novelty and contribution are limited.
summary_of_the_review: This paper proposes a new method for class-imbalanced semi-supervised learning. However, in my view, the assumption of the proposal is too strong to satisfy and the novelty of the proposal is limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies class-imbalanced semi-supervised learning. To handle this problem, a unified approach is proposed by combing distribution alignment (DA) and logits adjustment (LA). In particular, this paper proposes to apply DA and LA to both supervised and unsupervised losses, which is new. This method shows significant improvement over baselines on three datasets. 
main_review: ## strengths
1. This paper studies an important problem which is underexplored in literature;
2. The proposed method is simple yet effective, which shows substantial performance gains on multiple datasets;
3. This paper is well written and easy to understand;
4. The proposed method is reasonable that aligns the distribution of both supervised and unsupervised losses.

## weaknesses
1. The novelty is limited. As mentioned, the proposed method is pretty much based on existing works, i.e., DA and LA, which are used and demonstrated to be helpful in class-imbalanced tasks. Importantly, the improvements of the methodologies are limited. For example, it is very natural to extend LA to unsupervised loss by using pseudo-labels. In this regard, this paper does not examine the quality of the generated pseudo-labels. Does the distribution of pseudo-label make sense? For the DA in Eq. (10), it immediately follows previous work, with a minor difference by introducing an additional parameter $k$;
2. More baselines should be compared. In the current version, this paper mainly compares its proposal with CReST+. I am aware that CReST+ is a strong baseline, but it would be more convincing if more baselines could be compared, such as [1,2,3];
3. Hyperparameters. From Figure 3, it is observed that the performance of the model is quite sensitive to the choice of $\alpha_{min}$;
4. In Table 4, the results for Supervised models are not informative. To better quantify the gap, LA or other techniques that handles class imbalance, should be also applied.


[1] DISTRIBUTION-AWARE SEMANTICS-ORIENTED PSEUDO-LABEL FOR IMBALANCED SEMI-SUPERVISED LEARNING.

[2] ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning. NeurIPS 2021.

[3] Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning. NeurIPS 2020.
summary_of_the_review: My main concern is that the novelty is limited, which makes this paper like an incremental work. Therefore, I recommend rejection.
review_title: null

Meta-review: Thanks for your submission to ICLR.

Reviews were fairly mixed on this paper, with two reviewers advocating for accepting the paper and two advocating for rejecting the paper.  There were some concerns raised by the reviewers, most notably novelty and some issues with the experiments.  After rebuttal, the negative reviewers maintained their scores and the positive reviewers were somewhat less enthusiastic.  In the end, the paper is quite borderline and could really go either way, but it seems that the paper could use another round of reviewing, particularly to make sure the issues raised by the reviewers are adequately addressed.

Please do keep in mind their comments when preparing a future version of the manuscript.
Decision: Reject