Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce a semi-supervised learning algorithm (SSL) for class-imbalanced scenarios.  In particular, an Auxiliar Balanced Classifier coupled with a 0/1 mask is used to favor the cost regularization. The methodology is interesting, and the experiments demonstrate that the proposed method outperforms state-of-the-art techniques on image classification tasks, even for large datasets under challenging imbalance ratios.
main_review: Though the paper does not present an outstanding theoretical advance, the methodology seems to benefit real-world tasks for semi-supervised and imbalanced datasets. Overall, the manuscript is easy to follow, holding careful experimentation and method comparison.

Pros. 

-A consistency regularization approach is coupled within a deep learning framework for image classification under unlabelled data.

-In general, deep learning models treat class-imbalanced-datasets problems only from labeled data; this algorithm uses unlabelled data during training.

-The experiments include several scenarios (easy and complex), demonstrating that the author's proposal outperforms the included literature.

-An ablation analysis is presented, describing in detail the role of each of the algorithm stages.

Cons. and comments

-The mathematical foundation could be enhanced. In particular, the threshold hyperparameter tau seems to be crucial for the mask. 
However, neither experimental, not theoretical insights are provided.

-Please number all the equations in the manuscript.

-Please take care with the counters notations in lines 89-92. The authors mix i, n, and m indexes.

limitations_and_societal_impact: The proposed method is helpful for several artificial intelligence applications, in particular, scenarios including semi-supervised and imbalanced datasets. Then, societal impact, though not directly referenced, is clear concerning the presented approach.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new algorithm for semi-supervised learning in the class imbalanced setting by adding to the neural architecture an additional classifier head which is trained on the balanced datasets (through resampling).

main_review: Originality.
The idea of using random resampling inversely proportionally to class cardinality is rather standard. The architectures with an additional classification head as well as consistency regularization are also known, but their combination is quite interesting.

Quality.
The experimental results show that the proposed method improves the classification performance on three real datasets. The ablation study is performed.

Clarity.
The paper is in places hard to follow. The authors should add the pseudocode summarizing the proposed approach.  Some details that seem to have a non-neglectable impact on the results, like "gradually decreasing the parameter of B(·)for consistency regularization" are not strongly motivated and can be easily overlooked. The discussion of results is sometimes laconic.


limitations_and_societal_impact: The authors measure the classification performance only with accuracy on a balanced datasets and minority accuracy. It would be interesting to see other metrics appropriate for imbalanced data such as G-mean as well.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work takes class-imbalanced data distribution into account for semi-supervised learning, which is prevailing in realistic scenarios. The problem setting is reasonable and is under-explored.

main_review: The proposed algorithm, ABC, attaches an auxiliary balanced classifier to the representation layer of an existing SSL algorithm. The auxiliary classifier uses representations learned from all data points and is trained on a class-balanced subset so that it can avoid the bias towards the majority classes, thus improve generalization performance.

I have some concerns about this work:
a) This novelty is limited. As the authors point out, the superiority of ABC probably lies in that it is trained using unlabeled data, which is not adopted by previous methods. The performance improvement of ABC seems not surprising.
b) The failure of class-imbalanced learning (CIL) techniques (i.e., BALMS, LDAM) is obvious because they do not use unlabeled data for training. But what if we combine the existing CIL methods with SSL modules to make use of unlabeled data? I think it is a straightforward solution, and I wonder what is the superiority of ABC compared to such solutions.
c) The conclusion is too wordy. Some redundant explanations can be deleted to make it more concise.

>>

Thanks for the author response. It addresses my concerns on the superior performance for the proposal. I increase my score.

limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper deals with the class-imbalanced semi-supervised learning problem. The paper utilizes existing semi-supervised learning algorithms to acquire good representation, and additionally introduces an auxiliary balanced classifier called ABC, which is trained on a class-balanced subset of a minibatch to mitigate the biased results caused by class-imbalance. The ABC is trained using not only the balanced subset but also unlabeled data with consistency loss. The experimental results show that the proposed method outperforms existing methods by large margin.
main_review: The paper deals with a practically very important problem of class-imbalanced semi-supervised learning. I found the proposed method was reasonable and technically sound. Although the novelty of the method may not be outstanding given the prior works of [17], [3] and [4], the paper presents a good extension of these prior works to the practically important problem. The experiment is basically thorough and convincing. In addition, the paper is clearly and well written, thus easy to follow. 

Overall, I think this is a good paper and has merit to the community, but I have some concerns and questions as follows.

* Is there any specific reason of using mask based on Bernoulli distribution to create the balanced subset? I think it is more straightforward to sample the same amount of data in each class to create balanced subset. For example, how about sampling data from uniform distribution using Gumbel softmax?
* According to Table 4, incorporating DARP seems harmful. Do authors have any thoughts on the reason of this phenomenon? Moreover, given this phenomenon, I think it is more reasonable to compare the proposed method with “w/ cRT” variant rather than “w/ DARP+cRT” variant in Table 1-3.
* L316-319: I am skeptical about the contribution of end-to-end training to the accuracy improvement. Looking at Table 5, the techniques related to consistency regularization, or using unlabeled data to train ABC, has dominant effect. For example, the accuracy drops to the same level as an existing method if the consistency regularization is removed; the variant of “Without using the 0/1 mask for the consistency regularization loss $L_{con}$” achieves overall accuracy of 79.0, which is close to 78.5 achieved by an existing CISSL method, namely ReMixMatch w/ DARP + cRT. 

Minor comments and questions:
* L137-138: Is $\frac{N_{y_b}}{N_L}$ a mistake for $\frac{N_L}{N_{y_b}}$? The same doubt on L163-164, too.
* L163-164: How is $N_{\hat{q_b}}$ calculated? Is it calculated mini-batch wise?
* L252: “CRT” -> “cRT”
* L293: “ABC only (without backbone)”. Does this mean a single fully connected layer is attached directly on image input (I believe not)? The same question to L335-337, too. Please clarify the exact architecture in this condition.
* Table 5: Does each row mean the proposed algorithm + the condition in each row or does it mean all the conditions described earlier are also applied? For example, does “Without consistency regularization for the ABC” mean “Without gradually decreasing the parameter of B () for consistency regularization AND consistency regularization for the ABC”?
* [17] is published at ICLR 2020, not 2019

limitations_and_societal_impact: I believe the authors addressed these aspects in section 5.
review_title: null

Meta-review: This paper presents a method for 

This paper has four favorable reviews, describing beneficial methodology for semi-supervised learning for imbalanced data, and good experimental results for real data sets. In contrast, a reviewer pointed out that the combination of the existing class-imbalance learning (CIL) and semi-supervised learning is straightforward, and therefore the novelty is limited. The authors argue that it is not straightforward to utilize unlabeled data in CIL and that there are no such studies in the past. However, related research is not entirely absent; for example, 
T. Iwata, A. Fujino, N. Ueda, "Semi-supervised Learning for Maximizing the Partial AUC," proc. of AAAI 2020 has already been proposed. 
The authors should compare the proposed algorithm not only with CIL and naive combined methods, but also with other existing related works. 
Decision: Accept (Poster)