Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel imbalanced SSL method with pseudo-labeling.
The authors introduce a bias-adaptive module that can be trained to minimize
classification loss on a balanced dataset with bi-level optimization.
strength_and_weaknesses: ### Pros

1. The paper is easy to follow and well-written.
2. The problem is well-motivated and intuitively makes sense.
3. The reported performance exceeds the baselines significantly.

### Cons

1. The balance loss L_bal is just a subset of the original pseudo-labeling loss L 
   (balanced dataset B is a subset of the labeled dataset D_l).
   Thus, if L is minimized, L_bal will also be minimized (if we are overfitting L).
   The effect of introducing a balanced dataset here is unclear.
   If L_bal is a hold-out dataset (instead of "dynamically sampled from the labeled training set"), 
   there may be some differences.

2. A much simpler method is to keep the bias-adaptive module fixed when updating other parameters
   and update the bias-adaptive module on the balanced dataset. 
   I suggest the authors compare this baseline method and discuss the difference compared with
   the proposed method. The bi-level optimization may be overkill.

3. Theoretical analysis of the convergence of such a bi-level optimizing method is necessary,
   which may help understand the problem in 2.

clarity,_quality,_novelty_and_reproducibility: Clarity: easy to understand.

Quality: good.

Novelty: fair.

Reproducibility: good.
summary_of_the_review: The paper is overall good-looking, but the method does not make much sense to me.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a semi-supervised learning (SSL) method, especially for imbalanced classification. Since standard SSL methods based on pseudo labels often assign labels of head classes to unlabeled data, the obtained pseudo labels tend to have a small number of labels for tail classes, resulting in low classification performance. To cope with the problem, the bias adaptive classifier is proposed to reduce bias from imbalanced data. To train neural networks with the bias adaptive classifier, a bi-level optimization approach is employed. Through extensive numerical experiments, the proposed method outperformed various recent SSL methods. Also, the ablation study showed the usefulness of the proposed bi-level training with the bias adaptive classifier.
strength_and_weaknesses: ##### Strengths
- The idea of the proposed bias adaptive classifier is novel and has an impact on imbalanced SSL.
- The bi-level training is important to train a bias adaptive classifier and its effectiveness is carefully investigated through an ablation study.
- The superior performance of the proposed method was demonstrated on extensive numerical experiments and various recent SSL methods were compared with the proposed method.

##### Weaknesses
- The current explanation of the proposed method is not really clear. The presentation of the proposed method could be improved. 
- It seems that the proposed method requires intensive computation compared with the other methods because the proposed method requires a second-order gradient. If the computation time is provided during the rebuttal phase, I will reconsider this point.
clarity,_quality,_novelty_and_reproducibility: The explanation of the proposed method could be improved. Despite the novelty and quality of the proposed method, the explanation in Section 3.2 does not provide clear intuition why the proposed method works and the idea is effective. The current explanation says that a residual connection is key. However, the explanation, such as "the proposed bias attractor can assimilate a wide range of training biases", does not provide an in-depth understanding of why the proposed method works. My current understanding is that $\Delta f_\omega$ absorbs the bias from imbalanced distributions. I would be grateful if more in-depth explanations were provided for the proposed method.

Some notations are ambiguous. In Eq.(2), $\boldsymbol{I}$ and $\circ$ seem not to be defined. Also, $|_{\phi^t}^T$ in $G_i$ in Proposition 3.1.

In Figure 5, $\gamma_l=100$ and $\gamma_u=1$ were chosen for t-SNE visualization. It is interesting to see the results of the other values of $\gamma_l$ and $\gamma_u$. 
Such a result might show that a high-quality representation can often be obtained by the proposed method, i.e., the result in Figure 5 is not owing to the specific choice of the hyperparameters.

Appendices B and C seems to be useful information for reproducibility. Also, the authors declare that the source code will be made publicly available in Section 1. The results in this paper might be able to be reproducible by the code.
summary_of_the_review: This paper proposed the bias adaptive classifier for imbalanced semi-supervised learning. The idea looks novel, and the empirical results support its effectiveness. The reproducibility seems high. If the clarity of this paper is improved, the overall score will be increased.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a semi-supervised learning algorithm for unbalanced data. The key component is a bias adaptive classifier to alleviate the problem that the model is usually biased towards the majority classes. The bias adaptive classifier, together with the regular classifier, are learned through a bi-level training protocol, where the model us updated first and a sampled balanced set is used to update the bias attractor parameters. Experiments on several long-tailed dataset show the outstanding performance compared to existing solutions.

strength_and_weaknesses: This paper is well written and easy to follow. The idea of using another classifier to rectify bias introduced in the original classifier is intuitively sound. Moreover, the formula and design is actually very straightforward yet effective, allowing others to easily reproduce the proposed method and add the proposed module into any SSL framework. Technically, the approach of bi-level training makes a lot of sense to me, when considering the cost function to be enforcing the similarity between gradient of a sample and the average gradient of a sampled balanced set. In addition, this architecture seems works pretty well under various imbalance conditions. 

This work relaxes the assumption used by some existing works that unlabeled data and labeled data should follow similar distribution, and experiments show that it can handle both this case and reversed case (unlabeled data distribution is reversed from labeled data distribution). Apart from consistent gain over compared algorithms, this paper also provides solid ablation study to highlight a few important observations on how the proposed algorithm performs regarding different perspectives, such as pseudo-label quality improvement, capability of handling extreme label ratios. Such study is valuable and helps evaluate the contribution of this work, making this paper convincing.

There are several minor weakness, listed as follows:

- Although the related works section covers most of recent SSL algorithms for class imbalanced problem, there are more papers that should be cited:

   -  [1] Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding, 2022
   -  [2] Debiased Learning from Naturally Imbalanced Pseudo-Labels, 2022

  In particular, [2] also talks about debasing for pseudo-labels under class imbalance scenario, which is quite related to this paper. It would be good to include and discuss about them.

-  The motivation of the proposed bias adaptive classifier is still a bit vague to me, although the structure looks simple. I understand that it takes into account the second order gradient when computing the upper training loss $L_{bal}$, but could not draw a crystal clear connection between updating bias attractive parameters and enforcing model to make more prediction towards the minority classes. How such architecture, i.e., residual connection, linear and non-linear classifier design, and bi-level training helps alleviate classifier bias? More intuitive explanations would help. Additionally, how does the sampled balanced set affect performance of the bias adaptive classifier?

Some minor grammatical errors:

- Note that CReST+ (Wei et al., 2021) fail in … -> fails in
- In summary, such an nested training… -> such a
- design a be-level learning algorithm … -> a bi-level
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and organized. It proposes novel solution for class-imbalanced SSL and the proposed method is relatively straightforward. With detailed information on training recipe, I think it would not be difficult to reproduce it.
summary_of_the_review: In summary, this is overall a well-written, well-organized paper and the proposed method is technically sound. Despite its simplicity, the proposed method consistently outperforms existing works on SSL for class imbalance problem, and the improvement is quite significant. There are a few unclear pieces regarding motivation of the bias attractor parameters and missing references.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) The paper provides a new method to reduce bias towards the majority class in learning from labeled and unlabeled data.

(b) Strengths: The work is thorough and the authors responded to the reviewers carefully.

(c) The work is very incremental. It is a typical ML paper in the sense that it proposes a new algorithm that is a combination of known components, is merely heuristic, and is more complicated than previous methods. The experiments are merely on known datasets and the % gains are small. I don't see a new insight in the paper, or an indication that some task is now feasible that previously was not. 

After discussion within the program committee, this paper is accepted for publication at ICLR 2023. Two reviewers are definitely positive. Here are some additional comments that we hope will be useful to the authors and to readers.

- The thoroughness is admirable but ideally would lead to more insight. For example, reviewer juK4 asked for a convergence analysis, which the authors have provided, but this merely says that training converges at the expected rate, which provides no insight about generalization.
- Ideally, the authors would show that some task is feasible with the new method that wasn't before, and they would show success on a novel dataset or application of their own.
- A central topic is class imbalance. For decades, and despite the thouisands of citations of https://dl.acm.org/doi/abs/10.5555/1642194.1642224, many researchers have failed to compare against the simplest approach, namely shifting scores that are obtained using log loss by a constant. The missing comparison here against logit adjustment, "Long-tail learning via logit adjustment" https://arxiv.org/abs/2007.07314 is in this vein (none of the reviewers are connected to this paper).

Wrt insights, here are comments extracted from the original reviews and then from replies by reviewers. Score 5: "The paper is overall good-looking, but the method does not make much sense to me." Score 7 raised to 8: "The current explanation of the proposed method is not really clear." Later: " When I first read this paper, I read the figures and explanations several times to understand this paper, but it did not help a lot. To write this review, I found out my personal understanding/interpretation of why this method works. But, I would like to know the easy-to-follow explanations from the authors. The current one is a little bit better than the first one, but I hope that the explanations can be improved." Score 8: "The motivation of the proposed bias adaptive classifier is still a bit vague to me" Hopefully the final version of this paper will address this feedback successfully.

justification_for_why_not_higher_score: Incremental and not exciting to other researchers.
justification_for_why_not_lower_score: Thorough work and careful replies to reviewers.
Decision: Accept: poster