Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Imbalanced classification is a significant problem that many real-world datasets are imbalanced and the model trained on these datasets can yield poor performance. To tackle this problem, this paper proposes a novel re-weighting method using optimal transport (OT). Using a balanced meta set, the weight vectors are learned by minimizing the distance between an imbalanced training set and balanced meta set. This paper demonstrates the effectiveness on multiple benchmark datasets including image and text classification.

strengths_and_weaknesses: Strengths
- The idea of minimizing the distance between an imbalanced training set and a balanced meta set is intuitive, and easy to understand. 
- Re-weighting from optimal transport (OT) perspective is novel.
- The proposed method shows good performance on multiple benchmark datasets, including image and text classification. 
- The paper is easy to follow.

Weaknesses
- The motivation of using OT for re-weighting is not fully convincing. What other automatic re-weighting methods have not been able to solve? And so, how OT can benefit ‘re-weighting’? 
- Using a balanced meta-set seems unfair compared with other methods which do not use a balanced meta-set. For example, ten training samples per class are selected from the CIFAR dataset. In the case of IF=100, the least frequent class has only 5 samples on CIFAR-100-LT. It means, the meta-set has more samples than the training set, which can give a significant information for the minority classes. For fair comparison, in my opinion, other methods should also additionally use the meta-set in their training. 
- This paper missed some important recent baselines. For example, [1] automatically re-weights samples using the gradients and does not require a meta-set. [2] proposed Balanced Softmax to fix the label distribution shift between training and testing. 

[1] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV, 2021.

[2] Balanced meta-softmax for long-tailed visual recognition, NeurIPS, 2020.

questions: Please refer to the weaknesses.
limitations: The authors addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a reweighting method for class imbalance classification based on optimal transport. Experiments on both image and text data demonstrate the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths:
1. The paper is clear and well-written.
2. The paper provides lots of empirical evidence to demonstrate the effectiveness of the proposed method.

Weaknesses:
1. This paper looks quite similar to an existing work [1]. 
- In [1], class imbalanced problem (i.e., called class-prior-shift in [1]) is studied as a case study under the same setting (i.e., also using a small balanced meta set), and according to importance weighting, the optimal weights can be computed directly (i.e., w = p_{meta}(y)/p_{train}(y)). I'm wondering why not use this simple method to compute weights if such a balanced meta set is available? 
- Also, compared with [1], the core difference may be that this paper learns weights by minimizing an OT distance while [1] is by minimizing a maximum mean discrepancy (MMD). What is the critical contribution of this paper?
- In L2RW, their baseline RANDOM (i.e, using random weights from a rectified Gaussian distribution) is quite strong. It would be interesting to compare the proposed method with the RANDOM baseline. 

2. There is another crucial work worth to be referred and discussing in this area [2].

References:

[1] Fang et al. "Rethinking importance weighting for deep learning under distribution shift." In NeurIPS 2020.  

[2] Jiang et al. "Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels." In ICML 2018.

questions: 1. I'm wondering if the method proposed in this paper can handle other settings, e.g., label noise as in the baseline L2RW?
2. In Stage1 of the proposed method, if t1 is large, would it cause an overfitting issue by pre-training on an unweighted imbalanced dataset? How to decide the value of this parameter in practice? 

--------------Post-rebuttal comments--------------

Thank the authors very much for the rebuttal and the additional results!

After carefully reading the rebuttal, some of my concerns are resolved by the additional experimental results of baselines. But my major concern still remains, i.e., the novelty of this paper may not be so strong given [1] as prior work. I agree with the authors that the two papers are the same in high-level thinking but different in the tasks. [1] considers a more general setting of distribution shift and this paper focuses on class imbalance. Also, the contribution of this paper includes using OT loss in class imbalance and empirical contributions on many datasets. I will slightly raise my score for this and the additional experimental results. 

However, the authors said the key difference is that this paper disengages the dependence of the weight learning on the concerned classifier while [1] does not. This may not be true. If I understand correctly, in [1] the weight learning is independent of the classifier when using hidden-layer-output transformation as the non-linear transformation of data. I would suggest the authors carefully check this point to avoid any potential misunderstanding about the position of this paper with related work.



limitations: It seems that the authors haven't adequately addressed the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper argues that the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set. ​Besides, the general optimizing the weights by the classification loss may lead to inaccurate learning of the weights. Therefore, the authors formulate the learning of weight vector as the distribution approximation problem by minimizing the OT distance between to-be-learned distribution of samples in imbalanced training set and another balanced meta set. To show the effectiveness of the proposed method, they conduct extensive experiments on imbalanced classification tasks including image, text, and point cloud. 
strengths_and_weaknesses: Pros:
1. The idea of sample-reweighting is not new. But the method proposed by the author that learning the sample weights by minimizing the OT distance between the imbalanced training set and a balanced meta set seems reasonably novel.
2. The paper is well organized and easy to follow.

Cons:
1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights? A further explanation is required to highlight the reasonability of the proposed method.
2. The reasonability of the statement that "the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set" also needs further consideration or explanation.
3. An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.
questions: 1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights?
2. Why are the weights obtained when the imbalanced training set is approximated to a balanced meta data better for training a classifier?
3.  An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.
4. Missing the comparison with the highly related IB (Influence-balanced loss for imbalanced visual classification, ICCV 2021) and DisAlign (Distribution alignment: A unified framework for long-tail visual recognition, CVPR 2021.).
5. Typos:
1) $\theta$ in Eq.(2) should be bolded.
2) In Eq.(10), should $\eta$-->$\lambda$?
limitations: A hold-out balanced meta-data is required.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new sampling method to cope with imbalance learning, which is based on optimal transport. The advantages of the method are illustrated on several different types of datasets. Unlike the traditional reweight approach, the authors look at the problem from the perspective of distribution matching, which leverages the OT distance between the distributions to guide the learning of weight vector or weight net.
strengths_and_weaknesses: It is a very interesting idea to reweight the training samples based on OT in the training phase, this approach is therefore more flexible than the previous OT-based approaches to long-tailed problems. Whereas previous approaches have involved adjustments to the model's results in the post-processing stage, the authors go further and apply OT to the training stage and propose a corresponding solution. The authors' experiments are well developed and they validate the superiority of their approach on multiple data sets.

In addition to the above mentioned points, I have some concerns as follows:
1) The technique presented by the authors for reweighting with OT actually proposes a similar approach in many domains. For example, the usage of OT to address domain adaptation under conditional domain matching and label shift is proposed in [1]. Similarly, reweighting of samples using OT is described in [1]. A similar approach is suggested in [2] and [3]. I hope the author can explain the difference between them.
2) The authors do not seem to be comparing their experiments with the most advanced methods, and as far as I can tell, it seems that many recently published methods are better than the results in the authors' experiments. I wish the authors had compared their experiments with more methods than the ones in the paper.
3) Also, it seems that the author's method can only be applied to classification. I wonder if the author's method can cope with imbalance in object detection, or object segmentation, and also if the method can be extended to imbalance regression? The authors should explain more about the limitations of this work.

From the results on CIFAR and ImageNet, some of the authors' results seem to have very weak boosts, for example, almost less than 1%. The authors' method seems to be limited in terms of experimental effectiveness, and I would like to see the authors take multiple experiments and average them, and report the variance of the experiments. This is what the NeurIPS conference promotes the need to report. Of course, I note that the authors state that these values are reported for text classification, but I think the authors should report these values for all experiments.

In Table 2, the author has conducted comparative experiments on CIFAR-100 to compare which cost matrix is better, but some of the results do not appear to be very different from each other, so it is difficult to derive any significant differences between these different methods in terms of the results.

The writing of the paper also needs further improvement, some issues are as follows:
1) The description of OT is very inadequate, which is very difficult to understand for many people from outside this field. I also hope that the authors will explain how Sinkhorn can be used to solve OT problems.
2) The author should present Related Work after Introduction to make it easier for others to learn about related work in this field.
3) There is some confusion in the author's notation format, e.g. for scalars, vectors, sets, parameters, etc. The author should have followed some notational conventions. It will make the text look more rigorous.

In line 313, "... indicates the effectiveness" should be "indicate the effectiveness".



[1] Optimal Transport for Conditional Domain Matching and Label Shift.

[2] Multi-source Domain Adaptation via Weighted Joint Distributions Optimal Transport.

[3] Reweighting samples under covariate shift using a Wasserstein distance criterion.


questions: See above.
limitations: See above.
review_title: null

Meta-review: metareview: This work proposes a new re-weighting method based on OT: it views the training set as an imbalanced distribution over its samples and transported by OT to a balanced distribution obtained from the meta set. The paper is well written, and the proposed method is verified on multiple image and text datasets.

Most of the comments were addressed by the authors in the discussion period. Three reviewers raised their scores. 

The major remaining concern is regarding the relationship/novelty w.r.t. [1]. As clarified by the author, the problems, algorithms, tasks, and datasets are different. The novelty of this work is also supported by most reviewers. 

-- 4F4H: "Re-weighting from optimal transport (OT) perspective is novel."

-- DJ7Q: "The idea of sample-reweighting is not new. But the method proposed by the author that learning the sample weights by minimizing the OT distance between the imbalanced training set and a balanced meta set seems reasonably novel."

-- 8X8s: "It is a very interesting idea to reweight the training samples based on OT in the training phase, this approach is therefore more flexible than the previous OT-based approaches to long-tailed problems. Whereas previous approaches have involved adjustments to the model's results in the post-processing stage, the authors go further and apply OT to the training stage and propose a corresponding solution. "

LeMk also raised the score to 4 due to "the contribution of this paper includes using OT loss in class imbalance and empirical contributions on many datasets". 

Based on the above, the work meets the bar of a NeurIPS publication from the novelty point of view, and the AC recommends acceptance.

In the final version, please clearly discuss the position and contribution of this work w.r.t [1] based on LeMk's final comments.

-- "the two papers are the same in high-level thinking but different in the tasks. [1] considers a more general setting of distribution shift and this paper focuses on class imbalance."

-- “The authors said the key difference is that this paper disengages the dependence of the weight learning on the concerned classifier while [1] does not.... in [1] the weight learning is independent of the classifier when using hidden-layer-output transformation as the non-linear transformation of data. I would suggest the authors carefully check this point to avoid any potential misunderstanding about the position of this paper with related work.”

[1] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV, 2021.
confidence: Certain
Decision: Accept