Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies loss functions in deep learning under imbalanced data. It focuses on the training and testing behavior after the training error is zero. It systematically reviews the previous loss functions, builds a connection between various loss functions and linear (cost-sensitive) SVMs, and then proposes a general way to set loss functions that recovers previous loss functions as special cases. It is claimed that this generalization is beneficial in improving the performance in the balanced test set for deep learning under imbalanced data beyond overparameterization. Specifically, the paper analyzes the performance metrics in fair machine learning like balance error and difference in equal opportunity and shows that the combined strategy in the loss function setup is beneficial for achieving more fair results.


main_review: The technical contribution is novel and interesting. I do have many questions which I detailed below. However the writing of this paper, I would say, is a bit messy. I also detailed my suggestions below.
 
On the technical side, I mainly have the following detailed feedback and questions:
1. This is probably also due to writing issues but I generally have the question that how the two problems: imbalanced data and group sensitive (fair) learning connect with each other. Focusing on each of them could be a big topic. It turns out that, (at least my understanding is), good group sensitive learning results (when evaluated in those metrics) is a showcase/application of the proposed method. So the latter one is a by-product of a better understanding of the first problem. However, in the intro of this paper, it somehow seems you study the two problems in parallel. So if my understanding is correct, the writing should be changed. Or if my understanding is wrong, please clarify.

2. I am not quite familiar with the literature on connections between different loss functions and maximum margin classifiers. However, I guess one legitimate question is, how the analysis in the linear classifiers generalizes to the deep learning cases. And how useful these theories are. For example, in the analysis, there is a bounded norm constraint, do you also implement this constraint in practice? Why or why not?

3. This paper studies loss functions. My naive understanding is the optimization of these loss functions is all easily implemented easily by back-propagation. However, I feel it’s good to show how gradients or prediction forms would change when you use these losses. One related question is, how are they related to label smoothing?

4. The general form of the loss function is good but also has more parameters. And it seems in the experiments, they are just chosen by grid search (5.1). However, the connection between the proposed loss and the cost-sensitive SVM seems to suggest the loss should be set by user knowledge (after all in CS-SVM), like we penalize mistakes in certain classes more. Can you comment on how to choose those parameters?

5. On the group sensitive learning, an alternative approach is we just train for the performance metrics, which of course can be hard. But how learning from imbalanced data using specific loss functions is connected with group sensitive metrics? Is it connected by a certain way to set the parameters? Otherwise, we just can hope it works but we do not have guarantees or further understanding.

6. The terminology can also be confusing here. There is recent work on multiplicative gradient update in learning, (not in loss functions), so I was confused at first. 

7. A final question, how imbalanced data learning differs from balanced data after overparametrization? Did you try your method on balanced data?

I have some feedback on the writing: 
1. I found multiple instances of referring to equations or notations before definition, which makes this paper a bit hard to read. For example, in line 104, equation 2 is only introduced later. In line 88, \pi_y is not defined. My suggestion is, at the beginning of the paper, you should prepare the reader better but do not try to go to too many details and notations. 

2. The order of the content can flow better. I would suggest giving the conclusion and takeaway first and early in the paper before going to details. For example, section 3.1 is inductive, getting from observation to theory. But the observation part is too detailed and the important message is buried after those details.

3. The experimental section is filled with many experimental details, but I feel this paper needs more results and analysis, rather than details. 

limitations_and_societal_impact: I do not see the immediate negative societal impact. But as a paper discussing fair learning metrics, I think it could mention the "better" learning results on these metrics are not guaranteed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the training loss used by neural networks for class-imbalanced learning/group-sensitive. It wants to investigate among two terms, multiplicative and additive, added to the weighted cross-entropy loss, how do they contribute to the class-imbalanced/group-sensitive learning and in what phase. By studying this problem, the paper did some experiments to get some insights and proved the equivalence of the solution achieved by the weighted cross-entropy loss with the additional terms, and the solution achieved by the cost-sensitive SVM on a linear model. The conclusion is drawn that the multiplicative term is more effective than the other one. On the other hand, it discusses that at the initial phase, the additive term may take effect. By additional distribution assumption of GMM model, the paper also derives generalization analysis and the trade-off between balanced error and imbalanced error. Finally, it shows some experimental results.

============
Thank you for the rebuttal. I think the contribution of the paper is that it studies from the theoretical aspect a commonly met problem, and the insights drawn from theoretical studies can be used to design better loss functions.  In this way, I would keep my positive rate for this paper. 
main_review: The paper is a quite novel theoretical study of the imbalanced loss functions used. While it is intuitively correct that adding both the terms may make the weighted cross-entropy loss better suitable for class-imbalanced data, there are no theoretical studies on this aspect. Specially, the paper provides some quite novel insights from the theoretical studies, that one commonly used term is effective in the end phase, and another one is only effective in the initial phase of training. I believe such a conclusion has practical applicability in designing new algorithms. This paper, as a theoretical study, is significant not only theoretically but also practically. 

Technically, the paper also gives solid theoretical studies. Its claims are well supported by the theorems provided, as well as the experimental results. However, as the authors admitted in the conclusion, this is still far from deep-net practice while its focus is mainly on linear models. While we understand clearly that a linear model is not usually considered as an overparameterization model, the paper still needs efforts to make itself more “deep” from the theoretical point of view. 

The paper is generally clearly written, and there are a lot of discussions about the intuitions/insights drawn from the theoretical studies. On the other hand, it seems that the paper fails to follow the NIPS style. The layout is denser and the citations do not follow conventional form. The paper may need to give a clearer definition of some words used in the introduction, such as “favorable”/”favoring” and “individual mechanisms”. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigate the prediction problem under the settings of label-imbalance and group-sensitive. Previous literature propose several methods to solve it, such as weighted Cross-Entropy, additive/multiplicative adjustments on logits. This paper explore the mechanism and effect of these methods, and analyze the relationship to SVM and CS-SVM. It propose to combine additive/multiplicative adjustments to enjoy the benefit of both methods. The experimental results demonstrate the effectiveness of VS-loss which combine additive/multiplicative adjustments.
main_review: The problem of label-imbalance and group-sensitive has been studied by some previous literature. This work combines additive/multiplicative adjustments on logits to achieve improvement on balanced error and DEO. The related works is well cited.

The technical contribution is not very sigificant. But the statement is well supported by theoretical analysis and experimental results. Although the theoretical results is built on the setting of simple data generation process abount Gaussian mixture, experimental result shows that the method can generalize to complex scenarios. I think the author can try to be extend the analysis results to the setting under weak assumption in future work.

The paper is almost clearly written. However, since the theory is built on fixed feature setting, I am not clear that whether the representation of data is fixed and only the classification layer is trained in the experiements. I guess the answer is yes. It would be better to emphasize it in the experimental section.
limitations_and_societal_impact: The limitations are addressed in the paper. The potential negative societal impact of the method is not mentioned. It could be better to include it in conclusion section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper mainly shows two insights: 1) When optimizing in the terminal phase of training, multiplicative logit adjustment is critical. 2) Compared with multiplicative adjustment, additive adjustment can speed up the convergence at the beginning of the training. Motivated by the insights, a new vector-scaling (VS) loss is proposed to exploit the advantages of both multiplicative adjustment and additive adjustment, which is further introduced to group-sensitive settings. What’s more, a generalization analysis of the VS-loss on binary over-parameterized Gaussian mixtures is presented, revealing tradeoffs between balanced/standard error and equal opportunity.
main_review: Originality: This paper reveals how multiplicative and additive adjustment methods affect margins and proposes a new loss function which combines the well-known LA and CDT loss functions. Additionally, a further extension to group-sensitive classification is given.
Quality: Some insights and theoretical analyses are given in this paper, which is instructive but not sound enough. Specifically, to prove the multiplicative adjustment can be harmful at the initial phase of training, this paper claims that pushing the margin towards majorities requires Δ_+<Δ_-, which is derived from theorem 1. However, this conclusion may be improper for the following reasons: 1) Theorem 1 assumes feature mapping h(x) are linearly separable, which is unpractical. 2) according to the existing works, e.g., [1] and [2], reweighting methods could be harmful for the representation learning at the initial phase, thus it is not clear whether we should push the margin towards majorities at the beginning of network training. 3) This paper proves CDT loss function pushing the optimization in the wrong direction when f_w (x)  = 0, which can be inadequate for f_w (x)  = 0 is an extremely special case.
More experiments are required to prove the effectiveness of the proposed VS-loss. Specially, experiment results on iNaturalist, ImageNet-LT and Cifar-10/100-LT with imbalanced ratios 50 and 10 are required.
Clarity: This paper is clearly written, well-organized and provides adequate information to readers.
Significance: This paper provides some important results including the analysis and the extension to the group-sensitive learning. However, due to the lack of sufficient experiments, it if difficult to compare it with previous state-of-the-art methods.

[1] Zhou B, Cui Q, Wei X S, et al. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 9719-9728.
[2] Kang B, Xie S, Rohrbach M, et al. Decoupling representation and classifier for long-tailed recognition[J]. arXiv preprint arXiv:1910.09217, 2019.


limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper has obtained four favorable reviews. The reviewers laud the novelty of the theoretical study of the imbalanced loss functions used.  It is also practically valuable in the sense that it is useful to develop new algorithms. Many researchers would be interested in the result. The AC agrees with the reviewers. 
Decision: Accept (Poster)