Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed new cost-sensitive losses for training over-parameterized models. The losses are "calibrated" (see details in the main review). The experiments show the proposed method has gained in terms of a certain metric.
main_review: First of all, I have to claim that the paper is beyond my expertise and I'm not familiar with the literature. I asked to have the paper unassigned but it didn't work out. I'm not confident about my review.

Clarity & Significance: 

The paper is not so clear for me (perhaps it is just because I'm not an expert). 

I'm not quite familiar with the terminology "calibrated", does it mean "Fisher consistency" or something related? I mean, optimizing a certain surrogate loss can achieve the Bayes optimal classifier given infinite samples with a hypothesis of all measurable functions. It should be defined rigorously. 

The paper also presents some propositions. Only proposition 1 is quoted from existing work. I'm not quite sure about the others. Any of them is the main theory of this paper? They are the same and natural/simple to me according to the presentation. If there is some technical contribution, please highlight it.

Further, the discussion of the literature, especially the related ones is not sufficient. For instance, the authors claim that Eqn. 7 is a simple generalization of existing work. But in which sense? Making them "calibrated" or something else? I didn't see the significance and necessity.

As for the experiments, the results of the methods differ a lot under two different measures. The proposed method does not perform as well as the baseline in terms of the avg. acc. Then which one do we really care about and any deeper analysis? Lastly, the experimental settings are typical? It seems quite arbitrary to define a classification task. It would be better to consider some widely adopted settings and compare to a larger family of existing methods.


Originality & Quality: 

I cannot tell the originality of the paper as I do not know the literature at all. I give a temporal weak reject and update my score based on the authors' feedback and discussion with other reviewers. 


limitations_and_societal_impact: The clarity of the paper can be improved. See details above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use logit adjustments for class imbalanced training settings with over-parameterized models.
They introduce logit adjustments with general gain matrices (in constrast to only diagonal gain matrices).

main_review: The paper is well written, and nicely introduced.
The work is very interesting, however, it seems like the contribution of the work is limited considering the related works.

There are two very similar works (which have been cited by the paper). https://arxiv.org/pdf/2007.10740.pdf https://arxiv.org/pdf/2007.07314.pdf
Ren et al. (https://arxiv.org/pdf/2007.10740.pdf) also use logit adjustment on the same task, while they use a ResNet-34 instead of the here used ResNet-56.
It is not clear to me, why ResNet-56 is much more overparameterized than ResNet-34. I see that it is larger, but it is not that much larger.
The operation on overparameterized models seems to be the main contribution of this paper, it does not convince me that the proposed setting is overparameterized and (that at the same time) in the work by Ren et al. it is not overparameterized.

I acknowledge that this work is a generalization of those previous works. 
However, the empirical distinction of your work to the previous works is not strong.
As Menon et al. and Ren et al. train on the same task, I would strongly encourage you to include these baselines (as well as some of their baselines) into your paper. 

More clearly distinguishing between prior work and your contributions and using prior works as baselines would improve my score.

Minor:
l. 143 function -> functions

Comment on Confidence:
I carefully checked the math in the main paper. 
limitations_and_societal_impact: The empirical evaluation lacks baselines and the scope of contribution is limited.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author proposed a new approach for optimizing non-decomposable metrics that focuses on over-parameterized models (i.e., deep learning models). Rather than performing reduction and reweighting as in the traditional models for non-decomposable metrics, the proposed model modifies the logits of the over-parameterized models. The author also shows that the logit modifications are calibrated with respect to the original metric. Finally, the author shows the effectiveness of the proposed models on CIFAR and TIN datasets.

main_review: Overall, I like the idea of modifying the logits instead of reweighting the samples or classes, as it has been shown in previous literature that sample reweighting does not work as intended in the over-parameterized models. The author shows that the proposed logit modification can align the over-parameterized models with the original metrics better. I also like the consistency property of the proposed loss.
I have questions that I would like the author to address. 
1. Since the author mentioned that the proposed method works for complex metrics, could the author explain how generic the model is. Could the model handle other types of non-decomposable metrics? For example, the F-score or the MCC score, or ranking-based scores. If someone wants to apply the proposed model on a particular non-decomposable metric, how much modification is needed?
2. Related to the first question, how to construct the G matrix for a particular metric?
3. The logit adjustment technique presented in the paper is the inner step of the full training algorithm, but it was not so clear from the main paper. I suggest the author provide more explanation in the main paper rather than putting it in the appendix.
4. Some related papers are missing from the reference: e.g., Fathony & Kolter (2020) and Luo et. al. (2021)

Fathony, R. & Kolter, Z.. (2020). AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in PMLR 108:4130-4140

Luo, Junru, Hong Qiao, and Bo Zhang. "A Minimax Probability Machine for Non-Decomposable Performance Measures." arXiv preprint arXiv:2103.00396 (2021).
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers multiclass classification with "non-decomposeable" objectives: that is, objectives that are not the sum of individual terms over sample points. The focus is on problems that can be reformulated in terms of a single saddle point structure. These are often recast as a sequence of cost-sensitive problems, but this strategy can be bad when the learning method applied at each step tends to achieve zero error. The authors show that considering instead a sequence of problems with non-standard losses losses leads to significantly better empirical results. A partial justification for the non-standard losses is that they are calibrated (so their optima correspond to optimal classifiers). 
main_review: In a classification problem with $m$ classes, each classifier $h:\mathcal{X}\to [m]$ has an associated confusion matrix $C[h]$ that describes how often it predicts a $j$ when the true label is a $i$. The "non-decomposeable" objectives described in the title of the paper are losses that can be expressed in a saddle point fashion in terms of $C[h]$, like for example:
$$(\star) \max_{h\in mathcal{H}} \min_{\lambda\in S}\sum_{i,j}G_{ij}(\lambda)C_{ij}[h] ,$$
where the matrix $G$ depends linearly on $\lambda$. Such metrics are quite natural and include e.g. the minimum recall over classes for a given $h$.

The present submission notices that, for a fixed $\lambda$, $(\star)$ is a cost-sensitive classification problem that can be solved by standard techniques. Therefore, a solution to the saddle point problem $(\star)$ could obtained via a sequence of these cost-sensitive problems alternated with updates to $\lambda$. However, this strategy may perform poorly for finite samples. Intuitively, when the base learner tends to interpolate data, then the cost-sensitivity constraint is irrelevant, as training loss will be zero anyway. 

This paper presents an alternative approach whereby the cost-sensitive learning problems are replaced by certain learning problems. The salient feature of the latter is that the losses used are calibrated: that is, a classifier that achieves minimum population surrogate loss is the optimal one corresponding to an "ideal" iteration of the saddle point problem. Calibration is not quite a practical result, since in practice it is rarely the case that the population surrogate loss is exactly minimized. Still, this property lends some credibility to the procedure developed here. Moreover, the method performs well in experiments with CIFAR-10, CIFAR-100 and TinyImageNet. The authors also consider the effect of post shifting and distillation on the final result. 

This submission is quite well written. Mathematical statements and proofs are mostly well explained, and the authors give very precise descriptions of their experiments. As I explained above, calibration is not a decisive result proving their approach works, but it is nonetheless an interesting result. To this reader, the experimental results are convincing ; however, I would have liked to have access to code, and larger experiments would make a more compelling case. Overall, my evaluation of the paper is positive, although I also feel I am not a specialist on the topics of the submission. 

An additional (minor) comment is that the proof of Prop. 2 does not seem to work if there are repeated feature vectors with distinct labels in the dataset $S$.


 




limitations_and_societal_impact: They have.
review_title: null

Meta-review: The paper proposes new calibrated cost-sensitive losses for training over-parameterized models.  The reviewers were unanimous in their opinion that the paper is above the acceptance threshold.  The approach is well supported theoretically, and covers an interesting family of non-decomposable losses.  The authors have promised to release code at the time of publication in response to a reviewer comment.
Decision: Accept (Poster)