Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes
Authors: Sara Sangalli
Ertunc Erdil
Andeas Hötker
Olivio F Donati
Ender Konukoglu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Constrained optimization
class imbalance
augmented Lagrangian method

Abstract: Deep neural networks (DNNs) are notorious for making more mistakes for the classes that have substantially fewer samples than the others during training. Such class imbalance is ubiquitous in clinical applications and very crucial to handle because the classes with fewer samples most often correspond to critical cases (e.g., cancer) where misclassifications can have severe consequences.
Not to miss such cases, binary classifiers need to be operated at high True Positive Rates (TPRs) by setting a higher threshold, but this comes at the cost of very high False Positive Rates (FPRs) for problems with class imbalance. 
Existing methods for learning under class imbalance most often do not take this into account. We argue that prediction accuracy should be improved by emphasizing the reduction of FPRs at high TPRs for problems where misclassification of the positive, i.e. critical, class samples are associated with higher cost.
To this end, we pose the training of a DNN for binary classification as a constrained optimization problem and introduce a novel constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. We solve the resulting constrained optimization problem using an Augmented Lagrangian method (ALM).
Going beyond binary, we also propose two possible extensions of the proposed constraint for multi-class classification problems.
We present experimental results for image-based binary and multi-class classification applications using an in-house medical imaging dataset, CIFAR10, and CIFAR100. Our results demonstrate that the proposed method improves the baselines in majority of the cases by attaining higher accuracy on critical classes while reducing the misclassification rate for the non-critical class samples.

Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes
Sara Sangalli', Ertune Erdil', Andreas Hoetker", Olivio Donati', Enner Konukoglu
Computer Vision Lab, ETH Zürich
2 Institute for Diagnostic and Interventional Radiology. Universitätsspital Zürich
sara. sara. aaaanalll@iviii ee. ww..te..c... ch
Abstract
Deep neural networks (DNNs) are notorious for making more mistakes for the classes that have substantially fewer samples than the others during training. Such class imbalance is ubiquitous in clinical applications and very crucial to handle because the classes with fewer samples most often correspond to critical cases (e.g., cancer) where misclassification. can have severe consequences. Not to miss such cases, binary classifiers need to be operated high True Positive Rates (TPRs) by setting higher threshold, but this comes at the cost of very high False Positive Rates (FPRs) for problems with class imbalance. Existing methods for learning under class imbalance most often do not take this into account. We argue that prediction accuracy should be improved by emphasizing the reduction of FPRs at high TPRs for problems where misclassification of the positive, i.e. critical, class samples are associated with higher cost. To this end, we pose the training of DNN for binary classification as constrained optimization problem and introduce novel constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. We solve the resulting constrained optimization problem using an Augmented Lagrangian method (ALM). Going beyond binary, also propose two possible extensions of the proposed constraint for multi-class classification problems. We present experimental results for image-based binary and multi-class classification applications using an in-house medical imaging dataset, CIFARIO 10, and CIFAR 100. Our results demonstrate that the proposed method improves the baselines in majority of the cases by attaining higher accuracy on critical classes while reducing the misclassification rate for the non-critical class samples
Introduction
Deep Neural Networks (DNNs) perform extremely well in many classification tasks when sufficiently large and representative datasets are available for training. However, in many real world applications, it is not uncommon to encounter highly-skewed class distributions, i.e., majority of the data belong to only few classes while some classes are represented with scarce instances. Training DNNs on such imbalanced datasets leads to models that are biased toward majority classes with poor prediction accuracy for samples of minority class. While this problematic for all such applications, it poses an even greater issue for "critical" applications where misclassifying samples belonging to the minority class can have severe consequences. One domain where such applications are common and machine learning is having important impact is medical imaging.
In medical imaging, applications with data imbalance are ubiquitous [4] and costs of making some types of mistakes are more severe than others. For instance, in diagnosis application, discarding
Code is available at: http://ggtthuu..cssaaaaeeeaaeeee com/
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

cancer case as healthy (False Negative) is more costly than classifying a healthy subject as having cancer (False Positive). While the latter creates burden for the subject as well as health-care system through additional tests that may be invasive and expensive, the former, i.e. failure to identify a cancerous case, would delay the diagnosis and jeopardise the treatment success. In such applications, binary classifiers are operated at high True Positive Rates (TPRs) even when this means having higher False Positive Rates (FPRs). To make matters more complicated, there are usually significantly fewer samples to represent critical classes, where mistakes are more severe. For instance, in [6] authors found out that in prostate cancer screening only 30% of even the most suspicious cases identified with initial testing actually have cancer. Such class imbalance increases the FPR even higher in "critical" applications, because the models tend to misclassify minority classes more often. Useful algorithms need to achieve low FPR at high TPR operating points, even under class imbalance.
While various methods for learning with imbalanced datasets exist, to the best of our knowledge, these methods do not take into account the fact that "critical" applications need to be operated at high accuracy for the critical classes. We believe that for such applications ensuring low misclassification rate for the non-critical samples and high accuracy for the critical classes should be the main goal, to make binary classifiers useful in practice. This motivates us to design new strategies for training DNNs for classification.
Contribution: In this paper, we pose the training of DNN for binary classification under class imbalance as a constrained optimization problem and propose novel constraint that can be used with existing loss functions. We define the constraint using Mann-Whitme to maximize the AUC, but in an asymmetric way to favor reduction of false positives at high true positive (or low false negative) rates. Then, we transfer the constrained problem to its dual unconstrained optimization problem using an Augmented Lagrangian method (ALM) (2). We optimize the resulting loss function using stochastic gradient descent. Unlike the existing methods that directly optimize AUC, we incorporate AUC optimization in principled way into constrained optimization framework. We finally present two possible extensions of the proposed constraint for multi-class classification problems.
statisties [[6] in order
We present an extensive evaluation of the proposed method for mmage-basd binary and multi-class classification problems on three datasets: an in-house medical dataset for prostate cancer, CIFAR 10, and CIFARIOO 100 11. In all datasets, we perform experiments by simulating different class imbalance ratios. In our experiments, we apply the proposed constraint to 9 different baseline loss functions, most of which were proposed to handle class imbalance. We compare the results with the baselines without any constraint. The results demonstrate that the proposed method improves the baselines in majority of the cases.
2 Related work
Various methods have already been proposed to learn better models with class-imbalanced datasets. We group the existing methods into three categories: cost sensitive training- methods, sampling- based and classifier methods. Here, we focus on the first one and present related work for the other groups in the supplementary material for space reasons.
Cost sensitive training-based methods: This family of methods aims at handling class imbalance by designing an appropriate loss function to be used during training. In particular, they design loss functions to give more emphasis to the samples belonging to minority class, or the class with higher associated risk, than the majority ones during training [22]
25 proposes loss function, which we refer to as Weighted BCE (W-BCF), where samples from minority class are multiplied by constant to introduce more cost to misclassification of those samples. [10 proposes function that aims to learn more discriminative latent representations by enforcing DNNs to maintain inter-cluster and inter-class margins, where clusters are formed using k-means clustering. They demonstrate that the tighter constraint called inherently reduces class imbalance. In a more recent work, (3] proposes a loss function binary cross-entropy (CB-BCE) to weight BCE inversely proportionally to the class frequencies to amplify the loss for samples from the minority class. In similar vein, [13] modifies BCE and propose symmetric focal loss (S-FL) by multiplying it with the inverse of the prediction probability to introduce more cost to the samples that DNNs are not very confident, [[S]
introduces symmetric margin loss (S-ML) by introducing a margin to the BCE loss. 12 investigates different loss functions such as S-FL and S-ML, and propose their asymmetric versions, A-FL and

---

A-ML, by introducing margin for samples from the minority class to handle class imbalance. In different line of work, [33 proposes method called mean squared false error by performing simple yet effective modification to the mean squared enror (MSE) loss. Unlike MSE, which computes an average error from all samples without considering their classes, this loss computes mean error for each class and averages them. [ proposes a label-distribution-aaaa margin (LDAM) loss motivated by minimizing a margin-based generalization bound, optionally coupled with a training schedule that defers re- se-weighting until after the initial stage. [9]] introduces balanced meta-softmax for long-tailed recognition, which accommodates the label distribution shift between training and testing, as well as meta sampler that learns re-sample training set by meta- meta-learning. [2I] proposes a loss that ignores the gradient from samples of large classes for the rare ones, making the training more fair.
A particular group within the cost sensitive training- based methods focuses on optimizing AUC and our method falls into this group. AUC optimization is an ideal choice for class imbalance since AUC is not sensitive to class distributions [4. [8] proposes a support vector machine (SVM) based loss function that maximizes AUC and demonstrates its effectiveness for the class imbalance. [26] approaches the class imbalance problem from online learning perspective and proposes an AUC optimization-base loss function. 7 proposes a one-pass method for AUC optimization that does not require storing data unlike the previous online methods. Another online AUC optimization method proposed by [24] formulates AUC optimization as a convex concave saddle point problem. Despite their usefulness, all aforementioned AUC optimization- methods were applied to linear predictive models, as this allows to simplify the Mann- Whitney statistics [T6] for the definition of a small dataset for breast cancer detection where they also mention that extension to larger datasets may not be feasible. In very recent work called mini-batch AUC (MBAUC) ( authors extend AUC optimization to non-linear models with DNNs by optimizing AUC with mini-batches and demonstrate its effectiveness on various datasets.
AUC, and their performance DNNs is unknown. [20] applies online AUC optimization
The proposed constrained optimization method differs from the existing works in that enforces maximal AUC as constraint in way that favors reducing FPR at high TPR and can be used with existing loss functions.
Background - Augmented Lagrangian method (ALM)
A generic optimization problem for an objective function F(O) subject to the constraints C(0) {ci (0), C., (0)) can be expressed as 17
arg min F(0); subject to C(0)
8ee
Augmented Lagrangian method (ALM) 1. also known as methods of multipliers, converts the constrained optimization problem in Eq. ①1) to an unconstrained optimization problem. ALM is proposed to overcome the limitations of two carlier methods called quadratic penalty method and method of Lagrangian multipliers which suffer from training instability and non-convergence due to the difficulty of convexifying loss functions ? In ALM, the penalty concept is merged with the primal-dual philosophy of classic Lagrangian function. In such methods, the penalty term is added not to the objective function F F(0) but rather to its Lagrangian function, thus forming the Augmented Lagrangian Function:
= F(() + Σ (0)I + Σ (ic:(0)
(2)
1=1
In practice, this method consists in iteratively solving a sequence of problems as:
max min C,,,
E
(3)
Ak #
Where {X+} is a bounded sequence in RT, updated as th+1 X + (c.. {p²} is positive penalty parameter sequence, with 0 < n² I p 1²t + oc, which may be either pre-selected or generated during the computation according to a defined scheme. In ALM, increasing j² indefinitely
'Please consult supplementary material for more details of quadratic penalty method and method of La- grangian multipliers.
3

---

is not necessary in the quadratic penalty method. Thus, it does not suffer from training instabilities due to the constraint prevailing F(0). Furthermore, it does not require convexity assumption as the method of Lagrange multipliers to ensure convergence [
Proposed methods
4.1 Proposed constraint for binary classification
Let F(0) be a generic loss function that is used to train classification DNNs, fo(.), for binary problems. Let us also define p - {...... ز
and n _ {-r, x", as the sets of samples of the positive (critical) and negative classes, respectively. Note that we choose as the minority class in our description, i.e., [p| [ra,, and assume that this is a critical class associated with higher risk of making mistake. We define our constrained optimization problem as follows:
arg min F())
(4)
max (0, (o(o))))o(+=))+++- - 0, j (1,,..,,lll:
subject to
where fo(x) indicates output probability of the DNN on input æ. Note that the constraint states that the output of the DNN for each sample of critical class should be larger than the outputs of all of the
negative samples by a margin 5. Satisfying the constraint would directly ensure maximal AUC [16].
We define the equivalent unconstrained version of Eq. by writing it in the form given in Eq.
(4)
pl
lpl 2==1
9
(,(0, = F(0) + 2 |p
|
where, d3 - -
1= mmxx(,,((oox)))- [o(x)) + 8), μ is the penalty coefficient corresponding to the quadratic penalty term, Aÿ is the estimate of Lagrange multiplier corresponding each positive training sample j, and is the margin that we determine using a validation dataset.
The crucial aspect of the formulation is the asymmetry between positive and negative classes. A constraint is defined for each sample from the positive class, thus each positive sample gets a separate Lagrange multiplier. This form prioritizes the reduction of FPR at high TPR values as illustrated next.
We use Algorithm I to estimate the parameters of a DNN, 8, using the proposed loss function in Eq. 5) The parameters 0 are updated with every batch using gradient descent with learning rate a. Concurrently, ji increased using multiplicative coefficient p only when a chosen metric on validation is not improved, by margin to avoid training instabilities. The validation metric (ValMETRIC) is: Validation AUC for the binary setup; 2) Validation Accuracy for the multi-class version. We update Xÿ in each iteration for each positive sample zz..
Algorithm 1 ALM for Training DNNs
Input: ((0) (0), / p: for t 1, 1,.... do for each mini batch of TB with size B do yB = f(XB):
Calculate 0
b Vjj E [1, B] and yj = y
g(++1) + ((1) a (OC, (0)
x(++1) + (1) 11() 9;
Vÿ E [1. and y, y
end for
VVIMETRRCCO < <VIIMETRI( then
((+l) + 110) ps
else
[1(11] (1())
end if end for Return ((++1)

---

4.2 Toy example describing our design choice
Recall that our goal with the constraint and the final augmented loss function is to maximize AUC through minimizing FPR for high TPR. The design of the constraint crucial to achieve this goal. In Fig. [ we demonstrate this on a toy example. At the top we show 10 data samples and order them with respect to a classifier's 's out- put for the samples, i.e., samples on the right are assumed to yield higher output than those on the left. The gray area in the figure below shows the AUC for the toy data samples. Consider two dif- ferent optimizations to increase the AUC. One adds the red box and the other adds the green box to the gray area. Both optimizations lead to exactly the same AUC improvement, however, only adding the green box reduces FPRs at the highest TPRs.
Negative Postiive
Positiva semples
Negatiive class samples
Thus Poslive Rote
0
False Postive Rate
Figure 1: Toy example that illustrates different optimizations, which yield the same improvement in AUC. The one that adds the green box to the AUC however, leads to lower FPRs at the highest TPRs. The ALM given in Eq. (5) prefers adding
In the proposed method, we design the con- straint to achieve lower FPR at high TPR, such that the optimization can reduce cost more by adding the green box instead of the red box. To see this, let us assume that the distances between all the successive markers in the figure are the same, and we denote this by A. Note that the distances between markers indicate the differences between the outputs of the classifiers for the samples corresponding to the markers. Let us also further assume that all the Lagrange multipliers have the same value. In this case, one verify that if the optimization swaps the locations of the left most positive sample (red circle marker) and the negative sample to its immediate right (blue
the green box to improve the AUC rather than the red one.
triangle marker), the cost due to the constraint in Eq. 5 decreases by 39,4² 3AA Swapping the 50
locations of the right most negative sample and the positive sample to its left decreases the cost due
to the constraint by 19pa² + 3AA 3
So, from the cost perspective, the optimization should prefer the former swap over the latter, which corresponds to adding the green box to the AUC instead of red box. Therefore, the augmented Lagrangian cost would be decreased further when FPR at the highest TPR is reduced rather than increasing the TPR at the lowest FPR. Instead of defining the constraint for each positive sample in Eq. (5). if we were to define it for each negative sample in the exactly opposite way, i... Σ/T, maxx(0,((fele the situation would be reversed. The optimization would prefer adding the red box over the green box to decrease the cost further. If we were to define a constraint for each positive-negative pair, i.e., fo(x.") > [a(x)) ) Vj and k, then adding the red or the green box to improve the AUC would yield exacily the same decrease in the cost.
fa(x)) + 8) = 0, k {I...., [rl]]. then
4.3 Extensions to multi-class classification
The proposed constraint can also be extended to multi-class classification with slight modifications. Let us assume that we have a multi-class classification problem with C classes. In this case, there can be multiple critical and non-critical classes depending on the application. Let us define the corresponding family of sets, i.c., sets of sets, as P = {p'.... pl P and N = (n' ............ .n. NN| where the sets p' and n' are defined in the previous section. Also, note that P = C and PON ===0 0.
A main difference between binary and multi-class classification is the dimension of the output. While fa( was a single value for binary classification, in multi-class problem it is vector with one value for each class. Using the notation for the positive and negative classes, we write the output of the
'Derivations for this toy example and theoretical insights for our design choice are provided in the supple- mentary material for convenience.
5

---

network as fo(Ø) = ( Sb .0 ,. ... f00 ")
Based on this, we define our first constraint
for multi-class classification as
[NI In'l max (0.-(18" (x9)) 18 (1 + CE cI, .. (PI,, jE (1,....pp) (6)
where f (x) indicates the output probability of DNN for the critical class p². Note that the constraint in Eq. 66) enforces that output probabilities of the DNN for critical classes should be larger for critical samples than for samples of other classes. We refer to ALM with the first constraint as ALM
An alternative definition of constraint is also possible. The first constraint enforces that probabilities of the critical classes should be larger for samples of each critical class. We can go a step further and add penalty to enforce that probabilities of the non-critical classes should be smaller for critical samples than the non-critical samples belonging to the corresponding class. We denote ALM with the second constraint as m..
IN |n'|
0, (0. (88 (xt))
f6 18-((0 (1x +6)
det
max
(7)
+ max 0, (16 So (xx + +0) CE (1,...,, j E (1....,p})
In this formulation, the penalty does not view all non-critical classes as one, but also contributes to improving classification accuracy in those classes as well, by increasing the gap between the corresponding non-critical probability of critical and non-critical samples.
Experiments
In this section, we present experimental evaluations on image- based classification tasks. We perform experiments on three datasets: an in-house MRI medical dataset for prostate cancer and two publicly available computer vision datasets, CIFARIO and CIFAR100 [TI.
In our evaluation, we experiment with different existing loss functions, most which have been de- signed to handle class imbalance: classic binary and multi-class cross-entropy (BCE, CE), symmetric margin loss (S-ML) [5], symmetric focal loss (S-FL) [3], asymmetric margin loss (A-ML) and
focal loss (A-FL) 12), cost-weighted BCE (WBCE) (25, class-balanced BCE (CB-BCE) and CE (CB-CE) (5) wert---tttiuuuoo-aaaaaaa
margin loss (LDAM) 3 We first train DNNs for classification using only the loss functions and then using our method, which adds the proposed constraint to the loss function and solves Eq. (5). and we compare the classification performances. In addition, we also compare the proposed method with directly optimizing AUC using the mini-batch AUC (MBAUC) method proposed in (8] for the binary case.
The proposed method is implemented in PyTorch and we run all experiments on a Nvidia GeForce GTX Titan GPU with 12GB memory.
5.1
Datasets
Prostate MRI dataset consists of distinct cohorts: 1) a group of 300 multiparametric prostate MRI studies used for training and 2) another group of 100 multiparametric prostate MRI studies used for testing the trained DNNs. There is no overlap between the groups. Consent from the each subject obtained to use the data for research purposes. Two board-certified radiologists with 10 and years of experience in dedicated prostate imaging independently reviewed all examinations of the training set and test set and scored whether Dynamic Contrast Enhanced (DCE) sequences would be beneficial for cancer diagnosis. After completion of readings, consensus was reached by the two readers by reviewing all examinations with discrepant decisions. The goal of the binary classification here is to identify subjects who do not require additional DCE imaging for accurate diagnosis, so they can spared from unnecessary injection and cost and duration of the scanning can be reduced. In our experiments, we randomly split 20% of the training cohort validation set by keeping the class imbalance consistent across the datasets. In both training and testing cohorts, positive samples represent 13% of all the patients which leads to an inherent 1:8 class ratio.

---

CIFARIO and CIFARIO0 datasets For CIFARIO, in the binary experiments we randomly select 2 classes among 10 to pose binary classification problem. Additional experiments with different random selections are in the supplementary material. We use all the available training samples of the selected majority class, while we randomly pick a varying number of training samples for the minority class to obtain different class ratios, up to 1 1:200. We select 100 samples per class as validation set to determine hyper-parameters For testing, we use all the available test samples, which consist of 1000 images for each class. For CIFAR 100 in the binary setup we select one super-class and a sub-class of another super-category as minority one. For training we use all the available 2250 samples of the selected larger class, and we randomly pick a varying number of training samples for the minority class to obtain different class ratios, up to 1:200. We use 50 samples for each original sub-class validation set to perform the parameter search. For testing, we use all the available test samples, which consist of 100 images for each sub-class. For the multi-class experiments, loigetailed versions of CIFARIO 10 and CIFAR 100 are built accordingly to 3 and consider the smallest class we use balanced validation
majority one,
Rl as the only critical clas To be consistent with recent works 5 and test sets in all the experiments on CIFAR datasets.
5.2 Training details
Network architectures: For the prostate MRI dataset, we use a 3D CNN that consist of cascaded 3D) convolution, 3D max-pooling, intermediate ReI.u activation functions and Sigmoid in the final output. For the binary CIFARIO 10 and CIFAR1ON datasets, we use ResNet- 10, while ResNet-32 [
adopted for the multi-class experiments, consistently with 3 5). Further training details can be found in the supplementary material.
Ensembling for higher reliability: Model reliability is very crucial when training DNNs. Dealing with small datasets may lead to dataset-dependen results even with the random splits, which could completely hinder objective evaluation. To weaken this phenomenon, we adopt the following ensembling strategy on MRI and for consistency we apply it on the binary CIFAR10 and CIFAR 100 as well. Given dataset and class ratio, we create 10 random stratified splits of the dataset and train 10 models independently. The larger portions are used for training and the smaller portions for choosing ppper-parameters. During inference, all the models are applied on test samples and predictions are averaged in the logit space before the sigmoid function to yield the final prediction. We apply the ensembling to all the binary models we experiment with. This practice attenuates data dependency and we observed that the final AUC is improved when compared to the average of AUCs of different models, as presented in Table for MRI sequences and in supplementary material for CIFAR 10 and CIFAR100.
Hyper- prree-pprrmeters selection: Selection of the best seeeer-prramterrs is crucial both to ensure proper and fair evaluation of the methods and to understand the true performance of any model. To achieve this, we perform grid-search determine the hyper-paranee syper-pprrmmters that yield the highest AUC for the binary experiments. For the multi-class tests we select the model that achieved the best overall accuracy on the validation set, in order be consistent with the related works. The test sets in all experiments are not used for hyper-paramete selection. To reduce computational load, we select the common hyper-parameters such the optimizer, learning rate, and the activation functions in the DNNs based on their performance with BCE loss function for the binary experiments, and consistently with for the multi-class ones. Then, we keep them fixed in all experiments on the same dataset.
Besides the common hyper-parameters, the majority of the existing methods have hyper-parameterss that crucially affect their performance. Namely, these A-ML, exponent y for S-FL and A-FL, weight of the cost for WBCE and B for CB-BCE. We select best values for these information provided in the original papers for each of them.
are margin m for S-ML. and
from the respective candidate sets that we create based on the
In the proposed method, there are 4 parameters to be set: p(0), {(0),{(()) p. and 8. Thus, hyperparameters' search is an important aspect of the proposed method. (0( (0), and are stemming from ALM and we follow the guideline from 2 when setting them. We initialize all the Lagrangian multipliers
{ to O. We choose (0) from the set (10-7, 10-6, 10-5, 10-1, ; 10-3), as it is suggested to choose a small value in the beginning and increase it iteratively using the equation م ((++1) p (k), We
'xperrmment with multiple critical classes are presented in supplementary materials.
7

---

choose p from the set (2, as is suggested. Moreover, we do not increase beyond 4 to avoid potential dominance of the constraint on F()), since p is used to increase μ.. Once we find the best combination of and based on the chosen metrics on the validation set, we fix them and we search for 5 as final step. Please see supplementary materials for further details on the hyper-parameters selection.
Once the hyper-parameters for each model selected, the training is performed and models applied to the test set to yield the final results, which are described next.
Table 1: Results binary CIFARIO for class ratio 1:100 and 1:200.
Dataset
Binary CIFARIO, imb. 100
Binary CIFARIO, imb. 200
FPR
@
FPR @
FPR
@
FPR
@
FPR @
FPR @
Training method
Test AUC
Test
98%
95%
92%
98%
95%
92%
AUC
TPR
TPR
TPR
TPR
TPR
TPR
BCE
56.0
45.0
29.0
91.2
75.0
55.0
40.0
87.3
S-ML
59.0
40.0
26.0
91.7
75.0
54.0
35.0
87.4
S-FL.
59.0
40.0
27.0
91.7
78.0
59.0
43.0
85.7
A-MI
54.0
36.0
23.0
92.4
74.0
56.0
39.0
87.4
A-FL
50.0
38.0
24.0
92.3
76.0
59.0
40.0
86.2
CB-BCE
89.0
72.0
59.0
78.0
87.0
74.0
61.0
78.0
W-BCE
69.0
52.0
37.0
87.4
88.0
75.0
62.0
78.3
LDAM
65.0
48.0
34.0
89.0
78.0
63.0
45.0
86.4
MBAUC
86.0
71.0
56.0
74.0
89.0
83.0
69.0
67.9
ALM + BCE
52.0
34.0
21.0
93.1
70.0
54.0
39.0
86.7
ALM S-ML
50.0
37.0
24.0
92.5
72.0
52.0
39.0
87.9
ALM S-FL
55.0
39.0
25.0
91.5
74.0
55.0
41.0
86.9
ALM A-MI.
45.0
35.0
23.0
92.8
75.0
74.0
35.0
87.6
ALM + A-FL
49.0
37.0
23.0
92.7
78.0
57.0
37.0
87.0
ALM CB-BCE
67.0
51.0
36.0
88.1
85.0
69.0
53.0
80.0
ALM + W-BCE
66.0
48.0
31.0
89.3
83.0
69.0
54.0
81.0
ALM LDAM
60.0
42.0
31.0
91.0
73.0
61.0
43.0
85.6
5.3 Results
We present binary classification results CIFARIO, CIFARIO0, and the in-house medical imaging datasets in Tables T corresponding ALM+baseline and present the best result among the two by bold. Also, the underlined results indicate the best results among all. We evaluate the performance of the baselines and the proposed method (ALM) using FPR maximal levels of TPR minimal levels false negative rate (FNR)) and AUC on the test sets. In the binary experiments, ALM is overall able to consistently improve the performance of almost all the loss functions, with regard both AUC and FPRs at maximal TPR levels. Even in those cases when AUC is improved to a moderate extent there is still an improvement in FPRs, in accordance with our goal. Moreover, is noticeable that the higher the TPR, the higher the benefit of applying ALM, which is in accordance with our target applications. In fact, considering that such classifiers "critical" applications would operated high TPR (or low FNR), reduction in FPR in these settings is the effect we desired from the proposed approach. Lastly, we also observe in the tables that directly optimizing AUC via MBAUC does not provide the same improvements using the proposed ALM approach. We present additional binary classification experiments in the supplementary material.
2 and 3 respectively. In the tables, we compare each baseline with the
Additionally, we present the quantitative results of multi-clan clatss experiments in Table 4 A bold result for a baseline means that it is able outperform both constrained optimisation methods ALM,. and ALMm,,, otherwise the better constrained strategies are highlighted. As for the binary case, underlined results indicate the best method for each metric. Comparison with the other baselines are presented in the supplementary materials. In these experiments, computing the accuracy on the non-critical classes, at various levels TPR for the critical-class. For this purpose, the first step is find threshold on the logit of the critical class such that the desired
evaluate the performance by

---

Table 2: Results on binary CIFARI00 for class ratio 1:100 and 1:200.
Dataset
Binary CIFAR imb. 100
Binary CIFARIOO, imb. 200
FPR @
FPR @
FPR @
FPR
@
FPR @
FPR @
Training method
Test AUC
Test
98%
95%
90%
98%
95%
90%
AUC
TPR
TPR
TPR
TPR
TPR
TPR
BCE
93.0
63.0
47.0
81.8
94.0
77.0
61.0
79.1
S-ML
89.0
65.0
43.0
82.7
95.0
75.0
64.0
79.7
S-FL
89.0
62.0
44.0
82.6
90.0
78.0
50.0
80.1
A-MI
91.0
63.0
44.0
81.8
95.0
75.0
66.0
79.8
A-FL
88.0
63.0
45.0
82.8
91.0
78.0
50.0
80.0
CB-BCE
93.0
75.0
52.0
78.8
93.0
78.0
51.0
78.7
W-BCE
88.0
59.0
41.0
79.7
95.0
63.0
51.0
79.7
LDAM
84.0
70.0
42.0
82.8
80.0
67.0
45.0
82.1
MBAUC
81.0
62.0
41.0
82.3
88.0
63.0
48.0
80.3
ALM BCE
91.0
49.0
39.0
82.7
87.0
66.0
57.0
80.9
ALM S-ML
88.0
69.0
41.0
81.7
87.0
73.0
55.0
80.7
ALM S-FL
88.0
60.0
42.0
81.7
85.0
76.0
50.0
80.8
ALM A-MI.
89.0
55.0
37.0
82.7
92.0
63.0
45.0
81.0
ALM + A-FL
86.0
62.0
40.0
83.2
88.0
76.0
46.0
80.7
ALM CB-BCE
89.0
59.0
36.0
83.8
85.0
66.0
44.0
81.0
ALM + W-BCE
87.0
53.0
39.0
83.2
79.0
62.0
44.0
81.3
ALM+LM LDAM
80.0
59.0
40.0
83.2
84.0
61.0
46.0
81.5
TPR is obtained for the important class. All the test samples whose logit of critical class exceeds the selected threshold are assigned the important class. The remaining samples that are not assigned to the critical class are then classified based on the highest probability over the non-critical logits. In addition to this metric, we present overall classification accuracy on all classes. The quantitative results demonstrate that both of the multi-class strategies, ALMm, and ALMm, improve the baseline, reducing the error on non-critical classes, at high levels accuracy on the important class. Moreover, ALM,,, is able further improve the overall accuracy by larger margin almost all the experiments thanks to the additional term the constraint.
Table Results on in-house MRI dataset.
Method
FPR @0 FN
FPR @ FN
Avg AUC
AUC ens.
BCE
80.0
80.0
65.4
70.9
S-ML
81.0
77.0
67.347.0
71.5
S-FL
77.0
38.0
71.7+10.00
80.3
A-ML
77.0
73.0
68.019.0
74.2
A-FL
66.0
38,0
67.748.0
80.1
CB-BCE
100.0
34.0
72.0+5.00
77.7
W-BCE
56.0
42.0
68.8+6.0
80.5
LDAM
100.0
75.0
62.0+9.00
66.4
MBAUC
61.2
33.0
71.2+11.0
82.4
ALM BCE
54.0
38.0
76.819.0
85.4
ALM S-ML
81.0
33.0
72.511..0
80.3
ALM + S-FL
53.0
26.0
72.5++0.00
84.2
ALM A-ML
72.0
53.0
67.24...0
76.4
ALM A-FL.
62.0
46.0
74.7+7..0
81.5
ALM CB-BCE
86.0
34.0
73.019.0
79.5
ALM + W-BCE
59.0
40.0
72.416.0
81.4
ALM LDAM
59.0
53.0
66.5188.5
77.0

---

Table 4: Results on long-tailed CIFARIO for class imbalance 1:100 and 1:200. The Table shows the error on all the non-important classes, after setting a threshold on the logit of the important class to obtain 80, 90% TPR.
Dataset
Long-tailed CIFARIO, imb. 100
Long-tailed CIFARIO, imb. 200
Error @
Error @
Error
Error @
Training method
Overall Accuracy
Overall Accuracy
80%
90%
80%
90%
TPR
TPR
TPR
TPR
CE
29.80
34.67
70.35
37.81
42.37
64.03
FL
32.21
36.47
69.20
38.50
42.21
62.90
CB-BCE
31.04
33.44
72.90
35.11
39.13
65.77
LDAM
26.57
29.86
71.80
34.41
45.72
65.87
ALM m, + CE
28.89
33,93
70.90
36.14
39.90
65.13
ALM m, FL
29.59
34.94
69.74
36.87
41.87
64.23
ALM m, + CB-CE
27.89
30.27
72.10
33.09
35.44
65.34
ALM m. LDAM
25.73
28.52
72.86
31.94
37.41
65.65
ALM m. + CE
29.53
34.09
71.30
35.10
39.19
64.35
ALM m. FL
30.50
35.63
69.47
36.27
40.43
64.43
ALM + CB-CE
27.84
31.97
72.09
33.72
36.86
66.04
ALMm, m,2 LDAM
24.76
28.91
73.32
31.02
36.09
67.41
Conclusion
In this paper, pose the training of DNN for binary classification under class imbalance constrained optimization problem and propose a novel constraint that can be used with existing loss funetions. The proposed constraint designed to maximize the AUC, but in asymmetric way to favor the reduction of FPR at high TPR (or low FNR). Then, we transfer the constrained problem to its dual unconstrained optimization problem using Augmented Lagrangian method (ALM) 2 which we optimize using stochastic gradient descent. Additionally, we present two possible extensions of the proposed constraint for multi-class classification problems.
We perform an extensive evaluation of the proposed constraints for binary and multi-class image classification problems both computer vision and medical imaging datasets. We compare the per- formance the proposed constraints with different baselines by simulating different class imbalance ratio. The quantitative results demonstrate that the proposed constraints improve the performance of the baselines in the majority of the cases both binary and multi-class classification experiments.
Acknowledgments
The presented work was partly funding by: 1. Clinical Research Priority Program Grant Artificial Intelligence in Oncological Imaging Network, University of Zurich and 2. Personalized Health and Related Technologies (PHRT), project number 222, ETH domain.
10

---

References
[1] Bertsekas, D.P.: Multiplier methods: survey. Automatica 12(2), 133 145 (1976).
htt0:/77.tttt///ttt http::/wwww...ineeeereettttt
[2] Bertsekas, D.: Nonlinear Programming. Athena Scientific (1999)
[3] Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: distribution-aware margin loss. In: Wallach, H., Larochelle, H., Beygelzimer, A., d' A'Cche-Buuc, F. Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019),
Learning imbalanced datasets with label-
neurips www..pp..c./ppprr//aa...... pttppscccccppper/////////e/ pdf
[4] Cortes, C., Mohri, M.: Auc optimization
rate minimization. Advances in neural
information processing systems 16(16), 313-320 (2004)
[5] Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9268-926 9277 (2019)
[6] Djavan, B., Zlotta, A., Remzi, M., Ghawidel, K., Basharkhah, A., Schulman, C., Marberger, M.: Optimal predictors of prostate cancer repeat prostate biopsy: prospective study of 1,051 men. The Journal of urology 163 4, 11448; discussion 1148-9 (2000)
[7] Gao, W., Jin, R., Zhu, S., Zhou, Z.H.:: One-pass auc optimization. In: International conference on machine learning. pp. 904-914. PMLR (2013)
[8] Gultekin, S., Saha, A., Ratnaparkhi, A., Paisley, J.: Mba: mini-batch auc optimization. IEEE transactions on neural networks and learning systems 31((2), 5561-5574 (2020)
[9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 770-778 (2016)
[10] Huang, C., Li, Y., Loy, C.C., Tang, X.: Learning deep representation for imbalanced classifica- tion. In: Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5375-5384 (2016)
[11] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
[12] Li. Z., Kamnitsas, K., Glocker, B.: Overfitting neural nets under class imbalance: Analysis and improvements for segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 402-410. Springer (2019)
[13] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980-2988 (2017)
[14] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F,, Ghafoorian, M., van der Laak, J.A., van Ginneken, B., Sánchez, C.I.: A survey deep learning in medical image analysis. wrtp://ooo.o//1/001.0.0..00 media.20.......7.
Medical Image Analysis 42, 60-88 (Dec 2017). http:/ //dx doi org/ 10. 1016/j media. 00.
[15] Liu, W. Wen, Y., Yu, Z., Yang. M.: Large-margin softmax loss for convolutional neural networks. ICML. vol. 2, p. 7 (2016)
[16] Mann, H.B., Whitney, D.R.: On test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics pp. 50-60 (1947)
[17] Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York, NY, USA, second edn. (2006)
[18] Rakotomamonjy, A.: Optimizing under roc curve with svms. In: ROCAL pp. 71-80 (2004)
[19] Ren, J., Yu, C., Sheng. S., Ma, X., Zhao, H., Yi, S., Li, H.: Balanced meta-softmax for long- tailed visual recognition. In: Proceedings Neural Information Processing Systems(NeurIPS) (Dec 2020)
11

---

[20] Sulam, J., Ben-Ari, R., Kisilev, P:: Maximizing auc with deep learning for classification of imbalanced mammogram datasets. In: VCBM. pp. 131-135 (2017)
[21] Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., Yan, J.: Equalization loss for long-tailed object recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
[22] Thai-Nghe, N., Gantner, Z., Schmidt-Thieme, L.: Cost-sensitive learning methods for imbal- anced data. In: The 2010 International joint conference neural networks (IJCNN). pp. 1-8. IEEE (2010)
[23] Wang, S., Liu, W., Wu, J., Cao, L.. Meng. Q., Kennedy, P.J.: Training deep neural networks on imbalanced data sets. In: 2016 international joint conference on neural networks (IJCNN) pp. 4368 4374. IEEE (2016)
[24] Ying, Y., Wen, I., Lyu, S.: Stochastic online auc maximization. In: NIPS. pp. 451 (2016)
[25] Zadrozny, B., Langford, J., Abe, N.: Cost-sensitive learning by cost-proportionate example
weighting. In: Third IEEE international conference on data mining, pp. 435-442, IEEE (2003)
[26] Zhao, P., Hoi, S.C., Jin, R., YANG, T.: Online auc maximization (2011)
12


Reviews:
[{"review_id": "zftWnMCo6sr", "reviewer": "Reviewer_w5h3", "summary": "This paper proposes a very interesting approach for training DNNs on binary (and multiclass) classification tasks with critical under-represented classes, which is very typical in domains like healthcare. In this approach, the task is posed as a constrained optimization problem and a novel constraint is designed that emphasizes on reducing the false positive rate while maintaining a high true positive rate, which is then added to the loss function and solved using an Augmented Lagrangian Method. The main contribution of the paper is proposing a novel constraint optimization approach that can be added to any previous work, and that improves upon the state of the art results. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- The proposed approach is definitely very novel and interesting. I am not aware of other work in this area that pose AUC optimization as a constrained optimization problem. \n- The authors provided a very comprehensive and well organized related work section, and its clear how this approach distinguishes itself from the others. The related work is broadly split into 3 groups: the first proposes variants of common loss functions with added emphasis on minority classes, the second focuses on different sampling strategies to improve the occurrences of the minority class samples in the batches, and the third focuses on scaling and thresholding model outputs post training. In the first group, another subgroup of papers focus on AUC optimization. This paper falls in that subgroup, but differs from the other papers by proposing an approach that's scalable to large datasets and model architectures, and that can be used along with other loss functions. \n\nClarity: \n- The paper overall is very well written. Its clear, understandable, and very well organized. The code is also provided in the supplementary material, and the algorithms are very well explained in the paper. \n- It was very helpful that the authors included a toy example in section 4.2 to explain intuitively how their approach works. \n- One minor comment is, in Figure 1, the axis labels should be False Positive Rate and True Positive Rate instead. \n\nQuality: \n- The paper is technically very sound. The proposed constraint was given ample intuitive and mathematical explanation, and using a toy example in section 4.2 and detailed theoritical explanations in the appendix section A.8 is very helpful.  \n- The authors conducted very comprehensive experiments (including the results presented in the appendix) and provided compelling evidence for the improvements obtained by their approach by comparing directly with the results of several approaches from the related work. This approach improved the results considerably in most of the cases. \n- The authors also extended their work to the multiclass domain and provided the relevant results.\n\nSignificance: \n- The proposed approach produced considerable improvements in empirical results over several state of the art methods. The main appeal of this approach is that it can be integrated with any other approach very easily. I think this will make it very easy for other authors to include it as a baseline in their works. This is a complete piece of work with a several important details provided in the appendix, and code required to reproduce the results provided in the supplementary material.  \n- The proposed approach provides a unique way to reduce the FPRs of classifiers operating at a high TPR. This is a very wide spread problem faced especially in the medical domain, and a lot of approaches do lead to a high FPR which impacts their practical usability adversely. This approach is definitely a very welcome addition in tackling that problem.  \n\nWeaknesses: \n- It would have been helpful if the authors provided results in another public medical dataset like ADNI, etc.  \n- It would be helpful if the authors include some discussion on how they plan to extend this work in the conclusion section. \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- One main limitation that the authors discussed is that they performed their experiments on the small MRI dataset, which can make the results unreliable. They addressed this by using an ensembling strategy.\n- There is no obvious potential negative societal impact of this work. The authors further mention that fairness is one of the motivations of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "VCRTds7_WB-", "reviewer": "Reviewer_47qX", "summary": "The paper proposes a novel method to perform classification with high imbalanced classes. The aim of the method is to minimise the FPR at high TPR -- this is suitable for many safety-critical applications where a misclassification can have dire consequences. In order to solve the problem the authors formulate the classification problem as a constrained optimisation problem and introduce a constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a new method and they do a got job at explaining it through the usage of a simple example. I really liked the simple example, I think it really helps the reader in understanding what is the objective of the proposed constraints.  \n\nRegarding the simple example, I think the cost of the constraint should decrease by $\\frac{39\\Delta^2}{50}+\\frac{3\\lambda\\Delta}{25}$ in the first case, and by $\\frac{19\\Delta^2}{50}+ \\frac{11 \\lambda \\Delta}{25}$ in the second case, and not by $39\\Delta^2+3\\lambda\\Delta$ in the first case, and by $19\\Delta^2+ 11 \\lambda \\Delta$ in the second case.\n\nFurther the method has been tested on a variety of datasets, with different ratios between the critical and the non-critical class. \n\nRegarding the experiments, I have the following concerns: \n- In both the binary case and the multi-class case the distribution of the train set and the validation set are different. Indeed, we have highly imbalanced datasets for the training sets, but balanced datasets for the test sets. Would the results be the same if the authors picked test sets that are as imbalanced as the training set? (which is also what we expect to happen when we apply the method in the real world)\n-In the experiments on the MRI dataset, what is the reason of picking just FPR@0FN and FPR@1FN? I think it would have been interesting to show at least FPR@2FN, and maybe FPR@5FN (just to see how the performance varies when allowing for more false negatives) \n- In the multi class experiments, why do the authors report the overall accuracy and not the AUC score (maybe averaged in different ways)?\n- Is there any reason why the authors did not try MBAUC+ALM?\n\nRegarding the readability, the authors did a good job in explaining the intuitions behind the proposed method, however, the section about the experimental analysis can be confusing: \n- $\\rho$ is introduced only at page 7, and the reader does not really know where this hyperparameter stems from. I would encourage the authors to include Algorithm 1 (in appendix) to the main paper. \n- $ALM_{m,1}$ and $ALM_{m,2}$ are never formally introduced, and they are first mentioned at page 8. \n- the authors did not specify what does the bold and underlined in the tables mean. I assume that bold means that the ALM+method is able to perform better than the method alone, while the underline highlights the best method. I would encourage the authors to better explain the above, and also to bold the results of the methods that are able to outperform method+ALM (otherwise the tables can be misleading)\n- is there any reason why the Avg AUC in Table 3 has no bold or underlined results?\n\n**After reading authors' comments**: The authors addressed all my concerns. I hope they will include the results showed in the rebuttal phase in the appendix of the final paper. I changed my score from 6 to 7. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "59aMZdwBjbW", "reviewer": "Reviewer_hCqm", "summary": "Summary\nThis paper proposes a new optimization approach to classification problems with class imbalance, where correctly classifying the minority class is more important than correctly classifying the majority class. One specific motivation is in medical imaging diagnostics, where misclassifying the “critical” minority class (e.g., diseased) is a more severe mistake than misclassifying the majority class (e.g., healthy). The authors therefore use an augmented Lagrangian method (ALM) approach to minimize the FPR for the majority while maintaining a high TPR for the critical class in a binary classification problem, as well as suggesting an optimization formulation for multi-class classification tasks. The ALM approach tries to enforce a margin between the critical and non-critical samples in the output space, and is structured to penalize a critical misclassification more than a non-critical misclassification.\n\nThe problem is certainly important for medical diagnostic tools, as well as other problems where errors have different costs for different classes. The proposed method is a useful approach for these problems, and the theoretical analysis of the asymmetry and required imbalance of the problem is clear and interesting. The experiments show some benefit of the ALM-based approach and its ability to be coupled with many different loss functions. The writing is clear overall, but with various grammatical mistakes throughout (and particularly in the supplement). My major concerns are the significance of the result and the setting of the CIFAR experiments.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Comments\n1. The paper does not compare with any AUC-optimization-based approaches apart from MBAUC. In Section 2, the authors say that “... all aforementioned AUC optimization-based methods were applied to linear predictive models and their performance on DNNs are unknown”. Why not compare against any of these approaches? Or is it not just that the performance is unknown, but that their application to DNNs is nontrivial?\n2. Significance of the result. Statistical significance is not included in Table 1 and Table 2. Multiple replicates should be used to test the significance. Some numbers are really close in table 1,2 and still marked in bold text (better performance). Avg AUCs in table 3 have large standard deviation, which might not indicate significant improvement. Instead of using selected thresholds 98%, 95% and 90%, it is more informative to show a TPR- FPR curve plot.\n3. How to predefine critical and non-critical classes? In binary classification, it seems that the minority class is always the critical class. Is this always the case in real-world applications? In the multi-class setting with many classes, classifying each class into critical and non-critical seems to be difficult and needs domain knowledge. If a critical class is misdefined as a non-critical class, this might be problematic for many applications.\n4. Setting of class ratios in CIFAR experiments. Since this paper is motivated by clinical applications, the prostate cancer example makes sense to me. However, in the CIFAR experiments, very large ratios of 1:100 and 1:200 are used. This ratio seems to be unrealistic (e.g., a clinical dataset with 200 healthy patients and 1 sick patient). Is it because the method can only have good improvement on such a very imbalanced ratio? I suggest the author to either show improvement on CIFAR with smaller ratios (e.g., 1:5, 1:10) or find a real-world medical application that has such a class ratio close to 1:100.\n\nMinor Comments\n1. A slightly longer description of MBAUC, such as the framework it uses or other ways in which it differs from the proposed method, would be useful.\n2. In Section 3, the authors say that other methods “...suffer from training instability and convergence”. I think you mean non-convergence?\nIt would be useful to include a citation for the claim in Section 4.1 that “Satisfying the constraint would directly ensure maximal AUC.”\n3. In Figure 4.2, given the precise setup of the line plot I think that it would be useful for the grid axes to go from [0, 5], rather than [0, 1].\n4. In Section 5.1, what is the justification for CIFAR10 and CIFAR100 to use validation sets that have 100 samples per class? Why is this not also class-imbalanced?\n5. I find the tables difficult to read. First, when the original loss function achieves a lower FPR / higher test AUC than the ALM-optimization version, I think those results should be bolded for clarity. Given the quantity of bolded results, it is also difficult to distinguish the underlines of “best result”, particularly when it is not one of the bolded entries. Also, given my understanding of what “bold” means in these tables, I didn’t understand why the entry for ALM_{m, 1} + LDAM [Acc. Critical Class 1] was not bolded in table A5.\n6. The details of the multi-class classification experiment were not fully explained in the paper. Specifically, I was not fully clear that there was only one critical class (and multiple non-critical classes) in this problem formulation until Section A.4.\n7. I wish the authors provided slightly more discussion about the actual results, including why the ALM helped more with some methods than others. One particularly interesting result for me was the performance of BCE, which in theory does not take into account class imbalance, but was one of the best-performing baselines and, in some cases, outperformed or was very close to ALM + BCE (particularly on the CIFAR data). This wasn’t the case on the MRI dataset -- is this due to an easier problem (lower FPR/higher TPR to begin with)? Weaker requirements on TPR? Less extreme class imbalances?\n8. I would appreciate some kind of study on the impact of class imbalance ratios to the improvement attributed to the ALM optimization. This would help me understand more details about the kind of problem that the ALM optimization is most suited to, as well as potentially answer some of the questions in the point above.\n9. Showing the % improvement by incorporating ALM would also be a useful plot to include somewhere, and might be more readable than the tables.\n10. Tables A.3 and A.4 have less convincing improvements than the binary classification task (and particularly table A.4). A little more discussion here to connect these results with Table A.5 would be useful. One specific confusion with table A.5 was the low critical class accuracy -- given the paper’s motivation, could you increase the required TPR? Currently there is much lower accuracy on the critical class than the non-critical classes, which seems counter to the main goal. Is this mitigated at all with a less extreme class imbalance ratio?\nIn Figure A.2, what is Delta_N set to? Given the prior discussion about Delta_P and Delta_N, this would be useful. Also, is Delta_E just the Delta in the text?\n11. The tables (and table numbers) seem to be missing from Section A.10.\n12. The most consistent grammar mistake was “... critical class’ ….” The singular possessive of “class” is “class’s.”\n\nUpdate -- I raised my score after reading the response from authors. Thank you for the detailed explanation and additional results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't have any concern about the negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to train deep neural networks for classification tasks that involve high class imbalance, which is ubiquitous in medical problems.  The novelty of the proposed method is very high, and related works are presented comprehensively.  The paper is overall very well written and structured.  The potential impact of the proposed algorithm is high given that class imbalance is a common problem in many real-world applications.  Reviewers raised major concerns regarding various aspects of experiments, which were successfully addressed by the authors. Overall, this paper constitutes an important contribution to the field and passes the bar for the acceptance to NeurIPS.
Decision: Accept (Poster)