Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes
Year: 2022

Abstract:
Deep neural networks (DNNs) are notorious for making more mistakes for the classes that have substantially fewer samples than the others during training. Such class imbalance is ubiquitous in clinical applications and very crucial to handle because the classes with fewer samples most often correspond to critical cases (e.g., cancer) where misclassifications can have severe consequences. Not to miss such cases, binary classifiers need to be operated at high True Positive Rates (TPRs) by setting a higher threshold, but this comes at the cost of very high False Positive Rates (FPRs) for problems with class imbalance. Existing methods for learning under class imbalance most often do not take this into account. We argue that prediction accuracy should be improved by emphasizing the reduction of FPRs at high TPRs for problems where misclassification of the positive, i.e. critical, class samples are associated with higher cost. To this end, we pose the training of a DNN for binary classification as a constrained optimization problem and introduce a novel constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. We solve the resulting constrained optimization problem using an Augmented Lagrangian method (ALM). Going beyond binary, we also propose two possible extensions of the proposed constraint for multi-class classification problems. We present experimental results for image-based binary and multi-class classification applications using an in-house medical imaging dataset, CIFAR10, and CIFAR100. Our results demonstrate that the proposed method improves the baselines in majority of the cases by attaining higher accuracy on critical classes while reducing the misclassification rate for the non-critical class samples.1

heading: 1 Introduction
text: Deep Neural Networks (DNNs) perform extremely well in many classification tasks when sufficiently large and representative datasets are available for training. However, in many real world applications, it is not uncommon to encounter highly-skewed class distributions, i.e., majority of the data belong to only a few classes while some classes are represented with scarce instances. Training DNNs on such imbalanced datasets leads to models that are biased toward majority classes with poor prediction accuracy for samples of minority class. While this is problematic for all such applications, it poses an even greater issue for “critical” applications where misclassifying samples belonging to the minority class can have severe consequences. One domain where such applications are common and machine learning is having an important impact is medical imaging.
In medical imaging, applications with data imbalance are ubiquitous [14] and costs of making some types of mistakes are more severe than others. For instance, in a diagnosis application, discarding a
1Code is available at: https://github.com/salusanga/alm-dnn.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
cancer case as healthy (False Negative) is more costly than classifying a healthy subject as having cancer (False Positive). While the latter creates burden for the subject as well as health-care system through additional tests that may be invasive and expensive, the former, i.e. failure to identify a cancerous case, would delay the diagnosis and jeopardise the treatment success. In such applications, binary classifiers are operated at high True Positive Rates (TPRs) even when this means having higher False Positive Rates (FPRs). To make matters more complicated, there are usually significantly fewer samples to represent critical classes, where mistakes are more severe. For instance, in [6] authors found out that in prostate cancer screening only 30% of even the most suspicious cases identified with initial testing actually have cancer. Such class imbalance increases the FPR even higher in “critical” applications, because the models tend to misclassify minority classes more often. Useful algorithms need to achieve low FPR at high TPR operating points, even under class imbalance.
While various methods for learning with imbalanced datasets exist, to the best of our knowledge, these methods do not take into account the fact that “critical” applications need to be operated at high accuracy for the critical classes. We believe that for such applications ensuring low misclassification rate for the non-critical samples and high accuracy for the critical classes should be the main goal, to make binary classifiers useful in practice. This motivates us to design new strategies for training DNNs for classification.
Contribution: In this paper, we pose the training of a DNN for binary classification under class imbalance as a constrained optimization problem and propose a novel constraint that can be used with existing loss functions. We define the constraint using Mann-Whitney statistics [16] in order to maximize the AUC, but in an asymmetric way to favor reduction of false positives at high true positive (or low false negative) rates. Then, we transfer the constrained problem to its dual unconstrained optimization problem using an Augmented Lagrangian method (ALM) [2]. We optimize the resulting loss function using stochastic gradient descent. Unlike the existing methods that directly optimize AUC, we incorporate AUC optimization in a principled way into a constrained optimization framework. We finally present two possible extensions of the proposed constraint for multi-class classification problems.
We present an extensive evaluation of the proposed method for image-based binary and multi-class classification problems on three datasets: an in-house medical dataset for prostate cancer, CIFAR10, and CIFAR100 [11]. In all datasets, we perform experiments by simulating different class imbalance ratios. In our experiments, we apply the proposed constraint to 9 different baseline loss functions, most of which were proposed to handle class imbalance. We compare the results with the baselines without any constraint. The results demonstrate that the proposed method improves the baselines in majority of the cases.

heading: 2 Related work
text: Various methods have already been proposed to learn better models with class-imbalanced datasets. We group the existing methods into three categories: cost sensitive training-based methods, samplingbased and classifier-based methods. Here, we focus on the first one and present related work for the other groups in the supplementary material for space reasons.
Cost sensitive training-based methods: This family of methods aims at handling class imbalance by designing an appropriate loss function to be used during training. In particular, they design loss functions to give more emphasis to the samples belonging to minority class, or the class with higher associated risk, than the majority ones during training [22]. [25] proposes a loss function, which we refer to as Weighted BCE (W-BCE), where samples from minority class are multiplied by a constant to introduce more cost to misclassification of those samples. [10] proposes a function that aims to learn more discriminative latent representations by enforcing DNNs to maintain inter-cluster and inter-class margins, where clusters are formed using k-means clustering. They demonstrate that the tighter constraint inherently reduces class imbalance. In a more recent work, [5] proposes a loss function called class-balanced binary cross-entropy (CB-BCE) to weight BCE inversely proportionally to the class frequencies to amplify the loss for samples from the minority class. In a similar vein, [13] modifies BCE and propose symmetric focal loss (S-FL) by multiplying it with the inverse of the prediction probability to introduce more cost to the samples that DNNs are not very confident. [15] introduces symmetric margin loss (S-ML) by introducing a margin to the BCE loss. [12] investigates different loss functions such as S-FL and S-ML, and propose their asymmetric versions, A-FL and
A-ML, by introducing a margin for samples from the minority class to handle class imbalance. In a different line of work, [23] proposes a method called mean squared false error by performing simple yet effective modification to the mean squared error (MSE) loss. Unlike MSE, which computes an average error from all samples without considering their classes, this loss computes a mean error for each class and averages them. [3] proposes a label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound, optionally coupled with a training schedule that defers re-weighting until after the initial stage. [19] introduces balanced meta-softmax for long-tailed recognition, which accommodates the label distribution shift between training and testing, as well as a meta sampler that learns to re-sample training set by meta-learning. [21] proposes a loss that ignores the gradient from samples of large classes for the rare ones, making the training more fair.
A particular group within the cost sensitive training-based methods focuses on optimizing AUC and our method falls into this group. AUC optimization is an ideal choice for class imbalance since AUC is not sensitive to class distributions [4]. [18] proposes a support vector machine (SVM) based loss function that maximizes AUC and demonstrates its effectiveness for the class imbalance. [26] approaches the class imbalance problem from online learning perspective and proposes an AUC optimization-based loss function. [7] proposes a one-pass method for AUC optimization that does not require storing data unlike the previous online methods. Another online AUC optimization method proposed by [24] formulates AUC optimization as a convex-concave saddle point problem. Despite their usefulness, all aforementioned AUC optimization-based methods were applied to linear predictive models, as this allows to simplify the Mann-Whitney statistics [16] for the definition of AUC, and their performance on DNNs is unknown. [20] applies online AUC optimization on a small dataset for breast cancer detection where they also mention that extension to larger datasets may not be feasible. In a very recent work called mini-batch AUC (MBAUC) [8], authors extend AUC optimization to non-linear models with DNNs by optimizing AUC with mini-batches and demonstrate its effectiveness on various datasets.
The proposed constrained optimization method differs from the existing works in that it enforces maximal AUC as a constraint in a way that favors reducing FPR at high TPR and can be used with existing loss functions.

heading: 3 Background - Augmented Lagrangian method (ALM)
text: A generic optimization problem for an objective function F (θ) subject to the constraints C(θ) = {c1(θ), ..., cm(θ)} can be expressed as [2, 17]:
arg min θ∈Θ
F (θ); subject to C(θ) (1)
Augmented Lagrangian method (ALM) [1], also known as methods of multipliers, converts the constrained optimization problem in Eq. (1) to an unconstrained optimization problem. ALM is proposed to overcome the limitations of two earlier methods called quadratic penalty method and method of Lagrangian multipliers which suffer from training instability and non-convergence due to the difficulty of convexifying loss functions2. In ALM, the penalty concept is merged with the primal-dual philosophy of classic Lagrangian function. In such methods, the penalty term is added not to the objective function F (θ) but rather to its Lagrangian function, thus forming the Augmented Lagrangian Function:
Lµ(θ, λ) = F (θ) + µ m∑ i=1 ∥∥ci(θ)∥∥2 + m∑ i=1 λici(θ) (2)
In practice, this method consists in iteratively solving a sequence of problems as:
max λk min θ Lµk(θ, λk), θ ∈ Θ (3)
Where {λk} is a bounded sequence in Rm, updated as λk+1i = λki + µci(θ). {µk} is a positive penalty parameter sequence, with 0 < µk ≤ µk+1, µk → ∞, which may be either pre-selected or generated during the computation according to a defined scheme. In ALM, increasing µk indefinitely
2Please consult supplementary material for more details of quadratic penalty method and method of Lagrangian multipliers.
is not necessary as in the quadratic penalty method. Thus, it does not suffer from training instabilities due to the constraint prevailing F (θ). Furthermore, it does not require convexity assumption as in the method of Lagrange multipliers to ensure convergence [1].

heading: 4 Proposed methods

heading: 4.1 Proposed constraint for binary classification
text: Let F (θ) be a generic loss function that is used to train classification DNNs, fθ(.), for binary problems. Let us also define p , {xp1, · · · , x p |p|} and n , {x n 1 , · · · , xn|n|} as the sets of samples of the positive (critical) and negative classes, respectively. Note that we choose p as the minority class in our description, i.e., |p| < |n|, and assume that this is a critical class associated with higher risk of making a mistake. We define our constrained optimization problem as follows:
arg min θ F (θ) subject to |n|∑ k=1 max ( 0,− ( fθ(x p j )− fθ(x n k ) ) + δ ) = 0, j ∈ {1, ..., |p|},
(4)
where fθ(x) indicates output probability of the DNN on input x. Note that the constraint states that the output of the DNN for each sample of critical class should be larger than the outputs of all of the negative samples by a margin δ. Satisfying the constraint would directly ensure maximal AUC [16].
We define the equivalent unconstrained version of Eq. (4) by writing it in the form given in Eq. (2) Lµ(θ, λ) = F (θ) + µ ∑|p| j=1 q 2 j
2 · |p| · |n| + ∑|p| j=1 λj · qj |p| · |n|
(5)
where, qj = ∑|n| k=1 max(0,−(fθ(x p j )− fθ(xnk )) + δ), µ is the penalty coefficient corresponding to the quadratic penalty term, λj is the estimate of Lagrange multiplier corresponding to each positive training sample j, and δ is the margin that we determine using a validation dataset.
The crucial aspect of the formulation is the asymmetry between positive and negative classes. A constraint is defined for each sample from the positive class, thus each positive sample gets a separate Lagrange multiplier. This form prioritizes the reduction of FPR at high TPR values as illustrated next.
We use Algorithm 1 to estimate the parameters of a DNN, θ, using the proposed loss function in Eq. 5. The parameters θ are updated with every batch using gradient descent with learning rate α. Concurrently, µ is increased using a multiplicative coefficient ρ only when a chosen metric on validation is not improved, by a margin to avoid training instabilities. The validation metric (ValMETRIC) is: 1) Validation AUC for the binary setup; 2) Validation Accuracy for the multi-class version. We update λj in each iteration for each positive sample x p j .
Algorithm 1 ALM for Training DNNs
Input: θ(0), µ(0), λ(0)j , ρ; for t = 1, . . . , T do
for each mini-batch of xB with size B do yB = f(XB); Calculate q(t)j ; . ∀j ∈ [1, B] and yj = y+
θ(t+1) ← θ(t) − α · ∇θLµ(θ(t), λ); λ
(t+1) j ← λ (t) j + µ (t) · q(t)j ; . ∀j ∈ [1, B] and yj = y+ end for if V alMETRIC(t) < V alMETRIC(t−1) then
µ(t+1) ← µ(t) · ρ; else
µ(t+1) ← µ(t); end if
end for Return θ(t+1)

heading: 4.2 Toy example describing our design choice
text: False Positive Rate
Tr ue
P os
iti ve
R at
e
0 1
1
Positive class samples Negative class samples
PositiveNegative
distances between markers indicate the differences between the outputs of the classifiers for the samples corresponding to the markers. Let us also further assume that all the Lagrange multipliers have the same value. In this case, one can verify that if the optimization swaps the locations of the left most positive sample (red circle marker) and the negative sample to its immediate right (blue triangle marker), the cost due to the constraint in Eq. (5) decreases by 39µ∆ 2
50 + 3λ∆ 25 . Swapping the
locations of the right most negative sample and the positive sample to its left decreases the cost due to the constraint by 19µ∆ 2
50 + 3λ∆ 25 3. So, from the cost perspective, the optimization should prefer the former swap over the latter, which corresponds to adding the green box to the AUC instead of red box. Therefore, the augmented Lagrangian cost would be decreased further when FPR at the highest TPR is reduced rather than increasing the TPR at the lowest FPR. Instead of defining the constraint for each positive sample in Eq. (5), if we were to define it for each negative sample in the exactly opposite way, i.e., ∑|p| j=1 max(0,−(fθ(x p j )− fθ(xnk )) + δ) = 0, k ∈ {1, . . . , |n|}, then the situation would be reversed. The optimization would prefer adding the red box over the green box to decrease the cost further. If we were to define a constraint for each positive-negative pair, i.e., fθ(x p j ) > fθ(x n k ),∀j and k, then adding the red or the green box to improve the AUC would yield exactly the same decrease in the cost.

heading: 4.3 Extensions to multi-class classification
text: The proposed constraint can also be extended to multi-class classification with slight modifications. Let us assume that we have a multi-class classification problem with C classes. In this case, there can be multiple critical and non-critical classes depending on the application. Let us define the corresponding family of sets, i.e., sets of sets, as P = {p1, · · · , p|P |} and N = {n1, · · · , n|N |} where the sets pi and ni are as defined in the previous section. Also, note that |P ∪ N | = C and P ∩N = ∅. A main difference between binary and multi-class classification is the dimension of the output. While fθ(x) was a single value for binary classification, in multi-class problem it is a vector with one value for each class. Using the notation for the positive and negative classes, we write the output of the
3Derivations for this toy example and theoretical insights for our design choice are provided in the supplementary material for convenience.
network as fθ(x) = {fp 1 θ , · · · , f p|P | θ , f n1 θ , · · · , fn |N|
θ }. Based on this, we define our first constraint for multi-class classification as
qcj = |N |∑ i=1 |ni|∑ k=1 max ( 0,− ( fp c θ (x pc j )− f pc θ (x ni k ) ) + δ ) , c ∈ {1, · · · , |P |}, j ∈ {1, ..., pc} (6)
where fp c
θ (x) indicates the output probability of DNN for the critical class p c. Note that the constraint
in Eq. (6) enforces that output probabilities of the DNN for critical classes should be larger for critical samples than for samples of other classes. We refer to ALM with the first constraint as ALMm,1.
An alternative definition of constraint is also possible. The first constraint enforces that probabilities of the critical classes should be larger for samples of each critical class. We can go a step further and add penalty to enforce that probabilities of the non-critical classes should be smaller for critical samples than the non-critical samples belonging to the corresponding class. We denote ALM with the second constraint as ALMm,2.
qcj = |N |∑ i=1 |ni|∑ k=1 max ( 0,− ( fp c θ (x pc j )− f pc θ (x ni k ) ) + δ ) + max ( 0, ( fn i θ (x pc j )− f ni θ (x ni k ) ) + δ ) , c ∈ {1, · · · , |P |}, j ∈ {1, ..., pc} (7)
In this formulation, the penalty does not view all non-critical classes as one, but also contributes to improving classification accuracy in those classes as well, by increasing the gap between the corresponding non-critical probability of critical and non-critical samples.

heading: 5 Experiments
text: In this section, we present our experimental evaluations on image-based classification tasks. We perform experiments on three datasets: an in-house MRI medical dataset for prostate cancer and two publicly available computer vision datasets, CIFAR10 and CIFAR100 [11].
In our evaluation, we experiment with different existing loss functions, most of which have been designed to handle class imbalance: classic binary and multi-class cross-entropy (BCE, CE), symmetric margin loss (S-ML) [15], symmetric focal loss (S-FL) [13], asymmetric margin loss (A-ML) and focal loss (A-FL) [12], cost-weighted BCE (WBCE) [25], class-balanced BCE (CB-BCE) and CE (CB-CE) [5], label-distribution-aware margin loss (LDAM) [3]. We first train DNNs for classification using only the loss functions and then using our method, which adds the proposed constraint to the loss function and solves Eq. (5), and we compare the classification performances. In addition, we also compare the proposed method with directly optimizing AUC using the mini-batch AUC (MBAUC) method proposed in [8] for the binary case.
The proposed method is implemented in PyTorch and we run all experiments on a Nvidia GeForce GTX Titan X GPU with 12GB memory.

heading: 5.1 Datasets
text: Prostate MRI dataset consists of 2 distinct cohorts: 1) a group of 300 multiparametric prostate MRI studies used for training and 2) another group of 100 multiparametric prostate MRI studies used for testing the trained DNNs. There is no overlap between the groups. Consent from the each subject is obtained to use the data for research purposes. Two board-certified radiologists with 10 and 7 years of experience in dedicated prostate imaging independently reviewed all examinations of the training set and test set and scored whether Dynamic Contrast Enhanced (DCE) sequences would be beneficial for cancer diagnosis. After completion of readings, a consensus was reached by the two readers by reviewing all examinations with discrepant decisions. The goal of the binary classification here is to identify subjects who do not require additional DCE imaging for accurate diagnosis, so they can be spared from unnecessary injection and cost and duration of the scanning can be reduced. In our experiments, we randomly split 20% of the training cohort as validation set by keeping the class imbalance consistent across the datasets. In both training and testing cohorts, positive samples represent 13% of all the patients which leads to an inherent 1:8 class ratio.
CIFAR10 and CIFAR100 datasets For CIFAR10, in the binary experiments we randomly select 2 classes among 10 to pose a binary classification problem. Additional experiments with different random selections are in the supplementary material. We use all the available training samples of the selected majority class, while we randomly pick a varying number of training samples for the minority class to obtain different class ratios, up to 1:200. We select 100 samples per class as validation set to determine hyper-parameters. For testing, we use all the available test samples, which consist of 1000 images for each class. For CIFAR100 in the binary setup we select one super-class as majority one, and a sub-class of another super-category as minority one. For training we use all the available 2250 samples of the selected larger class, and we randomly pick a varying number of training samples for the minority class to obtain different class ratios, up to 1:200. We use 50 samples for each original sub-class as validation set to perform the parameter search. For testing, we use all the available test samples, which consist of 100 images for each sub-class. For the multi-class experiments, long-tailed versions of CIFAR10 and CIFAR100 are built accordingly to [5, 3] and we consider the smallest class as the only critical class4. To be consistent with recent works [5, 3, 19], we use balanced validation and test sets in all the experiments on CIFAR datasets.

heading: 5.2 Training details
text: Network architectures: For the prostate MRI dataset, we use a 3D CNN that consist of cascaded 3D convolution, 3D max-pooling, intermediate ReLu activation functions and Sigmoid in the final output. For the binary CIFAR10 and CIFAR100 datasets, we use ResNet-10, while ResNet-32 [9] is adopted for the multi-class experiments, consistently with [3, 5]. Further training details can be found in the supplementary material.
Ensembling for higher reliability: Model reliability is very crucial when training DNNs. Dealing with small datasets may lead to dataset-dependent results even with the random splits, which could completely hinder objective evaluation. To weaken this phenomenon, we adopt the following ensembling strategy on MRI and for consistency we apply it on the binary CIFAR10 and CIFAR100 as well. Given a dataset and a class ratio, we create 10 random stratified splits of the dataset and train 10 models independently. The larger portions are used for training and the smaller portions for choosing hyper-parameters. During inference, all the models are applied on test samples and predictions are averaged in the logit space before the sigmoid function to yield the final prediction. We apply the ensembling to all the binary models we experiment with. This practice attenuates data dependency and we observed that the final AUC is improved when compared to the average of AUCs of different models, as presented in Table 3 for MRI sequences and in supplementary material for CIFAR10 and CIFAR100.
Hyper-parameters selection: Selection of the best hyper-parameters is crucial both to ensure proper and fair evaluation of the methods and to understand the true performance of any model. To achieve this, we perform grid-search to determine the hyper-parameters that yield the highest AUC for the binary experiments. For the multi-class tests we select the model that achieved the best overall accuracy on the validation set, in order to be consistent with the related works. The test sets in all experiments are not used for hyper-parameter selection. To reduce computational load, we select the common hyper-parameters such as the optimizer, learning rate, and the activation functions in the DNNs based on their performance with BCE loss function for the binary experiments, and consistently with [3] for the multi-class ones. Then, we keep them fixed in all experiments on the same dataset.
Besides the common hyper-parameters, the majority of the existing methods have hyper-parameters that crucially affect their performance. Namely, these hyper-parameters are margin m for S-ML and A-ML, exponent γ for S-FL and A-FL, weight of the cost c for WBCE and β for CB-BCE. We select best values for these hyper-parameters from the respective candidate sets that we create based on the information provided in the original papers for each of them.
In the proposed method, there are 4 parameters to be set: µ(0), λ(0), ρ, and δ. Thus, hyperparameters’ search is an important aspect of the proposed method. µ(0), λ(0), and ρ are stemming from ALM and we follow the guideline from [2] when setting them. We initialize all the Lagrangian multipliers λ
(0) i to 0. We choose µ (0) from the set {10−7, 10−6, 10−5, 10−4, 10−3}, as it is suggested to choose a small value in the beginning and increase it iteratively using the equation µ(k+1) = ρ · µ(k). We
4Experiments with multiple critical classes are presented in supplementary materials.
choose ρ from the set {2, 3} as ρ > 1 is suggested. Moreover, we do not increase ρ beyond 4 to avoid potential dominance of the constraint on F (θ), since ρ is used to increase µ. Once we find the best combination of µ and ρ based on the chosen metrics on the validation set, we fix them and we search for δ as final step. Please see supplementary materials for further details on the hyper-parameters selection.
Once the hyper-parameters for each model are selected, the training is performed and models are applied to the test set to yield the final results, which are described next.

heading: 5.3 Results
text: We present binary classification results on CIFAR10, CIFAR100, and the in-house medical imaging datasets in Tables 1, 2 and 3, respectively. In the tables, we compare each baseline with the corresponding ALM+baseline and present the best result among the two by bold. Also, the underlined results indicate the best results among all. We evaluate the performance of the baselines and the proposed method (ALM) using FPR at maximal levels of TPR (or minimal levels of false negative rate (FNR)) and AUC on the test sets. In the binary experiments, ALM is overall able to consistently improve the performance of almost all the loss functions, with regard to both AUC and FPRs at maximal TPR levels. Even in those cases when AUC is improved to a moderate extent there is still an improvement in FPRs, in accordance with our goal. Moreover, it is noticeable that the higher the TPR, the higher the benefit of applying ALM, which is in accordance with our target applications. In fact, considering that such classifiers in “critical” applications would be operated at high TPR (or low FNR), reduction in FPR in these settings is the effect we desired from the proposed approach. Lastly, we also observe in the tables that directly optimizing AUC via MBAUC does not provide the same improvements as using the proposed ALM approach. We present additional binary classification experiments in the supplementary material.
Additionally, we present the quantitative results of multi-class experiments in Table 4. A bold result for a baseline means that it is able to outperform both constrained optimisation methods ALMm,1 and ALMm,2, otherwise the better constrained strategies are highlighted. As for the binary case, underlined results indicate the best method for each metric. Comparison with the other baselines are presented in the supplementary materials. In these experiments, we evaluate the performance by computing the accuracy on the non-critical classes, at various levels of TPR for the critical-class. For this purpose, the first step is to find a threshold on the logit of the critical class such that the desired
TPR is obtained for the important class. All the test samples whose logit of critical class exceeds the selected threshold are assigned to the important class. The remaining samples that are not assigned to the critical class are then classified based on the highest probability over the non-critical logits. In addition to this metric, we present overall classification accuracy on all classes. The quantitative results demonstrate that both of the multi-class strategies, ALMm,1 and ALMm,2 improve on the baseline, reducing the error on non-critical classes, at high levels of accuracy on the important class. Moreover, ALMm,2 is able to further improve the overall accuracy by a larger margin in almost all the experiments thanks to the additional term in the constraint.

heading: 6 Conclusion
text: In this paper, we pose the training of a DNN for binary classification under class imbalance as a constrained optimization problem and propose a novel constraint that can be used with existing loss functions. The proposed constraint is designed to maximize the AUC, but in an asymmetric way to favor the reduction of FPR at high TPR (or low FNR). Then, we transfer the constrained problem to its dual unconstrained optimization problem using an Augmented Lagrangian method (ALM) [2] which we optimize using stochastic gradient descent. Additionally, we present two possible extensions of the proposed constraint for multi-class classification problems.
We perform an extensive evaluation of the proposed constraints for binary and multi-class image classification problems on both computer vision and medical imaging datasets. We compare the performance of the proposed constraints with different baselines by simulating different class imbalance ratio. The quantitative results demonstrate that the proposed constraints improve the performance of the baselines in the majority of the cases in both binary and multi-class classification experiments.

heading: 7 Acknowledgments
text: The presented work was partly funding by: 1. Clinical Research Priority Program Grant on Artificial Intelligence in Oncological Imaging Network, University of Zurich and 2. Personalized Health and Related Technologies (PHRT), project number 222, ETH domain.

Review summaries:
The paper proposes a novel method to perform classification with high imbalanced classes. The aim of the method is to minimise the FPR at high TPR -- this is suitable for many safety-critical applications where a misclassification can have dire consequences. In order to solve the problem the authors formulate the classification problem as a constrained optimisation problem and introduce a constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. 

Summary This paper proposes a new optimization approach to classification problems with class imbalance, where correctly classifying the minority class is more important than correctly classifying the majority class. One specific motivation is in medical imaging diagnostics, where misclassifying the “critical” minority class (e.g., diseased) is a more severe mistake than misclassifying the majority class (e.g., healthy). The authors therefore use an augmented Lagrangian method (ALM) approach to minimize the FPR for the majority while maintaining a high TPR for the critical class in a binary classification problem, as well as suggesting an optimization formulation for multi-class classification tasks. The ALM approach tries to enforce a margin between the critical and non-critical samples in the output space, and is structured to penalize a critical misclassification more than a non-critical misclassification.  The problem is certainly important for medical diagnostic tools, as well as other problems where errors have different costs for different classes. The proposed method is a useful approach for these problems, and the theoretical analysis of the asymmetry and required imbalance of the problem is clear and interesting. The experiments show some benefit of the ALM-based approach and its ability to be coupled with many different loss functions. The writing is clear overall, but with various grammatical mistakes throughout (and particularly in the supplement). My major concerns are the significance of the result and the setting of the CIFAR experiments. 

This paper proposes a very interesting approach for training DNNs on binary (and multiclass) classification tasks with critical under-represented classes, which is very typical in domains like healthcare. In this approach, the task is posed as a constrained optimization problem and a novel constraint is designed that emphasizes on reducing the false positive rate while maintaining a high true positive rate, which is then added to the loss function and solved using an Augmented Lagrangian Method. The main contribution of the paper is proposing a novel constraint optimization approach that can be added to any previous work, and that improves upon the state of the art results.