Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers
Authors: Nathan Tsoi
Kate Candon
Deyuan Li
Yofti Milkessa
Marynel Vazquez
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Network
Binary Classification
Evaluation Metric
Confusion Matrix
Accuracy
F-Score

Abstract: While neural network binary classifiers are often evaluated on metrics such as Accuracy and $F_1$-Score, they are commonly trained with a cross-entropy objective. How can this training-evaluation gap be addressed? While specific techniques have been adopted to optimize certain confusion matrix based metrics, it is challenging or impossible in some cases to generalize the techniques to other metrics. Adversarial learning approaches have also been proposed to optimize networks via confusion matrix based metrics, but they tend to be much slower than common training methods. In this work, we propose a unifying approach to training neural network binary classifiers that combines a differentiable approximation of the Heaviside function with a probabilistic view of the typical confusion matrix values using soft sets. Our theoretical analysis shows the benefit of using our method to optimize for a given evaluation metric, such as $F_1$-Score, with soft sets, and our extensive experiments show the effectiveness of our approach in several domains.

Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers
Nathan Tsoi Yale University nathan. tsoi@yale.ed edu
Kate Candon Yale University
Deyuan Li
Yale University deyuan 1i0yale..... edu
kate. candon@yale.
Yofti Milkessa
Marynel Vázquez
Yale University
Yale University
yofti mokei.milkessa@yylee.e. edu
marynel. vazymeel.azzueeeyyale...cc edu
Abstract
While neural network binary classifiers are often evaluated on metrics such as Accuracy and Fi-Score, they commonly trained with cross-entropy objective. How can this training-evaluation gap be addressed? While specific techniques have been adopted optimize certain confusion matrix based metrics, it is challenging impossible in some cases to generalize the techniques to other metrics. Adversarial learning approaches have also been proposed optimize networks via confusion matrix based metrics, but they tend to be much slower than common training methods. In this work, we propose a unifying approach to training neural network binary classifiers that combines a differentiable approximation of the Heaviside function with probabilistic view of the typical confusion matrix values using soft sets. Our theoretical analysis shows the benefit of using our method to optimize for given evaluation metric, such as Fi-Score, with soft sets. Also, our extensive experiments show the effectiveness of our approach in several domains.
Introduction
Neural network binary classifiers output probability p E [o. 1] which often used at training time optimize model parameters using the binary cross-entropy (BCE) loss. The network's output can also be translated to binary value {0, 1 indicating set membership to the negative or positive class. To determine set membership, the Heaviside step function H is commonly used with threshold т, where 2 are considered positive classification outcomes. This notion of set membership is often used in evaluation metrics for binary classifiers.
is a common assumption that optimizing a network via the desired evaluation metric is preferable optimizing surrogate objective [12
Unfortunately, optimizing network using the desired evaluation metric, such as Fcoor,,, is typically not feasible. The Heaviside step function used compute confusion matrix set membership in terms of true positives, false negatives, false positives, and true negatives, has a gradient with properties not conducive to optimization via gradient descent. The Heaviside function's 's gradient is not defined the threshold and is zero everywhere else.
18
29 33
Empirical results (e.g.. [12] show that the training evaluatievaluation gap hinders evaluation performance. The gap means that in many applications, the metric differs significantly from the surrogate loss (33] and there may not be a strong correlation between minimizing surrogate loss and improving
evaluation metric [29]. Also, it could mean that classifiers are solving "the wrong problem" when optimizing surrogate loss, leading sub-optimal evaluation performance [18).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To address the gap between training and evaluating neural network binary classifiers, we propose method to make confusion-matiis matrix based evaluation metries usable for backpropagation. Specifically, we propose the use of differentiable approximation of the Heaviside step function along with confusion matrix values computed using the notion of soft set membership. desired evaluation metric can then be made differentiable by calculating it over the soft-set confusion matrix, rather than using the traditional confusion matrix values.
Our main contributions are: 1) a novel method for training neural network binary classifiers that allows for the optimization of confusion-matrio based evaluation metrics with soft (Sec. 33: 2) theoretical analysis of our method (Sec. 4 and 3) the application of our approach to various domains with varying levels of class imbalance, showing its flexibility and superior performance compared to several bascline methods (Sec. [5. We provide an open-source implementation of our method for reproducibilityy
Preliminaries
In binary classification via neural networks, step function is required to transform the network's output to binary value. A common choice is the Heaviside step function with threshold value T:
[1 0
P2T p<t
H(P,))
Confusion matrix set membership is then computed for a prediction and ground truth label y via:
(p.
y=1
H H(. T)
y =
tp(p.u.T) =
fn(p.
(0
otherwise
otherwise
(p)
==0
tn(p.
Hp. T)
y =
(0
otherwise
otherwise
Consider set of predictions p E [0, 1}. ground truth labels y E (0, 1} and threshold value T E (0. 1),
e.g., {(11, V1. r), (p2, 1/2 T), (Pr, ya, T)}. For n samples, the cardinality each confusion matrix
set is then computed as:
[TP] Σ tp(p..u.. W. [PN]
Σ fulp:,
FPI
|TN =
Σ tr(ps.
Common classification metrics are based on these cardinalities. For example, Precision TP//TPP + IPP) is the proportion of positive predictions that are true positive results. Recall = TP/( TP + FFN) indicates the proportion of positive examples that are correctly identified. These metries represent trade-off between classifier objectives; it is generally undesirable to optimize or evaluate for one while ignoring the other [17] This makes summary evaluation metrics that balance between these trade-offs popular. Commonly used metrics include:
TPI + ITN
2
Accuracy TPI + TN + FPI + |FN]
Fi-Score
(3)
precision + recall
Accuracy is the rate of correct predictions to all predictions. Fi-Score is a specific instance of Fa-Score, which is the weighted harmonic mean of precision and recall: Fy-Score + B²) (precision recall)/((?² precision + recall). The value 3 indicates that recall is considered , times more important than precision.
Some metrics like Fg-Score are usually computed at a specific threshold 7 whereas others are computed over a range of values. For example, the area under the receiver operating characteristic curve (AUROC) is a commonly used ranking performance measure defined in terms of the true positive rate (TPR) and false positive rate (FPR), each function of / (PP( (T) dr.
While metrics that rely on confusion matrix set values are commonly used for evaluation, as shown in Fig. I (right), it is difficult to use them as loss during training. These metrics rely on the Heaviside
Source code: https::/ggthiei http://gtthuu..ooc///aaatttt///s

---

Training with approach
Evaluation
1/
Let TP, PPp Leccuracs
F Accuracy
P
backprop
FN TN,
FN TN
Heaviside approox.
confasion: marrix soft sots
Heaviside
confusion matrix
function
Bridging the gape unifying training evalustion
Figure The proposed method. Binary classifiers are typically trained with the BCE loss and then evaluated confusion-mateee based metric. We propose to bridge this gap between training and evaluation by optimizing model parameters based on a metric that is computed over a soft-set set confusion matrix, which is differentiable. This is done using a differentiable approximation of the Heaviside step function.
step function, the derivative of which is undefined at the threshold and zero everywhere else. This means that these metrics do have derivative useful to backpropagate errors through the network.
When a binary classification neural network is trained on a loss function that is different from the evaluation metric, such as BCE, the network parameters are unlikely be optimal for the desired evaluation metric. As shown in our theoretical analysis and experimental results, when the goal is to balance various confusion matrix set values, e.g., with Fj -Score, performance is improved by using our proposed method to optimize Fi-Score.
Method
We propose method (Fig. [T that aims to better unify the training and evaluation steps of binary neural network classifiers whose performance is measured with metrics based the confusion matrix. Our method has two main steps. First, the Heaviside step function, H, is approximated with function H useful for optimization via gradient descent. Then, H is used compute soft version of set membership in the confusion matrix. With the soft-set confusion matrix values, we can then compute desired confusion-mattie based metrics, such as F1 Sc-ocoee using their standard formula. This approach makes the metrics end-to-end differentiable so that they can be used as a training loss.
3.1 Heaviside Approximation
A useful Heaviside approximation, H. has non-zero gradient: ()) 00, V Also, it ensures that for single example, the classifier predicts positive and negative probabilities which sum to 1. Like H, the approximation should meet the properties of a cumulative distribution function (CDF)
which ensures is right-continuous, non-decreasing, with outputs in [0, 1] following:
lim H(p, 7) Vr
lim H(p,T)=1 VT
Sigmoid approximation: One approximation method for H, proposed by 233, the sigmoid function so p) = kop)-- We reparameterize to account for r, so that it can be used for varying thresholds: H (k. that as k increases, the sigmoid function better approximates the Heaviside function, but the derivative is close to zero over a larger range of valid inputs. Another challenge is that when т is close to 1, H- does not approach H the input approaches or 1. These limitations as well as the hyperparameters and are further discussed in the Supplementary Material (Sec. 2
+ (a(p-)))-1_ A challenge with the sigmoid approximation is
Linear approximation: An alternative approximation of H is five-point linearly in- terpolated function (Fig. left). This approximation Hi, is defined over [0, 1] and is parameterized by given threshold т and slope parameter 8, which define three linear segments with slopes mi, m2, and mg. The slope of each line segment is:
1-26 25
8
T Im
m2
m2
Tmm
Ten with Tm min 7) in order ensure gradient suitable for backpropagation. The lincar

---

Heaviside approximation is thus given by:
p m1
p < Im 3
H
p- my + (1-6-man ma 22)
p + fm 2
(5)
p- m2 + (0.5 m27)
otherwise
Considering the threshold 7 in the formulation of 7' ensures H = T, 0.5 while maintaining the limits Eq. (4). See the Supplementary Material, Sec. 2.2 for the derivation of Eq. (
Recently, a similar linear function was implemented as an activation function to enforce meaningful logical outputs in logical neural networks [3). However, rather than using the linear function for activation within the larger model architecture, we use it in the training objective of classifier.
3.2
Soft Sets
We soft sets, a generalization of fuzzy sets [25], compose differentiable versions of confusion- matrix based evaluation metrics useful for backpropagation. A soft set μu in U, the initial universal set, is defined by the membership function μ. : U [0, I]. For U. the membership function (x) specifies a degree of belonging for x to set u.
We the notion of soft sets to compute confusion- matrix based metrics with Heaviside approxi- mation in place of the typically used strict sets. Soft set membership corresponds to the degree to which sample tuple (p, T) belongs to a confusion matrix set. We define the soft confusion matrix set membership functions for prediction and ground truth examples (p, y) relative to as:
tps(p. =
H(p.T) T)
==1
(n y.
H(p, r)
=
(0
otherwise
otherwise
(6)
fp-(p.u.7)
(((p.T)
y =
Inalp. y.7)
H(p, 7)
=
otherwise
otherwise
Whereas the values of the confusion matrix (tp, fn, fp, tn) are the sum of zeros and ones, and the values of the soft confusion matrix (tp., fn,. fP.. tn₂) are the sum continuous values in [0, 1], rather than integers.
After computing per-sample soft set values, specific metrics can be approximated by summing over the relevant elements of the confusion matrix. For instance, approximate precision as [TP.I/(] TP,I + |FPz)), where TP. = >n= (p, ys. r) and =P,|==' Σi= f fpx(Pt Vs. T) for n samples. In practice, this summation occurs at training time over mini-batches while optimizing via gradient descent. Because gradient descent and its variants expect small but representative sample of the broader data [5]. the proposed method also expects representative sample.
Decomposability: Metrics based on confusion matrix set values are considered con-decomposable Non-decomposaabe metrics cannot be calculated per datapoint and are not additive across subsets of data 19] We acknowledge that our optimization method of mini-batch stochastic gradient descent (SGD) does not provide an unbiased estimator
non-decomposable metrics. However, this is common neural network training [16]. In practice, we find that large enough mini-batches provide representative sample for confusion matrix based metrics, allowing the use of mini-batch SGD in our approach. Moreover, our method does not limit maximum batch size for training, unlike the adversarial approach to optimize F- -core proposed by [4].
Any metric composed of confusion matrix set values (TP, FP, FN, TN) can be approximated using our proposed method and used as a training loss. In our experiments, we train and evaluate on Aceuracy, FS-ccre, AUROC, and F3-Score (31]. chose these metrics to show the flexibility of our approach. Accuracy, -i-rr and Fy-Score may be selected based class imbalance and resulting tradeoffs. AUROC illustrates that our method can be used when computation over range of thresholds, is required.
Metrics and losses:
4 Theoretical Grounding
In this section, we first show the Lipschitz continuity variety of soft-set based metrics under the proposed Hcaviside function approximations. This indicates that when such metrics used as the

---

loss in neural network, the difference between successive losses is bounded across iterations of stochastic gradient descent. We then show. under certain assumptions, that metrics computed over the soft-set confusion matrix values are asymptotically similar to the true metric.
4.1 Lipschitz Continuity of Metrics Based Soft-Set Confusion Matrix Values
function f is considered Lipschitz continuous if there exists some constant K such that for all 21 in the domain, (x() - f(22) z K| -22l. Lipschitz continuous functions are always themselves continuous.
Theorem 4.1. The linear Heaviside approximation H. is Lipschitz continuous with Lipschitz constant M max mm, M2, my).
H is continuous because each piecewise linear component is continuous and Ht is defined to be continuous at the two points p = 7 + "2- Also, by construction, the slope of any secant line is positive and bounded by M = max 1 m2, ma). Thus, H' is Lipschitz continuous with Lipschitz constant M. Please see the full proof in Sec. 1.1 of the Supplementary Material. Note that the H" is also Lipschitz continuous (32].
Theorem 4.2. Every entry of the soft-sets confusion matrix based on the Heaviside approximations Lipschitz continuous in the output of a neural network.
The proof of Theorem 4.2 is in the Supplementary Material (Sec.
The value of Lipschitz continuous loss function composed of soft-set confusion matrix values is Lipschitz continuous. This is because compositions of Lipschitz continuous functions are also Lipschitz continuous. Following (9, note that the confusion-matie Balanced Accuracy, Fg.. Jaccard, and G-Mean are all Lipschitz continuous. These metrics are also Lipschitz continuous under our proposed method computed via soft sets.
based metrics Accuracy,
Lipschitz continuity of the loss function in neural network optimized via stochastic gradient descent indicates convergence without extreme variations in losses throughout training. Let K be the Lipschitz constant for the objective function f(w) on network weights w. Because of Lipschitz continuity, when updating Wi+1 - w; a;l'(w), (w:) using stochastic gradient descent with learning rate or small local change in the weights |W++1 wyl = az (w;) corresponds to a small local change in the value of the objective function of ([(w;+1) t(w:)] S (a,]e(w:))]
4.2 Approximation of Confusion-Matrix Based Metrics with Soft Sets
We provide statistical analysis showing that in the limit, as the number of examples goes to infinity, Fi-Score calculated with soft sets approximates the expected true Fi-score under a set of assumptions. Similar proofs for Accuracy and AUROC are provided in the Supplementary Material (Sec. [.2).
Consider dataset of size with (.1, .... Fn examples and {uI,.., Vn labels. Suppose network outputs a probability P: that the label y4 In this section, since the specific outputs p; are unknown and may change across iterations, we assume p: is random variable. When calculating Fi-S Fiore,, p; is passed through the Heaviside function H, which generates an output i!! Hps; T), where off (0, 1}. The Fi-ccoee can be expressed as:
Eim !
2 D. V.O." )(
F V. +
+ + v:)"""
Σ (v. + a1f
To calculate Fi- -Sore with soft sets (FF), P. is passed through the Heaviside approximation H. which generates output 0zf H(p,, 7), where úiHt [0. 1]. Fi-Score with soft-sets can be expressed as:
2 2 VAN."
F
(8)
> +
Consider a network trained on dataset with rn positive and (1 - r)n negative elements, where E [0. 1] is some constant. Suppose this classifier correctly classifies any positive example as true positive with probability u and any negative example as a false positive with probability v. Also, assume that all classifications are independent. Because Fi Score is calculated with discrete eff
5

---

assume that the classifier will classify examples a random variable o!" ~ (ernoull((ug,+o(((1y))) Thus, /f ~ Bernoulli(1) if y, = 1, and Dن ~ Bernoulli(v) if y: = 0.
Since Ff can take on continuous values in [0, 1), we consider that 0 is random variable drawn
from Beta distribution, which has support [0, 1]. In particular, assume o Beta( + a
3:), Puy: + B1 3:)). Hence, 0.th Beta (a.. 3., if yE 1, and y Beta(a 3,) if yi 0.
Let au and a.+o. U, for any i. E [ u if y, 1, and E [ if 0.
Under the above assumptions, both F1 and Ff have the same average classification correctness: for any given i, [ = E [u.+]. Also, there exists some Cu, Pu, Ge 9o such that the distributions of 9."h H (ps. 7) and 0/f = Mp,, r) can both hold simultaneously under the same network for all i.
If we let U Binomial(wrr, u) and V ~ Binomial( (n(1 -r), v) be the independent random variables denoting the number of true positives and false positives a sequence of independent predictions, then F1 from Eq. 77 becomes:
2 2 B. Fi
n Di=ı 11
nr + U + = (
Ufn r+ V/n
(9)
Σ (Nt 1.1))
vor
By the Strong Law of Large Numbers, _ # and m((->) V S both converge with probability 1
0C. Hence, U " ru and V. ** r)v. We therefore have, from the Continuous Mapping Theorem, that as n + 0o:
U/n Fi r+ U/n V/n
$2
ru
2ru
(10)
r
For Fí. let U
1/2h be the independent random variables denoting the total amount of true positives and false positives in the soft set case. Then, Ff from Eq. ( becomes:
-1 1ith -
and V
-
D DO Ff Σn (vi DN)
Σ y. nr Ei D7!
U U v. ) =2 U+U+ (
U
=2
(11)
Since U is the sum of nr i.idd random variables distributed as Beta(Qu, Su). which has mean by the Strong Law of Large Numbers, U ass. autp converges with probability 1 as - Hence, U' "$ ru and from the Continuous Mapping Theorem, that as n + oc,
au a.+
Similarly, xi1ー V 25
aut2. also
Va "$(1 r)v. We therefore have,
"
U r r+U*/n+Ve/nn
"3
ru
2ru
(12)
r r)0
ru
Thus, Fi and Ff both converge almost surely to the value
Since 0 S r+rute 1 is bounded, E[Fi], E[F?] - p-rayyrr by the Bounded Convergence Theorem. This means that the Ff value is an asymptotically unbiased estimator for the expected true Fi-Score, and we expect average Fi -Soore values to converge to F as n - oo, under setup. However, the Fi -c-Se computed from soft sets is not an unbiased estimator for the expected true Fi -Sooe for finite n.
2r
2ru
The proof of almost sure convergence generalizes for any metric that is continuous function in the ratio of each entry of the confusion matrix to n, as shown the Supplementary Material (Sec. T.2). This suggests that optimizing over the desired evaluation metric using our proposed method (e.g.. Fi-Score computed with soft sets) justified, as the loss should follow our true final loss closely for a large enough n. However, note that some metrics may be poor indicators of true classifier performance, such as Accuracy when data imbalanced [7]. In such cases, it may not desirable to optimize for such metries using method, as further discussed in the next section.

---

5 Experiments
This section presents three experiments to (1) evaluate the performance of our approach against several baselines tabular data, (2) evaluate approach higher-dimensional image data, and (3) evaluate the ability of our method to balance precision and recall during training.
Experiments were conducted on five publicly available datasets in variety of domains and were chosen for their varying levels of class imbalance as explained later. All datasets were minimally pre-processed and split into separate train, test, and validation sets. See Sec. 4.1 of the Supplementary Material for details.
Datasets:
Architecture and training: Our experiments aim to fairly evaluate our method using different confusion-matrix based objective functions. Therefore, the same network architecture and training scheme was used unless otherwise noted. Performance was evaluated over 10 repeated trials to control for the effects of random weight initialization. We report the mean of results evaluated over threshold values T {0.1, 0.2, ..., 0.9} } for metrics that require threshold choice at evaluation. The details of the network architecture and training scheme are Sec. 4.2 of the Supplementary Material.
We trained networks using the typical BCE loss as well as two existing approaches for optimizing specific confusion matrix metrics: an adversarial approach for Fi-score [14), and using
Baselines:
an approximation of the Wilcoxon-Mang Mann-Whitney Whitney (WMW) statistic for AUROC [39]. Also, the Sup- plementary Material provides comparisons with other, less related methods for binary classification.
5.1 Experiments on Tabular Data
We evaluate our proposed approach using four tabular datasets with different levels of class imbalance. The CocktailParty dataset [40] has 30.29% positive class balance making it the most class-balanced dataset of those considered in this experiment. Salary classification data in the Adult dataset, from the UCI Machine Learning Repository [II, has a 23.93% positive class balance. Classifications of microcalcifications in the Mammography dataset 38 is heavily skewed with 2.32% positive class
balance. Lastly, the Kaggle Credit Card Fraud Detection dataset (36) has the most extreme class balance with only 0. 0.7 positive class balance.
We compare bascline methods against the performance of neural networks trained using our method to optimize Fi-Score, Accuracy, and AUROC. We instantiated our method using the Heaviside approximations and soft sets. Results using the linear approximation are presented in Table See the Supplementary Material, Sec. [5, for results using the sigmoid approximation, which were comparable.
Fi computed with soft sets. Our method of optimizing Fi over the soft-set confusion matrix out- performs baselines when evaluated on Fi-Score for all datasets. In Table [ line (1) has a higher F Fi-Score than lines (2) and (6). Additionally, even when evaluated on the other metrics (Accuracy and AUROC), networks trained using our method on Fi perform similarly or better than BCE.
Accuracy computed with soft sets. Our method of optimizing Accuracy over the soft-set confusion matrix has comparable performance to the BCE baseline when evaluated on Accuracy for all datasets. In Table line (3) has comparable Accuracy to line (6). Note that Accuracy can reward prediction of only the dominant-clat in imbalanced datasets [17), potentially resulting in no positive predictions and Fi -Scoe of O, in line (3) of Table for for the Mammography dataset.
Our theoretical analysis (Sec. 4.2p indicates that our proposed method of optimizing confusion-matris. based metrics using soft-set Fi soft-set Accuracy should converge to the evaluation metric (F) or Accuracy respectively) in the limit. The alternative of using BCE loss performs comparably to our proposed method in some cases; however, BCE - > (y, log p: + (1 y.) log(1 p.)) and Accuracy = (ITP)= + |TN))/n are different expressions that don't generally converge to each other as 00.
AUROC computed with soft sets. Neither optimizing AUROC using our method nor optimizing AUROC via the WMW statistic (39 consistently outperforms traditional BCE. In Table lines (4) (6). This may be due in part to
and (5) have lower Fi-Score, Accuracy, and AUROC than line
AUROC metric's challenge with scale assumptions [15]. Moreover, AUROC is not linear-fractions
21, unlike FS-Scor and Accuracy. Therefore, AUROC does not necessarily have an unbiased estimator of gradient direction (I, which may also contribute to lackluster performance.
7

---

Table 1: Losses (rows): Fi, Accuracy, and AUROC via the proposed method (*) using the linear approximation; F1 Fi-Scoref via adversarial approach [14] and AUROC; via WMW statistic 39]. indicates performance better than equal to the BCE baseline.
Bold
CocktailParty (μ σ)
Adult (p + o)
Loss
-Sore
Accuracy
AUROC
Fi-Score
Accuracy
AUROC
(1)
F1
0.75 + 0.01
0.85 + 0.01
0.82 + 0.01
0.63 + 0.02
0.78 +: 0.04
0.78 -: 0.02
(2) F
0.30 1:3000.00 0.06
0.06
0.60 0.02
0.16
0.78 + 0.00
0.55 +.55+0001
Accuracy
0.70 0.02
0.00
0.35 0.01
0.81
0.01
0.61 0.02
(4)
AUROC
0.01
0.01
0.00
0.01
0.32 0.02
0.01
(5) AUROC;
0.01 1001::00 0.03
0.70
0.03
0.50
0.50
0.00 1 0.00
0.76 1 0.00
0.50 1 0.00
(6) BCE
0.70 - 0.02
0.85 1 0.01
0.08
0.26 1 0.06
0.80 1 0.01
0.58 1 0.02
Mammography (1x + Ø)
Kaggle (μx + g)
Loss
Fi-Score
Accuracy
AUROC
Fi-Score
Accuracy
AUROC
(1) F
0.63
0.04
0.98 + 0.00
0.78
0.03
0.83
0.02
1.00 + 0.00
0.90 + 0.02
(2) Fif
0.46
0.08
0.98 - 0.00
0.66 = 0.04
0.76 0.06
1.00
0.83 = 0.04 0.04
(3)
Accuracy
0.00
0.00
0.97=0.00
0.50 - 0.00
0.62
1622:.00 0.33
1.00
0.15
(4) AUROC
0.11 0.01
0.18
0.57 0.02
0.06
0.11
0.50
(5)
AUROC:
0.00
0.12
0.50 0.00
0.00
0.13
0.50
(6)
BCE
0.56
0.99 0.00
0.71 0.06
1.00 0.33
0.73
Table 2: Losses (rows): Fi and Accuracy via our proposed method. Bold indicates performance
better than or equal to BCE baseline.
IIFAR-10-Transportatin (fx + σ)
CIFAR-10-Frog (J+ + σ)
Loss
Fi Scoree
Accuracy
Fi Score
Accuracy
Fi*
0.91 + 0.00
0.93 + 0.00
0.73 0.01
0.95 + 0.00
(2)
Accuracy*
0.92 + 0.00
0.93 + 0.00
0.74 0.01
0.95 + 0.00
(3) BCE
0.88 = 0.01
0.91 1 0.00
0.59 0.04
0.94 + 0.00
Note that in comparison to 14] the adversarial approach Fit did not perform as well in our experiments. The difference could be due to different datasets, and because we used PyTorch implementation provided by the author while [14] utilized Julia and Flux ML. Training networks for the tabular datasets on Fi-Score using our method on CPU took median time of 2.3 0.16 minutes whereas the adversarial approach on CPU took median time of 70 + 1 118 minutes.
5.2 Experiments Image Data
We conducted experiments similar to the those in Sec. 5.1 with higher dimensional data using two different binary image datasets created from the CIFAR-10 dataset [22]. The CIFAR-10- Transportation and CIFAR- CIFAR-10-Frog datasets had a 40% and 10% positive class balance, respectively.
We focused on Accuracy and Fi-Score, given the limitations with AUROC found in earlier experi- ments. We also excluded the adversarial approach to optimize Fi-Score [4] due to long runtime on tabular data. Otherwise, we use the same set up from Sec. 5.1
Overall, our findings with image data Table are consistent with tabular results from Sec. 5.1 Our method of optimizing Fi-S Fi-Score over the soft-set confusion matrix performed better than traditional BCE for both datasets when evaluated on both Fi-Score and Accuracy. In Table line (1) has higher the soft-set confusion
scores than line (3). Results from method of optimizing Accuracy matrix (line (2) in Table are better than the results for BCE (line (3) in Table (bble)..
5.3 Experiments on Graph Data
We applied our method to a more structured representation of the examples in the CocktailParty dataset [40]. Following Thompson al. [35,, constructed fully-connected graph in which each
8

---

Table 3: Graph data from the CocktailPPaty (a + a) dataset classified using a Graph Neural Network (GNN) following Thompson et [35. Fj and Accuracy and via our proposed method. Bold indicates performance better than or equal to BCE baseline.
Loss
Fi-Score
Accuracy
Fi
0.83 + 0.01
0.77 0.01
Accuracy
0.77 + 0.02
0.72 + 0.02
BCE
0.81 + 0.03
0.75 + 0.03
Table 4: Mammography (u + σ) dataset: Fg (3 {1, 21 33) loss using the proposed method, to balance between precision and recall while maximizing Fi-Score.
Loss
Fi -coor
F2 Score
FooSe
Precision
Recall
(1)
Fi
0.61 + 0.06
0.57 = 0.06
0.56 0.06
0.70 0.06
0.55 0.07
(2)
P2
0.63 + 0.04
0.67 0.04
0.69 + 0.04
0.57 + 0.05
0.71 0.04
(3)
0.57 + 0.03
0.69 0.02
0.75 + 0.02
0.44 + 0.04
0.81 + 0.03
person corresponded to a node in the graph and edges held distances between people. We then used a pesssage-passingg Graph Neural Network (GNN) 2 to predict pairwise affinities corresponding to whether two individuals were part of the same conversational group. This task was similar to the experiment on tabular data (Sec. [.I] using the CocktailParty dataset, but instead of predicting interactions among pairs of potential interactants independently, we made predictions for all the pairs seen in a frame of the dataset simultaneously. Using the hyperparameters proposed by Thompson et al. (35]
we obtained results (Table 3 in line with those for the tabular data (Sec. 5.1
5.4 Balancing Between Precision and Recall
Our method also allows training-time optimization that balances between precision and recall using Fg-Score. This approach training is particularly useful in real-world scenarios where there is high cost associated with missed detections. This type of metric is difficult to optimize effectively for using a typical BCE loss because BCE is not aware of any preference towards precision or recall.
Results in Table 4 show that using the proposed method to optimize Fg-Score is an effective way of maintaining maximum classifier performance while balancing between precision and recall at ratio appropriate for a given task. Increasing values of 3 correspond to an increased preference toward recall with small loss of total performance measured by Fi Fi-Score. Similar results optimizing for Fg-Score with additional datasets are provided in Sec. 6.2 of the Supplementary Material.
6 Related work
Our work is inspired by research on the direct optimization of evaluation metrics for binary classifica- tion. This includes plug-in methods that empirically estimate threshold for a classifier on metric. For example, [26] demonstrated the applicability of plug-in classifiers to optimize Fi-Score with linear models. For metrics based on linear combinations of confusion matrix set cardinalities, [2T identified an optimal plug-in classifier with metric-dependent threshold. Also, 20 explored the Precision @ K metrie for linear models the context of ranking. Our approach is not a competitor to plug-in methods, but rather approach to train a neural network classifier on a differentiable approximation of metric based on the confusion matrix. As such, it could be used in conjunction with plug-in method, if desired.
Works such [9] optimized specific metries like Precision@ K and Fi-Score in online learning, which is characterized by the sequential availability of data. Our work does not address online learning, but batch learning methods. Additionally, other work has focused on optimizing AUROC 18 39. F-score [1], and AUPRC in the context of ranking [12]. Rather than focusing specific metric, we provide a flexible method for optimizing a neural network using approximations of varied metrics based on the confusion matrix. For example, our experiments in Sec. show how tradeoffs between precision and recall can made by adjusting the / parameter the Fy-score during training with our method.
single

---

In the field of computer vision, differentiable surrogate losses have been proposed for the Fi -S-or
6 [) Jaccard Index [ 28 29, and Dice score 4 27 34] However, since these methods are applied to difficult problems in computer vision, they incorporate particular surrogate, such as surrogate for the Fi-Score, into larger composition of losses. Mean average precision (mAP) 13, (IoU) values at different thresholds
for example, is composed of a number of intersection-overuung for multiple classes. We leave the study of metrics which require complex aggregation to future work. In our experiments, we focus evaluating the impact of individual losses approximating desired evaluation metric. We also evaluate the theoretical merits of optimizing a differentiable surrogate for confusion-matiio based metrics using approximation of the Heaviside step function and soft-set confusion matrix values.
Recently, adversarial approaches have emerged as another related area of research. Wang et al. 337 used structured support vector machine and reported performance on Precision as well as Fi-Score. Fathony and Kolter [14), which we compare against, improved performance via marginalization technique that ensured polynomial time convergence. The authors evaluated their adversarial approach Accuracy and F -S-oc,,, among other metrics, while reporting performance relative to BCE [1]. Downsides of the latter approach are that it is limited to small batch size, on the order of 25 samples, and has cubic runtime complexity. Our approach does not limit batch sizes and has a worst-car case runtime complexity equivalent to the runtime complexity of the confusion- based metric being used loss.
7 Broader impact and ethics
Our work was motivated by the potentially wide ranging impact that more flexible and robust binary classifiers could have across application domains such as social group dynamics, sociology, and economics, which we explored in our experiments. Improving the tools used in these areas of research has the potential to positively impact human quality of life, but are still susceptible to data bias. These tools should be used with care when building safe and responsible artificial intelligence systems.
8 Conclusion
We proposed a novel method to optimize for confusion-matii based metrics by using a Heaviside function approximation and soft-set membership. Our method addresses the common training- evaluation gap when working with binary neural network classifiers: these networks are typically trained with BCE loss, but evaluated using confusion-matrix based metric, such as Fi-Score.
Our theoretical analyses showed that soft-set confusion matrix based metrics, such as Ff, are Lipschitz continuous and are likely to converge in expectation to the true metric's 's expectation. Additionally, our experiments showed the feasibility of using our method to optimize confusion matrri based metrics. While many factors play into final classifier performance, such as balance of samples in the dataset, the desired evaluation metric, the approximation of H, and network hyperparameters, we found that in many cases, final classifier performance can be improved by bridging the training-evaluution gap using our method. In fact, optimizing model parameters for F1 with soft sets resulted in better evaluation results with Fi Fi-Ccore and better or comparable evaluation results with Accuracy and AUROC. Our approach also outperformed other methods that directly optimize for a specific metric. For example, Fathony and Kolter's 's [14] adversarial approach applied to SScore had much longer training time and resulted worse performance than our approach.
In the future, we are interested in further investigation of other aspects of our approach such as analyzing statistical sample efficiency well as the approximation error and the generalization error. Further, we believe studying how our method can be applied to multi-class classification problems is an interesting area for future work.
Acknowledgements
The authors thank Andre Wibisono for helpful discussions and feedback. This work was supported by the National Science Foundation (NSF), Grant No. (IS--124802)) The findings and conclusions in this article are those of the authors and do not necessarily reflect the views of the NSF.
10

---

References
[1] Han Bao and Masashi Sugiyama. Calibrated surrogate maximization finear-faactional utility in binary classification. In International Conference Artificial Intelligence and Statistics, 2020.
[2] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkmer, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1vXvv11806.0126, 2018.
[3] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: tractable surrogate for the optimization the intersection-oveeruniin measure in neural networks. In CVPR, 2018.
[4] Jeroen Bertels, Tom Eelbode, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Biss- chops, and Matthew B Blaschko. Optimizing the dice score and jaccard index for medical image segmentation: Theory and practice. In International Conference on Medical Image Computing and Computer-Assiseed Intervention, 2019.
[5] Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pages 221-436. Springer, 2012.
[6] Ilker Bozcan, Jonas Le Fevre, Huy X Pham, and Erdal Kayacan. Gridnet: Image-agnostic conditional anomaly detection for indoor surveillance. IEEE Robotics and Automation Letters, 6(2):1638-164) 2021.
[7] Róbert Busa-Fekete, Balázs Szörényi, Krzysztof Dembczynski, and Eyke Hüllermeier. Online f-measure optimization. In NeurIPS, pages 595-603, 2015.
[8] Rail Chamidullin, Milan Šulc, Jifí Matas, and Lukáš Picek. A deep learning method for visual recognition of snake species. 2021.
[9] Krzysztof Dembezyíski, Wojciech Kotlowski, Oluwasanmi Koyejo, and Nagarajan Natarajan. Consistency analysis for binary classification revisited. In International Conference on Machine Learning, pages 961-969. PMLR, 2017.
[10] Krzysztof Dembezynski, Willem Waegeman, Weiwei Cheng, and Eyke Hüllermeier. An exact algorithm for f-measure maximization. In NeurlPS, 2011.
[11] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[12] Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan Rifkin, and Gal Elidan. Scalable Learning of Non Non-Decomposable Objectives. In AISTATS, 2017.
[13] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(():98-136,, 2015.
[14] Rizal Fathony and Zico Kolter. Ap-perf: Incorporating generic performance metrics in differen- tiable learning. In AISTATS, pages 4130-4140, 2020.
[15] Peter Flach and Meelis Kull. Precision-recall-aain curves: Pr analysis done right. In NeurIPS, pages 838-846, 2015.
[16] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT' press Cambridge, 2016.
[17] Haibo He and Edwardo A Garcia. Learning from imbalanced data. TKDE, 2109)) 2009.
[18] Alan Herschtal and Bhavani Raskutti. Optimising area under the roe curve using gradient descent. In ICML, page 49, 2004.

---

[19] Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Online and stochastic gradient methods for non-eecomposable loss functions. In NeurIPS, pages 694-702, 2014.
[20] Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Surrogate functions for maximiz- ing precision at the top. In ICML. pages 189-198, 2015.
[21] Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon. Consistent binary classification with generalized performance metries. In NeurlPS, pages 2744 2752, 2014.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Nikolay Kyurkchiev and Svetoslav Markov. Sigmoid functions: some approximation and modelling aspects. LAP LAMBERT Academic Publishing, Saarbrucken, 2015.
[24] Fausto Milletari, Nassir Navab, and Seyed- Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), 2016.
[25] Dmitriy Molodtsov. Soft set theory-first results. CAMWA, 1999.
[26] Harikrishna Narasimhan, Rohit Vaish, and Shivani Agarwal. On the statistical consistency of plug-in classifiers for non-decomposable performance measures. In NeurIPS, pages 1493-1501, 2014.
[27] Marcus Nordström, Han Bao, Fredrik Löfman, Henrik Hult, Atsuto Maki, and Masashi Sugiyama. Calibrated surrogate maximization of dice. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 269-278. Springer, 2020.
[28] Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-uiie in deep neural networks for image segmentation. In International symposium on visual computing, pages 234-244. Springer, 2016.
[29] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[30] Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, et al. Logical neural networks. arXiv preprint arXiv:2006. 13155, 2020.
[31] C. J. Van Rijshergen. Information Retrieval. nutterworth-Heinemnnn. USA, 2nd edition, 1979.
[32] Yuzhang Shang, Bin Duan, Ziliang Zong. Liqiang Nie, and Yan Yan. Lipschitz continuity guided knowledge distillation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 10675-10684, 2021.
[33] Yang Song, Alexander Schwing, Raquel Urtasun, al. Training deep neural networks via direct loss minimization. In ICML., pages 2169-2177, 2016.
[34] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Gener- alised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep learning medical image analysis and multimodal learning for clinical decision support, pages 240-248. Springer, 2017.
[35] Sydney Thompson, Abhijit Gupta, Anjali W Gupta, Austin Chen, and Marynel Vázquez. Conversational group detection with graph neural networks. In Proceedings of the 2021 International Conference on Multimodal Interaction, pages 248-252 2021.
[36] Machine Learning Group UBL. Credit card fraud detection. https: / www. kaggle com/
mlg www.uu/bczee httpb//rrrdttcarrrrrdddd Accessed: 2022.
12

---

[37] Hong Wang, Wei Xing, Kaiser Asif, and Brian Ziebart. Adversarial prediction games for multivariate losses. NeurIPS, pages 7728-2736, 2015.
[38] Kevin Woods, Jeffrey L Solka, Carey E Priebe, Chris Doss, Kevin Bowyer, and Laurence P Clarke. Comparative evaluation of pattern recognition techniques for detection of microcalcifications. In Biomedical Image Processing and Biomedical Visualization, 1993.
[39] Lian Yan, Robert Dodier, Michael Mozer, and Richard Wolniewicz. Optimizing classifier performance via approximation to the wilcoxon-mamme mannn-whtteyy statistic. In ICML, 2003.
[40] Gloria Zen, Bruno Lepri, Elisa Ricci, and Oswald Lanz. Space speaks: towards socially and personality aware visual surveillance. In Proceedings of the 1st ACM international workshop on Multimodal pervasive video analysis, pages 37-42, 2010.
[41] Ming-Jie Zhao, Narayanan Edakunni, Adam Pocock, and Gavin Brown. Beyond fano's inequal- ity: bounds on the optimal f-score, ber, and cost-sensitive risk and their implications. JMLR, 2013.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Sec.
7 addresses potential negative societal impacts due to data bias and the importance of building safe and responsible artificial intelligence systems.
(d) Did you describe the limitations of your work? [Yes] In Sec. 4.1 we discuss which metrics are Lipschitz continuous under our proposed method computed via soft sets. In Sec. 4.2 we state the assumptions of independent classifications, with samples drawn from Binomial and Beta distributions for F1 and Fý, respectively. In Sec. we discuss limitations due data bias.
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] As mentioned in com nathantsoi /btg (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. Of the paper and the Supplementary Material.
the paper, the source code is available at bectps:/ggithi
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the mean and standard deviations in our results. We run each experiment for 10 trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the Supplementary Material, Sec.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] We cite code for binary classification methods we compare against. For example, upon personal communication with authors of the Fathony paper [14], we received PyTorch implementation and suggested hyperparameters.
(b) Did you mention the license of the assets? [Yes] All assets are open source.
(c) Did you include any new assets either in the supplemental material or as URL? [Ys]] As mentioned in the paper, the source code available hitps: : //gi thub com/ nathantsoi/btg
(d) Did you discuss whether and how consent was obtained from people whose data you're 'ou're using/curating? [N/A] We use publicly available datasets. We describe datasets and provide links in the Supplementary Material.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We use classification datasets with no PII or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Did not conduct research with human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Did not conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Did not conduct research with human subjects.
14


Reviews:
[{"review_id": "u_hKUtEUs3", "reviewer": "Reviewer_Mx4J", "summary": "The paper proposes a method to directly optimize evaluating metrics, such as F1 and AUROC, at training time. The method includes a piecewise linear approximation of the 0-1 loss (heaviside function), and soft versions of F1 and AUROC metrics based on the diffierentiable approximation. Experiments show that the proposes method can achieve superior F1 and AUROC compared to models trained with binary cross entropy loss. ", "questions": "1. Based on my understanding, I think the technical part of proposed method includes a piecewise linear approximation of the heaviside function, and approximations of confusion-matrix metrics based on soft sets. I'd like to see an ablation study where these two metrics are evaluated separately. For example, if the heaviside function is approximated with sigmoid function, does the proposed \"accuracy\" metric coincides with BCE? \n\n2. Sec. 4.2 presents a consistency result. Isn't that also true for the BCE loss, at least for accuracy? I think to firmly demonstrate the advantage of the proposed method, consistency alone might not be enough, and one might want to consider the statistical (sample) efficiency?", "limitations": "Limitations an societal impact are sufficiently addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- Binary classification is an important and frequent problem. \n- The proposed method can have wide application potential.\n- The proposed method is simple and effective.\n\nWeaknesses\n- The novelty might be somewhat thin: approximating non-diffierentiable functions with differentiable ones are common. The theoretical grounding section might not be too superising (continuity, consistency). Proposing to directly optimize evaluating metrics at training time might be novel, but I think it is also studied, as those mentioned in related works and are compared against.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vocby5Ynvyl", "reviewer": "Reviewer_x14X", "summary": "This work focuses on the loss function of the binary classification problem and proposes the use of the differential Heaviside Function Approximation $\\mathcal{H}$ and soft set membership as a loss to directly optimize desired confusion matrix-based metrics such as accuracy, F1-score, etc., instead of the commonly used surrogate objective BCE. The study focuses on the analysis of a sigmoid function and a five-point linearly interpolated function as an approximation, with some assumptions providing theoretical underpinning. Experiments are conducted on several unbalanced level datasets, and the results are consistently good or comparable in some scenarios.", "questions": "Results are observed when the evaluation measure is F1 and the loss is also F1*, however practically all studies on accuracy are comparable to BCE (tabular datasets) or the same improvement is reached using F1* and Accuracy* (image datasets). Also, it appears that F1* performs better on numerous evaluation metrics; does this contradict what the theory predicts?", "limitations": "Did not conduct research with human subjects", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The strategy suggested in this article provides a number of advantages over earlier attempts to directly improve measures. It alleviates the batch size limitation and being faster for the adversarial approach [11] (cubic runtime complexity), and it supports a wider variety of metrics as opposed to WMW [33]. \n\nSpecifically, the method can optimize metrics such as $F_1$, $F_\\beta$, Accuracy, AUROC, etc. directly and balance the tradeoff of Precision and Recall, without increasing the training time complexity (linear). \n\nThe authors provide a theoretical foundation for this method and empirically validate it using a variety of datasets with varying levels of imbalance. The linear approximation function was introduced in [27] as an activation function, but this work seems the first to use it in the training objective.\n\n\nWeaknesses: The experimental results show that only some of the metrics have a significant improvement, while others do not. For example, AUROC has no unbiased estimator which leads to bad performance. This approach seems less directly applicable to multiclassification tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GeR3HJw0fCZ", "reviewer": "Reviewer_MHi1", "summary": "In binary NN classifiers, we train the models on BCE objective but at the test time, we evaluate the model's accuracy or AUROC instead. The authors propose a differentiable approximation to the test time metrics (like Accuracy/AUROC/F1 score) so that we can use them for training the model as well. The authors evaluate the new loss function on 5 datasets with varying levels of class imbalance and show that the models trained with proposed loss have better generalization.  ", "questions": "The following are my major concerns, it would be great to discuss this \n-\tHow expensive is the method? In terms of compute and time efficiency.\n-\tHow robust is the model to slight OOD perturbations? \n-\tIs this method scalable to large datasets? \n", "limitations": "n/a", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The introduction of a new optimization objective.\n- The investigations seem thorough.\n\nWeaknesses:\n- Did not justify enough why this new optimization objective is useful. I understand this improves the metrics we are interested in measuring but are there any other benefits.\n- The paper is hard to follow in some places.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5YQQD33Tf6f", "reviewer": "Reviewer_7RmE", "summary": "This paper proposes to use a differentiable approximation of the Heaviside step function (which determines where the prediction should be positive according to the input and threshold) to build a loss function that theoretically approximates Accuracy or F-score.  The motivation is to bridge the gap between training loss and evaluation metrics in binary classification.\n\nIn terms of the method, it has two parts. The first is to approximate the Heaviside function with a sigmoid or piecewise linear function with hyperparameter tau. Then this paper introduces the soft set membership to measure the degree of belonging to a certain confusion matrix set. In terms of theoretical support, the author proves that the new approximation of the Heaviside function is Lipschitz continuous and the loss constructed by their combination also makes the variation of each step in SGD optimization small. Then the paper proves that the loss that approximates the F1 metric converges to the F1 metric when the number of samples is infinite. Finally, in the experimental part, the paper experiments on tabular and image data and also explores the balance of precision and recall. The approximated loss of F1 score achieves higher performance than BCE loss and other losses.\n", "questions": "1. I suggest that this paper can give a figure that pairs and compare the original version and soft version and highlight the difference. Formula 2 and Formula 6 look identical (the difference is the small s).\n\n2. Line 124, “By [11]”, I recommend not making numbers the subject of a quote.\n\n3. What are the challenges of extending this research to multiclass classification?\n\n4. I suggest that this article can also experiment on more kinds of data, such as binary classification on graph data.\n", "limitations": "The paper discusses limitations scattered throughout the paper. Besides the limitation summarized in checklist 1 (d), they also acknowledged other limitations.\n\nIn Line 119, the paper acknowledges that the mini-batch stochastic gradient descent (SGD) optimization method does not provide an unbiased estimator on non-decomposable metrics.\n\nIn Line 202, the paper acknowledges that the F1-Score computed from soft sets is a biased estimator for the expected true F1-Score for finite n.\n\nIn Line 249-254, they find that the loss that approximates AUROC does not work.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n1. The paper is novel and compares some related methods in related work and experiments.\n2. This is a completed work, with both theoretical support and experimental results, and the authors honestly point out various deficiencies.\n3. The presentation of this article is very good; the overall structure and specific details are relatively clear.\n4. This paper provides a training loss that approximates the evaluation metric. The difference between training and evaluation is a problem of great interest.\n\nWeakness：\n\n1. Although this paper has compared adversarial-based methods in experiments, comparisons with other differentiable surrogate losses are lacking.\n\n2. This paper can compare previous methods and the method in this paper for the difference in approximation error.  Previous methods include BCE loss and the previous methods to solve the difference between training loss and test indicators.\n\n3. What kind of theoretical problems will the gap between the two-class training loss and test indicators bring about; this paper lacks the analysis in this regard.\n\n4. There are some differences between the test sample and the training distribution. The closeness of the loss at training time to the metric at test time does not seem to solve this problem, and this article does not consider generalization error.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept