EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification
Authors: Ben Dai
Venue: ICML 2025 poster (2025)
Keywords: classification-calibration
ensemble learning
statistical consistency
surrogate loss
stochastic regularization
Primary area: general_machine_learning->supervised_learning
TL;DR: We introduce EnsLoss, a novel ensemble method combining loss functions within the empirical risk minimization framework, preserving consistency and calibration of combined losses.

Abstract: Empirical risk minimization (ERM) with a computationally feasible surrogate loss is a widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maximizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we propose a novel ensemble method, namely *EnsLoss*, which extends the ensemble learning concept to combine loss functions within the ERM framework. A key feature of our method is the consideration on preserving the "legitimacy" of the combined losses, i.e., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, *EnsLoss* enables loss ensembles through one training process with doubly stochastic gradient descent (i.e., random batch samples and random calibrated loss-derivatives). We theoretically establish the statistical consistency of our approach and provide insights into its benefits.  The numerical effectiveness of *EnsLoss* compared to fixed loss methods is demonstrated through experiments on a broad range of 45 pairs of CIFAR10 datasets, the PCam image dataset, and 14 OpenML tabular datasets and  with various deep learning architectures. Python repository and source code are available on our Github (https://github.com/statmlben/ensLoss).

ENsLOSS:
Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification
Ben Dai
Abstract
produces predicted class sign(](xx) to predict the true class Y [-1,++1. The performance of the classification
Empirical risk minimization (ERM) with a com- putationally feasible surrogate loss is widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maxi- mizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we pro- pose a novel ensemble method, namely ENSLOSS, which extends the ensemble learning concept combine loss functions within the ERM frame- work. A key feature of our method is the con- sideration on preserving the "legitimacy" of the combined losses, i.., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, ENSLOSS enables loss en- sembles through one training process with doubly stochastic gradient descent (i.c., random batch samples and random calibrated doss-derivatives). We theoretically establish the statistical consis- tency of our approach and provide insights into its benefits. The numerical effectiveness of EN- sLOSS compared to fixed loss methods is demon- strated through experiments on a broad range of 45 pairs of CIFAR1O datasets, the PCam image dataset, and 14 OpenML tabular datasets and with various deep learning architectures. Python repos- itory and source code are available on GITHUB.
function f is typically evaluated using the risk function based on the zero-one loss:
R() = ((I 5)),
(1)
where 1(-) is an indicator function, and the classification accuracy defined as Acc(f)
R(f). Clearly, the
Bayes decision rule [*) sgn 1|X x) - 1/2) is minimizer of the risk function R(f). Due to the discon- tinuity of the indicator function, the zero-one loss is usually replaced by a convex and classification-caliraaee loss & to facilitate the empirical computation (Lin, 2004; Zhang, 2004b; Bartlett et al., 2006). For example, typical losses including the hinge loss d(z) = (1 =)+ for SVMs (Cortes & Vapnik, 1995), the exponential loss <<2) exp(-z) for AdaBoost (Freund & Schapire, 1995; Hastic al., 2009), and the logistic loss 6(2) log(1 + exp(-=)) for logistic regression (Cox, 1958), and others (Lin et al., 2017; Leng et al., 2022). Then, the risk based on x(-) is defined as:
R&( = (((
Note that convexity and classificationcaaibraane (hereafter referred to as calibration for simplicity) are widely accepted requirements for loss function o(2). The primary motiva- tions behind these requirements are that convexity facilitates computations, while calibration ensures the statistical con- sistency of the empirical estimator derived from Rs, as formally defined below.
Definition 1.1 (Bartlett et al. (2006)). A loss function Φ(-) clasification-calibbaaee if for every sequence of measurable function fr, and every probability distribution X' x {=1),
1. Introduction
Rs (f.) + inf R&( implies that R(fx) + inf R(f),
The objective of binary classification to categorize each instance into one of two classes. Given a feature vector X E X' Idd classification function ʃ Rd IR
approaches infinity.
According to Definition 1.1, calibrated loss function guarantees that any sequence Sn that optimizes R2 will eventually also optimize R, thereby ensuring consistency in maximizing classification accuracy. To achieve this, the most commonly used and direct approach is ERM, which directly minimizes the empirical version of RS to obtain
'Department of Statisties, The Chinese University Hong Kong. Correspondence to: Ben Dai <bendai cuhk. cuhk.edu.hk>> hku.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
I. Specifically, given a training dataset (x,, the d-classification framework is formulated as:
article is to also propose novel regularization method EN- sloss, which differs from existing regularization methods, or rather, regularizes the model from a different perspective.
Jn = argmin Ro(s), Ra( = _ Σ (xx((x))), (2) FEF
The primary motivation for ENSLOSS stems from ensemble learning, but it specifically focuses the perspective of loss functions, applying the ensemble concept to combine various "valid" losses. As mentioned previously, numerous CC losses can act as a valid surrogate loss in (2), yielding favorable statistical properties in terms of the zero- one loss in (1). Yet, pinpointing the optimal surrogate loss in practical scenarios remains a challenge. A potentially effective idea is loss ensembles, by implementing an ensem- ble of classification functions fitted from various valid loss funetions. However, for large models, particularly those involving deep learning, the computational cost associated with multiple training sessions can often be prohibitively expensive. It is worth mentioning that a similar computation challenge is also prevalent with model ensembles or model combination. This issue, has been effectively addressed by Dropout (Srivastava et al., 2014): by randomly taking dif- ferent network structures during each SGD update, thereby achieving the outcome akin to model ensembles. In our content, we employ a loosely analogous of Dropout, adopt different surrogate losses in each SGD update to achieve the objective of loss ensembles. This motivating idea behind ENSLOSS is roughly outlined in Table 1.
Our motivation.
where F F= (fo E } is a candidate class of classi- fication functions. For instance, F can be specified as, Reproducing kemee Hilbert space (Aronszajn, 1950; Wahba, 2003), neural networks, or deep learning (DL. models (Le- Cun et al., 2015). Notably, most successful classification methods fall within the ERM framework of (2), utilizing various loss functions and functional spaces.
Given a functional space F. the training process of ERM in (2) focus on optimizing the parameters f within O Stochas- tie gradient descent (SGD; (Bottou, 1998; LeCun et al., 2002)) is widely adopted for its scalability and generaliza- tion when dealing with large-scale datasets and DL models. Specifically, in the i-th iteration, SGD randomly selects one or a batch of samples (xi,. "4i.:Vi,55 Vi.)b= with the index set IB, and subsequently updates the model parameter as:
1 Σ
0(++1)= = (+)) -? BBB
VerlviSow (xi))
VEIB
0
1 ?
Σ
y:0p(y (x:)) (x;)). (3)
B
iEIs
2. Related Works
where 7 > 0 represents a learning rate in SGD, the second equality follows from the chain rule, and (x) can be explicitly computed when the form of fe or F is specified.
This section provides literature review of related works on regularization methods for mitigating overfitting, well as related ML literature focused on the loss function.
The ERM paradigm (2) with calibrated losses, when com- bined with ML. models and optimized using SGD, has achieved tremendous success in numerous real-world appli- cations. Notably, with deep neural networks, it has become a cornerstone of supervised classification in modern datasets (Goodfellow et al., 6; Krizhevsky et al., 2012; He et al., 2016; Vaswani, 2017).
Dropout. One simple yet highly effective method for pre- venting overfitting is dropout (Srivastava et al., 2014). The key advantage of Dropout lies its ability to simulate an ensemble approach during SGD updates, thereby mitigating overfitting without substantial computational costs. Thus, Dropout has become a standard component in many DL ar- chitectures, and its advantages have been widely recognized. Notably, the direction of ensemble in Dropout is achieved through different model architectures, whereas our method achieves ensemble through different "valid" loss functions. Thus, the proposed method and Dropout exhibit a comple- mentary relationship and can be used simultaneously, as implemented in Section 4.4.
Overfitting. The overparameterized nature of deep neural networks is often necessary to capture complex patterns and various datasets in the real world, thereby achieving state-of-the-ar performance. However, one of the most per- vasive challenges in DL models is the problem of overfitting, where a model becomes overly specialized to the training data and struggles to generalize well to testing datasets. Par- ticularly, DL models can even lead to a phenomenon where they perfectly fit the training data, achieving nearly zero training error, but typically, with significant gap often persisting between the training (close to zero) and testing errors, discrepancy attributable to overfitting. Given this fact, many regularization methods (c.f. Section 2) have been proposed, achieving remarkable improvements in alleviating overfitting in overparameterized models. The purpose of this
Penalization methods. Another approach to mitigate over- fitting is to impose penalties or constraints (such as a / norm) on model parameters, which aims to reduce model complexity and thus prevent overfitting (Hoerl & Kennard, 1970; Tibshirani, 1996; Zou & Hastie, 2005). Similar ap- proaches include weight decay during SGD (Loshchilov & Hutter, 2017). The underlying intuition is to strike balance
2

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
between model complexity and data fitting, thereby miti- gating overfitting through the bias-variance tradeoff. Thus, these methods can also be seamlessly integrated with the proposed method, as demonstrated in Section 4.4.
Table I. Stochastic calihrated loss ensembles under SGD. Upper: A standard SGD updates (based on a fixed surrogate loss). Lower: Informal outline for the proposed loss ensembles method (please refer to Algorithm for the formal details).
Clasffication-calirrtion Note that the zero-one loss (or accuracy) cannot be directly optimized due to its discon- tinuous nature, and thus, surrogate loss introduced to facilitate the computation. A natural question that arises is: how can we ensure that the classifier obtained under the new loss performs well in terms of accuracy? The answer leads to the defimition of cesilficcationccaibrattte for loss function (Definition 1.1). Meanwhile, a series of works (Lin, 2004; Zhang, 2004b; Lugosi& & Vayatis, 2004; Bartlett et al., 2006) have finally summarized loss calibration to a simple if- and-only-if condition, as stated in Theorem 3.1. Calibration is an extensively validated condition through both empirical and theoretical consideration, and is widely regarded as necessary minimal condition for loss function.
SGD + Fixed Loss
For each iteration:
batch sampling from a training set; implement SGD on batch samples and a fixed sur- rogate loss.
SGD + Ensemble Loss (ENSLOSS; our)
For each iteration:
batch sampling from a training set;
randomly generate a new "valid" surrogate loss; implement SGD on batch samples and the generated surrogate loss.
Post loss ensembles. A straightforward approach to con- structing loss ensembles is to fit separate classifiers for each calibrated losses (e.g.. SVM and logistic regression) and then combining their outputs using simple ensemble meth- ods, such as bagging, stacking, voting (Breiman, 1996; Wolpert, 1992). This approach has empirically achieved satisfactory performance; however, its major drawback is that it requires refitting a classifier for each loss, resulting in substantial computational costs that make it impractical for large complex models.
the notable advantages of our proposed method over fixed losses. The experimental results in Figure/ 1 and Section 4 are highly promising, suggesting that ENsLOss has the potential to significantly improve the performance of the fixed loss framework, with this improvement exhibiting a universal nature across epochs and various network mod- cls. The detailed setup and comprehensive evaluation of our method's empirical performance, along with related exploratory experiments, can be found in Section 4.
Loss Meta-learn. A recently related topic is the learning of loss functions via the meta-learn framework in multiple-task learning or domain adaptation (Gonzalez & Miikkulainen, 2021; Bechtle et al., 2021; Gao et al., 2022; Raymond, 2024). These methods primarily employ a two-step approach: first, learning loss function from source datasets/tasks via bilevel optimization under AgnssiAg Meta- Detaa-ee (Finn et al., 2017), and then applying the learned loss to tra- ditional ERM in the target tasks/datasets. While they share some similarities with our approach in relaxing the fixed loss in ERM, they are mainly applied in transfer learning and typically require additional source tasks or datasets to learn the loss, differing from our setting and objectives.
Based on the empirical evidence of its promising perfor- mance, we are now prepared to discuss the proposed loss ensembles method in detail. Certainly, the generation of surrogate loss functions is not arbitrary: it must still satisfy the requirements for consistency or calibration (Lin, 2004; Zhang, 2004b; Bartlett et al., 2006). Furthermore, the im- pact of the loss function under SGD is solely reflected in the loss des-vveriv,,vve, as discussed in subsequent sections. Thus, our primary focus is on the development of conditions of "valid" losses or loss-derivatives, as well as how to randomly produce them.
3.1. Calibrated loss-derivative
In this section, we aim to list the conditions for a valid loss and transform them into loss-derivative conditions, therehy facilitating the usage in SGD training; ultimately, these con- ditions directly inspire the implementation of our algorithm, with the overall motivation illustrated in Figure 2.
3. Calibrated Loss Ensembles
As previously mentioned in the introduction, the motiva- tion behind ENsLosS lies in incorporating different loss functions into SGD updates. The proposed method can be delineated roughly within an informal outline in Table (refer to Algorithm for the formal details).
As indicated in the literature (c.f. Section 2), the well- accepted conditions for a valid surrogate loss Φ are: (i) convexity; (ii) calibration. The key observation of SGD in (3) is that the impact of the loss function & on SGD or other (sub) suu)graddient-aar based algorithms is solely reflected in its loss- derivative dΦ. Interestingly, the convexity and calibration
Key empirical results. Before delving into the technical de- tails of our method, we hegin by providing representative tetpchh----ttttcccraaae curve (Figure 1), to underscore
3

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
MiDe Nosteker
model SeeNer24
moSe VGG15
1.00
2.75
2.70
ICCLo
Hege
3.55
25
50
75
300 125
250
375
200
25
30 75
203
325
150
375
200
25
20 75
100
125
350
175
230
spoch
op407
Figure 1. Coomparison epoch-vs proch-vs-test.aete accuurayyy curves for various models on CIFAR2 (cat-dog) dataset using ENSLOSS (ours) and other fixed losses (logistic. hinge, and exponential losses). The training accuracy curves are omitted, as they have largely stabilized at I after few epochs. The pattemn shown the figure, where ENsLOss consistently outperforms the fixed losses across epochs, is a phenomenon observed in almost all CIFAR1O 10 lahel-pairs and the PCam dataset, and with different scales of ResNet, MobileNet, and VGG architectures.
aΦ o nondecreasing ('(0) 0 Superlinear raising-tail
ALGO Lines 8-11 (sorted negative r.v.s. as 24)
Corollary D.I. Secondly, although some unbounded below losses are proved calibrated in certain specific data distri- butions, yet they may introduce instability in the training process when using SGD in our numerical experiments, see more discussion in Appendix D. Therefore, we impose the bounded below condition for loss s, or a form of regularity condition on the loss- poss-gradient 0Φ, see Lemma 3.2.
Convex
Calibration
Bounded
Lines 12-13
below
(rescale loss derivalivee)
Figure 2. The overall motivation behind generating "valid" loss- derivatives in our algorithm: first transform the loss conditions (left) into loss-derivative (middle), thereby bypassing the loss and directly generating loss-derivatives in SGD algorithms (right).
Lemma 3.2. Let o is convex and calibrated. If there exists continuous function g(z) > 0 such that S g(z)dz con- verges, and ( is nondecreasing when = 2 20 for some 20 > 0. Then, @ is bounded below.
conditions for & can also be transformed to aó: (i) convex- ity can be ensured by stipulating that its loss-derivative is non-decreasing, and (ii) a series of literature (Lin, 2004; Zhang, 2004b; Lugosi & Vayatis, 2004; Bartlett et al., 2006) is finally summarized in the subsequent theorem, which provides a necessary and sufficient condition for calibration.
Lemma 3.2 offers an implementation to translate the bounded below condition of loss functions into require- ments on the loss-derivatives. Without loss of generality, we set 2g in the subsequent discussion. This is analogous to the cut-off point in the hinge loss, which can be nullified through scaling fx) and does not significantly affect per- formance. Additionally, g(z) can be chosen as p-integrals, i.e. g(=) = 1/2" for p > 1. Naturally, a smaller value of p provides more flexibility to dø. Figure 3 illustrates the rights tails of derivveervsssves for some widely-used loss functions. Intuitively, Lemma 3.2 essentially indicates that the right tail of a valid loss-derivative needs to rise rapidly from d' (0) < 0 towards zero, either surpassing zero (as in the case of squared loss), or vanishing faster than 1/2 when z is large (ignoring the logarithm). We refer to this condition in Lemma 3.2 as a superlinear raising-taile loss-derivative.
Theorem 3.1 CCaasification-caliraadd
(Bartlett et al., 2006)). Let de be convex. Then & is classfication-caliiraedd if and only if is differentiable at 0 and d() < O.
Theorem 3.1 effectively transfers the properties of con- vex calibration from @ to its loss-derivative 0ø, offering a straightforward and convenient approach to validate, design and implement a convex calibrated loss or loss-derivative under SGD implementation.
3.2. Superlinear raising-tail
We now present all the conditions for the loss-derivative of bounded below convex calibrated loss. Convexity implies that the loss-derivative is nondecreasing, while calibration requires the loss to be differentiable at O with s' p0(0) < 0. Additionally, the bounded below assumption yields that the derivveriv exhibits a superlinear raising-tail. We re- fer this particular form of derivveeisss as superlinear raising-tailed calibrated (RC) derivvvrisss.ves.. The follow-
Notably, there one condition that could be easily over- looked yet remains crucial: the surrogate loss function ø must be nonnegative or bounded below (since we can al- ways add a constant to make it nonnegative loss, without affecting the optimization). Its importance lies in two-folds. Firstly, it directly influences calibration: the bounded be- low condition is necessary condition of calibration, see
4

---

ENSLOss: Stochastic Calibrated Loss Ensembles
Algorithm (Minibatch) Calibrated ensemble SGD.
2.0
name
exp
1: Input: a train set D = (Ti, ys) a minibatch size B;
1.5
hinge
ogisttr
2: Initialize 0.
square
for number of epoches do
1.0
/+ Minibatch sampling +/
5: Sample minibatch from DD without replacement:
0.5
B = {(xx Mi), ". (Xig: Mig
6: Compute z = (21 .ZB)T, where 26 = fo
0.0
for b - 1, 1..., B.
7: /* Generate random RC loss-derivs */
-0.5
1/2
/x calibration and convexity */
-1.0
9:
Generate g = (9I,". where db -5,
1.0
1.2 1.4 1.6 1.8 2.0
where & is a positive random variable (accomplished through Algorithm 2)
Figure 3. Plot of loss-derivatives of existing losses when z >> Conclusion. Lemma 3.2 essentially indicates that the right tail of the loss derivatevss needs to rise rapidly from 6'(0) < towards zero, either surpassing zero (as in the case of squared loss) or vanishing faster than 1/2 when = is large (ignoring the logarithm).
10:
Sort z and g decreasingly, that is
(I(1) > > BB( 9r(1) >.. -...))
11:
(the derivative corresponding to 26 is (())))
12:
/x bounded below */
ing lemma indicates that RC doss-eerivatives essentially correspond to bounded below convex calibrated loss.
13: For b 1....,............. B,
9o(+-1(6) + (o()- if 2b 1.
Lemma 3.3. Given set of samples (x:, yi Bi))im,.... .n. and classification function f. let = (xi), and denote g- (g1, ( as RC derivvaeis,,,ves, that is, satisfying the sollowing conditions:
14:
/x Update parameters */
15: Compute gradients and update
B
1. (Convexity) 9. S 9j 2 < 2j,, and 9: = 9j if 2zi
7 B b=1
(n Vol
2. (Calibration) g: Of 24 S 0;
3. (Superlinear) g. S 2/ if S < 2j. for p >>> 1.
16: end for
17: Return the estimated 0
Then, there exists a bounded below convex calibrated loss function 6, such that 0Φ(z.) 9: for all i =,,... B.
Lemma 3.3 sheds light upon the conditions for RC loss- derivatives (or its implicitly corresponding CC loss). Hence, it allows us to bypass the generation of loss and directly gen- crate the loss-derivatives in SGD, thereby inspires doubly stochastic gradients in Algorithm 1.
loss-derivatives by sampling from a positive random vari- able g, then sorting and rescaling. Specifically, in Algorithm 1', the lines 8 to 11 (sampling and sorting) are dedicated to satisfy Conditions 1 and 2; the line 13 (rescaling) is for Con- dition 3. This implementation approach, which builds upon stochastic gradients by adding an another level of "stochas- ticity", is thus referred to as doubly stochastic gradients.
Given that Conditions and require least two samples to demonstrate the properties, our primarily focus on imple- menting our algorithm using mini-batch SGD. For sake of simplicity in implementation, we directly choose p = 1, as the numerical difference between 2 and >P when is very close to is exceedingly tiny. Our empirical experiments also demonstrate that p = does not significantly affect the performance compared with p close yet not performing superlinear raising tail adjustment on loss doss-derivasives can significantly impact the performance.
Note that Algorithm I does not explicitly implement Con- dition 2. In fact, for simplicity, condition that g. 0 for all i 1.. B. This adapta- tion, made only for implementation simplicity. does not fundamentally alter the framework inspired by Lemma 3.3. Furthermore, the choice of the positive random variable E impacts the diversity of the random loss-derivatives. To address this, we propose Algorithm 2 to generate distribu- tion of & using the inverse Box-Cox transformation (see discussion in Appendix A).
consider sufficient
Doubly stochastic gradients. The most important impli- cation of Lemma 3.3 is that it provides a guideline for gen- erating RC doss-deriva,ives, as Conditions 1-3 are straight- forward to satisfy. For example, we can obtain a set of RC
We assume 2; A zj. otherwise we duplicate the derivative by merging and treating them as one sample.
5

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
3.3. Enhance loss diversity via the inverse Box-Cox
Algorithm 2 Inverse Box-Cox transform of loss-derivatives.
transformation
1: Input: minibatch size B, a hyperparameter À (default 0)
In addition to considering the RC regularity and stochastic generating of loss-derivatives, it is also interesting further boost the diversity of the loss functions or loss-derivatives generated from our proposed method (Algorithm 1). This could potentially improve the performance of loss ensem- bles, based on previous experience with ensemble learning (Breiman, 1996; 2001; Wood et al., 2023).
/ Generate normal grad x/
3: Generate g = (g1. .....,,,
" (B) where Ib id N()
/x Inverse Box-Cox transformation +/
9b f- -invBCx
5: Return the generated loss-derivatives (01, (0..""992) ".BB..
g
In fact, the diversity of loss functions in Algorithm is partly associated with the loss-derivatives generated from the vari- ety of positive random distributions. Consequently, a crucial consideration is to generate a sufficiently diverse range of positive random distributions. This naturally invokes the concept the Box-Cox transformation (Box Cox, 1964), which transforms (any) positive data via a power transfor- mation with hyperparameter À such that the transformed data closely approximates normal distribution.
3.4. Statistical consistency of loss ensembles
In this section, we establish theoretical framework to ana- lyze the statistical behavior and consistency of the proposed loss ensemble framework. Our idea comprises two primary steps: first, aligning the proposed method with a novel risk function; second, leveraging learning theory to evaluate the calibration and consistency of the risk function.
X -- 1),
if X + O,
BCx(E)=
To proceed, we introduce relevant definitions construct the corresponding risk function for the proposed method. Specifically, we denote L as measurable space consisting of bounded below convex calibrated (BCC) losses. A ran- dom surrogate loss is considered as a C-valued random variable, where loss á represents an observation or sample of Φb Note that in our analysis, is assumed independent of X, Y); for detailed probabilistic definitions of random vari- ables in functional spaces, refer to (Mourier, 1953; Vakhania et al., 2012). On this ground, we introduce the calibrated ensemble risk function as follows:
log(E).
if A 0.
Interestingly, the Box-Cox transformation is to transform a diverse range positive random distributions into normal distribution. This represents our exact "inverse" direction: we need to generate sufficiently diverse range of positive random distributions. Consequently, we propose the inverse Box-Cox transformation, defined as invBCx(.) in (4).
(1 1
A + 0,
(E) =
(4)
R() Ex,y (E+)(Y)())).
exp(E),
if A O.
where Eg is the expectation taken with respect to T. In this content, given a classification function fe. with mini- batch data (x,,)) se and the sampled loss o, the stochastic gradient of R in SGD is defined as:
On this ground, the final loss-derivatives can be generated in this manner, see Algorithm 2. First, we generate derivatives from a standard normal distribution, then using the inverse Box-Cox transformation (4) (with a random X) transforms them into arbitrary positive random distribution. This guarantees variety in the loss function during the ensemble process over SGD. Here, the randomness of À is used to control a diverse range of -oss-derivatives.
1
g
(od(v.f((x))).
(6)
IB
EEIR
Thus, the proposed method (Algorithm 1) can be regarded as the SGD updating based on g, and R(-)) is an appropriate risk function for characterizing the proposed method.
In our numerical experiments, we fix = 0 (no hyperpa- rameter tuning) for a fair comparison, which corresponds to an exponential transformation of normally distributed random variables. In practice, to further enhance loss di- versity,
Next, we discuss the loss space L. To ensure the calibration of the proposed method, we require that a measurable subspace of the collection of all BCC losses:
implement random sampling of X every T epochs. This approach presents beneficial in certain cases, as demonstrated in our ablation studies presented in Ta- ble 6. However, determining a reasonable value of T a straightforward manner likely requires further investigation future research.
L {Ø is convex inf ((2) -0, o'(0) < 0}.
Assumption 3.4. Let (2, be a probability space and be measurable space. Suppose t S - is a L-valued random variable satisfies following conditions.
6

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
1. (+(=) exists and (=) oo for any given E R.
(Moulines & Bach, 2011; Shamir & Zhang, 2013; Fehrman et al., 2020; Garrigos & Gower, 2023). Given that the stochastic gradient g in (6) for the proposed method offers an unbiased estimate of V. (fo), many existing SGD con- vergence results can be extended to our ensemble setting, ensuring that R(fn.) - i R(f).
2. There exists random variable U f -> IR such that (=) 2 U a.s. for all and EU > -00;
3. There exist a random variable G : Q + R and a constant do > O, such that (0)( < G a.s. for any 8 E -og 6o] and EG < co.
By combining the convergence results from SGD and the calibration established in Theorem 3.6, we demonstrate that the proposed loss ensemble preserves statistical consistency for classification accuracy. We illustrate the usage of the theorem by a toy example in Example C.I of Appendix C.
Assumption 3.4 characterizes the feasibility of the loss space L and probability measure μs, thereby establishing the scope of applicability for our proposed method. Notably, a finite loss space automatically satisfies Assumption 3.4.
Lemma 3.5. If the loss space & is finite, then Assumption 3.4 is automatically satisfied.
The proposed method appears to have an analogous effect to bagging (Bühlmann & Yu, 2002; Soloff et al., 2024). We next provide theoretical insights into a natural question: what advantages do loss ensembles offer over a fixed loss approach? We partially address this question by examining the Rademacher complexity. Specifically, given a classi- fication function space F, the Rademacher complexity of @-classification are defined as follows:
Indeed, & can be extended to a more general functional space, subject the mild uniform assumptions in Assump- tion 3.4, which are necessary to preserve the completeness of BCC properties. For example, limiting over L without uniform assumptions may violate the calibration condition. In practice, during SGD training, the number of epochs is typically fixed, which implies that the number of ensemble losses implemented is also fixed. As a result, our analy- sis of the proposed method in practical scenarios can be exclusively focused on finite L case.
1
Rad = sup FEF
T (Y)((X))))
Now, we have aligned the proposed method with the pro- posed ensemble risk (5). Hence, we can infer the statistical behavior of our method by analyzing R. Of primary require- ment is the calibration or cosher-consisteny,, as formally stated in the following theorem.
where (TT) are i.i.d. Rademacher random variables in- dependent of (X;, Y:) The Rademacher complexity plays a crucial role in many existing concentration inequali- ties (Talagrand, 1996a;;; Bousquet, 2002), determining the convergence rate of the excess risk (Bartlett & Mendelson, 2002) (with smaller values yielding faster rate). On this ground, the corresponding Rademacher complexity for the proposed ensemble loss method can be formulated as:
Theorem 3.6 (Calibration). Suppose Assumption 3.4 holds, and R(.) is defined as in (5r for any distributions Px,y and Pa. then for every sequence of measurable function fn.
iin R(f) implies that R(fx) + inf R(f). R(fa) + inf
Rad(F) = sup 1 IEF n
f
TEE + (Y.S(X)))
(7)
Moreover, the excess risk bound is provided as:
S Ex (R)d
(9)
R(}) - inf R(J) s W- inf R(f)),
(8)
where the inequality follows from the Jensen's 'ss inequality. This simple deduction yields positive result, that is, the Rademacher complexity of the ensemble loss is no worse than the average based on the set of fixed losses. Yet, (9) only partially showcases potential benefits of loss ensemble, but it does not provide conclusive evidence of its superiority over fixed losses, as a comparison of their excess risk bounds is also crucial. In fact, ensemble loss appears suboptimal in terms of distribution-fii excess risk bounds. Furthermore, achieving definitive and practical conclusions across specific datasets or distributions remains longstanding challenge for statistical analysis. The development of more effective combining weights, as in ensemble learning (Yang, 2004; Audibert, 2004; Dalalyan & Tsybakov, 2007), may provide a promising solution for future research.
vh. where 4 is the inverse fanction the and 1/2 is defined as:
inf EE( 1 ((a) 1 1- 0
(0)) = ((((((())))
2 (a)).))
ocR
2
Theorem 3.6 ensures the classification clasiffiationncalibaatiin of the proposed method, indicating that minimizing the ensem- ble calibrated risk R would provide a reasonable surrogate for minimizing R(f)) Furthermore, the excess risk bound is also provided in (8), which enables presenting the re- lationship between R(f)) - R" and R())- - R² when the distribution of & is given.
In addition, there is a substantial amount of literature that discusses the convergence results based on (batch) SGD
7

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
4. Experiments
dence suggesting that A outperforms B. Pairwise hypothesis tests are performed for each pair of methods. If method ex- hibits statistical significance compared to all other methods, it will be highlighted in bold font in the tables. Our primary focus is on accuracy, but we also provide AUC results in our accompanying W&B projects and GitHub repository.
This section presents experiments comparing ENSLOSS with fixed loss methods and assessing its compatibility with regularization methods. All Python codes is openly acces- sible at our GITHUB. All experimental results, up to the epoch level, are publicly available on our W&B projects ENSLOSS-IME and ENSLOSS- enabling transparent and detailed tracking and analysis.
4.3. ENSLOSS vs fixed loss methods
This section presents our experimental results, wherein we examine the numerical performance of ENSLOSS and com- pare it with other fixed loss methods across various setups.
4.1. Datasets, models and losses
Image datasets. We present the empirical results for image benchmark datasets: the CIFARIO (Krizhevsky et al., 2009) and the PatchCamelyon (PCam; (Veeling et al., 2018)). CI- FARIO was originally designed for multiclass image classifi- cation. In our study, we construct 45 binary CIFAR datasets, denoted as CIFAR2, by selecting all possible pairs of two classes from CIFARIO, which enables the evaluation of our method. PCam is a binary image classification dataset com- prising 327,680 96x96 images from histopathologic scans of lymph node sections, each annotated with a label indicating the presence or absence of metastatic tissue. CIFAR and PCam datasets are widely recognized benchmarks in image classification, frequently employed in various studies, such as (He al., 2016; Huang et al., 2017; Srinidhi et al., 2021).
Design. The experiment design is straightforward: we com- pare various methods on 46 image datasets (including 45 CIFAR2 datasets and the PCam dataset) and 14 OpenML tabular datasets and using different network architectures. The implementation settings for each method are identical, with the only difference in loss functions.
Results. Due to space limitations, we only present the summary statistics of significant testing results across all datasets in Table 2 (for 14 tabular datasets) and Table 3 (for 45 CIFAR2 datasets). Moreover, some highlighting detailed empirical results are provided in Appendix B, and all performance metrics for all experiments are publicly accessible at the epoch level via our W&B projects.
Tabular datasets. We applied filtering (n 2 1000, d 2 1000) across all OpenML (Vanschoren et al., 2014) binary classification dense datasets, resulting 14 datasets: Biore- sponse, guillermo, riccardo, christine, hiva-agnostic, and 9 OVA datasets: Breast, Uterus, Ovary, Kidney, Lung, Omen- tum, Colon, Endometrium, and Prostate.
Conelusion. The key findings are summarized as follows.
Image data. Tables 9 10 and Figure 4 demonstrate that ENSLOSS consistently outperforms existing fixed loss meth- ods. (i) The improvement is universal across experiments. As shown in Table 9, ENSLOSS achieves non-inferior per- formance in all 45 CIFAR2 datasets, and significantly out- performs ALL other methods in at least 60% of the datasets. (ii) The improvement is remarkable, with substantial gains of 3.84% and 3.79% observed in CIFAR2 (cat-dog) and PCam, respectively, surpassing the best fixed loss method paired with the optimal network architecture.
Models. To assess the effectiveness of the proposed method, we explore a range of commonly used neural network struc- tures, including Multilayer Perceptrons (MLPs; (Hinton, 1990)) with varying depths for tabular data, as well as VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), and MobileNet (Sandler et al., 2018) for image data.
Tabular data. Table 11 reveals that: (i) When dealing with overparameterized models, ENSLOSS tends to be a more desirable option compared to fixed losses, whereas for less complex models, ENSLoSS may underperform outper- form certain datasets compared to the optimal fixed loss, yet it remains viable alternative worth considering overall. (ii) The effectiveness of ENSLOSS exhibits a clear upward trend as model complexity increases, as evident from the performance comparison from MLP(1) to MLP(5).
Fixed losses. ENSLOSS is benchmarked against with the ERM framework (2) using three widely adopted fixed classi- fication losses: the logistic loss (BCE; binary cross entropy), the hinge loss (HINGE), and the exponential loss (ExP).
4.2. Evaluation
All experiments are replicated 5 (for image data) or 10 (for tabular data) times, and the resulting accuracy values are reported. Moreover, to evaluate the statistical significance, p-values are calculated using a one-tailed paired sample I-test, with the null and alternative hypotheses defined as:
The superiority of ENsLOSS is more pronounced in image data than in tabular data, likely attributable to the increased risk of overfitting associated in high-dimensional inputs and complex models characteristic of image datasets.
Ho: Acc I Aceg, H1 Acc > Aceg,
(10)
where Acc and Aceg are accuracies provided by two com- pared methods. A p-value of I 0.05 indicates strong evi-
The improvement is also prominently reflected at the epoch level, particularly after the training accuracy for the pro-
8

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
posed ENSLOSS reaches or approaches one, as suggested by the epoch-vs-test accuracy curves in Figure (and those for all experiments in our W&B projects). This is crucial for practitioners: by specifying relatively large number of epochs, ENSLOSS is promising choice compared to fixed loss methods; furthermore, its training accuracy can some- times serve as a key indicator for early-stopping, obviating the need of validation set.
Table 4. The averaged accuracy (with AUC included in our Github repository, exhibiting similar patterns) and their standard errors (in parentheses) for all methods with various regularization on the CIFAR2 (cat-dog) image dataset are presented. "HP" indicates the corresponding hyperparameler for each regularization method.
REG
BCE
EXP
HINGE
ENsLoss
REG:
baseline
67.99(0.30)
60.09(0.19)
68.19(0.40
69.52(1.38)
WEIGHTD
Se-5
67.64(0,14)
60.43(0.23)
71.01(1.04) 68.26(0.65)
Table 2. The summary statisties of datasets exhibiting statistical significance when comparing the proposed ENSLOSS against all other fixed loss methods in CIFAR2 binary classification datasets (provided by pairwise labels subset of CIFARIO) are pre- sented. The significance of "better", "no diff", and "worse" are suggested by the significance test, as described in Section 4.2.
Se-4
67.59(0.35)
61.57(0.56)
72.14(0.355 67.57(0.28)
5o-3
68,00(0.31)
62.26(0.45)
70.84(0.67) 68.26(0.35)
DROPOUT
0.1
67, 67,50(0,39)
60.70(0.34)
67.89(0.30)
72.48(0.22)
68. 68.13(0,54)
60.02(0.52)
67.78(0.44)
70.08(1.28)
67.65(0.29)
59.70(0.46)
67.78(0.49)
72.44(0.68)
DATAAUG
79.22(0.12)
58.96(0.31)
80.47(0.26)
83.00(0.25)
(EnsLoss)
(vs BCE) (ys ExP)
(vs HINGE)
MODELS
(better, no diff, worse) with p < 0.05
ResNer34
(41, 0)
(45, 0, 0)
(36,9, 9,0)
benefits in mitigating overfitting. Moreover, the advantages of ENSLOSS is further demonstrated by its consistent su- perior performance compared to other fixed losses, even with additional regularization methods. Another benefit of ENSLOSS is its relative insensitivity from time-consuming hyperparameter tuning, as a simple strategy of setting a large epoch often yields improved performance.
ResNet50
3,
(45,0, 0)
243, 0)
ResNet101
6,
(45,0, 0)
(40, 0)
VGG16
(36, 9,0,
045.0,0)
(29, 16, 0)
VGG19
9,
0,
(27, 18, 0)
MobileNet
(45,
0, 0)
(44, 1,
MobileNatV2
(45, 0, (45,
(45, 0)
Table 3. The summary statisties of datasets exhibiting statistical significance when comparing the proposed ENSLOSS against all other fixed loss methods in 14 OpenML datasets are presented.
5. Conclusion
ENSLOSS is framework designed to enhance ML per- formance by mitigating overfitting. The proposed method has shown potential to improve performance across a wide variety of datasets and models, particularly for overparame- terized models. The primary motivation behind consists of two components: "ensemble" and the "calibration" of the loss functions. Therefore, this concept can be extensively applied to various ML problems, by identify the specific con- ditions for loss consistency or calibration. Fortunately, some consistency conditions have been extensively studied in the literature, including Section 4 in (Zhang, 2004a), Theorem in (Zou et al., 2008), gamma-phi losses in (Wang & Scott, 2023), Theorem in (Tewari & Bartlett, 2007) and encoding methods in (Lee et al., 2004) for multi-class classification, Theorem 2 in (Gao & Zhou, 2015) for bipartite ranking or AUC optimization, and Theorem 3.4 in (Scott, 2012) for asymmetric classification. In addition. new discussions re- garding consistency, such as H-consistency (Awasthi et al., 2022), would also be intriguing in the context of ensembles with specific functional spaces.
(ENSLOsS)
(vs BCE) (vs EXP) (vs HINGE)
MODELS
(better, no diff, worse) with p < 0.05
MLP(1)
(9,4, 1)
27,
5)
MLP(3)
(7,7,0)
58,5,
2,
MLP(5)
(11.3.0)
211,
(13, 1)
4.4. Compatibility of prevent-overfittnng methods
As discussed in Section 2, the proposed ENSLOSS comple- ments most existing prevent-overfittnng methods, suggesting the potential for their simultancous use. In this experiment, we empirically investigate the compatibility of ENSLOSS with widely used regularization methods: DROPOUT, L2- regularization (or equivalently weight decay, denoted as W WEIGHTD),, and data augmentation (DATAAUG; (Wong et al., 2016; Xu et al., 2016)).
Design and results. To illustrate the compatibility, we conduct on the CIFAR-2 (cat-dog) dataset and ResNet50, using the same experimental setup as the main experiment. Specifically, we compare the performance of ENSLOSS with fixed losses under various regularization methods, and the results are presented in Table 4, which illustrates their compatibility and effectiveness in preventing overfitting.
A limitation of ENSLOSS is that it often requires more epochs to achieve stable training (see Table 8), resulting in longer training times. This issue is also seen in stochas- tic regularization methods like Dropout. Additionally, the selection of the positive random variable & during random loss- derivativv generation, currently done using the inverse Box-Cox transformation, needs further investigation.
Conclusion. According to Table 4, our prior hypothesis is confirmed: ENSLOSS is compatible with other regular- ization methods, and their combination yields additional
9

---

ENSLOss: Stochastic Calibrated Loss Ensembles
Acknowledgements
Breiman, L.. Random forests. Machine Learning. 45:5-32, 2001.
We thank the anonymous Area Chair and some reviewers for their valuable feedback, suggestions and support. This work was supported by the Hong Kong RGC-ECS Grant 24302422 and Hong Kong RGC Grant 14304823.
Bühlmann, P. and Yu, B. Analyzing bagging. Annals of Statistics, 30(4))927-96) 2002.
Cortes, C. and Vapnik, V. Support-vector networks. Ma- chine Learning, 20:273-297, 1995.
Impact Statement
Cox, D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statisti- cal Methodology, 20(2):215-22 232,, 1958.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Dalalyan, A. S. and Tsybakov, A. B. Aggregation by expo- nential weighting and sharp oracle inequalities. In Inter- national Conference on Computational Learning Theory, pp. 97-111. Springer, 2007.
References
Aronszajn, N. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68(3):337-404, 1950.
Evans, L. Measure Theory and Fine Properties of Functions. Routledge, 2018.
Audibert, J.-Y. Aggregated estimators and empirical com- plexity for least square regression. In Annales de I I'IHP Probubilités et Statistiques, volume 40, pp. 685-736, 2004.
Fehrman, B., Gess, B., and Jentzen, A. Convergence rates for the stochastic gradient descent method for non-convex objective functions. Journal of Machine Learning Re- search, 21(136):1-48 2020.
Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. H- consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning, pp. 1117- 1174. PMLR, 2022.
Finn, C., Abbeel, P., and Levine, S. Model-agnostie meta- learning for fast adaptation of deep networks. In Interna- tional Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Bartlett, P. L. and Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-453 -82,, 2002.
Freund, Y. and Schapire, R. E. desicion-theoretic general- ization of on-line learning and an application boosting, In European Conference on Computational Learning The- ory, pp. 23-37. Springer, 1995.
Bartlett, P. L., Jordan, M. I., and McAuliffe, J. D. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):
Gao, B., Gouk, H., Yang, Y., and Hospedales, Loss function learning for domain generalization by implicit gradient. In International Conference on Machine Learn- ing, pp. 7002-7016. PMLR, 2022.
2006.
Bechtle, S., Molchanov, A., Chebotar, Y., Grefenstette, E., Righetti, I.., Sukhatme, G., and Meier, Meta learning via learned loss. In 2020 25th International Conference Pattern Recognition (ICPR), pp. 4161-4166 IEEE, 2021.
Gao, W. and Zhou, Z.-H. On the consistency of auc pair- wise optimization. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, pp. 939-945, 2015.
Bottou, L. Online algorithms and stochastic approximations. Online Learning in Neural Networks, 1998.
Garrigos, G. and Gower, R. M. Handbook of conver- gence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.1123:: 2023.
Bousquet, O.
A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Mathematique, 334(6):495-51)) 2002.
Gonzalez, S. and Miikkulainen, R. Optimizing loss func- tions through variate taylor polynomial parameteri- zation. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 305-313, 2021.
Box, G. E. and Cox, D. R. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statisti- cal Methodology, 26(2):211 122223,, 1964.
Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. http:/ /www. deeplearni www.rriingbooooor....
Breiman, L. Bagging predictors. Machine Learning. 24: 123-140, 1996.
10

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
Hastie, T., Rosset, S., Zhu, J., and Zou, H. Multi-class adaboost. Statistics and its Interface, 2(3):349-600, 2009.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, arXiv preprint arXiv: 2017.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lugosi, G. and Vayatis, N. On the bayes-risk consistency of regularized boosting methods. Annals of Statistics, 32(1): 30-55, 2004.
Moulines, E. and Bach, F. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in Neural Information Processing Systems, 24, 2011.
Hinton, G. E. Connectionist learning procedures. In Ma- chine Learning, pp. 555-610. Elsevier, 1990.
UtrittttUuruty, J.-B. and Lemaréchal, C. Convex Analysis and Minimization Algorithms I: Fundamentals, volume 305. Springer science & business media, 1996.
Mourier, E. Eléments aléatoires dans espace de banach. In Annales de I' l'institut Henri Poincaré, volume 13, pp. 161 161-244, 1953.
Hoerl, A. E. and Kennard, R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1))55-67, 1970.
Raymond. C. Meta-learning loss functions for deep neural networks, arXiv preprint vrXiv:2406.09715. 2024.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510- 4520, 2018.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
Scott, C. Calibrated asymmetric surrogate losses. Electronic Journal of Statistics, 6:958-999 008899922221 2012.
Shamir, and Zhang, T. Stochastic gradient descent for non-smooth optimization: Convergence results and opti- mal averaging schemes. In International Conference on Machine Learning, pp. 71-79. PMLR, 2013.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Effi- cient backprop. In Neural Networks: Tricks of the Trade, pp. 9-50. Springer, 2002.
Soloff, J. A., Barber, R. F., and Willett, R. Bagging provides assumption-fiee stability, Journal of Machine Learning Research, 25(131):1 2024.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Na- fure, 571(5553) 2015.
Lee, Lin, Y., and Wahba, G. Multicategory support vector machines: Theory and application to the classification of microarra data and satellite radiance data. Journal of the American Statistical Association, 99(46)):67-88 2004.
Srinidhi, C. L., Ciga, O., and Martel, A. L. Deep neural network models for computational histopathology: A survey. Medical Image Analysis, 67:101813, 2021.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov. R. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-199) 2014.
Leng, Z., Tan, M., Liu, C., Cubuk, E. D., Shi, X., Cheng, S., and Anguelov, D. Polyloss: polynomial expansion perspective of classification loss functions. arXiv preprint arXiv:2204. / /25 2022.
Talagrand, M. New concentration inequalities in prod- uet spaces. Inventiones Mathematicae, 126(3):505-50) 1996a.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988, 2017.
Talagrand, M. A new look at independence. Annals of Probability, pp. 1-34, 1996b.
Lin, Y. A note on margin- based loss functions in classi- fication. Statistics & probability letters, 68(1):73-82, 2004.
Tewari, A. and Bartlett, L. On the consistency of multi- class classification methods. Journal of Machine Learn ing Research, 8(5), 2007.

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology. 58(1):2672288, 1996.
Zhang, T. Statistical behavior and consistency of classifica- tion methods based convex risk minimization. Annals of Statistics, 32(1):56-85, 2004b.
Vakhania, N., Tarieladze, V., and Chobanyan, Probability distributions on Banach spaces, volume 14. Springer Science & Business Media, 2012.
Zou, H. and Hastie, Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
Zou, H., Zhu, J., and Hastie, T. New multicategory boost- ing algorithms based on multicategory fisher-consistent losses, Annals of Applied Statistics, 124): 2008.
Vanschoren, J., Van Rijn, J. N., Bischl, B., and Torgo, L. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-66 60, 2014.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathol- ogy. June 2018.
Wahba, G. An introduction to reproducing kernel hilbert spaces and why they are so useful. In Proceedings of the 13th IFAC Symposium on System Identification (SYSID 2003), pp. 228,528,220 2003.
Wang, Y. and Scott, C. On classification--caiiratii of gamma-phi losses. In The Thirty Sixth Annual Conference on Learning Theory, pp. 4929-4951. PMLR, 2023.
Wolpert, D. H. Stacked generalization. Neural Networks, 5 (2):241-259, 1992.
Wong, S. C., Gatt, A., Stamatescu, V., and McDonnell, M. D. Understanding data augmentation for classification: when to warp? 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1-6. IEEE, 2016.
Wood, D., Mu, T., Webb, A. M., Reeve, H. W., Lujan, M., and Brown, G. unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24 (359):1-49, 2023.
Xu, Y., Jia, R., Mou, L., Li, G., Chen, Y., Lu, Y., and Jin, Z. Improved relation classification by deep recurrent neural networks with data augmentation. In Proceedings of COL- ING 2016, the 26th International Conference on Compu- tational Linguistics: Technical Papers, pp. 1461-1470. The COLING 2016 Organizing Committce, December 2016.
Yang, Y. Combining forecasting procedures: some theoreti- results. Econometric Theory, (0(1):176-220) 222, 2004.
Zhang. Statistical analysis of some multi- multi-category large
margin classification methods. Journal of Machine Learn- ing Research, 5(Oct): 1200::222222115111 2004a.
12

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
A. Ablation studies
The necessity of the bounded below condition. The bounded below condition is an important yet easily overlooked condition. Here, we present ablation studies demonstrate its significance. Our experimental setup is straightforward simply remove the superlinear raising-tail condition induced by the bounded below condition (Line 13 in Algorithm 1).
Table 5 presents typical training and test performance metrics at 20-epoch intervals for CIFAR2 (dog-cat) using VGG16, comparing models with and without this condition. Similar patterns are consistently observed across all datasets and network architectures.
epoch
20
40
60
80
100
120
140
160
180
200
w/o B-below
Train Acc
0.498
0.503
0.499
0.498
0.491
0.499
0.501
0.506
0.502
0.502
Test Ace
0.488
0.508
0.507
0.495
0.508
0.488
0.507
0.491
0.492
0.493
w/ B-below
Train Acc
0.961
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Test Acc
0.741
0.826
0.827
0.827
0.827
0.827
0.829
0.827
0.826
0.828
Table 5. Comparison training and testing accuracy with and without the bounded below (B- below) condition across different epochs. The results indicate that the absence of the bounded below condition leads to model fail to learn.
According to Table 5, without the bounded below condition, the model fails to learn, achieving only random-level performance (~500% accuracy) throughout training. In contrast, with the bounded below condition implemented, the model achieves high training accuracy (100%) and demonstrates good generalization with approximately 83% test accuracy.
From a loss-derivative perspective, the explanation is straightforward: Line 13 implements a superlinear loss-derivative that effectively discounts gradients for correctly classified samples = (x) > 1). This focuses optimization efforts on misclassified or boundary samples, a common design principle in classification loss functions.
When this condition (Line 13) is reflected into loss function as boundedness below constraint, the result is somewhat counterintuitive and unexpected. However, within the framework of convexity and calibration, the boundedness below condition appears to take on additional significance: Lemma 3.2 suggests that this condition provides essential regularity for the right tail of the loss-derivative. Additionally, in the Appendix D, we provide a more comprehensive theoretical investigation of the importance of the bounded below condition for loss calibration.
Randomly updating the inverse Box-Cos transformation. The performance of ENSLOSS, based on ResNet50 and CIFAR2 datasets, with varying values of T, is presented in Table 6, under the same experimental setting as described in Section 4.3.
DATASETS
fixed À 0
(used in Section 4)
T = 10
T' 20
T = 50
CIFAR2 (cat-dog)
70.04(1.21)
70.87(0.72)
71.48(0.62)
70.22(1.11)
CIFAR2 (bird-cat)
81.12(0.28)
80.45(0.30)
80.58(0.38)
80.63(0.47)
CIFAR2 (cat-deer)
83.11(0.299
83.05(0.12)
82.82(0.25)
83.47(0.15)
Table The averaged classification Accuracy and their standard errors (in parentheses), for different periods T of randomly updating inverse Box-Cox transformation based ResNet50 reported for CIFAR2 datasets.
These preliminary experiments suggest that randomly updating over epochs can be beneficial in certain cases; however, the effects are not particularly significant, and identifying the optimal value for this tuning parameter seems challenging. Conse- quently, we implement experiments in Section 4 with a fixed O, which corresponds to an exponential transformation of normally distributed random variables, and defer further investigation of this topic for future research.
Baseline of post-ensemble over fixed losses. We supplement our comparison by evaluating our approach against baselines of post-ensembling fixed losses. For illustration, we report test performance for model averaging (AVERAGE) and majority
13

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
BCE
EXP
HINGE
AVERAGE
VOTING
ENSLOSS
(Ace)
67.53 (1.05)
67.30 (0.38)
67.90 (1.33)
70.61 (1.09)
69.67 (1.59)
72.37 (1.04)
(AUC)
74.31 (0.73)
73.94 (0.75)
74.39 (0.93)
77.63 (0.37)
79.81 (1.14)
Table The averaged classification Accuracy and AUC and their standard errors (in parentheses) of model combination methods in the image dataset CTFAR2 (cat-dog) based on ResNet50 are presented. Bold font is used to denote statistical significant improvements over ALL other competitors.
voting (VOTING) CIFAR2 (dog-cat) using ResNet50 trained with three fixed losses. Similar patterns observed across all datasets and network architectures. The results are presented in Table 7.
As shown, post-ensembling indeed improves single fixed loss methods, but the proposed ENSLOSS still performs better. The challenge with post-ensembling is that selecting too many fixed losses requires substantial computational costs of training, while combining too few losses produces less effect. A key advantage of ENsLOSs that it requires only a single model training. Therefore, ENsLoss differs from post-ensembling methods in practical application.
B. Highlighting empirical results
In this section, we highlight some detailed empirical results of our proposed method, ENsLOSS, and compare it with other fixed loss methods on various image and tabular datasets. All detailed performance metrics for all experiments are publicly accessible at the epoch level via our Github repository and W&B projects.
Training time comparison. To more accurately compare the computational differences between ENsLOSS and fixed loss training, we provide detailed time comparisons between BCE, Hinge, and ENsLoss in Table 8. Specifically, report the minimum number of epochs required for training accuracy to stabilize (defined as remaining within an error margin of 0.005 thereafter) across different neural network architectures the CIFAR2 (cat-dog) dataset.
Loss
MobileNet
MobileNetV2
ResNet101
ResNet34
ResNe150
VGG16
VGG19
BCE
90
80
80
70
45
15
45
HINGE
40
35
60
25
35
25
35
ENSL.OS
110
90
160
145
150
55
150
Table 8. Comparison of minimum epochs required for training accuracy to stabilize (within an error margin of 0.005) across different neural network architectures on CIFAR2 (cat-dog) dataset.
As indicated in Table 8, ENsLoss training typically requires 2-3 times more epochs than traditional BCE or Hinge loss training to stabilize the training procedure. This computational overhead acceptable for practical applications, and we are actively exploring methods reduce the training time in future work.
Performance on CIFAR2 datasets. Figure 4 illustrates the overall pattern of performance of all methods in 45 CIFAR2 binary classification datasets. Table 9 presents the performance results (accuracy and AUC values) for representative single CIFAR2 dataset (CIFAR2 (cat-dog)). Additional detailed results for other network architectures and datasets available in our accompanying W&B projects and GitHub repository.
Performance on PCam dataset. Table 10 presents the performance results (aceuracy and AUC values) for the PCam dataset for all different network architectures.
Performance on OpenML datasets. Table 11 presents the accuracy results for the 14 OpenML datasets using MLP(5), with additional detailed results for other network architectures available in our accompanying W&B projects and GitHub repository.
C. Technical proofs
C.1. Auxiliary definitions
To proceed, let us first introduce or recall the definitions and notations:
14

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
3.00
4150006 Prrec BCC
EXP
5.05
5.50
2.05
5.00
6.75
5.70
8.65
ceross
crius
owe2
covaz3
cevass
Figure 4. The overall pattem of performance (Accuracy) of ENsLoss against all other fixed loss methods in 45 CTFAR2 binary
classification datasets (provided by pairwise labels subset of CIFARIO), based on VGG16, is illustrated. The r-axis represents label-paired binary CIFAR datasets, where, for example, CIFAR35 corresponds to CIFAR2 (cat-dog) dataset.
Classification probability: -x()) == P(YY = 1(X).
The Bayes classifier: (x) = sgn(n(x)) - 1/2).
The pointwise minimization of R:
Co(n, (a 2 0) + (1- s 0), Ho(r) inf Co(n, a).
aeR
The pointwise minimization of Rg:
Co(1. = nε(a) + (1 (1- ()--aa,, Ha(r) = inf Co(n, a), aER inf
H (n) 1350
inf
Co(n,o), H*)1) a=)22
Co(
Proof of Theorem 3.6
Proof. We begin by defining the ensemble loss, denoted by (2)), as the expected value of (2)) with respect to the distribution of Φ i.e., d(z) == E.g ((z). Note that the expectation is well-defined according to Assumption 3.4. Consequently, the calibrated ensemble risk function can be reformulated as follows:
R(f) Ex,y (x(Y)
Thus, we can rewrite the ensemble risk in the classical form of a fixed loss, namely, R(f) = Rf),), thereby enabling us to leverage Theorems 1 and 2 in (Bartlett et al., 2006) facilitate statistical analysis of the proposed method. Therefore, it suffices to verify the BCC condition of the ensemble loss G. that is, convexity, the bounded below condition, and having a negative derivative at O.
To clarify, we sometimes denote ((2) as ((2,w) to emphasize that Φ : Ω + L is a L-valued random variable, equipped with a probability measure μ. Now, we start with considering a simple finite space case, which provides insight into the underlying proof strategy.
Specific case: finite space. Suppose L = (oi,... Φ2) is finite space, 5(2) can be simplified as x(2) >g (2), where ny = P(O = o,)) > 0 and ΣΦ T. = 1. Next, we check the BCC conditions of Φ. (i) T is a convex combination of convex functions, thus & is convex; (ii) since Øy is bounded below by Cg. thus 3 is bounded below by milg=1,,... Q Cyi (iii o, is differentiable at 0 for all g, thus & is also differentiable at 0, and (0)) = TB'1(0) (0) - T, (0) < O. Therefore, & satisfies the BCC conditions.
15

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Table The averaged classification Accuracy and AUC and their standard errors (in parentheses) of all methods in the image dataset CIFAR2 (cat-dog) are presented. Bold font is used to denote statistical significant improvements over ALL other competitors.
MODELS
BCE
ExP
HINGE
ENsLOss
(Acc)
ResNet34
68.94(0.45)
60.10(0.44)
70.39(0.37)
72.03(0.71)
ResNet50
67.59(0.35)
61.57(0.56)
72.04(0.355 67.57(0.28)
ResNet101
67.32(0.23)
53.57(0.41)
67.12(0.22)
70.07(0.90)
VGG16
77.36(0.68)
69.27(0.44)
78.13(0.87)
81.13(0.77)
VGG19
76.96(1.03)
66.38(1.17)
78.06(0.68)
80.57(0.76)
MobileNet
66.77(0.86)
55.89(1.69)
67.66(1.03
69.98(1.08)
MobileNetV2
73.34(1.12)
62.94(1.09)
73.45(1.02)
78.40(1.56)
(AUC)
ResNet34
75.97(0.44)
(4.39())52)
76.02(0.32)
79.24(1.09)
ResNet50
73.96(0.29)
65.52(0.60)
74.37(0.23)
79.40(0.31)
ResNet101
73.35(0.24)
54.88(0.34)
68.61(0.66)
76.60(0.87)
VGG16
85.42(0.71)
76.13(0.94)
86.20(0.63)
89.54(0.29)
VGG19
85.61(0.97)
72.88(1.12)
84.47(1.07)
87.55(1.08)
MobileNet
73.12(0.97)
58.51(1.77)
73.96(1.34)
76.61(1.23)
MobileNetV2
81.29(0.96)
67.53(1.11)
81.31(0.74)
86.22(1.37)
Table 10. The averaged classification Accuracy and AUC and their standard emrors (in parentheses) of all methods in the image dataset PCam are presented. Bold font is used to denote statistical significant improvements over ALL other competitors.
MODELS
BCE
EXP
HINGE
ENsLoss
(Ace)
ResNet34
76.91(0.52)
73.78(0.52)
77.20(0.18)
82.33(0.30)
ResNet50
77.23(0.51)
74.10(0.49)
77.96(0.34)
82.00(0.07)
VGG16
80.97(0.25)
77.11(0.50)
82.69(0.30)
85.77(0.35)
VGG19
81.58(0.25)
76.13(0.35)
82.77(0.41)
85.91(0.19)
(AUC)
ResNet34
88.69(0.34)
83.30(0.57)
76.11(0.37)
92.24(0.13)
ResNet50
88.75(0.30)
77.24(0.67) 83.51(0.46)
92.07(0.49)
VGG16
93.35(0.26)
88.77(0.59)
86. 18(..66)
95.44(0.24)
VGG19
93.49(0.17)
87.89(0.46)
84.09(0.60)
(0.51(
General cases. Now, suppose C a general measurable BCC subspace satisfying Assumption 3.4. The crucial issue is whether the BCC conditions are complete in the space L over limiting. Let check the BCC conditions of Ø. (i) Convexity. Note that
7( + +1 A)22) / @(x21 + A)22, w) du(()) X00(21 + (1
A0(21) A)0(22).
where the inequality follows from the fact that L is BCC subspace and thus ((z,w) w) is convex for any E Ω. Thus, Ø is convex function. (ii) Bounded below.
τ(2) = (22,w)dp(() 2 ( ((w)dp(w) EU -0,
where the inequality follow from the second condition in Assumption 3.4 such that $(2,w) 2 U(w) for any 2 almost surely. (iii) Calibration. For every sequence 5n with &n O, without loss of generality, assume 1ô,l S 5o. which be satisfied for sufficiently large We now check the definition of differentiability of
0:
((6m) -50)
4 ((n,,)) w) 4(0,w) du(w) = lim Gn(w)dd(w)),
lim
n 700
8n
n1 10 jo
on
no0
16

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Table 11. The averaged classification Accuracy and its standard errors (in parentheses) of all methods with ML.((5) in 14 OpenMI. datasets are presented. Bold font is used to denote statistical significant improvements over ALL other competitors.
MLP(5)
(1.")
BCE
Exp
HINGE
ENsLoss (our)
Bioresponse
(3.75, 1.78)
76.84(..33)
77.49(1.44)
76.03(0.67)
77.18(1.18)
guillermo
(2..0,4.30)
70.35(0.44)
70.26(0.67)
69.67(0.63)
70.34(0.78)
riccardo
(20.0, 4.30)
98.68(0.21)
98.69(0.13)
98.62(0.23)
99.14(0.23)
hiva-agnostic
(4.23, 1.62)
91.02(0.85)
91.65(1.30)
95.55(0.53)
90.61(1.49)
christine
(5,42, 1.64)
69.62(1.07)
69.42(1.30)
67.48(0.72)
69. (9.99((..3)
OVA-Breast
(1.54, 10.9)
94.27(1.33)
94.38(1.41)
92.61(1.75)
95.45(1.30)
OVA-Uterus
10.9)
80.54(1.54)
82,09(1.50)
84.22(1.75)
86.68(1.66)
OVA-Ovary
(1.54, 10.9)
81.83(1.69
82.82(2.19)
82.76(1.69)
87.16(1.40)
OVA-Kidae
(1.54, 10.9)
97.59(0.83)
97.72(0.65)
96.47(0.95)
98.06(0.48)
OVA-Lung
(1.54,
88.17(1.70)
89.31(2.36)
89.76(1.53)
93.00(1.31)
OVA-Om
(1,54, 10.9)
71.42(3.33)
74.91(1844
79.25(2.19)
82.00(1.98)
OVA WWA-Colon
(1.5,
95.73(0.87)
95.73(0.88)
95.15(0.79)
96.27(0.63)
OVA-En
10.9)
71.66(3.79)
74.33(1.73)
81.68(1.87)
83.19(2.01)
OVA Prostate
(1.54,
97.39(0.51)
96.96(0.77)
97.22(0.84)
97.93(0.60)
where G,, (w)= (#(6,, w) (0, (0,w))) Next, the proof involves usage of the Dominated Convergence Theorem (c.f. Theorem 1.19 in (Evans, 2018)) to exchange the limit and the integration. To proceed, since 4(2, w) is differentiable at O for any w E Q, it follows that limn-+oe G.(w) = & (0, w) pointwise. Note that 4 w) is convex w.r.t. 2 for any w E Q, then by the definition of the subderivative, we have,
(( - ((0,w) 2 w',,,))n. and (00 - D( w) 2 -00 (6n, w))n=
which yields that
Gn](w) S max (j00(6n: w),, (0,,w) S sup w)) G(w), 16|560
J. G(w)dp(w) = EG < co.
Thus, G₁ dominated by G, then the Dominated Convergence Theorem yields that the limit exists and equals to,
lim (lōn) -
( lim Gn(w)du(w)- n000
/ Φ' (((')) (0) < 0,
5n
where the last inequality follows from that 4'(0,w) < 0 for all w E Ω.
In summary, we have proved that Ø- RR -> R satisfies the BCC conditions. Consequently, according to Theorem 1 and Theorem 2 (part 2) in (Bartlett et al., 2006), we can draw the following conclusions:
1. Φ classffcationn-calibbaaedd as stated in Definition 1.1.
2. The excess risk bound of R((-) is provided
R(f) R S (B(( -R'),
where v defined as:
1 1+0 1+0) B(O(0)) H 1+0
(00)= - ((0) H
where (n)) == infaeR an)
This completes the proof.
17

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Proof of Lemma 3.2
Proof. Since & is convex and calibrated, (0) O, and ((z) ( + 0' 2 ((0) when 2 S 0, thus d(z) bounded
below when I 0. Next, we aim establish lower bound of ((2) for 0.
CASE 1. If there exists constant 20 > 0 such that 0x(=0) 2 0, then ((z) 2 6(2)) + 00(2)) 20) 2 ((20) for 2 2 20. For O S = S zo. ((2) is bounded according to the boundedness theorem. Then, <(z) is bounded below.
CASE 2. If 00(()) < 0 for all 2 > Then, for any 2 > O, we partition [0, as n intervals with do 0, z/n. dn-1 (n 1)2/n, dn 2, for the i-th interval, we have
o(d++1) 2 ((d) 00(d))//n
Taking the summation for both sides, and (x((2) is nondecreasing according to Lemma C.2,
do(d,)z ((0) + (((0)z
0p(d;)²
d(=) = ((d,)) 2 ((dg) +
I'" 1=
2 (0) + Ø (0)z
0b(z)dz 2 (00) o' (0)z
@b(z)dz,
where the second last inequality follows from the integral test, and the last inequality again follows from Lemma C.2. Taking the limit for + oc, for any z > we have
((2) = lim (zz) 2 ((0)= + 1- Do(z)dz=dhl + 10 00(2)dz +
1 0p(z)
g()dd
9(2)
00(20) ( g(z)dz 2 200) + (0)
6'(0)
(0) (0) g(00)
g(2)d2
-00.
g(2))
The desirable result then follows.
Proof of Lemma 3.3
Proof. Before proceed, we add = 0 into the batch points as {= _.+1). Without loss generality, we assume (i) 2 F Zj, as we can duplicate the gradient by merging and treating them as one point; and assume that (ii) 21 < 22 ... < BB+1 and 2be = 0, as we can always sort the batch points. Next, we also design and add the loss-derivative for 2b, = 0 into the given dersvveeiss g to form a new gradient vector g. specifically,
g : Di: if 20: Ôi = 9:-1 if 20,
where The Ibn-1 S 9bu 9b
12, min -1/, (ghp-1
Dog +1 22), if bo 1: Don min min(Tho (0h 1), if by 1. Hence, Tho 0 and
Then, we define (z1 + /2, corresponding loss function d can be formulated as follows.
(z; + 1/2, ..,, uB = (EB + 2B+1)/2, uB41 2841 + 1, and the
(x)) 912, l2(2) = ğz(zz - -u1) + (ur).
if S U1,
if 01 < 2 S u2
Φ(z):
(x) = Ino (z - ubo 1) + 1b0 11400
if the -1 < S Ubg
(2) ØB-1(z up) + +l
uf S UB+1,
(B42(z) = max(āe+1 1)(z uB+1) (B+1(1)))), if > UB+1
By definition, & provides the loss-derivatives for original sample points, that is, ('(z+) gi for all * by Now, we verify that d(z) is a BCC loss. Specifically, (i) Φ(z) is a continuous piecewise linear function with coefficients Ō 9 < ... < 08+1 S max(ō 1) according to Condition 1 and the definition of g. o(2) is a convex function. (ii) Note that 2bo = O E (ubo-1, I. hence o(x) is differentiable at 0 and ø'(0) = 14,, (0) = goo < O. & is calibrated. (iii) d(z)
increasing function when u thus bounded below. This completes the proof.
18

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Proof of Lemma 3.5
} is finite space, then (+(=) is defined as E0(z) = Eg= m00 < co, where 3g PP( 04) > 0 and Σ. Fy = 1, which leads to Condition Since &g is bounded below, that is, there exists constant Ug -0,, such that (=) 2 Ug. thus (=))) 2 Biiig=,...,,QUU. U > -o,, which leads to Condition 2. Finally, d4 is convex and differentiable at O, then (z) converges to %% when = - 0, see Corollary 4.2.3 in (Hiriart-Urruty & Lemaréchal, 1996). Therefore, for each
Proof. Suppose L (ol, Φ2)
1, ..... Q. there exists constants o, > 0 and G, > 0, such that (2)] 1 Gg for all E [-6g. 5gl, which leads to Condition 3 by the fact that [0)(22) - max .Q Gg =: < x0
for 2 E -o: do] with on mi... 6, for all = I, : Q. This completes the proof.
Auxiliary lemmas
Lemma C.1. For C)(n,,), Ho(n), H (n) and H (n) defined in Appendix C.I with a convex and bounded below 6. then
a a) is convex with respect to a.
b. For any n # 1/2, H (n) H4(r) is equivalent Ho (n) d6)0) < 0, then H (n) = 60))
(n) > H (n)
Proof. (a). Since 0 S 7 S 1, Co(y. a) no(a) + (1 n))(-a) is a convex combination of Φ(a) and d(-a). Then, a) is convex with respect a since both Φ(a) and (--a) are both convex.
(b). Denote A= fa]a(2n 1) < 0}, then (T) = infoRлuse Co(n, a). H (n) = infoEe Co(n, a), and H (n) inf C) Given that 7) * 1/2, both A and A are nonempty.
We first show that H²(n) = min (HT (1), H (17)) Note that H²(1) S Hz()) and H²(1) S H then Ha(r) < min (M),
(n), H (n)). On the other hand, for any E R, then either E A or E A thus Co(n.a) 2 min (H, (n), H3) (n)), and Ha(r) 2 min (H2 (1), (n)).
Now, Hg (1) > Hz(n) min (Hg (1), H (m)) if and only if H (1) < (n).).
(c). Since é convex, Co a) = no(a) 1--)((-aa) 2 (2n 1) 1)o:). Hence,
(0) Co 0) 2 H (n)
inf Co a) ž inf (()) c(0), a(20-1)50
=50
where the last equality follows from the fact that o is convex and (b=) 2 (0) + (0) (()) when S 0. The desirable result then follows.
Lemma C.2. Let o R + I be convet Then
a. The sub-derivative dΦ is nondecreasing.
b. If ((x)= > (ly) for some (x. y), then for any between x and y (excluding x and y). 6(x) d(z).
Proof. (a). By the definition of sub-derivative, for any > I, we have
d(y) 2 ((x) + (') - x), <(x) 2 2(0)+ v),
providing that (y - ()(())))) (2)) S O. Thus, ø' is a nondecreasing function.
(b). For any between x and y. there exists A > O, such that 2 = Ar + (1 A)y, then
d(z) = o(Ar + (1 - A)y) S xΦ(x) + (1 - A)o(w) < Φ(x).
19

---

ENSLOSS: Stochastic Calibrated Loss Ensembles
An example illustrating Theorem 3.6
Let C { Φ2). where (2) exp(-2) is the exponential loss, and dz(2) log(1 + + e==) is the logistic loss, and P(@ = og) = Ty O, then R(-) is clasification-caliiraee and the excess risk bound is provided as:
((((f) R²) - R(f) R, where (0) = T1 vI 8 + 12/2((1 0) log(1 8) (1 + 8) log(1 + +)) for 0 E [o, 1), which can be simplified as:
2
R(F) Rd <
R(f) - R
2 + T2
D. Calibration for unbounded below losses
In this section, we discuss the relationship between the unbounded below condition and calibration for surrogate loss functions. First, present that the bounded below condition is indeed necessary condition for classifiattinn-caiiaattio
Corollary D.1. Suppose a comvex loss function <(-) unbounded below, then it not classifiction-ccaibreee
Proof. Since ((2) is convex and unbounded below, then d(=) -oo for + co. Let's consider the case where there exists domain Un such that 1 P(X Xn) and n)xx) for Mn. Then, define [i(x) I when E Ag, and fi(x) 0 when x & Ng. In this case,
Ro(fi) E(I(XE Xo)o(1) E(1(X & xo)o(0) + -00 R
however, is clear that R(fi) - R", since ft fails to match the Bayes rule when x & Xg. Thus, if d is calibrated, it must be bounded below.
The concept of clasification-calibratio is considered "universal", as it requires the loss function to ensure consistency for all data distributions. We then investigate weak version of calibration: can unbounded below loss functions guarantee calibration for some specific data distributions?
To address this question, we start by extending the if and only if condition for the equivalent definitions of classification- calibration in (Bartlett et al., 2006).
Lemma D.2. For any loss function IR + (-00, 00). R7 -oo if and only if the statements (a) and (b) are equivalent, where statements (a) and (b) are defined as:
a. For any n + 1/2, (n) > Hn(1).
b. For every sequence of measurable function ft X - IR and every probability distribution on x {+1}.
Ro(fi) - Rs implies that R(fi) + R".
Proof. The necessity (=). Suppose R > -o, then (a) - (b) can be proved by following the proof of Theorem I (the last paragraph) in (Bartlett et al., 2006). Next, we tend to prove (b) = (a) by contradiction. Suppose there exist a sequence } and some probability distribution, such that
Rolfi) + Ry but R(fi) + R
(11)
Define
2o = lim Co(r(X), (x(() 10000
Ω
x lim (o()(X), Ho Ho(x()))
20

---

ENSLOSs: Stochastic Calibrated Loss Ensembles
Since (0) 2² > X; and > n² 2 O, (11) implies that P(Rg)) and P( < 1, and thus P(2 (g)) 2
P(22)) ( > 0. Therefore, there exists a subset A C !2 S2 with (P(A) 0, such that for any E A, we have
Co(n. on) + Ho(n), but Co(n, az) + Ho(n),
where v- : 7(x) and aq := fi(x). This implies that 3e 0, VL.. 31 > Lg, such that, Cu(n, > Ho)r), and thus (nn2n- - 1) S O. Now, (n) = (1) follows from by taking limits of (n) H S Co(n, ar), and the fact that aa) + (n). which contradicts to the calibration definition of Φ in (a). This completes the proof of the necessity.
The sufficiency (=). We construct the proof by contradiction. Suppose R = - -o, we can find an example that (a) does not imply (b). Specifically, assume that Φ is classification calibrated and Rg (fi) + R = -00, then there exists sequence {fi) such that 1P(22) = 1, and a set A with P(AA) = c > O, such that Ha (7(x)) = - 0o, for x E A. Now, we split A as two disjoint sets A1 and A2 each with probability measure c/2, and define a new sequence fi(x) == fi(x) for x E AG, and fi(x) == 0 for x E A1. In this case,
Rs (a) = P A1)Ø(0) + ((1 E A2) + 1(X & ()) -00= Rig
However, R(fi) - R since x E A1, Co(n(x),, f(xx)) min(n(x), n(x)) -Ho(7(x)) and P(A1) c/2 > 0.
This completes the proof.
Lemma D.2 relaxes the non-negativeness or bounded below condition for calibration in (Bartlett et al., 2006), which helps extend the if and only if conditions of classification-calibeee to more general loss functions. For example, if we work with specific data distributions, many calibrated surrogate losses do not need to be unbounded below, as illustrated in the following examples.
Hinge loss with varying right tails. In this example, we examine the impact of various right tails on calibration of loss functions. Specifically, we adopt the shape of hinge loss function x(2) = 2, when 2 < 1, and explore different right tails when = 2 1, then discuss calibration.
When = < 1, 6(2) =
When x > (zero tail) (02) = O; (exponential tail) 6(z) (x-11) 1; (inverse tail) d(z) 1/2 (inv-log tail) ((=) = e/ log(z + 1) e; (logarithm tail) <0=) log(z).
Lemma D.3. Suppose (X) E [c. 1 c) almost surely for any fixed e > 0, then the hinge loss, with the zero, exponential, inverse, inverse-logaritmm, and logarithm tails, are all clasification-callbrueeeld
Proof. We can check that R2 -o for all pre-defined losses, and & is convex with s0(0) < then & is classification- calibrated.
Therefore, some unbounded below loss functions can still provide calibration for particular data distributions. Unfortunately, unbounded below loss functions often exhibit instability during training with SGD, even if they are calibrated in simulated cases (see our GITHUB). One possible explanation is that the batch sampling may disrupt the distribution assumptions necessary for calibration of unbounded below losses. Conversely, bounded below calibrated loss functions do not rely on data distribution assumptions to maintain calibration.
This appendix delineates the relationship between the bounded below condition and calibration. First, the bounded below condition is a necessary condition for calibration. Next, by introducing additional distribution assumptions, it is demonstrated that certain unbounded below loss functions can still be calibrated. Lastly, through numerical experiments, it is suggested that unbounded below loss functions may exhibit potential instability during SGD training. In conclusion, the unbounded below condition (or superlinear raising tail) is identified critical yet often underestimated criterion for loss functions, both theoretically and in numerical implementations.
21


Reviews:
[{"review_id": "HXn1BPREhr", "reviewer": "Reviewer_zDea", "summary": "This paper introduces EnsLoss,  a stochastic ensemble learning method specifically designed to mitigate overfitting in classification tasks. The main idea is is to ensemble different surrogate loss functions during SGD. The authors provide theoretical analysis and empirical results across various datasets. The results indicate that EnsLoss consistently outperforms fixed-loss methods, particularly for overparameterized models like ResNet.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (No ablation studies on invBC transformation)\n\n**Reply.** Thank you for the comment. Table 5 partially shows our invBC ablation studies: (1) Fixed case ($\\lambda=0$) used in main experiments, where invBC is somewhat \"deactivated\" with gradients sampling from a fixed log-normal distribution; (2) Dynamic case with varying $\\lambda$ sampled at different periods, where invBC is activated. This comparison reveals invBC's impact. \n\nWe agree these results are insufficient and will include expanded analysis in our revision.\n\n> (Limited to binary classification)\n\n**Reply.** We agree and will discuss that ensLoss can be extended to more ML tasks with calibration conditions.  We focused on binary classification (BC) because:\n1. The if-and-only-if condition for calibration in BC is well-studied and straightforward to implement, allowing us to directly validate our ensemble idea without additional complications.\n2. BC itself is a fundamentally important problem with many applications. We believe that improvements are also highly significant.\n\n> (No evaluation of computational costs of ensLoss with additional epochs)\n\n**Reply.** Agree. We now provided time comparisons: report the *min epoch* required for training to stabilize (after which train accuracy remains within an error margin of 0.005) on the CIFAR2 (cat-dog) as a demo.\n||BCE|Hinge|ensLoss|\n|-|-|-|-|\n| MobileNet|90|40|110|\n| MobileNetV2|80|35|90|\n| ResNet101|80|60|160|\n| ResNet50|45|35|150|\n| VGG16|15| 25|55|\n| VGG19|45|35|150|\n\nAs indicated, *ensLoss* training typically requires 2-3x more epochs to stabilize, which remains acceptable.\n\n> (Theorem 3.6 assumes finite loss space via Lemma 3.5)\n\n**Reply.** To clarify, Theorem 3.6 relies solely on Assumption 3.4 (A3.4) and NOT on finite loss space. Lemma 3.5 provides only a sufficient condition for A3.4.\n\nE.g., a loss space $\\phi(z)=(1-\\beta z)\\_+$ with $\\beta \\sim$ U(1,2), contains infinitely many losses, yet satisfies A3.4. We will clarify it in our revision to prevent confusion.\n\n> (invBC can produce unbounded gradients)\n\n**Reply.** Thanks for this key insight. In our implementation, we applied clipping at -1 for the loss-derivatives (Line 46 in `loss.py`). This was done initially because BCE and hinge gradients are both bounded by 1, ensuring fair comparison by eliminating any potential effect from differences in loss-derivative magnitudes. We will add the clipping step between Lines 9-10 in Algo 1.\n\n> (Theory assumes static losses while algo uses dynamic losses)\n>\n> ($p=1$ is used but theory requires $p>1$)\n\n**Reply.** This concern may be related to the finite loss space assumption mentioned earlier. Our framework (4) requires only that losses satisfies Assumption 3.4, not enumeration of all possible losses.\n\nFor our implementation, Algo 1's generated loss-derivatives naturally satisfy conditions 1 and 3. \n\nWe agree a small gap in condition 2 between theory $(p > 1)$ and algo $(p=1)$, but it really has a negligible effect on numerical results. E.g., using $p=1+10^{-10}$ would satisfy theoretical conditions with negligible numerical difference, as $z^{10^-10}$ falls beyond `float32` precision even for huge $z$ values.\n\n> (About insensitivity of tuning)\n\n**Reply.** We appreciate the careful reading. To clarify,\n- We used fixed $\\lambda=0$ in all main experiments, achieving strong performance without tuning. \n- Table 5 merely explores invBC effects, not suggesting necessary tuning. Unlike methods requiring careful adjustment of parameters, *ensLoss* needs minimal tuning effort.\n\nWe will clarify that Table 5 is for ablation study of invBC not suggesting tuning necessity.\n\n> (Comparison with ensembling fixed losses)\n\n**Reply.** Agree. To illustrate, we report test performance for model averaging (m-ave) and majority voting (m-vote) on CIFAR2 (dog-cat) using ResNet101 over three fixed losses:\n\n||Acc|AUC|\n|-|-|-|\n|BCELoss|67.33(.23)|73.35(.24)|\n|EXP|53.62(.48)|54.86(.35)|\n|Hinge|67.12(.32)|68.60(.66)|\n|**m-ave**|68.14(.25)|74.35(.26)|\n|**m-vote**|67.93(.26)|---|\n|ensLoss|**70.07(.90)**|**76.60(.87)**|\n\nAs shown, ensembling indeed improves single fixed loss methods, but enLoss still performs better. We observed these conclusions across datasets and will add details in the revision. \n\nThe challenge with ensembling methods is that selecting too many fixed losses requires substantial computational costs of training, while combining too few losses produces less effects. A key advantage of ensLoss is that it requires only a single model training.\n\n> (Need evidence that EnsLoss offers advantages beyond just running more epochs)\n\n**Reply.** We clarify that all methods used the same number of epochs. We also provided epoch-level comparisons, as shown in Fig 1, where performance stabilizes after 100-150 epochs, with ensLoss maintaining its advantage. This pattern appeared across all datasets. We'll include more results in our revision. The persistent gap confirms ensLoss benefits aren't simply from long-epoch training.", "claims_and_evidence": "Strenghts:\n1. The authors provide empirical results that show consistent improvements over fixed losses.\n2. EnsLoss is compatible with prevent-overfitting methods (section 4.4).\n\nWeaknesses:\n1. The alrogithm/practial experiments and theoretical framework (some conditions and proof) mismatch, see detailed questions in Theoretical Claims.\n2. Limited scope of experiments (only binary classification, and small-scale image and tabular datasets) and also need to provide more ablation studies and baselines, see detailed questions in Methods And Evaluation Criteria and Experimental Designs Or Analyses.", "methods_and_evaluation_criteria": "Strenghts:\n1. The conditions for loss derivatives are derived from established theory\n2. The ensemble risk framework (Theorem 3.6) connects the proposed method to statistical consistency.\n3. It is pratical that the EnsLoss can be compatible with exisinting SGD based pipelines.\n\nWeaknesses:\n1. No ablation studies on Inverse Box-Cox transformation (9 in Algorithm 1)\n2. All experiments only focus on binary classification, which limits the scope of the proposed methods.\n3. The authors mentioned the limition of EnsLoss requiring more epochs in general, but there is no evaluations on computational costs.", "theoretical_claims": "Weaknesses:\n1. The proof of Theorem 3.6 assumes a finite loss space via Lemma 3.5, but the algorithm generates loss derivatives stochastically which implys an infinite space. The authors argue that fixing the number of epochs makes finite in practice, but this is not formally proven.\n2. The inverse Box-Cox transformation in Algorithm 2 can generate gradients from exponential distributions, so produces unbounded gradients, which violates the condition 3 in Assumption 3.4.\n3. The theoretical framework assumes pre-defined losses (possibly random but static) as Eq 4, while the algorithm dynamically generates losses via SGD. The stochasticity of loss sampling is not addressed in the proofs.", "experimental_designs_or_analyses": "Weaknesses:\n1. The authors choose $p=1$ for simplicity in the actual implementation in line 263, but in Lemma 3.2 the authors propose taking $g(z)=1/z^p$ with $p>1$. It may lead the loss unbounded below.\n2. The authors mentioned \"Another benefit of ENSLOSS is its relative insensitivity from time-consuming hyperparameter tuning\" in Lines 405-406, but the $\\lambda$ is still a hyperparameter affecting the results (Table 5).\n3. Since authors are inspired by ensemble ideas, there should be more baselines to be compared, such as ensembling fixed loss\n4. Longer training with traditional loss functions might mitigate overfitting similarly, so it is better to provide more detailed analysis on whether EnsLoss truly provides advantages beyond simply running more epochs.", "relation_to_broader_scientific_literature": "EnsLoss extends the idea of dropout to the loss space while grounding it in calibration theory. However, the authors only conduct experiments on binary classficiation, which leave open questions about its broader applicability.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "If retaining $p=1$, it is better to provide additional explicit conditions or constrants on the random loss derivatives, or clearly state the theoretical limitation of Lemma 3.2", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2i2pEdPZq0", "reviewer": "Reviewer_UHGu", "summary": "The paper introduces EnsLoss, a novel ensemble learning method that applied the idea of ensemble to loss functions during model training within the empirical risk minimization (ERM) framework. Specifically, instead of explicitly using one loss functions, the author proposes to randomly sample loss functions on the fly during ERM. In practice, the authors propose transforming convexity and calibration conditions into loss-derivatives, allowing for direct generation of calibrated loss-derivatives. The authors verifies the validity of the proposed method through theoretical analysis. They also conduct experiments across multiple datasets and deep learning architectures to demonstrate its superiority over fixed loss methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (Empirical verification weak - only tested on CIFAR2, not CIFAR10.)\n\n**Reply.** To clarify, our current work focuses specifically on *binary classification*. Since CIFAR10 is a multi-class dataset with 10 categories, we derived binary classification problems from it by creating all possible pairwise combinations of the original classes, which resulted in 45 distinct CIFAR2 datasets (calculated as ${10}\\choose{2}$ = 45 pairs).\n\n> (The accuracies of the CIFAR2 results seems rather low compared to CIFAR10 in literature)\n\n**Reply**. The performance gap you've noted stems from several differences between CIFAR10 and CIFAR2:\n\n1. **Dataset size**: Each of our CIFAR2 datasets contains only 10,000 training images (5,000 from each of the two selected classes), whereas the full CIFAR10 dataset contains 50,000 training images. This significant reduction in training data naturally impacts model performance.\n\n2. **Classification difficulty variation**: As shown in Figure 4 of our manuscript, the accuracy of CIFAR2 varies substantially depending on which pair of classes is selected, ranging from 68% to 96%. Some binary classification tasks (e.g., distinguishing between cats and dogs) are inherently more challenging than others (e.g., distinguishing between automobile and horse).\n\nTherefore, the 90%+ accuracy for CIFAR10 is a 10-class classification task with full training data. Our experimental setup is fundamentally different, making direct accuracy comparisons is not meaningful.\n\nIn addition, our experimental design aims to provide a fair comparison across different loss functions and the proposed ensLoss. The implementation settings for each method are identical, with the only difference being the loss functions used. Additionally, we have provided anonymous GitHub open source code to enable replication of our results.\n\n> (Algorithm 2 from Appendix A should be in main text for better understanding)\n\n**Reply**. Thank you for the suggestion. In our revision, we will move Algorithm 2 and the relevant explanatory content from Appendix A into the main text, specifically after Section 3.2 before Section 3.3 where we introduce our methodology.\n\n> (Need discussion of prior works on modified loss functions/gradients for neural networks)\n\n**Reply.** Thank you for highlighting these important references on loss function modifications. We agree that our work could benefit from being positioned within the broader context of gradient-based loss function innovations.\n\nThe papers you've mentioned represent significant contributions to loss function design. Focal Loss addresses class imbalance by down-weighting well-classified examples. PolyLoss provides a unified framework through polynomial expansions. Generalized Cross Entropy and Normalized Loss Functions tackle noisy label scenarios through robust formulations. As you mentioned, our approach differs in that we dynamically sample from a collection of loss functions rather than proposing a fixed modified loss.\n\nIn our revision, we will expand Section 2 to include these references and others, providing a more comprehensive discussion of loss function design from a gradient perspective.\n\n> (Unconvinced about theoretical and empirical advantages of randomly sampling loss functions)\n\n**Reply.** For empirical evidence demonstrating superior of *ensLoss*, please refer to our reply to the first two points, which clarifies why CIFAR10 (and its performance) is not appropriate to be used or referenced in binary classification.\n\nFor theoretical justification, we supplement the theoretical advantages of *ensLoss* through the lens of Rademacher complexity. The Rademacher complexity plays a crucial role in determining the convergence rate of the excess risk (with smaller values yielding a faster rate).\n\nSpecifically, given a classification function space $\\mathcal{F}$, the Rademacher complexity of $\\phi$-classification are defined as follows:\n$$\n\\text{Rad}_{\\phi}( \\mathcal{F} ) := \\sup\\_{f \\in \\mathcal{F}} | \\frac{1}{n} \\sum\\_{i=1}^n \\tau_i \\phi (Y_i f(\\mathbf{X}_i) ) |,\n$$\n\nwhere $\\tau_i$ are i.i.d. Rademacher random variables. On this ground, the corresponding Rademacher complexity for the proposed ensemble loss method can be formulated as:\n$$\n  \\overline{\\text{Rad}}( \\mathcal{F} ) := \\sup\\_{f \\in \\mathcal{F}} | \\frac{1}{n} \\sum\\_{i=1}^n \\tau_i \\mathbb{E}_{\\Phi} \\Phi (Y_i f(\\mathbf{X}_i) ) | \\leq \\mathbb{E}\\_{\\Phi} ( \\text{Rad}\\_{\\Phi}(\\mathcal{F}) ),\n$$\n\nwhere the inequality follows from the Jensen's inequality.\n\nThis simple deduction yields a positive result, that is, the Rademacher complexity of the ensemble loss is no worse than the average based on the set of fixed losses. Notably, identifying an effective fixed loss is often challenging due to the varying data distributions across different datasets, which highlights the potential of the ensemble loss as a promising solution.", "claims_and_evidence": "I think the authors of the paper provides some good theoretical justification to the proposed method. However, I feel that empirical verification is weak. Specifically, according to Table 4, the authors of the paper only conduct experiments with CIFAR2, which is puzzling to me. Why don't we use CIFAR10? Moreover, the accuracies of the CIFAR2 results seems rather low. For instance, according to Table 6, the accuracy of CIFAR2 with ResNet34 is approximately 76%. Prior literature has shown that similar architecture achieves 90%+ accuracy on the CIFAR10 dataset. What is causing the huge gap? As such, I am not confused that the proposed method is preferred over BCE loss.", "methods_and_evaluation_criteria": "As argued above, I don't think the evaluation criteria makes sense for the problem.", "theoretical_claims": "I briefly went over the claims, but did not have time to go through all the proofs provided in the appendix. In general, the claims makes sense theoretically, and seems technically sound.", "experimental_designs_or_analyses": "Yes. As argued above, I am not confident about the experiments conducted by the authors.", "relation_to_broader_scientific_literature": "The proposed method is a general loss function that can be applied to training of all sort of machine learning models.", "essential_references_not_discussed": "I think the authors needs additional discussion about different loss functions proposed to train neural networks. While the idea of dynamically sample loss functions is new ,a lot of prior works have been proposed that changes the gradients of the loss functions. Some of the examples include: \n\n1. Lin, Tsung-Yi, et al. \"Focal loss for dense object detection.\" Proceedings of the IEEE international conference on computer vision. 2017.\n2. Leng, Zhaoqi, et al. \"Polyloss: A polynomial expansion perspective of classification loss functions.\" arXiv preprint arXiv:2204.12511 (2022).\n3. Zhang, Zhilu, and Mert Sabuncu. \"Generalized cross entropy loss for training deep neural networks with noisy labels.\" Advances in neural information processing systems 31 (2018).\n4. Ma, Xingjun, et al. \"Normalized loss functions for deep learning with noisy labels.\" International conference on machine learning. PMLR, 2020.", "other_strengths_and_weaknesses": "I think the idea of randomly sampling loss functions, while incremental arguably, is interesting. However, I am not fully convinced why, both theoretically and empirically, that the proposed loss function is better.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "Why do we want a loss ensemble instead of a fixed loss function from a theoretical standpoint? Any benefits that we can show?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9tMsoRZGN", "reviewer": "Reviewer_ecqi", "summary": "This paper considers the problem of binary classification, where it proposes a method called EnsLoss that combines with stochastic gradient descent (SGD) to obtain the optimization objective to train a classifier. The key idea of EnsLoss is based on the convenient classification-calibrated condition (Bartlett+, 2006) that the derivative at 0 is negative for a bounded-from-below convex loss is sufficient to state that a loss is calibrated. Based on this information, Ensloss tries to obtain a reasonable derivative that gives rise to a classification-calibrated loss. Theoretical analysis of classification and excess risk bound confirms the validity of the proposed method. Experiments show that with SGD, the proposed Ensloss outperformed using a fixed loss function (e.g., Logistic, Hinge, EXP) significantly, while it can also combine with other regularization schemes (e.g., dropout).\n\n## update after rebuttal\nI have read the rebuttal and appreciate the authors' response to my concerns. I think this paper has merits that outweigh its flaws, and I want to keep my score on the acceptance side. The contribution is limited to binary classification is acceptable to me as it is an important problem setting, and I feel the extension to the multi-class might require a further significant amount of work as the classification calibration condition is not as simple as binary classification.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (No clear explanation of why EnsLoss outperforms fixed losses)\n>\n> (Intuitive explanation)\n\n**Reply.** We appreciate the opportunity to clarify why ensLoss outperforms fixed losses.\n1. An intuitive explanation:\n- ensLoss is a method conceptually similar to Dropout, offering benefits through ensemble effects.\n- More importantly, for fixed losses, the only truly \"necessary\" condition is calibration, with potential overfitting arising from extra structures (e.g., hinge loss pushes classifiers to exactly +1/-1). ensLoss *not* require to satisfy a specific risk minimizer, instead seeking a good classifier across various random losses while maintaining calibration. This can be considered a form of *regularization*—we avoid pushing the classifier to excessively satisfy any particular loss, thereby reducing overfitting.\n\n2. While calibration and excess risk bounds are crucial to verify consistency, they are not necessarily informative about comparative performance on specific datasets. Distribution-free upper bounds are often not tight under particular datasets, making it difficult to determine which loss will perform better on a given dataset.\n\n3. We add theoretical support for ensLoss through Rademacher (Rad) complexity. The Rad for ensLoss is:\n$$\n\\overline{\\text{Rad}}(\\mathcal{F}):=\\sup\\_{f \\in \\mathcal{F}}|\\frac{1}{n}\\sum\\_{i=1}^n \\tau_i \\mathbb{E}_{\\Phi} \\Phi(Y_i f(\\mathbf{X}_i) )|\\leq \\mathbb{E}\\_{\\Phi}(\\text{Rad}\\_{\\Phi}(\\mathcal{F})),\n$$\nwhere $\\tau_i$ are i.i.d. Rad random variables, and $\\text{Rad}\\_{\\Phi}(\\mathcal{F})$ is the Rad of a fixed $\\phi$-classification. Hence, Rad complexity of ensLoss is no worse than the average of a set of fixed losses, showing at least one theoretical benefit.\n\n> (Ablation studies on \"bounded below\")\n\n**Reply.** We greatly appreciate this comment. This is precisely an important step when developing ensLoss: we also initially overlooked this condition, but discovered that is indeed critical.\n\nTo demo its effect, we reported typical train/test Acc for CIFAR2 (dog-cat) using VGG16 w/wo the bounded below (BB) condition (showing select epochs).\n|Epoch|w/o BB||w/ BB||\n|-|-|-|-|-|\n||Train Acc|Test Acc|Train Acc|Test Acc|\n|20|0.50|0.49|0.96|0.74|\n|60|0.50|0.51|1.00|0.83|\n|120|0.50|0.49|1.00|0.83|\n|160|0.51|0.49|1.00|0.83|\n|200|0.50|0.49|1.00|0.83|\n\n*Finding*\n- w/o: model fails to learn (random-level train accuracy)\n- w/: effective learning and prediction\n\n*Explanation*\n\nFrom a loss-derivative perspective, Line 13 discounts gradients for correctly classified samples ($z=yf(x)>1$), focusing optimization efforts on misclassified/boundary points—a standard loss design principle.\n\n> (Limited to binary classification)\n\n**Reply.** Agree, we will discuss the limitation and potential extensions, ensLoss can be extended to more general ML tasks with calibration conditions. Also, we focus on BC because:\n- The iff condition for calibration in BC is well-studied and straightforward to implement, allowing us to directly validate our ensemble idea without additional complications.\n- BC itself is a fundamentally important problem with tremendous applications. We believe that improvements are also highly significant.\n\n> (ensLoss requires longer time for stabilization)\n\n**Reply.** We agree this limitation and provide precise time comparisons. We report the *min epoch* for stabilization (after which train acc remains within an error margin of 0.005) on CIFAR2 (dog-cat).\n||BCE|Hinge|ensLoss|\n|-|-|-|-|\n|ResNet101|80|60|160|\n|ResNet34|70|25|145|\n|VGG16|15|25|55|\n|VGG19|45|35|150|\n\nAs indicated, ensLoss typically requires 2-3x more epochs to stabilize, which remains acceptable.\n\n> (Discussion on derivative-design for SGD)\n>\n> (Limited to SGD)\n> \n> (Suggestions on writing)\n>\n> (typos of $z$ in Algo 1)\n\n**Reply**. Agree, we will: (1) move Algo 2 into the main text; (2) add a description of dropout; (3) discuss derivative-design methods; (4) discuss the limitations of relying on SGD; and (5) correct typos: such as replacing $z$ with $\\gamma$.\n\n> (Superlinear for existing losses)\n\n**Reply.** All the mentioned losses (exp, hinge, logistic, and square) satisfy the superlinear tail condition (proofs omitted due to space constraints).\n\n> (The \"bounded below\" may be unnecessary if f is bounded)\n\n**Reply.** Yes, if $f$ is bounded, the condition can be relaxed.\n\nIn fact, for a convex loss, boundedness of $f$ is a sufficient condition of boundedness of $\\phi$. Assume $|f(x)| \\leq B$, $|z| = |y f(x)| \\leq B$. Since $\\phi$ is convex, it is continuous, thus $\\phi$ is bounded for $|z| \\leq B$.\n\n> (risk minimizer of the Ensloss)\n\n**Reply.** Thank you for this insightful question. Unlike fixed losses, ensLoss has no exact form of risk minimizer, as the minimizer would depend on the distribution of the underlying losses. Yet, as indicated in Theorem 3.6, the method is calibrated, ensuring that the minimizer of ensLoss shares same sign as $P(Y=1|x) - 0.5$, although we cannot determine its exact form.", "claims_and_evidence": "1. Methodologically, the proposed method is clear and convincing.\n2. The experimental results showed superiority of the proposed method. But I still feel there is no clear explanation why this method is better than a fixed loss. Since a fixed loss can also be classification-calibrated and has a reasonable excess risk bound.", "methods_and_evaluation_criteria": "1. Many datasets were used in the experimental result section. The evaluation criteria: accuracy and AUC (in the appendix) are reasonable.\n2. This paper kindly provides the section of compatibility of the proposed method with other regularization methods, which nicely shows the usefulness of Ensloss.", "theoretical_claims": "1. I briefly checked the proof and think the result is reasonable.", "experimental_designs_or_analyses": "1. Experimental results are reasonable.\n2. Unfortunately, it would be nice to have more ablation studies to highlight the importance of each condition, e.g., whether Line 13 is important (bounded below). In the empirical sense, I am not sure if this is needed, as function class is already somewhat bounded in practice.", "relation_to_broader_scientific_literature": "It is called Ensloss, but I also feel it is more like a design of objective that bypasses the loss design but goes straight to the derivative design for using stochastic gradient descent. Perhaps additional discussion on derivative-design for SGD for several problems might be nice.", "essential_references_not_discussed": "I do not have any additional suggestions.", "other_strengths_and_weaknesses": "Strengths\n1. Theoretically justified.\n2. Easy to implement under SGD framework.\n3. Paper is easy to follow.\n4. Many datasets were used in the experimental results, which I found impressive.\n\nWeaknesses\n1. Limited to binary classification\n2. Limited to SGD optimization algorithm\n3. The algorithm seems to require more training time to stabilize the training accuracy as noted in the conclusion section.\n4. I still feel it is unclear why Ensloss can significantly outperform logistic loss and hinge loss.", "other_comments_or_suggestions": "Overall, I think the paper is easy to follow.\n\nSuggestions on writing:\n1. Please include the discussion of Algorithm 2 if possible as it is important for understanding Algorithm 1. Or try to rewrite Algorithm 1 in a way that doesn't rely too much on Algorithm 2.\n2. I know dropout is very popular and common, but I think in the discussion of related work, there is no description at all how it works, i.e., randomly deactivates a subset of neurons during training.\n3. What is Assumption 1 in Theorem 3.6? Do you mean Assumption 3.4? Assumption 1 is also referred to in the appendix section.\n\nCurrent score before the rebuttal:\nOverall, I think the idea to make classification-calibrated loss without directly defining a loss for binary classification is interesting. The proposed method is reasonable and achieves good experimental results. I listed some weaknesses in the weaknesses section above.", "questions_for_authors": "1. I still feel it is unclear why Ensloss can significantly outperform logistic loss and hinge loss. Could you explain intuitively why this is the case? \n2. Do exp, hinge, logistic, squared losses satisfy superlinear raising tail condition? If not, then is there any well-known fixed loss that satisfies this condition? \n3, What is z (not bold) in line 15, I failed to follow this part. Is it a constant for all data points in the mini-batch?\n4. Regarding bounded below condition, I believe it might not be a big concern if we assume f is bounded. For example, there is an unhinged loss which is simply a linear loss $\\phi(z)=1-z$ that has robustness and is classification-calibrated if function is bounded (proposition 5: Learning with Symmetric Label Noise: The Importance of Being Unhinged: https://arxiv.org/abs/1505.07634). Do you find this condition highly important in the experiment?\n5. Can we know the form of the risk minimizer of the Ensloss? For example, hinge loss risk minimizer will be sign(p(y=1|x)-0.5), 2p(y=1|x)-1 for squared loss.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dty0iqR0Rv", "reviewer": "Reviewer_cfo1", "summary": "This paper proposed a novel ensemble framework, EnsLoss, for mitigating overfitting issue for binary classification task. The EnsLoss is motivated by Calibration property studied in the literature and combined with the idea of ensemble, which makes the 'equivalent loss' also calibrated for ensemble training process. Excess risk bound and empirical experiments are provided to justify the proposed method.\n\n## update after rebuttal\nI keep the same score as weak acceptance as before. The proposed method looks novel to me. And there are some theoretical justification and empirical experiments for supporting the method. Given that there are still some space for further improving from i) multiclass setting; ii) more theoretical (or better) justifications; iii) more empirical baselines for comparison, I don't raise my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (No theoretical evidence EnsLoss reduces overfitting - missing analysis of generalization error bounds or excess risk)\n\n**Reply.** Thank you for this insightful question. We do not explicitly provide the generalization bound, yet following your suggestion, we can show the advantages of *ensLoss* through the lens of Rademacher (Rad) complexity. The Rad complexity plays a crucial role in many concentration inequalities, so it will be helpful to infer the benefits in generalization bounds. \n\nThe Rad complexity of $\\phi$-classification is:\n$$\n\\text{Rad}\\_{\\phi}(\\mathcal{F}) := \\sup\\_{f \\in \\mathcal{F}}|\\frac{1}{n} \\sum\\_{i=1}^n \\tau_i \\phi(Y_i f(\\mathbf{X}_i))|,\n$$\nwhere $(\\tau_i)$ are i.i.d. Rad random variables. The corresponding Rad complexity for ensLoss is:\n$$\n\\overline{\\text{Rad}}( \\mathcal{F}) := \\sup\\_{f \\in \\mathcal{F}} |\\frac{1}{n} \\sum\\_{i=1}^n \\tau_i \\mathbb{E}\\_{\\Phi} \\Phi (Y_i f(\\mathbf{X}_i) )| \\leq \\mathbb{E}\\_{\\Phi} (\\text{Rad}\\_{\\Phi}(\\mathcal{F})).\n$$\n\nHence, Rad complexity of ensLoss is no worse than the average of a set of fixed losses, showing at least one theoretical benefit. Yet, it does not provide conclusive evidence of its superiority over fixed losses. We will honestly report the potential advantages and limitations in our revision.\n\n> (No numerical evidence EnsLoss reduces overfitting - training results should be shown alongside test results)\n\n**Reply.** We apologize for not including the training results. This was because almost all methods achieved a train accuracy of 1 (or close to 1), as demonstrated below for CIFAR2 (dog-cat) dataset. \n\n| Dataset  | Model    | Loss    | Train Acc | Test Acc |\n|----------|----------|---------|-----------|----------|\n| CIFAR2 | ResNet34 | ensLoss | 1.00 ± .00  | 0.72 ± .02 |\n|          |          | Hinge   | 1.00 ± .00  | 0.70 ± .01 |\n|          |          | BCE | 1.00 ± .00  | 0.69 ± .01 |\n|          |          | EXP     | 1.00 ± .00  | 0.60 ± .01 |\n|          | VGG16    | ensLoss | 1.00 ± .00  | 0.81 ± .01 |\n|          |          | Hinge   | 1.00 ± .00  | 0.78 ± .01 |\n|          |          | BCE | 1.00 ± .00  | 0.77 ± .01 |\n|          |          | EXP     | 1.00 ± .00  | 0.69 ± .00 |\n\nMoreover, in Table 6, the accuracy appears to decrease when the model becomes more complex (e.g., from VGG16 to VGG19). This is also a typical phenomenon associated with overfitting.\n\nTherefore, we reasonably conclude the improvement by *ensLoss* is mitigates overfitting not simply fits the data better.\n\n> (Limited to binary classification - extend to multiclass where calibration theories exist)\n\n**Reply.** Thank you for this valuable comments. We agree that extending our work would be beneficial. Our current focus on binary classification is deliberate for several reasons:\n\nFirst, the if-and-only-if condition for calibration in binary case is well-studied and straightforward to implement, allowing us to directly validate our ensemble idea without additional complications.\n\nSecond, as indicated in Theorem 7 of [1], the iff calibration conditions for multiclass case are substantially more complex. Implementation would require additional specifications in the formulations and sufficient conditions for calibration, as shown in Section 4 of [2] and Section 5 of [1]. While extension is possible using simplified conditions and specific formulations (Prop 15, [1]), yet there remains ongoing debate regarding optimal formulations. Binary case allows us to avoid these debates and directly compare the core ensemble idea.\n\nFinally, binary classification itself is a fundamentally important problem with many applications. We believe that improvements are also highly significant.\n\n**References:**\n1. Tewari & Bartlett (2007). On the Consistency of Multiclass Classification Methods. JMLR.\n2. Zhang (2004). Statistical analysis of some multi-category large margin classification methods. JMLR.\n\n> (No comparison with existing ensemble methods)\n\n**Reply.** Agree. To illustrate, we report test performance for model averaging (m-ave) and majority voting (m-vote) on CIFAR2 (dog-cat) using ResNet101 over three fixed losses:\n\n||Acc|AUC|\n|-|-|-|\n|BCELoss|67.33(.23)|73.35(.24)|\n|EXP|53.62(.48)|54.86(.35)|\n|Hinge|67.12(.32)|68.60(.66)|\n|**m-ave**|68.14(.25)|74.35(.26)|\n|**m-vote**|67.93(.26)|---|\n|ensLoss|**70.07(.90)**|**76.60(.87)**|\n\nAs shown, ensembling indeed improves single fixed loss methods, but enLoss still performs better. We observed these conclusions across datasets and will add details in the revision. \n\nThe challenge with ensembling methods is that selecting too many fixed losses requires substantial computational costs of training, while combining too few losses produces less effects. A key advantage of ensLoss is that it requires only a single model training.\n\n> (the fonts on Fig 1 and 5 are too small)\n\n**Reply.** Thank you for comments. We will revise them in the revision.", "claims_and_evidence": "One claim that is not convincing to the reviewer is: the EnsLoss can mitigate overfitting issue.\n\nDoes EnsLoss enjoy better generalization error bound? Or is the excess risk bound better than fixed single loss?\n\nEven for the empirical experiments, there is no demonstration that EnsLoss mitigates overfitting or simply fits the data better. Maybe empirically the training performance should also be presented for supporting the claim.", "methods_and_evaluation_criteria": "Yes, the evaluation generally makes sense to me. \n\nThe only concern is that there is still a gap to the paper claim 'mitigate overfitting issue'.", "theoretical_claims": "I didn't check the theoretical proof thoroughly in the appendix but the flow in the main contents looks correct.", "experimental_designs_or_analyses": "The experimental design and analyses generally make sense to me. \n\nHowever, as mentioned previously, I don't see much connection and justification to 'mitigating overfitting issue'.", "relation_to_broader_scientific_literature": "The paper contributes a novel ensemble method for binary classification motivated by Calibration property.", "essential_references_not_discussed": "I didn't noticed any missing essential references.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper writing is well organized and the flow is clean. \n2. The motivation is clear and the proposed ensemble method should be novel.\n3. Some theoretical justification and empirical results are provided.\n\nWeakness:\n1. Neither the theoretical nor empirical results can clearly support that the proposed method can mitigate overfitting issue. There are still a gap to justify the claim (also see previous 'Claims And Evidence').\n2. The binary classification application is restricted. Given that the multi-class classification also have been developed calibration theories in the literature, this work can enjoy higher impact if can be extended.\n3. As a novel ensemble method, there is no comparison with other existing ensemble methods. We don't know whether the proposed ensemble method is better, nor if it is compatible with existing ensemble methods.\n4. Minor concern: the fonts on Fig 1 and 5 are too small to be viewed clearly.", "other_comments_or_suggestions": "Based on my previous comments:\n1. It would be better for the paper to justify the claim 'mitigate overfitting issue' more clearly.\n2. It would be more impactful to extend the method on multi-class classification setting.\n3. It would be better to study/show the relations with existing ensemble methods, in order to fully demonstrate the impacts or limitations for the proposed method.", "questions_for_authors": "Please see previous comments, thanks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper extends classification-calibrated losses by allowing the "stochasticity" in the choice of loss functions. In other words, a binary classification can be a random variable so that we generate its gradient (based on the Box-Cox transform) in backward. This can be thought as the "ensemble" of loss functions. The authors provided conditions to ensure this loss is classification-calibrated, and extensive experiments to demonstrate superior performances over conventional "non-ensemble" loss functions.

During the author-reviewer discussions, there were several important discussions sparked. Specifically, some reviewers argued that it is restrictive if we can only apply to binary classification; others argued that it is theoretically unclear why the ensemble loss is a better choice. Even so, AC believes the core idea to introduce the stochasticity to a loss function itself deserves the acceptance. Indeed all reviewers agree on this point. Hence we suggest for the acceptance.

The authors are expected to reflect the rebuttal discussions to the camera-ready version carefully. (Nit: AC is not fully convinced with the argument on the Rademacher complexity as well, but this does not mean a bad thing. To speak of extremes, we could make the loss Lipschitz constant arbitrarily small by multiplicative scaling. Probably we need a better analysis in future)
Decision: Accept (poster)