Reweighting Improves Conditional Risk Bounds
Authors: Yikai Zhang
Jiahe Lin
Fengpei Li
Songzhu Zheng
Anant Raj
Anderson Schneider
Yuriy Nevmyvaka
Venue: Accepted by TMLR ()

Abstract: In this work, we study the weighted empirical risk minimization (weighted ERM) schema, in which an additional data-dependent weight function is incorporated when the empirical risk function is being minimized. We show that under a general ``balanceable" Bernstein condition, one can design a weighted ERM estimator to achieve superior performance in certain sub-regions over the one obtained from standard ERM, and the superiority manifests itself through a data-dependent constant term in the error bound. These sub-regions correspond to large-margin ones in classification settings and low-variance ones in heteroscedastic regression settings, respectively. Our findings are supported by evidence from synthetic data experiments.

Published in Transactions on Machine Learning Research (01/2025)
Reweighting Improves Conditional Risk Bounds
Yikai Zhang
Yikai hwn.::/hannnnnngggnannnne......
Machine Leorning Research, Morgon Stanley
Jiahe Lin
Juate.Lin@mmorggnellnlleyy
Machine Learning Research, Morgan Stanley
Fengpei Li
Fengpei. Fenques. Bimpgpps_LBBmmrrgnnsssaaneeeee com
Machine Learning Research, Morgon Stanley
Songzhu Zheng
Songhha.ZZeng@immyyanndddaagg
Machine Learning Research, Morgon Stanley
Anant Raj
Bhant.RRgfBmorrgnnstanneyyy com
Machine Learning Research, Morgon Stanley
Anderson Schneider
Snderoo..SShaeddrrrrorrynssseeee com
Machine Learning Researeh, Morgon Stanley
Yuriy Nevmyvaka
Machine Learning Research, Morgan Stanley
Reviewed on OpenReview: https: // openreview. net/fooune Ld-ModddudHuu
Abstract
In this work, we study the weighted empirical risk minimization (weighted ERM) schema, in which an additional data-dependent weight function is incorporated when the empirical risk function being minimized. We show that under a general "balanceable" Bernstein condition, one can design a weighted ERM estimator to achieve superior performance certain sub- regions over the one obtained from standard ERM, and the superiority manifests itself through data-dependent constant term in the error bound. These sub-regions correspond to large-margin ones in classification settings and corvaarianc ones in heteroscedastic regression settings, respectively. Our findings are supported by evidence from synthetic data. experiments.
1 Introduction
The empirical risk minimization (ERM) schema plays important role in tackling modern machine learning tasks. Given set of samples S, {z, & (æ,, b)) that are often assumed i.i.d., ERM estimates the target hypothesis within some hypothesis class F by minimizing the empirical risk based on Sn, that is, I 2 arg min 1 Σz,es, zi), where é is some loss function. The method is widely adopted due to its case of use and generalization power.
Let A (x.v) and f A arg min E(((f,))) 2)], the encess risk defined as:
Excess Risk A
Under suitable conditions, ERM achieves low excess risk with high probability. Specifically, it is known to achieve a minimax optimal rate of Onn11/2 for lcarning tasks under general settings (Vapnik & Chervonenkis,
Correspondence Yikai Zhang. (zhangyile) i Saagmail.com. #: equal contribution

---

Published in Transactions on Machine Learning Research (01/2025)
1974; Talagrand, 1994; Boucheron et al., 2005), and an O(n fast rate in more restrictive ones (Koltchinski & Panchenko, 2000; Mendelson, 2002a; Bartlett et al., 2005; Boucheron et al., 2005), where matching lower bounds have also been established (Massart & Nédélec, 2006; Zhivotovskiy & Hanneke, 2018).
More recently, alternative risk minimization schema that outperforms ERM on certain sub-regions has been proposed in several works. Notably, Namkoong & Duchi (2017) introduces a distributionally robust. optimization schema that is provably superior to ERM in scenarios where ERM is susceptible to noise; e.g., of the slope has sizable impact to the solution. In Xu & Zeevi (2024), carefully designed moment penalization function is introduced and it achieves superior rate than ERM within the large-variance hypothosis class. Bousquet & Zhivotovskiy (2021) introduces a rejection option to achieve fast rate of convergence in the absence of margin conditions; note that these conditions are typically required for establishing fast rate for ERM in settings absent of the rejection option. The aforementioned studies point to different directions of improvement over the standard ERM.
when the loss is piecewise linear and the thus under monder/overeestmatte
In this work, we leverage the problem-dependent structure to improve upon ERM, which provably outperforms in "high confidence" regions. Specificallyy, instead of minimizing directly the empirical loss, re-weighted version based on some weight function w(-) is considered, which leads to an estimator that minimizes the weighted empirical risk of the following form:
1 Σ FRRRM - arg min Jer - 1= w(xx)e(fiz;)
(1)
We show that such an estimator improves the conditional excessive risk, given by
Ex (((F:))
> et).
with the conditioning region characterized by level set associated with the data-dependent weight. Such an improvement can be of interest to practitioners in settings akin to those considered in the selective learning literature, where the focus is on model outcomes associated with selected sub-regions.
Our approach is inspired by an analysis of a Berustein-lype condition of the form Var[h(=)] S [E[] for all h in some function class H (e.g., Bartlett & Mendelson, 2006), which is often required for local Rademacher complexity-based analysis (Bartlett et al., 2005). In particular, one way to derive the O(n fast rate is to leverage some variance-aware inequalities such the Bernstein inequality (Boucheron et al., 2005) and Talagrand's inequality (Talagrand, 1994; Bartlett et al., 2005), which gives rise to intermediate results of the form:
-((((=))((((+)))) < _ Σ - 21) + +av Varl((f.)) e(f", x)] b Z.ES.
(2)
for some problem-dependent constant a and b. Under Berustein-type condition, the variance term on the [[[",))] The multiplier B is usually chosen in a
right-hand side of inequality (2) is replaced by [E, conservative way; for example, it is chosen as the inverse of the minimum margin present in the Massart noise condition (Massart & Nédéloc, 2006) for classification problems or the uniform upper bound of regression loss in bounded regression settings (Boucheron et al., 2005; Bartlett et al., 2005).
On the other hand, a vanilla conservative choice of in Var[h] < BE[4] is not necessarily optimal. Under the weighted ERM schema where carefully designed weighted empirical risk in (1) is considered, the Bernstein condition could be "balanced" as: (arlw(x)/((z))) < (l=()))((z))). Crucially, the constant B can be eliminated, which subsequently leads to tighter bound (up to some problem-dependent constant) in eacess risk in certain sub-regions. Specifically, within the context of classification and heteroscedastic bounded regression settings, these regions can be characterized as the large-margin and the low-variance ones, respectively. More generally, the conclusion is applicable to learning tasks whose loss function satisfies Balanceable Bernstein Condition when the weight function can be designed accordingly.
Contribution
The major contribution of this paper can be summarized as follows.
2

---

Published in Transactions on Machine Learning Research (01/2025)
We consider the weighted ERM schema and investigate its theoretical properties; in particular, we show that it admits a tighter bound on excessive risk up to some problem-dependent constant, conditional on specific sub-regions. The enhanced conditional excessive risk bound also implies an improved unconditional excessive risk bound, provided that the noise satisfies certain conditions. See Table 1 for comparison.
achieve superior performance in selected regions, respectively for heteroscedastic regression and classification tasks.
Empirically, we demonstrate that with properly designed weight function,
The proofs of the theorems rely on an alternative version of Theorem 3.3 from Bartleit et al. (2005), which is based on a relaxed Bernstein-tyy type condition that allows for an E additive error. The exact arguments and strategies adopted are technical tools that of independent interest to the community.
As a by-product, the O(1/n) fast rate for learning the variance function (²x) derived in the regression example (Theorem 4.6) improves over existing results in Zhang et al. (2023) that has an O(1/Vn) rate.
ERM
Weighted-ERM
Classification
risk function
+(f=2) A ((((m) of v)
w w.(((z)) (f()) M
general Setting
5
- s
large margin region
P R| PIR
A10(f=))((('1 =) PIRT
2 (w=( > c)
low-margin diminishing
[.[((]:)) -000 ;('"; $
= [(f"= 5
(1 - P[R()))) Se
Regression
risk function
8(:=) A (f(x)) - v)²
w w(((f=)) e (f(=)) (a²
general Setting
E.[((fi=) ((f"; S Se/7
E.]((f:=)- -(=;)) [["" $ s/7
low variance region
[a(((f=) :]][]] S $ F())
=.(((==)-(( = & fo F(R)
è (a**(a) 1/c) largo-variance diminishing P(R))e S
Table 1:
Comparison between existing results and ours. Let v), represents results that are implied by Massart & Nédélec (2006), where the excessive risk bound is minimax optimal, represents results that are implied by Bartlett et al. (2005), represents results in this work. Throughout the table, all results are derived given sample size 6()). classification task, (*) is the Bayes optimal classifier and (=xx)) the margin function calculated as 2P [x)]]= 1. @(x) is the approximated margin and represents the minimum margin. For the regression task, (=) B[y|a=]. o²* (=)) Var(ylæ= and (x)) is an approximation of (xx). 1/7 is taken to be the maximum possible variance. We denote PP-) as the probability mass of a set of events. For the general region, weighted ERM recovers least the same rate ERM. For the large-margiu region low-variance region, weighted ERM improves ERM by (c/T) > 1 on the conditional excessive risk bound. In the case of classification we provide a lower bound construction showing the tightness of the conditional risk (Theorem 4.4). The improved conditional excessive risk of weighted-EEie ERM also implies an improved original excessive risk under low-margin or large- variance diminishing condition. To clarify, not necessary for the value of e to be 'large? Instead, having e as a constant, such as 0.1, is sufficient, especially when 7 approaches a vanishing value, e.g., when is of order VE.
2 Related Work
In this section, we provide brief literature review of related work. On the theory front, review some representative results in standard ERM, under both the slow rate and the fast rate regimes; on the methodology front, we establish connections to existing literature where Gaussian maximum likelihood estimation (MLE) is performed in heteroscedastic regression settings, and note that MLE coincides with weighted ERM when & loss is used with the inverse variance function being the weight. Finally, given the connection between the results established in this work and those selective learning, we also briefly review results for the case where a rejection option is allowed.
3

---

Published in Transactions on Machine Learning Research (01/2025)
Empirical risk minimization The combination of VC-tool and Rademacher complexity analysis (Vapnik & Chervonenkis, 1974; Koltchinskii & Panchenko, 2000; Mendelson, 2002b; Boucheron et al., 2005; Bartlett. et al., 2005) constitutes one of the most widely-adopted frameworks in deriving risk bounds for ERM. Under the "slow rate" regime, its generalization error bound is at the rate of O(n (/n=1/3), and such rate can be established using a uniform convergence argument (Vapnik & Chervonenkis, 1974; Talagrand, 1994; Boucheron et al., 2005) over an "unrestricted" function class. On the other hand, fast rate can be achieved if one moves away from an "arbitrary" hypothesis (Mendelson, 2002a; Boucheron et al., 2005; Bartlett et al., 2005). By restricting to subset of the function class that has small variance and considering the Rademacher averages associated with this small subset (i.e., "localized") as a complexity term, sharper bounds can be obtained vis-à-vis the case where global averages are used. In particular, under a Bernstein-type condition (Massart & Nédélec, 2006), fast, rate up to O(r²²) can be obtained in well-specified (or realizable, equivalently) settings where the optimal model lies within the hypothesis class. For binary classificatinn, this condition can be satisfied by imposing Massart or Tsybakov noise condition (Mammen & Tsybakov, 1999; Tsybakov, 2004; Massart & Nédélec, 2006). The same fast rate can be obtained for bounded regression (Bartlett al., 2005; Massart & Nédélec, 2006) or learning problems whose loss function satisfies strong convexity and Lipschitz condition (Klochkov& & Zhivotovskiy, 2021), or when it is self-concordant (Bach, 2010) exp-concave (Koren & Levy, 2015).
Maximum likelihood estimation and weighted ERM The weighted empirical risk minimization has been adopted to tackle several challenges in machine learning including distribution shifting (Cortes et al., 2008; 2010; Ge et al., 2024), censored observation (Ausset et al., 2022) and reinforcement learning (Jiang & Li, 2016; Xie et al., 2019; Min et al., 2021). In particular, in heteroscedastic regression settings where the variance depends on the imput, a negative Gaussian tikelihood-basee formulation coincides with weighted ERM, when the latter uses &a loss and the inverse of the conditional variance as the weight. In particular, in the view of weighted ERM, samples with higher conditional variance (and potentially more noisy) are doin-weighted and therefore contribute less to the overall empirical risk. The conditional variance can be estimated using either parametric models (Daye el al., 2012; Zhang et al., 2023) or kernel methods (Cawley et al., 2004). Despite the wide applicability of such formulation (Kendall & Gal, 2017; Lakshminarayanan et al., 2017; Shah et al., 2022; Seitzer et al., 2022), little has been done in comparing the sample efficiency between estimators based on the weighted and those of the standard ERM. In this work, we aim to fill this gap, by explicitly analyzing the sample efficiency in the weighted ERM case and juxtaposing its performance to that in the standard ERM.
Reject option and selective risk Along a line of work that slightly deviates from the ERM is the learning with rejection option; e.g., Chow's reject option model (Chow, 1970). The model allows one to refrain from making predictions on "hard" instances at the inference stage with some abstention cost; better precision is obtained in exchange for lower coverage, and the selective risk is only evaluated on the covered subset. Such learning schema has applications in domains such as finance (Pidan & El-Yaniv, 2011) and health care (Hanczar & Dougherty, 2008), and can be generalized to other tasks (Mozannar & Sontag, 2020). Extensive work has been done to understand the trade-off between coverage ratio and precision (Herbei & Wegkamp, 2006; Bartlett & Wegkamp, 2008; Yuan & Wegkamp, 2010; EI- Yaniv et al., 2010; Cortes et al., 2016), along with the statistical properties (Bousquet & Zhivotovskiy, 2021; Zhang et al., 2024) associated with the learning procedure. At the conceptual level, weighted ERM can be viewed as a "soft" counterpart to learning with the rejection option. In particular, instead of adopting a hard "in-or-out" rule to improve selective risk, performing weighted ERM can lead to similar benefit in soft manner, with the weight typically coming from plug-in in estimates (Herbei & Wegkamp, 2006; Bartlett & Wegkamp, 2008; Franc et al., 2023). In practice, the weight can be the estimated margin or the inverse variance function, in classification and regression settings, respectively.
3 A Road Map
Notation We denote vectors by bold-faced letters (e.g., x)) and scalar variables by lower-case regular ones (c.g., y). Let B) be a probability space, where X' denotes the sample space, the probability measure and B the Borel ffeld. The expectation with respect to the probability distribution of random
4

---

Published in Transactions on Machine Learning Research (01/2025)
vector X' is denoted by Exlt]; the subscript is omitted whenever there is no ambiguity in the respective context. For random vector z (at, v) defined on the product sample space 2 _ X x y (equipped with the corresponding Borel o-field), its probability measure is denoted by I.. We use 2 ~~ D to denote that = is sampled from data generative process (DGP) D. Throughout the remainder of this manuscript, we use æ, with a subscript to denote the ith random sample drawn from X; ys and 2, are analogously defined. We also denote W X' >> [0, ci] as hypothesis classes with finite peudo-dimension dp < oc. Finally, we use Ô(.) and Ó()) to denote counterparts of the standard ())), notation while suppressing the poly-logarithmic factors; the notation f 2 g means f 2 eg for some universal constant c, and & is analogously defined. We say 9 if cg S S 0g for some universal constants ē and
Problem setup Consider sequence of i..dd samples 21.n (z, (a;; v:)) drawn from sample space Z, with a;'s being the imput and yi 's the output; let f: X' + V E F and be some loss function that is bounded and Lipschitz. In this study, we focus on analyzing the excess risk of the ERM and the weighted ERM estimators (see definitions below) on certain sub-regions.
Definition (Empirical risk and the FRM estimator). Given Ziza and f FEF, for loss function F:Fx 2 +
[0, d), we define the empirical loss as
LERM(fiZLM) = 1T 1,E,, (ffiz));
the ERM estimator is given by
fEUM e arg min LERM arg min fer
n En ((fizi).
(3)
fEF
Definition 2 (Weighted empirical risk and the weighted ERM estimator). Given Zimn f E F and some weight function w E W: X + [0,ei], for loss function é: F x Z + [0,a], we define the weighted empirical loss as
TEERM (f:
Σ (f:))((;
the weighted FRM estimator is correspondingly given in the form of
FWERM A arg min LWERM = arg min + ( ((fiz))(((z))
(4)
JEF
JEF
i=l
We provide brief overview of the main results established in Section 4. As briefly mentioned in Section 1, under the weighted ERM schema, an improved bound can be derived by encapsulating the weight function inside the appropriate terms of the Bernstein-type condition, namely,
Var [M] < BE
Varlw()))(<z)) s ([w()))(()))
(vanilla Bernstein-type condition)
(balanceble Bernstein condition)
Such an "encapsulation" step does not alter the rate of O(n"1) in the risk bound; however, the constant which the bound depends will change, which then leads to tighter bound for selected regions, manifested through an improved problem-dependdnt constant. This selected region is determined by the ratio B/(Bww(x)) both B and B' polentially depend on some y that characterizes the uniform lower or upper bound of the weight, depending on the setting (classification or regression). Finally, note that a Bernstein-type condition can be satisfied by imposing some margin condition or boundodness, classification setting with 0-1 loss and heteroscodastic regression settings with & loss, respectively.
To derive the desired risk bounds, adopt the following road map and formalize the arguments in Section 4.
We first establish the risk bounds of the weighted ERM estimator (in Theorem 4.1 for classification and in Theorem 4.5 for regression, resp.), assuming that the weight function (xx) used in the estimation is sufficiently close "true" one w² (z). Note that w--) coincides with the margin function in the case of classification and the inverse of the variance function in the case of regression.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Subsequently in Theorem 4.3 (for classification) and Theorem 4.6 (for regression), we provide the risk bound for (x)), which shows that the sample complexity for estimating w(x)) is comparable to that of learning the (-). This justifies the aforementioned assumption on W in that such an assumption can be operationalized without rendering the cstimation "harder".
Finally, for the classification case, we additionally show that the established conditional excess risk bounds for the weighted ERM estimator is tight, by providing lower bound result that matches its conditional (i.e., in sub-regions) excess risk upper bound. To contrast, the ERM estimator is provably inferior to the weighted one in terms of the lower bound in such sub-regions, despite of its minimax optimality in the entire region.
Note that the above result points to the fact that the weight function (i.c., the margin function) we consider is optimal, as manifested by a minimax optimal conditional excessive risk bound by adopting such a weight function.
Note that results point to the possibility of leveraging weighted ERM to achieve superior performance in certain regions, provided that the weight function is carcfully designed and the region is chosen accordingly. See also results presented in Section 5 based on synthetic data experiments that altests to our theoretical claims, as well as how an estimate of the weight function can be readily obtained from data, and the weighted ERM schema be operationalized in two steps.
4 Main Results
Before stating our main results that applicable to more general setting, we first present. results for two specific ones, namely, classification under margin condition (Section 4.1) and bounded heteroskedastic regression (Section 4.2) scttings. Our result suggests that weighted ERM can improve the standard ERM error bound by a problem-dependent constant, in regions with high margin in the former and those with low variance in the latter. In Section 4.3, we present our main results for a more general setting; specifically, we show that a similar property for weighted ERM holds, as long as the loss function under consideration satisfies "balanceable" Bernstein type condition.
4.1 Classification with/without margin condition
We formalize the classification problem considered in this paper, which largely follows from the general setting benign baesart/benign label noise (Massart & Nédélec, 2006; Hanneke, 2009; Diakonikolas et al., 2019).
of Massart/
Let F (f: X' [-1,1}} be a hypothesis class with finite VC-dimension* d < xc, g a {n X [o, }}} be a hypothesis class with finite pseudo-dimension dr( < 0o, and W & {w : + [7, 1]} be some other hypothesis class with some fixed constant S 7 1.
The data generative process (DGP) for & (x.y) can be characterized as follows:
with probability ()(xx) +
{11
CEM X;
g/
with probability 1 1 (x)) where n²(x)) a P(y =1/x).
(5)
For the family of DGP described in (5), (x) captures the conditional probability. The label y can be oquivalently viewed as satisfying y/ır 2Ber (Ber(1"(x)) (z)) - 1, where Ber(p) denotes a Bernoulli random variable with success rate p. Let f be the target hypothesis the Bayes optimal classifier, defined as
f (x) e max Ply = cc cE(-1,1)
"The VC dimension of F (f X (-1,1)} (al1))in is the largest integer d such that Sx(d) 2, where S)(k) is the value of the growth function, namely, the largest cardinality (o))..) f(x)) I F} among all æ1. E X (Vapnik & Chervonenkis, 1971).
'The pseudo-dimension of G (g: :g:x+ x+ u]] is the VC-dimension of the hypothesis class (h X (-1,1) [M(x))) = sign(g(>) f).g E G,t. E R. E X3 (Pollard, 1990).

---

Published in Transactions on Machine Learning Research (01/2025)
that is, (x) is the value of E {-1,1} that maximizes the conditional probability, which also satisfies (a(x)) = 21 (m) 2 }} 1. Let w* be the feature-dependent margin function given by w=(x) _ 2 PPy = (*))) - and note that (x(x) = (x) - 1/2 Throughout the remainder of this subsection, we consider the well-specified sctting (Massart & Nédélec, 2006; Bousquet & Zhivotovskiy, 2021) by assuming f E F, y" EG and w² E W.
We choose ((f = =(f(a) + y) as the loss function and propose to use the margin function w" (x) as the weight to perform weighted ERM; this leads to a weighted risk of the form w" (f() # y). In practice, when (x) is not available, one can use its estimate while still achieving similar results under mild assumptions.
We first give an informal overview of the results established. Typically, a standard margin condition (Massart & Nédélec, 2006; Bousquet & Zhivotovskiy, 2021) requires the minimum margin to satisfy y - inf (x) > O. For standard ERM, Bernstein-type condition (Bartlett et al., 2005) is satisfied in the form of:
| + f(x)) - 1 + (x))) & (1/4) E2[(1 (v + [())) - fx)))
whereas in the case of weighted ERM, it is satisfied in the following form:
Var (x) + f(x)}
1y # ('()i s E=(w"()) + f(x)) - 1 f*
(6)
Crucially, in the latter case, the 1/7 factor is removed, which subsoquently leads to an improved bound. In particular, Equation (6) does not require the margin condition 7 > 0 (Massart. & Nédélec, 2006), i..., Y can be zero; this suggests that even if the vanilla empirical risk does not satisfy the Bernstein condition, it is still possible to utilize a weight function to establish a Bernstein-type condition. Theorem 4.1 formally states this result.
Theorem 4.1 (Risk Bound for the case of Classification). Suppose that we have w(-) E W s.t. Bal( - (x))) S & is salisfied. Let Sn {(a,, y)) be i.i.d. samples drawn according to the DGP described in (5), and they are independent of the ones used for estimating w. Let FWERM be the weighted ERM estimator as defined in (4), with the weight. subsliluted by (xx;)) 's. Then the following hold simullaneously with probability at least - ô for some E > 00,8 0:
2) ((f" Se,
E.( = SE.
(7)
provided that the somple size satisfies 2 dyc(F) +lgg!!
The next theorem shows that there exists a DGP that conforms with the description in (5), such that when the estimation is performed based on samples drawn from it, the risk of the weighted ERM estimator on some sub-region can be arbitrarily close to zero, provided that the sample size grows commensurately; on the other hand, the risk of the ERM estimator on the same region is bounded below by a constant, irrespective of the sample size. Here the sub-region is characterized by large- margin level set, and the risk this sub-region can be viewed as the selective risk.
Theorem 4.2. There ezists a DGP that belongs to the DGP family satisfying (5), such that under the same assumplion as in Theorem 4-1, when the estimation is performed based on i.dd. samples S, = {( v)) W)) drawn from it, the following hold simultaneously for the ERM estimator (as per defined in (3)) and weighted ERM estimator (as per defoned in (4) with the weight function substituted by ((x.) 's), with sample size (4dlog( (og(
n 78
with probobility at least 0.1, Ex EL Hw) 7 2 0.015;
with probability at least 1 1 8, Ex [[[FEERM # f Hw (x) > > S
Remark 1 (On the bounds established). Some discussion of the implications of the bounds established in Theorems 4.1 and 4.2 is provided next. First note that the low oow/hgh margin region can be characterized through {a: (xx) < c} (and fa: w=x)= > c), resp.): the encess risk bound, given by [2 + - * yłl, can be further derived for weighted ERM based on (7). This enables a direct comparison between the bound of the weighted ERM and that of ERM, depending on the property of Pa) S c).
7

---

Published in Transactions on Machine Learning Research (01/2025)
Improved bounds in the large-margin region. For any CE [0, given large-margin region characterized by (æ: wx( > of with P(w*(()) > c) > 0 and ( samples, the risk bound of an
ERM estimator (e.g,, Massart & Nédélec, 2006, equation (7)) implies the following excess risk within the region:
Ex[A[fERM(z)) + v)-11ff(at & vx) 2 20s 7P((w(x)) c)
E
for the weighted ERM estimator, it achieves
Eg (fMEENN()))) & リ) f 2 cP(w" (x) > c)*
(8)
which improves the ERM bound by a factor of (7/c). In particular, in Theorem 4.2, a lower bound construction for the conditional risk on the largo-margin region presented, to demonstrate that there exists scenario where the ERM estimator fails with constant probability, while the weighted ERM one achieves standard PAC guarantee (Valiant, 1984). Note that under Chow's rejection rule (Chow, 1970) which optimally balances the trade-off between coverage and accuracy, the non-reject region coincides precisely with the large-margin region described above.
These analyses establish the benefit of the weighted ERM procedure where data is weighed by the margin function w (x)) In particular, its major advantage lies in the region where e is large compared to 7/ and such advantage vanishes as (7/c) + 1. Later in Theorem 4.4, a pathological scenario achieving a matching lower bound is presented, which implies the minimax optimality of the bound presented in (8).
Improved bounds under the "low-margin diminishing" condition. The low-margin dimin- ishing condition holds when there exists e such that (*= S c)e² S. One can view the set {=/W> (z) VE} as the collection of æ with "low margin", whose corresponding y's have high label noise. The condition (x) S c)c² < & describes settings where the low margin set has diminishing mass. Under such a condition, Equation (8) implies the following improved excessive risk bound:
Ex [L +) 11 + s/cc,
which enjoys improvement of factor 7/c when compared to the excessive risk of ERM. The condition P(w (x) c)c² < could be satisfied when e S VE and P(w (x) I c) 0.1. The derivation is presented in Appendix A.2.
Fast rate with/withon margin condition. Under standard margin condition 7 0, based on the Corollary 3 from Massart & Nédélec (2006), both ERM and weighted ERM achieves a E=lw((x) + y) 1(f + y})] S excessive risk with @()) samples, which is the information- theoretic optimality (Massart & Nédélec, 2006; Zhivotovskiy & Hanneke, 2018). Notably, Theorem 4.1 implics such result given the fact that wn (a) 2 7.
When the standard margin condition is not satisfied, c.g., for cases x E X with 1)) = 0.5, the corresponding effectively becomes a pure noise. In this case, a fast rate of 0(1/n) cannot be attained due to these "extremely noisy" data points. However, it is still possible to attain a fast rate with a slight modification of the vanilla empirical risk. Specifically, in Theorem 2.1 of Bousquet & Zhivotovskiy (2021) establishes risk bounds that enjoy a fast rate under Chow's rejection option framework (Chow, 1970), without requiring margin condition. This is achieved by introducing an "artificial" margin through the inclusion of rejection option, which allows for the removal of those "extremely noisy data. points" during both the learning and inference processes. Similarly, the weighted-ERRe ERR approach follows similar principle by utilizing a weight function wxx to down-weigh such extremely noisy data points with (x) 0.
Both Theorems 4.1 and 4.2 require that the surrogale @ be reasonably close to w", and such a condition is presented in the form of (x) - - - ) < E. One may wonder the task of approximating wt is statistically more challenging than lcarning the classifier itself in terms of sample efficiency. Theorem 4.3 addresses this concern and suggests that under standard assumptions, the required condition holds with high probability and the sample complexity of learning (-) is comparable to that of learning f*())
8

---

Published in Transactions on Machine Learning Research (01/2025)
Theorem 4.3 (Risk bound for estimating w*). Given i.i.d. samples Sy {(x ye) drown from the DGP described in (5), let fj arg min n. Zz,s, (n(a;) - v.)2. Then the following holds with probability at least. 1 1-0 ô for some E > 0,8 8 > 0:
Em (m())) 1/" (x)))² S,
dp(S) log( :) Further, let ((x) = 17(x)) 1/21, and the
provided that the sample size satisfies n 2 above result further implies that
Em (() - wn (x)) Se.
Remark Both Theorems 4.1 and 4.2 effectively assume the availability of a sufficiently-well estimated weight function (.),, and that the subsequent weighted estimation procedure is carried out on samples S, independent of those used for obtaining (-)). In practice, this can be operationalized via sample-splittin: namely, one equally splits the training set into two halves at random, and uses one half for weight estimation and the other half for obtaining f. Such a procedure would increase the generalization error bound by a factor of two, in exchange for the problem-dependent constant improvement in the large margin sub-region.
The following corollary be derived by combining Theorems 4.1 and 4.3, which takes into account all the randomness embedded in the training samples:
Corollary 1. Under the assumptions in Theorems 4.1 and 4.3, let w be the defined in Theorem 4.3, then the following hold simultannoosly with probability at least 1 - 25 for some E > 0:
(z) z)
s.
Ez (e((())(((F 2) ;(f] S..
(10)
Next we present our lower bound results for the conditional excessive risk bound.
Theorem 4.4. There erists set of DGPs that is in accordance with the description in (5), such that for i. samples S, = {(x,, " drawn from (arey) such DGPs with = }, the following inequality holds:
inf supe + 1)-1 + y w (x) 2 22
jer D
Theorem 4.4 suggests that for all f F, there exists D where the following holds:
[n)D(1(f(x)) + 5) (x) + vllw (x) 2 2 cP (x) c)²
in other words, there exists a family of data generating process satisfying (5), such that the conditional excessive risk any estimator irrespective of the learning procedure cannot improve upon the bound established in (8), the absence of any additional assumptions. The theorem implics that the conditional excessive risk bound of weighted ERM estimator in (8) tight.
Sclective inference in practice While the optimal weight function w"(x) is not available in practice, Theorem 4.1 admits an e-approximation of wxx)) in the PAC seuse, that is, one can have control over the selective risk of the weighted ERM estimator reasonably well using any "good" estimates of the margin function. Note that using the plug-in estimate is a standard procedure adopted in the literature related to selective classification (Herbei & Wegkamp, 2006), where users have the option to abstain from prodicting data with high uncertainty or low margin. This is pertinent in scenarios where prioritizing accuracy in the low uncertainty region (conditional risk) takes precedence over accuracy across the entire domain (unconditional risk). The result in Theorem 4.1 suggests the same, that by using good estimate of the margin to re-weight the empirical risk, one can improve the conditional risk at the inference stage.
4.2 Bounded heteroscedastic regression
The regression problem considered in this section is formalized next. Let y E RR be generated according to
y f') 20 E, x xEX,
(11)
9

---

Published in Transactions on Machine Learning Research (01/2025)
where (x) & E(ylz), (x)) & Var(Y[æ), and & (-c2,0y) is some bounded noise with zero mean and unit variance. (a) is effectively the target hypothesis.
1/>]],0 c3 < 1 be hypothesis classes with finite pseudo-dimeniions dp(F)) < co and dp(g) < oc, respectively. Note that for the range of a², we assume that C3 is bounded away from and therefore the variance is non-vanishing; on the other hand, the upper bound satisfies 1/7 2 and thus we do not preclude scenarios where the variance dominates the mean function, similar to the settings considered in Zhang et al. (2023). Throughout this subsection, we consider the well-specifted learning setting, namely, f*) F, (ax G. Additionally, let W & {w X X+ 1/ca]
Without loss of generality, let F 2 [f= X' + + (o x
be some other hypothesis class for the weight function w* (xx), which satisfies wn (a) = 1/()2)((x)
To perform weighted ERM, we adopt the mean-squared-error loss given by ((f,z) - (v - [a=)) and a data-dependent weight that coincides with the inverse of the variance function '- (x); this leads to weighted loss function of the form (--((=)²²_ Note that in practice, (²x( is unavailable and one can replace it with some estimate, while still achicving similar results, provided that the estimate approximates the truth reasonably well.
d=((x)
We first give high-level account of the results established. Let (**()) S 1//7 be the maximum variance defined in Zhang et al. (2023), then the following holds (sce derivation in the appendix):
Var [(v (w
- (-
(8/7) E- (w)
(y
(12)
The expression in (12) suggests that the Bernstein-type condition is satisfied with B == 4/7; analysis similar to Corollary 3.7 in Bartlett et al. (2005) further leads to the following risk bound Eg [x((f (z) {x)) =
O( ((y4a) For weighted ERM, o² (xxx)) (or w" (xx 1/02* (xx), equivalently) is introduced to "balance" the
inequality in (12), resulting in the Bernstein-type condition to hold in the following form:
Cly f(x)² (2)(
Cly- f (²*(x)) xr)
C(y CCv--f(e f(x)) C(y f(y-f(x))
Var=
< Ez
(13)
(x²)
σ2 (2x(x)
where C- 111 4/c3). Once again, leveraging the results in Bartlett et al. (2005) gives Ex Exlastaa (xe)
- (VERM(x)))1 = - (2), which effectively removes the 1/y factor. These statements are formally stated next.
+
Theorem 4.5. Suppose we have 8² E G s.t. En (1100(((xx) - (x))) S e/cī. Given Lid samples Sn = -(x,;,))) Wi) that are drawn according to the DGP and independent of those used for obtaining o², let FWERM be the weighted ERM estimator defined in Equation (4) with the weight function substituted by 1/)²();; then the following holds simultaneously with probability at least 1 5 5 for some e > 0,8 > 0:
(2*(2) ((uErea(a)) - [ Se and Enp (2(xx)
1
- Se,
Enp
provided that the sample size satisfies 2 d(F)(lo(()) +oog(1/)))
Remark 3.
The risk bounds in Theorem 4.5 could be made analogous to those in Theorem 4.1. A standard
analysis using techniques in Bartlett et al. (2005) implics that the ERM estimator achieves E_ [(F
[*(x))?] <E e using (+++e) samples whereas the weighted ERM one achieves Ex [= (= <
with é( ((2) samples. Similar to the conclusions in the classification task, weighted ERM achieves sample efficiency at least comparable to ERM in the general region, and is superior in the small-variance region as depicted by o²* (z) < 1/c, with problem-dependent constant 7/c improvement. Additionally, by using the negative log-likelihood loss, the sample complexity of learning (²*(.) is comparable to that of learning [*()), as presented next in Theorem 4.6.
Next we provide some guarantees for learning the <²(x) function. Here we seek to obtain 2² by minimizing the negative log-likelihood loss, while restricting (2, ) to be in the hypothesis class F x J. FFF
that the normalized residual square is bounded: (f -tl)- I 4c2 P (2), Vo X.
o (2))
10

---

Published in Transactions on Machine Learning Research (01/2025)
Theorem 4.6 (Risk bound for estimating r²t). Let Sh ( be i.dd. somples drown according to the DGP in (11) and
log(o (yi ju(x;))2
( 52) arg min
E
( Fx
(
(#..)))ES
Then for any > 0,6 > O, the following holds with probability at least 1- 8:
)
2
E
SE
(2(2)) x2*
provided that the somple size satisfies
n 2z TIT2 ((dp()) + dp( + Ta TT))
where Ti- = (1+ +2 T2 = (c2 + log
log2(111)) (1/7)), Ta log log(1//3 2/ + 2/637) T4 llo Ts = log( ))
Note that the PAC guarantee for learning (x) has been studied in Zhang et al. (2023) which admits a bound at the rate of Ō (1)),, whereas the bound in Theorem 4.6 is of the order ( See Appendix A.9 for
discussion that compares the two and the proof for the above theorem.
The following corollary combines Theorems 4.5 and 4.6 and takes into account all the randomness embedded in the samples:
Corollary 2. Under the setting considered in Theorem 4.5 and Theorem 4.6, with (xx) be the one defined in Theorem 4.6, the following hold simultaneously with probability at least - 26:
Eg
(2+ f())))(())))
Se and Eg (2 - fz SE.
4.3 General case
Next, we generalize the classification and regression examples presented in Sections 4.1 and 4.2 above, Consider hypothesis class F F {f X' - [-1, 1]} with complexity mcasure d(FF < co, and W & {w X' + Iy, a]] with complexity measure d(W) < co, and c 0,7 OF. Assume that the target hypothesis f* E F and the true weight w² E W.
A balanceable Bernstein-lype condition is given in the following assumption, together with some other technical assumptions required for the loss function.
Assumption 1. Let D F x F x X' + [0, be uniformly bounded function that captures the excess risk. The following are assumed to hold for the loss function ((.
Lipschitzness and uniform boundedness.
Vz, (e({2) e((", a; Vz, f1, fa 1e(f).z) - =) - Llft fal-
(14)
Under semi-random noise label. Suppose the DGP satisfies semi-random noise label (Diakonikolas et al., 2021; Pia et al., 2022) and the following are satisfied:
-l(", E. En (z)D( -_("
(15)
D(f*
Note that here effectively assume that is bounded away from zero accommodate both the case of classification and regression. However, note that in the case of classification, the assumption being bounded away from is equivalent the margin condition from Massart Nédélec (2006). The weighted ERM framework actually agnostic this condition and allows be zero.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Balanceable Bernstein condition. Under the same semi-random noise label assumption for the DGP, there exists a uniformly bounded w(x) that the following holds:
Varzlu(<))((1f 2) [('' +))] = E,lw(z)(((f ((f)
(16)
For (16), if one replaces w" by its uniform lower bound, standard Bernstein-type condition Var[h(>)] <
BE, [[[]]]] can be recovered with w(x) and B = 1/7. On the other hand, there exists wa:) that balances the ratio between Varlw(()) (x(x]] and ([w(x))((x), such that Bernstein-type condition holds with an improved multiplier. In particular, one can set w(+) _ wn()) so that B = 1, as in (6) and (13).
Remark 4 (Assumption 1 in classification and regression settings). We provide concrete examples for how expressions in Assumption 1 manifest in classification and regression settings, respectively.
For classification settings, let ((f: Å L[f(z) + and D(f ((f + f(z)}. }. It is easy to verify that (14) is satisfied with L = 1,a = 1; (15) is satisfied with w²(x)) derivation in Appendix A.1); and (16) holds as cstablished in (6)
_ 12n) - 1| (see
For regression settings, let {(f: z) _ (y- - ( and D(S", J. x) constant C. (14) is satisfied with a + (8c2/ VT), L == L / where b b= 4/(Cr), = f (15) holds with wn (x) - )म and (16) holds as established in (13).
(m) (f"(a) for some
C
Theorem 4.7. Suppose Assumption 1 holds. Let f E E and wt E W, and suppose we have w E W s.t. E (x((((())-a w (x))²² S 5. Given ii.d. samples Sn = (<x,,,)):) v:)l.- drawn from the dala generative process, let fWERM e arg min Ez,s, ((z)e(f:zi). Then for any E > 0, 0, the following holds simultaneously with probabilily at least 1- d:
E (x) J m0 Se, E SWRDS,,WRRMM, x)] se, and (#))("" (a)D((",FTRRRM S Se,
as long as the sample size requirement is satisfied:
ciaa (d(F) ( + log(ei +lg(()) + c
Remark 5. The proof of Theorem 4.7 is deferred to the appendix. A major hurdle in completing the proof comes from the inaccessibility of (xx and thus one needs to use ((x) instead; this is out of practical consideration as one can only access an estimated version. To overcome this challenge, it suffices to require [=]((((== w (=))²] S e/b, with which one can show the weighted empirical risk satisfies an additive error version of the Bernstein type condition, namely, Var[h] S BE[[]] + s. To this end, prove an alternative version of Theorem 3.3 in Bartlett et al. (2005) under this relaxed Bernstein-type condition, and show that the weighted ERM achieves a fast rate in the generalization error bound.
5 Synthetic Data Experiments
We present results from synthetic data experiments to support our theoretical claims, respectively for regression and classification settings. For both settings, we follow a two-step procedure, in which we first perform ERM to obtain estimates for the mean and the weight, followed by a reweighting step. Subsequently, from ERM and weighted ERM-in terms of their selective risk,
we compare two sets of astimates-rests where the selective set is chosen over a range with varying coverage determined by the variance or the margin, depending on the setting under considcration. For both experiments, the size of the training set is set at 204, to ensure that the algorithm has access to adequate number of samples and circumvent any potential issues due to lack of fit, although empirically the conclusion broadly holds even with much smaller sample sizes.
5.1 Experiments under regression settings
We consider regression settings in the presence of heteroscedasticity, similar to the ones used in Skafte et al. (2019); Seitzer et al. (2022). The true data generating process is given by a univariate regression with x E R of the form
y (x) (x) E, B(E) O; Var(s) 1.
12

---

Published in Transactions on Machine Learning Research (01/2025)
The mean (*(2) _ sin(z) is simusoidal function; the scale function of the additive noise & depends on the value of x, and is given by (x)) - (0.09) + 2²). The regressor x is sampled uniformly from [o, 10] and the noise& & is standard Gaussian. Figure 1a provides a visualization of the data resulting from this DGP.
y= fix) + o(xiE
selective risk
0.5
FRM
lne 20dd
0.00
WBBM
0.1
0.10
086
-19
0.00
0.7
coveage
(a) Visualization of the DGP and the estimates. The true mean (*( and the two-standard deviation bands are in black dolted lines. The mean estimates from ERM (blue) and wERM (orange) are in solid lines, and the shaded area corresponds to the two- standard deviation derived from the variance estimate of the ERM step.
(b) Risk over the selective set with varying coverage ag the solid lines correspond the risk on the test (sub)set averaged over 10 runs of the experiment, and the shaded area correspond to 1 standard deviation.
Figure 1: Regression setting: underlying true data, estimates from ERM and weighted ERM, and the selective risk
We consider the following estimation procedure using l loss; ((x) and (xr) are both parametrized by multi-layer perceptrons (MLP):
An ERM step that gives rise to the mean and the variance estimates, that is, Σ [logo"(x) + (ui PERM( ))2 (x)
(Æ)M(()= = arg min Σ f(x))) and (x) == arg min mig
i=1
reweighting step, with the weight given by the precision estimate from the ERM step:
(W) arg min (w (v:))) 2 where w(x)): = 10:==11033a 1102(x.).
Once (x) and [WERM(x) are obtained, on the test set, we consider evaluating their risk over a range of selective set with varying coverage [0, 1]. Coneretely, at evaluation time, the selective risk is calculated as Ra = E (f(= I (0²(x) S (a((2)))
where qa- (o2) is the a quantile of the variance over the entire domain; a low-coverage (i.e., small a) selective set corresponds to the low-variance region. Empirically, the risk is obtained by substituting f by either fERM or FWERM for each test data point in the selective set then taking the average, with @(r) coming from the ERM step. The cut-off qa (a²) is determined by the empirical quantile of the estimated o² on the validation set.
Figure la presents the mean estimate from ERM (blue) and weighted ERM (orange) respectively, although the quality of the fit is satisfactory for both cases and therefore they largely overlap with the truth and bocomes hard to distinguish, visually. Figure 1b displays the risk over the selective set with varying coverage a. As it can be seen from the plot, weighted ERM has an advantage over the ERM estimate in the low-coverage region, as manifested by a lower selective risk, and the advantage diminishes as the coverage a increases. This is in accordance with the theoretical results established in Section 4.2.
As a remark, the practical implication for such results is that in certain scenarios (e.g., some finance applications) where one takes actions only when there high confidence and abstains otherwise (and therefore being "selective"), a weighted ERM procedure can be leveraged to obtain more refined estimates for the region of interest where actions would take place.
13

---

Published in Transactions on Machine Learning Research (01/2025)
5.2 Experiments under classification settings
For classification, we consider the following data-generating process for illustration purpose, in which extremely noisy data points are present. The features 2, E I2² are sampled from class- conditional Gaussian with equal covariance, that is P(c; = k) pai P( = k) ~ ( Σ) Here we consider 4 "clusters" with k E {0. 0', 1, 1,1', where Po = 0.5, Pu = 0.25. P1 = 0.20, pr = 0.05; Po' = (-10,0) an = (-3, ;1 = (3, 0) pv = (12,0),, E= [0.s 0.5]. Let
のつ P(c (1,1') æ.) EEne(i,n Pe pla; Pks E))
( EKE Ph p( M4s E)),
where p(z;: pkg E) denotes the pdf corresponding to Mu,, Σ) evaluated at æ4, and the equality follows from Bayes rule. Set y (iitial) & (x;));; > 1/2]. Further, we inject noise to the class labels for points that are in cluster 0', such that their final labels are flipped with probability phips that is:
y: = - giittial (w.p. paip) if c = 0' otherwise y; iiitial
It is worth noting that given that above-mentioned DGP, the theoretical decision boundary is linear; additionally, the theoretical margin is given by wn (x;;) (x,) - 1/21, where 17" (x;) [6 > 1/2) if
& + 0' otherwise Phip- In this setting, the flipping probability Pmip is set at 0.49.
Similar to the case of regression, we consider the following procedure that entails two steps, using the cross-cntropy loss:
An ERM step: we obtain the estimated margin ((x;) = (7x;)) - 1/2| where 7(x,) the estimated Bayes-optimal classifier, and a linear decision boundary that gives rise to the class labels f(x; ERMM
reweighting step with the weight set ((x,), which then gives rise to an updated decision boundary and the associated class label f(z;)wERM.
The selective risk is then evaluated
R. = E E 1 2. (w])))]
1==10">005)
where (a(w) (w) is the a quantile of the margin over the entire domain; a low-coverage (i.c., small a)) selective set corresponds to the large-margin region. Empirically, the evaluation done by averaging the risk the corresponding test (sub)set, and is substituted by either f(x,)ERM f(z;)WERM; the cut off (w)(w) is empirically determined by the quantile of the estimated margin on the validation set.
true y with nolay tabel
estimated label
risk
WERM
-10
107
w wERM estimated class label
02-
011
00
-5
-5
0.2
15
(a) Visualization of the DGP. The top panel displays the class labels after injecting label noise, and the bottom shows class labels based on the Bayes optimal classifier, respectively
(b) Estimated class labels based on
(c) Risk over the select set with vary- ing coverage the solid lines corre- spond the risk on the test (sub)set averaged over 10 runs of the exper- iment, and the shaded area corre- sponds to standard deviation.
ERM (top) and weighted ERM (bot- tom), respectively
Figure 2: Classification setting: underlying true data, estimates from ERM and weighted ERM and the selective risk
14

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 2a provides a representative view of the data resulting from this DGP; Figure 2b displays the estimated class labels from the FRM (top) and the weighted ERM (bottom) step, respectively. As it can be seen from the figure, the latter largely aligus with the class labels dictated by the Bayes-optimal classifier, whereas the former has a noticeable number of points near the decision boundary being mis-labeled. Figure 2c provides a comparison of the selective risk for the two sets of estimates. In this specific setting, weighted ERM dominates.
6 Discussion, Limitation, and Future Work
To conclude, this work investigates the generalization error bound of weighted ERM under the fast-rate regime. We show that by additionally incorporating carefully-designed weight function in each loss term, estimators based on weighted ERM can achieve tighter bound in selected regions by problem-dependent constant, when compared with the one from standard ERM. This finding leads to practical applications where one can use plug-in estimates of the weight function to obtain superior performance in sub-regions through a two-step procedure, as demonstrated in our synthetic data experiments.
It is worth noting that recent work Zhai et al. (2023) considers a generalized reweighting scheme where samples are reweighted dynamically during training; in spirit, this is similar to the procedure considered in our work, yet the two differ in the following aspects: (1) the starting point of Zhai et al. (2023) is the generalization of distributional robust optimization (DRO) algorithms; in particular, under a DRO setting where distributional shift is present yet the test distribution is "close" to the training one, at the conceptual level, training should focus on the "hard" cases, In our work, on the contrary, the setting under consideration can be made analogous to soft abstention, and thus the weight schema further upweights the "easy" cases. (2) The result established in Zhai et al. (2023) states that the generalized reweighting procodure leads to a solution that is close to the ERM one, in that the points they converge to are close; this largely done by analyzing the properties of the estimates between successive iterates. On the other hand, this work is concerned with the statistical crror bound of the weighted ERM estimator- in particular, its superiority over the standard ERM one in the high-confidence region.
There are several limitations of this work and directions that could be further explored. Throughout the paper, we consider well-specified sengs There are several difficulties in regards to the exlension to mis-specified settings where the target hypothesis can potentially live outside of the hypothesis class in question. Possible extension includes exploring mis-specified settings with surrogate losses using tools developed (2022); Mao et al. (2023), and leveraging several recent results which show that fast rate could be achieved under mis-specified setting via model selection aggregation (Tsybakov, 2003; Bousquet & Zhivotovskiy, 2021; Kanade et al., 2022). Other recent work that touched upon this issue includes Puchkin & Zhivotovskiy (2022), where to establish the desired results the authors require both the diameter of the hypothesis class and the star number to be finite. Alternatively, lools that can work in specific setups may be introduced to characterize the approximation error; e.g., if one considers a setting similar to that in Kohler & Langer (2021), namely, the hypothesis class being a set of fully connected DNNs and the target hypothesis being in the class of (p, C')-smooth functions, then the approximation error bound can be calibrated. The results developed in this work can potentially be extended to these settings, which however requires more involved analysis to handle various components (e.g., the approximation error of the weight function (x)) and very specific assumptions on the exact setup.
Awasthi et al.
Separately, our analysis is limited to Bernstein-type condition (Lee et al., 1996; Bartlett et al., 2005) of the form Var[h] < BE[A]. To study classification problems under Tsybakov noise condition (Mammen & bsybakov, 1999; Tsybakov 2004) and regression with lp risk (Bartlett Mendelson, 2006), a more generalized form 2 S ((E B E (0, 1] is required. Finally, for some other settings such as Offset Rademacher Complexity (Liang al., 2015) and "small-ball condition" (Mendelson, 2018), where the fast rate has been established, we are optimistic they can also enjoy some problem-depenndent constant improvement by exploring the structure of
noise label with a properly designed weight function.
15

---

Published in Transactions on Machine Learning Research (01/2025)
References
Ausset, G., Clémengon, S., and Portier, F. Empirical risk minimization under random censorship. Journal of Machine Learning Research, 2022.
Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. H-cousistency bounds for surrogate loss minimizers. In International Conference on Machine Learning, pp. 1117-1174. PMLR,, 2022.
Bach, F. Self-coneordant analysis for logistic regression. Electromic Journal of Slatislics, 4:384 414, 2010. Bartlett, P. L. and Mendelson, S. Empirical minimization. Probability Theory and Related Fields, 135(3):
311 334, 2006.
Bartlett, P. L. and Wegkamp, M. H. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8), 2008.
Bartlett, P. L.., Bousquet, O., and Mendelson, S. Local Rademacher complexities. The Annals of Statistics, 33(4))1497 1537, 2005.
Boucheron, S., Bousquet, O., and Lugosi, G. Theory of classification: A survey some recent advances. ESAIM: probabilily and statistics, 9:323 375, 2005.
Bousquet, O. and Zhivotovskiy, N. Fast classification rates without standard margin assumptions. Information and Inference: A Journal of the IMA, 10(4):1389-1421) 2021.
Bousquet, O., Boucheron, S., and Lugosi, G. Introduction to statistical learning theory. In Summer school machine learning, pp.
Springer, 2003.
Cawley, G. C., Talbot, N. L., Foxall, R. J., Dorling, S. R., and Mandic, D. P. Heteroscedastic kernel ridge regression. Neurocomputingg 57:105- 11..05-11,,,0 2004.
Chow, C. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16 (1):41 46, 1970.
Cortes, C., Mohri, M., Riley, M., and Rostamizadeh, A. Sample selection bias correction theory. In Algorithmic Learning Theory: 19th International Conference, ALT 2008, Budapest, Hungary, October 13-16, 2008. Proceedings 19, pp. 38 53. Springer, 2008.
Cortes, C., Mansour, Y., and Mohri, M. Learning bounds for importance weighting. Advances in Newral Information Processing Systems, 23, 2010.
Cortes, C.,, DeSalvo, G., and Mohri, M. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Boly, October 19-21, 2016, Proceedings 27, pp. 67-82. Springer, 2016.
Daye, Z. J., Chen, J., and Li, H. High- dingh-imensional heteroscedastic regression with an application to cQTL data analysis. Biometrics, 68(1):316 326, 2012. Diakonikolas, I., Gouleakis, T., and Tzamos, C. Distribbutoon-indeeendenu PAC learning of halfspaces with Massart noise. Advances in Neural Information Processing Systems, 32, 2019.
Diakonikolas, I., Park, J. H., and Tzamos, C. ReLU regression with Massart noise. Advances in Neural Information Processing Systems, 34:25891 25903, 2021.
Dudley, R. M. Uniform Central Limit Theorems, volume 142. Cambridge University Press, 2014.
El-Yaniv, R. et al. On the foundations of noise-free selective classification. Journal of Machine Learning
Research, 15).,, 2010.
Franc, V., Prusa, D., and Voracek, V. Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11):1 49, 2023.
Ge, J., Tang, S., Fan, J., Ma, C.,, and Jin, C. Maximum likelihood ostimation is all you need for well-specified covariate shift. In The Twelfth International Conference Learning Representations, 2024.
Hanczar, B. and Dougherty, E. R. Classification with reject option in gene expression data. Bioinformatics, (8117))889-1895 2008.
Hanneke, S. Theoretical Foundations of Aclive Learning. Carnegie Mellon University, 2009.
Haussler, D. Sphere packing numbers for subsets of the boolean n-cube with bounded Vapnik-Chervonent Chervonenkis dimension. Journal of Combinatorial Theory, Series A, 69()):217-22)) 232, 1995.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Herbei, R. and Wegkamp, M. H. Classification with reject option. The Canadian Journal of Statistics/ La Revue Canadienne de Slatistique, pp. 709-721, 2006.
Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652-661. PMLR, 2016.
Kanade, V., Rebeschini, P., and Vaskevicius, Exponential tail local Rademacher complexity risk bounds without the Bernstein condition. arXiv preprint (.ft::2222.11466 2022.
Kendall, A. and Gal, Y. What uncertainties do need in Bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Klein, P. and Young, N. E. On the number of iterations for Dantzig Wolfe optimization and packing-covering approximation algorithms. SIAM Journal on Computing, 44(4):1154 1172, 2015.
Klochkov, Y. and Zhivotovskiy, N. Stability and deviation optimal risk bounds with convergence rale o(1/n). Advances in Neural Information Processing Systems, 14:5065-5076, 2021.
Kohler, M. and Langer, S. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231 2249, 2021.
Koltchinskii, V. and Panchenko, D. Rademacher processes and bounding the risk of function learning. In High dimensional probability 11, pp. 443 457. Springer, 2000.
Koren, T. and Levy, K. Fast rates for exp-concave empirical risk minimization. Advances in Neural Information Processing Systems, 28, 2015.
Lakshminarayanan B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Lee, W. S., Bartlett, P. L., and Williamson, R. C. The importance of convexity in learning with squared loss.
In Proceedings of the Ninth Annual Conference on Computational Learning Theory, pp. 140-146, 1996. Liang, T., Rakhlin, A., and Sridharan, K. Learning with square loss: Localization through offset Rademacher complexity. In Conference on Learning Theory, pp. 1260 1260-1285. PMLR, 2015.
Mammen, E. and Tsybakov, A. B. Smooth discrimination analysis. The Annals of Statistics, 27(6):1808 -1118808--199,,, 1999.
Mao, A., Mohri, M., and Zhong, Y. H-consistency bounds: Characterization and extensions. In shirth Conference on Neural Information Processing Systems, 2023.
Massart, P. and Nédélec, É. Risk bounds for statistical learning, The Annals of Statistics, 34(5):2326 2366, 2006.
Mendelson, Improving the sample complexity using global data. IEEE transactions on Information Theory, 48(7):1977-1991, 2002a.
Mendelson, S. Rademacher averages and phase transitions in Catrenko-Cantell classes. IEEE transactions Information Theory, (1(1))251-26)) 2002b.
Mendelson, S. Learning without concentration for general loss functions. Probability Theory and Related Fields, 171(1-2):450-50 502, 2018
Min, Y., Wang, T., Zhou, D., and Gu, Q. Variance-aware off-policy evaluation with linear function approximation. Advances in Neural Information Processing Systems, 34:7598-7610, 2021. Mozannar, H. and Sontag, D. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pp. 7076 7087. PMLR, 2020.
Namkoong, H. and Duchi, J. C. Variance- based regularization with convex objectives. Advances in neural information processing systems, 30, 2017.
Pia, A. D., Ma, M., and Tzamos, C. Clustering with queries under semi-random noise. In Loh, P.- L. and Raginsky, M. (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 5278 5313. PMLR, 02 05 Jul 2022. URL https: //proceedings mlr, press/v118/pia222. html...hmm..
Pidan, D. and El-Yaniv R. Selective prediction of financial trends with hiddeu markov models. Advances in Neural Information Processing Systems, 24, 2011.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Pollard, D. Empirical processes. volume 2, pp. 43-50. Institute of Mathematical Statistics, 1990.
Puchkin, N. and Zhivotovskiy, N. Exponential savings in agnostic active learning through abstention. IEEE Transactions on Information Theory, 68(7):4651-400
2022. doi: N0.1109/TTT.202.,. 2022.1152225
Seitzer, M., Tavakoli, A., Antic, D., and Martius, G. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. Tn International Conference on Learning Representations, 2022. URL http:/foouummidaPPXXiii
betps://openreviie
Shah, A., Bu, Y., Lee, J. K., Das, S., Panda, R., Sattigeri, P., and Wornell, G. W. Selective regression under fairness criteria. In International Conference on Machine Learning, pp. 19598 19615. PMLR, 2022.
Skafte, N., Jørgensen, M., and Hauberg, S. Reliable training and estimation of variance networks. Advances in Newral Information Processing Systems, 32, 2019.
Talagrand, M. Sharper bounds for Gaussian and empirical processes. The Annals of Probabilily, pp. 28 76, 1994.
Tsybakov, A. B. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics, 32(1):
135-166 166, 2004.
Tsybakov, A. B. Optimal rates of aggregation. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pp. 303-313. Springer, 2003.
Valiant, L. G. A theory of the learnable. Communications of the ACM, 27(11):1134 1142, 1984.
Van Der Vaart, A. W., Wellner, J. A., van der Vaart, A. W., and Wellner, J. A. Weak Convergence. Springer, 1996.
Vapnik, V. and Chervonenkis, A. Theory of pattern recognition, 1974.
Vapnik, V. and Chervonenkis, A. Y. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications, 16(2):264-28)) 1971. Vidyasagar, M. Learning and Generalisation: with Applications to Neural Nelworks. Springer Science Business Media, 2013.
Xie, T., Ma, Y., and Wang, Y.-.. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in Newral Information Processing Systems, 32, 2019.
Xu, Y. and Zeevi, A. Towards optimal problem dependent generalization error bounds in statistical learning theory. Mathematics of Operations Research, 2024.
Yuan, M. and Wegkamp, M. Classification methods with reject option based on convex risk minimization.
Journal of Machine Learning Research, 11(1), 2010.
Zhai, R., Dan, C., Kolter, J. Z., and Ravikumar, P. K. Understanding why generalized reweighting does not improve over ERM. In The Eleventh International Conference on Learning Representations, 2023. Zhang, Y., Lin, J., Li, F., Adler, Y., Rasul, K., Schneider, A., and Nevmyvaka, Y. Risk bounds on aleatoric uncertainty recovery. In International Conference on Artificial Intelligence and Statistics, pp. 6015 6036. PMLR, 2023.
Zhang, Y., Zheng, S., Dalirrooyfard, M., Wu, P., Schneider, A., Raj, A., Nevmyvaka, Y., and Chen, C. Learning to abstain from uninformative data. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Zhivolovskiy, N. and Hanneke, S. Localization of VC classes: Beyond local Rademacher complexities. Theoretical Computer Science, 742:27-49, 2018.
18

---

Published in Transactions on Machine Learning Research (01/2025)
A Proofs and Discussions
In this section, we include proofs for the results established in Section 4 and the corresponding discussions. We introduce additional notation and definitions that are used in the ensuing development.
Let [n] $ (1,...,,) n). We use {2) _ Fim denote samples of of size n; {z)ig|n] = 21:1 ( y:) is analogously defined, and they are i.i.d. samples drawn from P. Given {z] we use P, to denote the empirical measure. We use || Ilp to denote the lp norm vectors and lll, to denote the Lp norm of random variables under The indicator function 1 (xr E A} equals when the condition otherwise. Denote b max(a, b) and A b min(a, b). Let II.,. on be independent Rademacher random variables. For function h: Z + R, define:
is true and
Pnh -
Σ
hzz,), Bxpph(z).
For a family of functions H a (h: 2 + IR} the Rademacher Complexity and Rademacher Average is defined as:
R.H Bor..
sup Nih
Σ ozh(zi)
sup hEH
o,,((z))
We further define the Local Rademacher Complezity and Local Rademacher Average with radius r as Rh H.P,, S and Rish E H.Ph2 rr..
Definition (Star Hull; see also Bousquet et al. (2003); Bartlett al. (2005)). The star hull of set of functions F is defined as
+F =F f E [0,
Definition (Sub-Root Function (Bousquet et al., 2003; Bartlett et al., 2005)). A function v R is sub-root if
& is non-decreasing
db is non-negative
v(r\/VF is non-increasing
And we say r" is a fixed point of di if (()()=))>)) -
The following definition for VC-class could be found in (Vapnik & Chervonenkis, 1971; Van Der Vaart et al., 1996). We include them here for the sake of completeness.
Definition 5 (VC-dimension; Vapnik & Chervonenkis (1971)). The VC-dimension dyc(F) of a hypothesis class F = {f i (1, -11} the largest cardinality of any set C X' such that Vš C S, 3f F
1 if x
f(x)
-1 if zesss
Definition 6 (Psendo-dimensson:; Pollard (1990)). The Pseudo-dimension dr(F) of real-valued hypothesis class G = [g: X' + 11 u}} is the VC-dimension of the hypothesis class
H = (h: X R (-1,1) Max, sign(g(æ:)
19

---

Published in Transactions on Machine Learning Research (01/2025)
A.1 Derivation of Equation 6
The following derivation establishes the connection between [2(((]=)) - which could be found in Boucheron et al. (2005). We include here for completeness.
==)] - and (x(w"())I + M.),
- -
-Em,y/1 {v + - - 10
-Eay - [0000000000000000000000000--1000 f()))]
f
(17)
= Po
(x) + (|)
Next, derive Equation 6:
Var (00)
SEaylu (000
=Exylw" (1) # ( 11
A.2 Derivation of Equation 9
To see Equation 9, we first bound Ez (x) 2 (x) + f"]]. By leveraging Equation (17) and
+y 11 # yb)l S æ the follow inequality holds:
Bm (z)
(x) (1 (1 き 10 * w
B. (0)
(18)
which implies the following inequality:
E (w (ar) 2 (x)) ( 2
# S
(19)
thus E (m( 2 clw" (x) [') E. In addition,
f")]
(20)
(*) (P) <c
2))
(=)) <c)
Pc-P(w(<z) <)
Equation (19) and Equation (20) combined gives
En,y|-iff + -
+ ( +, (c0
=E,[1 (w'(
<P(W (x) < c) :.
A.3 Derivation of Equation 12
Equation 12 is a standard result, we include the derivation here for the sake of completeness.
Var (x)) (y - [=))) 1ett:
-f())
(y-
(x))² 40 (x))
8
[(f (x) - v)² (y- - f(z))²))
7
20

---

Published in Transactions on Machine Learning Research (01/2025)
A.4 Proof of Theorem 4.1
The proof readily follows from invoking Theorem 4.7 with {(f:)) è [[[(a: # y)), (= 12n)" (a) 1/ D(fi, x) = 1[fi # f2}. It can be easily verified that Assumption is satisfied with L 1,a 1,6 1, = inf (x)
A.5 Proof of Theorem 4.3
Proof. Let R(n, zi) A (n(x;) v1)². We define following function class:
H = Ao AR(m R(m z) =(]"==); 5
which is a composite function of hypothesis class G, loss function R and the difference operation A. For simplicity we let An. = AR(T ;"; æ) when there is no ambiguity.
By definition of i7, we have R( 2) s P,, R(n" 2), also by definition of A n we have P,,A, R, S
Next we bound PA PARÁ n. (aRm) & (aanF((AR)) To invoke Theorem 3.3 Bartlett et al. (2005), we need to find a subroot. function (r) such that
P, n. Since ly f(x)] (0,1).n(x)) [0, it can be easily verified that
((r) 2 2EP,. 2IEP, R. E H
To this end, we show some analysis on the Local Rademacher Average ERR RR E H: B[h2] S r).
ER, (A Rog,r) -Es,oLn
sup n
of
R.
NEG
Sr
R.
The following analysis largely follows from the proof of Corollary 3.7 in Bartlett al. (2005). Since A RR. uniformly bounded by 2, for any r 2 vl(r,, Corollary 2.2 in Bartlett et al. (2005) implies that with probability at least 1. {h E # Pn r) S (h #H P,h² < 2r}. Let & th E * 1hh2 S r S (h E * pn < 2r}. then the following holds:
ERR, (=H, ph2 3 - P (*H, PI S P Ph STle8
P < 2 2
Since r"= = =>((r), r" satisfies the following
22 log 20BEM P,H, 2r
(21)
n
Next we leverage Dudley's 's entropy integral (Dudley, 2014) to upper bound EsR (*H, S 2r"}, using the integral of covering number. Specifically, by applying the chaining bound, it follows from Theorem B.7 (Bartlett et al., 2005) that
'2r"
(RH, S 2r*)) const
B
v log NogNN((e *H, 1n.))
(22)
where const represents some universal constant. Next we bound the covering number log N2(5, H, æi.n) by log As, G, w.m). We show that for all Bia, any e2ooo of G is e-cover of H. Specifically, let c 1]
an g-cover of H. on 2is so that for all E 9, 30nm E V so that a / (n(x.) v,)² < E. Now we show
21

---

Published in Transactions on Machine Learning Research (01/2025)
that for any 2i:1: the family of (v; y1)² - (n, ur),, E [nx] is e-cover for H. where m - P(y: 1 1x2;)::
1
Σ ((e, y1)2 (1) -y))² ARn)) n ie(n)
((v, y-)² (m((2)) y/))
n
VE(N]
0 - (7;2))) (((x)) 2y.)² = 4e.
n
VEIN
Combine the above incquality with Corollary 3.7 from Bartlett et al. (2005), we have the following upper bound on the entropy number of *HH
8 2: H, 11m
)
(121+))} log N2 (5.00,21m)) 21. (121+1)}
log (2 *H, xim) S log N2
Next we bound coust Vī
E So 2r
v Jlg.NN((. +H, xi.n de from (22). Note that by Haussler's covering number
bound (Haussler, 1995), the following inequality holds: lo g, G, S edpplo log
we can therefore
derive the following inequalities:
const nn
E.
V2
log Nole, *H, (i ddm
const
E /
V2r
V VoggN22 No 2. H. mmmm
( de
J
const
E
V2r
5 log.1
8 G. xin
(12) de
Vn
dp(
V2r-
log (1)
dr(g)r* log(1/r*))
consty
V
sconsty
S consty
(P)(9)
dr(g)r* log(n/edr
consty
n²
where const represents some universal constants that may change from line to line. Together with (21) one can solve for
dp(g) log()
n
Since PAR,, Ee, we have Pl - 17-12 s E, given that the following equality holds:
PAR., = [R( 2) R(vn; [f, ;", Em.y (n((z) y)² (7)((x) (ax) - Ex [x(((())- 17" (x:) 21. (as 17" (x) & Elyl
Note that since In 21 In 2 S In 17 1,, the following readily follows:
Em (17(2=) ! --nn(()) Se.
(23)
A.6 Proof of Theorem 4.4
Proof. The proof is by construction. We construct the full support of to be X1 X₂. For all æ E Xi,
let w² (xx))= (==) 21 2 and X2, let w*()) (x: 1 0. By decomposing the excessive risk
22

---

Published in Transactions on Machine Learning Research (01/2025)
have
E
- - X]] P| E X%2] E
E Xb2 IP E X22 =0,since for E No. Pvy 1| = P(y -1/0)
The lower bound of term Eg,g [1 # v) (f * w}| E in Massart & Nédélec (2006).
x1] E
could be established by Theorem
on
A.7 Proof of Theorem 4.2
Throughout this proof, we use superscript j to index the jth coordinate of a vector (e.g., x²), and this is to be distinguished from the subscript i that indexes the samples.
Proof. The proof is by construction. Let y be the minimum margin. Let. F (1 T sign(æ 3]13 E ] X== - 15m18, 8 = a. e a E {1- - - (- u [1, j e [d] where eº is j-th standard basis. Let
1/" (x)) = 1 1( 11 sign(x) 2 0f{11 1 ||ll] 0.13 + 1[|al| 2 17}
(24)
Note 1 sisinna 3) could be viewed a composition class of the d-dimensional half-space and the interval function, and its VC-dimension could be bounded as loog(d) (Vidyasagar, 2013). Consequently, the choice of n implies that E, [ # f=" > 7 S E for any given @ that satisfies E,[(w(z) (x)))) S
Consider following data generative process:
j ~ Unif[1, 21,2,..,. d.
0.1,
with prob 32
-0.1, with prob 3
Unif(1,2),
with prob 3y
32
Unif(-2,-1
with prob 3
= a e
1, with prob 17" (x)
with prob 1 17" (xx)
Note that the following version of the Chernoff inequality will be applied multiple times in the proof:
PI Σ X; mE[[ 2 [mE[[ S 2e EmEx/
(25)
By setting & 0.5 in inequality (25) and taking a union bound over glid we have
p 3j, s.l., |8 n Sal 64 d log(1/5)
32 log d log((15))
78
< 2de - log dl log(1/6)
< 2de 3ogg(d/)) <
(26)
The inequality above implies that with probability at least 1 - 6, Vj E [d], we have
32d log((()) (d) log(1/5)
96d log(d) log(1/8)
< 1 U Sal S
(27)
EE
7E
Next we show that for each j [d], for sufficiently small 7 with constant probability, 3 * 5². We first present argument for the case where d = 1. W.O.I.G, we focus on the case where X' X1. Given
23

---

Published in Transactions on Machine Learning Research (01/2025)
aims let x(-m))(-1) U x (1) be an ordering of on with x4, S x x' I 'x'1 < 0 x S x
mhite #hytng ang. where ((():())) 0.1 and Ti + n2 ng n. Since 32 12g(
I S 16
by Chernoff inequality in (25) with ç 0.5 and picking T y, we have log(1/6) 7
7t
< n2 S 3log(1/0)
and
r90log(1/d) that hold simultaneously with probability at least 1 - 26. Consider fraction of positive samples: ((net))((ra))) Given n2, by Lemma 2, we have,
(ogg(/1 S Tha I 76
75
(rns))
P
Σ i
186( = (z)))) s ( 22 + - +
1 -3)((-))TT)) 2 0.2e P0e-00 re
(28)
s4y lng(d) loget 1 /8x(
By inequality (28) with & 37, we have that with probability least 0.2e
(ngg rmall)
Σ
1860 = (ax()) s ( + 7)(1 - -37)n3
1
1=2+1
(mg (mna))
Σ
1Vy()) (((())) S (1 21) -37) 113
i=02+1
(m2 (rma))
(set 7 < 12
Σ
11M00 (()())) $11 2)
i=
Therefore,
[.ttl) Σ
n2
1(3()) f (x + Σ Hy60 ((()))
+1++
2) 113
7m3
- n33
$ (1
+
(29)
2
4
2
(12+17nal)
Σ
18() # (x))))
i=
It can be easily verified that inequality (29) implies that BERM 2 ((rns) To ensure that
s4.2 log(e)) loo( /
0.2e
0.12,
it suffices to pick 7 v log(d) 1og(1/8)
Since with probability 0.12 we have 3 PERM (Traa) 0.1, which implies
55c
that E [ # f* (xx) > x, E [*] 2 0.5. With markov inequality, we have with probability at least 0.1, 0.03 fraction of glid has BERR > 0.1, which implies that [x * f Hw)((x) > 7, E 80 2 0.5. We have with probability at least 0.1, E [1 (FRRM *
7) 0.015.
A.8 Proof of Theorem 4.5
The proof readily follows from Theorem 4.7 with {(J:2) 2 (y- - f(x))) wx)) = ()) and D(J (a), f(x)) =
(m)
f)2. It can be easily verified that Assumption is satisfied with 8 + sc3 bz b
A.9
Proof of Theorem 4.6 and discussion
Our analysis of learning (ar) is based on the following negative log-likelihood loss:
(NLL(O2, f(z) log(o²(x)) + (y f(x))
(30)
(²)
24

---

Published in Transactions on Machine Learning Research (01/2025)
In particular, we seek to minimize risk of the form in Equation (30) while restricting f and o² to be in hypothesis classes F,G C g that satisfy (w-f(=))²
d²( S 46 uniformly for all 2 = (x, For learning (x)) For simplicity, we assume Egl(E² 1)a] being a constant.
Verification of Bernstein-type condition We show that the risk function in Equation (30) satisfies some Bernstein-type condition. Since (リ-
(=)) I 4c², it then follows that [ =) (NII, , =)1 s 5c3 + 2ogg( )) In following analysis, we let C. = 5c3 + l))) The expectation term then satisfics:
E- (NNLL (a (o2, (O ,f,,
(f(z) - (xx)) - 1
0²* (²x(xx) (x)
2' (x))
Ex
log
(31)
02(x)
x2
0²(x)
(x))
2
(J(x) - (
2 E
(32)
x²(x)
max (5)
2
the last inequality leverages the fact that for all t>
log( 1 2 max 1, t)
(t - 1)2
(33)
For the variance term, the following holds:
Varz((CnLL ffz) lo2 ,f",,)))
= Ex((ENLL f.²) 25) - Ez [(NLL (o² f.z) - (NLL(O2t ,f,,
- (ax))) 52 (2))
2 2o
([(x) (
2-
[x)))
(2)((0
27
S FFn.
-log 0²(xz)
d²(x)
d²(x)
x²(xx
this can be further decomposed as
0 (x) (x)
26 2 (1(x)) (x))
Ex.
(x 1)
(2(x)
(J(x) - 8' (x)))
σ (x)
log (²)(a))
+ Ex
-1
O2(x)
(2())
(2(x)
(UV(o) -f)))
0 2 ( log
(
+ 2Em,
(x)
(x))
(x)
(52 1)
0 (1)
-1) 1)
28 2E 1EV022 2 (z (f(z) - (x))
(x(x)
0²(x)
< 2Ex. (162-19 1)2
2 (x)
(x) (f(x) f*)
1
+ 2EExr
0² (x))
Term
Term
+ C'EZIENLL f.z) (02 (0t
(34)
Term III
Bounding Term I: Due to the fact that for all t 0, max max log(1) - (t - 1)2, the following
holds:
(x)
2²*(2)
2 (xx
Ex max
1,
(²(x) / 1+
log
x²(xx)
x²(x))
25 max 1, (xx) /
0² (x
0 (z))
の² (3( x )
(0)((0)
} )}
Exgg
d²(x) /
-
log
2 P,
(x
25

---

Published in Transactions on Machine Learning Research (01/2025)
On the other hand, since for all f E F, ب. -,v=)(() could be bounded as:
I 4cz, we further have (x) x=
2 4/cg. Term
v-(x)
n. (15-1)0 (2)())) -1 (2())
|
27
x²*(x²)
x2t log
(2()(0)
x²(xz) )
462 +
E Exs
+1
2
x2(x)
3
Bounding Term II: Since for all E 0² 2 E - Consequently, we have
I 4c2, further have / /
S 202 2/c3.
(²))
a<(x))
m (
(x)) ( <
4 Ex og
(f(x)-1 (x)))
4
x2(x))
(2
As result, the following holds:
Var [ar=L(( 512) (0, - INLL
x c(++
+ 1 2 Ex(lNLL (1,) (NLL(O? (",,,
Next, move to the proof of the theorem statement.
C c+ oz + ) the Bernstein constant. Define the following function class:
Proof- Let B
== A
AlNuuloP. f: =xx ENLL - ( o
which is a composite function of hypothesis class F x 9. loss function L and the difference operation A. For simplicity we let A = MENLL(02, f: , 2) when there is no ambiguity. By definition of (f, we have (o F.z) s Pn ff, and therefore P S O.
Next bound PAr,o PnAee,o,, According to (34), have Var= |E To invoke Theorem 3.3 in Bartlett et al. (2005), we need to find a sub-root function wrr) such that
(r) > 2BEP,, (A E # BBA)
To find (r)), we show some analysis on the Local Rademacher Average EM,, (De E H: E[h?] S <r). Note that
(H,,H) = Es..ii oi
sup 1sr
and we have h(x:) E C- By leveraging some analysis from the proof in Corollary 3.7 in Bartlett et al. (2005), we bound {h E *H Ph2 <r} using (h E #H : P P,, I r} where the latter could be applied in the entropy integral. Since Ar,o² uniformly bounded by 2C, for any r 2 (b(r), Corollary 2.2 in Bartlett et al. (2005) implies that with probability least 1 1.
nt {h *H. : Ph <r) S (h E *H p2h s 2r). Let be < <) C {h E A p,, < 2r} holds, then we have
event that {h E *H ph
ER, Ph < r < P R HH.
2C
E[ (+H, P.A < 2r
Since V((*), satisfies
r < 20BCER, (=H, < 2r 22C log n
(35)
26

---

Published in Transactions on Machine Learning Research (01/2025)
Next we leverage the entropy integral (Dudley, 2014) to upper bound (*H, P,h² S =2r"} using the integral of covering number. Apply the chaining bound, it follows from Theorem B.7 (Bartlett et al., 2005) that
V2 fram
(*H, Sa..h"$2r' 2(r")]
const
E
Vlog *H, #1n dee,
(36)
where const is some universal constant.
Next we bound the covering number log.N2(E, x1:n) by log.N((E, G, vi.m + log. aim). We show that for all Fims the composition of an E-cover of F and an E-cover of 9 gives rise to a an E-cover of H. Specifically, let V c [ca, 4
be an s-cover of 9 on #i. so that for all o² E g, 3rn V so that
V / (o u;)² S 10//cst1/705 u C [-1,1]" be an s-cover of F on Uiah so that for all f E F,
Buim L so that, V LiE() (f(x.) - VTaec///16 Next, we show that for any Zims the family of
(u,-u13 (()))))-)) + log ( (m)) i E [n]] is an e-cover for H (x.)
(u: 11)a
(f=():) - y:)² log
Ui
A A
(2*(1.)
02 (2,))
iE[n]
(u+ - 11)2
(f(x;) - v1)2
+ loog 02(2)))
(2(xx)
24
iE[n]
(u: y:)2
(f(x;) - v1)2
+ (f(x;) vi)2 (f(z;) u1)
+ +log (2((z))
1i
27
Vi
1//
0² (²(x))
iE|n]
(u-- (wx))) + s(x;) - 2y,)
(f(z)) n1)²
-v;) (o²(x;; - vi)
+
+ llog (xx)))
U/3
0² (अ{z))
0i
1E|m]
1/2((2/7+16)6 - f(x,))2 (4e3/c3 + -))(x)))(),,)
:E|m]
<8c.
Clearly, above inequality implies that that N2(e, H. xim) S N2
I x N2 / &,
V1/G302//+10 F F1 F,0
Combining the above inequality with Corollary 3.7 from Bartlett et al. (2005) we can bound the entropy number of #H
(()+1))))))
E
Nog * #H, s log
H. x1m No
2
F. 01n ) (121 }
< <log Ne
" 11m No
E
4cg/c3 + 1/703"
Vi/e////+ 16)
Next we bound comst E Jo 2r
Nog *H, Iin from (36). Note that
log No 4c//0s cs + 1/7os"
J, No
8
F, F, 2i:n
/1/2//2// + 16)
sc(dr(ā) + (r((F)) log 2
+1 +
/ 1
27

---

Published in Transactions on Machine Learning Research (01/2025)
Consequently,
/rr
const
E
v Vlog log.Na N2(,N( *H, 1i de
vn
2
const
E log 2. H.
[ + de
E
21H 21m
Vn
(dp(g) + dF(F))
//2
const
log
(dp(e) + dd(pp)) F" log(1/63
+
const
(dr(E) +
+ (()())
c
n
where const corresponds to some universal constant. Together with (35) one can solve for
+dp (/0
Since PAENII f::0"*, f) E
(()))
o (w)
log )-(x)
one can leverage the S :/c
inequality 2 ++
(1-1)² e2) to conclude that
2 Sc.
En
(2) 02*(z))
This completes the proof.
Discussion Note that the risk bound of learning the variance function using NLL loss is also studied in Zhang et al. (2023), wherein Theorem suggests a rate of order ( On the other hand, the bound in Theorem 4.6 of this work is of the order (()). Such improvement of learning )²* might be of independent interest. Compared to risk bounds in Zhang et al. (2023), the major improvement comes from an application of the Local Rademacher Complexity (Bartlett et al., 2005) analysis under a Bernstein-typp condition.
A.10 Proof of Theorem 4.7 and discussion
Proposition 1. Suppose Assumption 1 holds. Let f E F. and w E W, and suppose we have W E W s.t. E. (()) wn (x)) < g. Given ii.d. samples S. = ((1; vi) ) drawn from the data generative process, let n Ex.s, (())((f:z)) Then for any > 0, > 0, the following holds simultaneoussy with prodability at least 1 0:
f e arg min 1
f()))) Eu F())) s,, E (x)(((())))) se, (4 Se,
as long as the sample size requirement is salisfied:
(o((()))) log(L) + log(c)) log(
+ (gg()
Proof. We start by defining the following function class:
H = loF= (ttp:-aa)-_at00aaaa-00000aaaaaa1
28

---

Published in Transactions on Machine Learning Research (01/2025)
which is composite function of hypothesis class F, loss function é and the difference operation A. To simplify the notation, we denote De, Aā(x)((f; f", x) when there is no ambiguity. Due to the fact that ())) is the empirical minimizer, we have:
Pno(z)((f:2) S (f
and thus PRA, S O. Leveraging such fact, next we show that PA of small via an empirical process argument. Note by assumption we have
PA 2.
- - - -.0000000000
and
PA t. (PA, ,)2
-Var Var=[[(a\(((.:=) - 21)]
<E.((())(((1))) - -((f, 2))²
(ED() f(x))))
Since (x)() (')))² s i. and (z)* f())) S we have - (() s e, which implies that (a)(()) [(x)) < 2 find subroot function (r) that
+e. To apply Lemma 1 next.
(r) 2 2FP 1 1A r. E H: EM r
Note, we have h(x)= E [-c1a, cia].
To find vfr), we first analyze on the Local Rademacher Average ER { , H. E|h2 S
1
(A
E.g
sup JEF. , n 1=1
By Lemma 3.4 from Bartlett al. (2005), it suffices to choose (r) - 10ctaER, HH2 < r]+ + Hefa log The following analysis largely follows from the proof in Corollary 3.7 in Bartlett et al. (2005) which aims to bound ER, (*,, PA22 < r} using ERR, (*H, P. < <r. Since A,7 uniformly bounded by 201 for any 2 v)r), Corollary 2.2 in Bartlett et al. (2005) implies that with probability at least 1 1 4. {h E *H PA < r S (h *H P2hh² < 2r). Let be event that {h # Ph2 < r} S (h E *H P,h s 2r3 holds, above implics
n
EDR (*H, Ph <t < PE HH PA < <r) PI (*H, Ph STHe
20
E + 7, P < 2r
Since x" = =b((r), x" satisfies
< a00R (*H, < 2r*) } 50c log
(37)
n
Next we leverage Dudley's 's chaining bound (Dudley, 2014) to upper bound ER, (*H, (P,, P2, < 2r*} using the integral of covering number.
Apply the chaining bound, it follows from Bartlett al. (2005) Theorem B.7 that
v2r
const
E (=H, 2
2r
E
/logNN N2(,,*H. *H, H1m))
(38)
29

---

Published in Transactions on Machine Learning Research (01/2025)
where const is some universal constant. Next we bound the covering number log,N2 H. x1:n) by F, We show that for all Fim, any e/c, T-cover of F is e-cover of H. Specifically, let V. C
[-1, be an Lcy-errv of F on Cin so that for all f E F, 30im E V so that // [iEm v,)² S c. Now we show that for any Zim, the family of ((x)) zi) (m;)) 2,)), x E [n] is an e-cover for H:
2 (( Ar.))
Σ ((a))(((,,,))))((11
Σ
((2s)(((0) - - {(((x)), {(((a)))
Y2 iE(m]
n2 rE(m]
L
(a:)( ( s 48.
(E(M)
Next, combining the above inequality with Corollary 3.7 from Bartlett et al. (2005), we have the following incquality on the entropy number:
logNN(E, *H, xim) s log N/2 ( Hr ,,, xin ((21+1) = log f Na 80, L.
((?+1))
E F. xi:n
Next we bound coust V
E So 22
/log.Nl(e *H, 11.) de from (38). Note that by Haussler's covering number
bound (Haussler, 1995), we have: log L F, an:n
I ed
Plugging such bound into the
entropy integral yields:
const
2r v JogNN(e, 21: $
const
22
E H, H1, 21:1 2
2 (121 +1 de
E
E
log N2
Vn
Vn
0
const
2r
E log.
(121 +1 de
E
F, 2im
Vn
8'
d(F)
2r
C
const
log
n
d(F)r* log(c)L/r*)
K const
n
(FFF) n²
d(F)r* log(c, Ln/ed(3 [F((F) FF)
IN const
n
where const represents some universal constant. Together with (37), can solve for
Since F,A, ,, S O, by Lemma we have PA ,, < 28 with probability at least 1 - 8 where the randomness comes from the training data S..
By assumption we have ([(x) - - s i. and (z) f(x)) < with probability at least 1 5, (x)((()) - - (z)) (I))) f(x)) se.
which implies that
Fm " (0, + + 38
Ex En f(x))) < 2E, (lo)) +
+ 3e.
30

---

Published in Transactions on Machine Learning Research (01/2025)
It can be easily verified that
32 Ex (L) 3e - (aID()(() 2d (P(([()) cc)
B Technical Lemmas
Lemma 1. Let F be class of functions ranging in [a, and assume that there some functional T - Bh+ and some conslant B such that for every h E H, Var(h) S Th) S BP[h] e. Let to be subroot function and r' be the fized point of tb Assume the vp satisfies, for any r rr,
(r) 2 BER (h E H: T(h) rr)
Then with C1 704 and 02 26, for any K > 1 and every t > 1 with probability at least 1 - e
K
ciK_
1(11(b a) c2)
vh EEH, H.P
P puh
const
K-1
B
n
Also with probability at least -
K+1
K
t(11(b - a) + cK)
vh E H. P,P.I S
Ph +
r"
+ const
K
B
where Pf = (m.(((a))) and P., = - Н M(x;))
Proof. The proof is similar to the proof of Theorem 3.3 from Bartlett et al. (2005) except that here we are modifying some step so as to apply the argument under the condition T(h) s BPh + E, instead of the original condition T(h) & f. We introduce notations and concepts: given class H, > and r > 0, we let wh) = min(r> CEN,, r 2 T(h) and set
r h G. = =={ure w(1))
And similar to Bartlett et al. (2005) we define
sup
(Pg Png), Vr sup (PPngg Pg). gEG.
gEG,
Next modify the proof step of Lemma 3.8 from Bartlett et al. (2005). Suppose K> > 1,A 0 and 0. We aim to prove the following two claims;
K P.f
if V ABK then yfEFPfS
(39)
K
ABK K-1
ir V,
XBK then Aʃ FF
K
+
1
PS
E XBK
(40)
K
When T(h) < use the same conclusion the one in Lemina 3.8 in Bartlett et al. (2005):
Ph S P, V S P,le T
ABK
In the case T(h)= > r, we have whh) = r with & > and T(h) E (r2 Moreover, g = h
XE,
Pg < P.g + V thus
Ph A
Pub x + V Since T(h) > -- we have:
XT(h)V+ S P, +
Ph E
Ph < Pah + V < P
K K
K
Ph <
P h
K 1
K K-1
XBK
31

---

Published in Transactions on Machine Learning Research (01/2025)
Let 2 r", applying Theorem 2.1 from Bartlett et al. (2005), we have for all 0 < 8 s 1, with probability at least 1 - 8:
V s 2(1 + a)ER + 2r log(1/0) + (b --) + 2 log(1/0)
n
Let H(u, u) - (h. E MH: u S T(h) I w) and define & to be the smallest integer that r 2 Bb + e. By assumption we have Th) z BE[b]+ +e, and ((r) a sub-root function that 6(r) 2 BER, {h H. T(h) s we have:
r
ER, &
+ EE
(cH(O. w(h)
sup
Rinh
k
h Σ x /=
s ERHH(0,r) +
Σ
E
hE heH(rb,rrdii) w(h) Rinh = EXH((0,)) +
sup
sup he he
Rah
(())
* (r2it11)
$
B
1=0
Since v is sub-root function, we have for all B 2 (b(rr) S (BD((r) Hence,
ER, S (1 VA 113))
1
B
1=0
Similar Bartlett al. (2005) we can setting À to bound RHS by 5w(r) Since r r2 v1()s (r/r" = Vrrt, we have:
V < 10(1 + a)
Vrr + +V 2rx + +(b +b--a) 1 log(1/8)
B
Next we set A A= 10(1 aa) vF B + 210g(1/6) and C= (b ()33 11/3+1/00 1/a) 1og(1/5)/n so that V + < AVF+C. It can verified that r can be chosen such that V,+ S TAK The largest solution of AvF + XBK denoted as rg. One can verify that ro no less than X B² which is no less than r*. Meanwhile ro S (ABK)2A² 2XBKC, by the claims in (39) and (40), one can show that for all h E H,
K
Ph <
K - Fuh + ABKA2 +20
K
K
20(1 a)
2log(1/5)r
2 2log(1/0)
K Ph + ABK(100 a)
B
B
n
n
+ (b - (b- + ) 1og(1/0)
E
+
K
Setting a = 1/10 use the fact that 2/uv S + av completes the proof.
Next lemma is largely from Lemma 5.2 Klein & Young (2015) and Zhang et al. (2024). We include the proof here for completeness.
Lemma (Chernoff type lower bound). Let be the average of k independent, 0/1 random variables (r.v.).
For any E 1/2] and p E (0, 1/2], assuming epk 2 pk 2 6, S 3 we have:
If each r.v. is with probability p, then
51- > 0.2e 6c2pk
If each r.v. with probability p, then
P[X > (1 +ppp
0.2c 66
32

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. With Stirling's approximation, with i! approximated by v2ri(i/e)'e> with A E [1/(12i + 1),1/12 one can show:
k i
k k-t
(41)
eV2T
k
Since P[X& & (1 e)pl = E P[X # it suffices to provide a lower bound for >i 2i-(1-22)p PIx = 1l
where PfX = El 1:
p² -)- (Klein & Young, 2015).
To this end, let 6 = 1(1 p. Given the fact that epk 2 6, we have (1 - 2c) - 15 2e)pk. We
have E(T-e 21=(1-2e)p P[X t1 at least
epkp(x 1== sph
ev2ml (1))
( & k-E ) pp(1 =e)
p)m
From Equation (41) know that we need to bound A = cpp Pitl/V22T and B =1) (3)(xx1)) (01))~-(((1
Df -p)8- 2 - Since 31
(--1)
For term A, since epk 6, s (1 - 2s)pk: thus we need pk 2 90 to gel
v
e1/2m(1-2e)
it suffices to have ph 2 16. To bound B we need to show:
k
k
k-1
p'(1-p)** 2 4
k
Since (t)' Df 2 (1-26)) )' and (1 - p)h-1
)
k-1
(1-p)h
we have:
k-1
h(1 (11-p)+1+2ppp
(1 20)8 (1+
1 + 227k
k-t
2x2g2 T-p Tep
+2spk 22akk 2e pk+2 76 16
e
k(1 p)
33


Reviews:
[{"review_id": "omWTsz6P94", "reviewer": "Reviewer_7fek", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents a new viewpoint for achieving fast rates that does not depend on the minimal margin.\n2. The weighted ERM rule provides a natural way of abstention on hard instances, which may be useful in practice.\n3. The general \"Balanceable Bernstein condition\" may be interesting on its own, allowing more flexibility in selecting the weight function.\n\n**Weaknesses**\n\n1. The main weakness, in my opinion, is that the results rely heavily on the assumption that the weight function can be estimated under the $L^2$ norm. This is questionable in practice. Although the authors mention that this can be estimated by assuming the weight function comes from a class of finite pseudo-dimension, this seems quite restrictive to me. Can the authors provide any natural hypothesis class for the weight function in specific real-world tasks? (Your experiments choose a very specific function, $0.09(1+x^2)$, which seems artificial to me.)\n\n2. It appears to me that the technical originality is minimal, as most of the proofs boil down to local Rademacher complexity analysis. (This is not crucial given the TMLR acceptance criterion.)\n\n3. The technical writing is not very good, especially for the proofs in the appendix. I have some specific comments below:\n   - The equation (7), as stated, is unclear whether it is a consequence or an assumption. I suggest the authors clarify that it is a consequence and provide a proof in the appendix (although it is similar to the standard Massart noise case), as the condition plays a central role in the main results.\n   - I suggest the authors re-define the symbols used in the proofs in the appendix to save readers from constantly checking back to the main text. For example, in the last equation on page (18), I do not understand why $\\hat{f}$ suddenly becomes $f$.\n   - I recommend the authors provide more intuition at the start of each proof and refrain from using words like \"clearly\" (as it is not always clear, especially given the typos).\n   - I suggest the authors do a thorough proofreading to catch the typos, especially since the subscript for $\\mathbb{E}$ is sometimes inconsistent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of PAC learning with fast rates when the margin condition of the noise (such as Massart's noise) is not available. The paper shows that if one can estimate the margin up to a small error under the $L^2$ norm, then an empirical risk minimization weighted by the margin achieves fast rates on the same weighted population risk. The authors further demonstrate that such a guarantee on the weighted population risk implies population risk on the original loss restricted to a level set defined by the weight. In particular, this provides a PAC learner with a rejection option that outperforms bounds depending on the minimal margin. The paper then generalizes the classification scenario to regression and introduces a general condition on the weight named the \"Balanceable Bernstein condition.\"", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the points in weakness section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gpiA2B9cUD", "reviewer": "Reviewer_ZpQh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The overall idea is novel (as far as I can tell) and interesting.\n\n- The authors have derived several theorems to demonstrate the non-triviality of their theoretical findings. Theorem 4.1 shows that reweighting can eliminate the slow-decaying term in the generalization gap, though this result might not be entirely new (I am uncertain). However, this comes at the cost of limiting the analysis to large-margin regions. Theorem 4.2 illustrates that standard ERM has a non-decreasing lower bound everywhere, while the error of the weighted ERM can become arbitrarily small in large-margin regions given sufficient samples. Theorem 4.3 provides theoretical guarantees that the weights can be well-approximated using a relatively small number of samples. The remaining theorems, including Theorem 4.7, which offers the most general result in this paper, share a similar nature, though they vary in their assumptions and scopes.\n\n- There are several technical discussions and remarks interspersed between the theorems that provide valuable insights to the reader.\n\n**Weaknesses**:\n\n- The presentation of the paper could be significantly improved. Some explanations are repeated several times within the first 4-5 pages.\n\n- In line with the previous comment, the order in which the theorems are presented adds to the ambiguity. For example, in Theorem 4.1, the authors initially discuss a seemingly deterministic setting (where the randomness inherent in $\\widehat{\\omega}$ is implicit) and then suddenly assume that an inequality holds \"with probability at least $1-\\delta$\". Probability with respect to what randomness? This becomes clearer later when Theorem 4.3 is introduced. While it's understandable that the authors want to present the main results early, this has come at the cost of readability.\n\n- **The main weakness**: The results seem adequate, but honestly not too surprising without Theorem 4.3. Therefore, this theorem plays a crucial role in the overall technical contribution of the paper. However, there is an assumption in Theorem 4.3 that I find difficult to accept as mild or trivial. The authors assume i) knowledge of and access to a set $\\mathcal{G}$ with low complexity, such as a bounded pseudo-dimension, and ii) that this set simultaneously contains the optimal weights $\\eta^*$ (Bayes probabilities). This assumption facilitates the use of highly desirable bounds from the realizable setting in classical learning theory, enabling the rest of the theorems to function. However, this might be equivalent to assuming that the entire problem is being solved in a realizable setting where we already know a low-complexity hypothesis set that contains the true hypothesis. This makes the result somewhat equivalent to many prior works. I would like the authors to clarify this issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The aim of this paper is to propose a reweighting procedure that improves the generalization gap of standard (non-weighted) Empirical Risk Minimization (ERM). However, this improvement comes at the cost of the bound being applicable to more restricted regions of the feature space, rather than being global as before. The work is primarily theoretical, with some experimental examples on synthetic data, which I have not verified. The authors theoretically demonstrate that both the optimal weights (and thus the corresponding regions) and the new ERM estimators can be approximated for any given small error $\\varepsilon$, with a significantly reduced sample complexity of only $O(1/\\varepsilon)$, while still ensuring reliability (i.e., with high probability).\n\n**Core Idea**:\n\nThis work builds upon existing variance-based bounds that leverage Bernstein-like properties, such as assuming $\\mathrm{Var}(h(z))\\leq B\\mathbb{E}[h(z)]$ for all $h \\in \\mathcal{H}$, where $z$ denotes the feature-label pair $(x, y)$. In this context, each $h$ typically represents the difference between the empirical/statistical loss of a given classifier in the hypothesis set $\\mathcal{F}$ and the optimal classifier, denoted by $f^*$. These bounds consist of a fast-decaying term of $O(n^{-1})$ and a slower-decaying term of $O(n^{-1/2})$, where the latter is scaled by the constant $B$. While $B$ is a constant, it can be large in some cases. To address this, the authors propose using a weight function $\\omega(x)$ for the feature vector $x$ such that the inequality $\\mathrm{Var}(\\omega(x)h(z))\\leq B'\\mathbb{E}[\\omega(x)h(z)]$ holds for a much smaller $B'$ compared to the original $B$. This results in a tighter bound, albeit applicable only to more specific regions of the feature space—those where the weight function $\\omega(x)$ is significant.\n\n**General Assessment**:\n\nI have not yet verified all the proofs, but I have not identified any notable mathematical errors. The overall framework appears sound, and the results seem to be mathematically rigorous. However, the presentation of the paper leaves significant room for improvement. I have a major concern about the paper that I expect the authors to address. If this issue is resolved, I believe the paper would be a good candidate for TMLR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Minor comments**:\n\n- \"can be equivalently view\" -> can be equivalently viewed\n\n- In the statement of the some theorems: What do you mean by $g\\\\,\\mathrm{or}\\\\,\\omega\\in\\mathcal{F}$? $\\omega$ is the weight function and $f$ is the classifier. Do you assume they belong to a same hypothesis set?", "broader_impact_concerns": "I have no concerns regarding this issue.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ru9z2xibTj", "reviewer": "Reviewer_BugY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n\n1) The paper is well-written, the literature review considers various aspects of the problem, the motivation is clear, the derivations of the main results are well-structured and easy to follow. The proof technique is quite solid though significantly relies (Bartlett et al., Ann. Statist., 2005).\n\n2) The results show that standard ERMs, though staying minimax optimal, may be outperformed by the weighted counterparts in some subregions. This refutes the argument of Zhai et al. (ICLR, 2023) on futility of weighting schemes.\n\n\nWeaknesses.\n\n3) The main limitation of the present paper is that the authors focus on restrictive well-specified settings only. The agnostic case study is left for the future.\n\n4) In the literature review, the authors discuss that reweighting is widely used for distribution shift adaptation and that it is also related to learning with reject option. However, the authors do not discuss how Theorem 4.6 relates to distribution shift nor compare weighted ERM with other algorithms in the context of classification with reject option.\n\n\nDespite the fact that the authors consider restrictive realizable setup only, I am mostly positive about the present paper. However, I must mention that extension of the results to the agnostic setting may be extremely non-trivial and may require different techniques. The assumption that $\\omega^*$ is bounded away from zero is very similar to the Massart margin condition. Fast rates of convergence for ERM under such a condition usually require $\\eta^*$ to belong the  class of concepts $\\mathcal F$. The only exception I am aware of is the paper of Puchkin and Zhivotovskiy (IEEE Trans. Inf. Theory, 2022), where the authors argued that the assumption $\\eta^* \\in \\mathcal F$ is not necessary once $\\mathcal F$ has finite both star number and combinatorial diameter (see Section IV). These assumptions are also quite restrictive and I am afraid that the authors will need similar requirements in the agnostic setting.\n\n\nMinor remarks.\n\n(a) Please, add references to the definitions of VC-dimension and pseudo-dimension. I think, it would be useful for those readers, who are not very familiar with learning theory. \n\n(b) Please, describe the meaning of the inverted $\\simeq$ sign (see Th. 4.4). \n\n(c) (p.3) In the paragraph on reject option, the authors mention the paper of Klochkov and Zhivotovskiy (NeurIPS, 2021) on algorithmic stability, which is not related to prediction with reject option. Probably, the authors meant another reference.\n\n(d) (p.7, Theorem 4.2) Misprint in the formula for $S_n$: there should be $i = 1$, instead of the subscript $i$.\n\n(e) (p.8, Theorem 4.3) $d_P(\\mathcal G) \\leq \\infty \\quad \\rightarrow \\quad d_P(\\mathcal G) < \\infty$.\n\n(f) (p.9, eq. (13))  Misprint: a closing bracket is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1)  In the paper under review, the authors study statistical properties of weighted empirical risk minimizers. They are interested in situations when reweighting helps to outperform standard ERMs. In (Zhai et al., ICLR, 2023), the authors argue that generalized reweighting algorithms lead to estimators which are very close to ERM. However, they miss the dependence on problem-dependent constants (e.g., margin). In the present paper, the authors show that weighted ERMs provably have better sample complexity compared to ordinary ERMs in several scenarios. The improvement is characterized by the factor $1/\\gamma$, where $\\gamma$ is a margin parameter.\n\n2) The main result of the paper is Theorem 4.7, considering a general case. It is preceded by two particular examples: binary classification (Theorems 4.1 and 4.3) and heteroscedastic regression (Theorems 4.5 and 4.6), where reweighting naturally arises. The upper bound in Theorem 4.1 essentially matches the lower bound form Theorem 4.4. Besides, the authors show that wERM performs better than ERM in large margin regions  (Theorem 4.2).\n\n3) The key instrument for the theoretical analysis is a weighted counterpart of the Bernstein condition called balanceable Bernstein condition. It plays essentially the same role as the standard Bernstein condition in derivation of fast rates of convergence for ERMs. However, reweighting helps to avoid distribution-dependent constants appearing in the ordinary Bernstein condtion and, as consequence, leads to tighter bounds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I encourage the authors to address the following comments before I can recommend acceptance.\n\n1) Please, elaborate on implications of Theorem 4.6 in the context of distribution shift and classification with reject option.\n\n2) Please, discuss the importance of the weighting function $\\omega^*(x)$ to be bounded away from zero. Is it possible to prove similar bounds if $\\omega^*$ is a $[\\gamma, 1]$-valued function but the other functions from $\\mathcal G$ can take values in $(0, 1]$?", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There are two core critiques of this work.

First, all reviewers argue that assuming that the weight function comes from a class of finite pseudodimension, as the authors do, is quite strong. In spite of the strength of this assumption, two reviewers do still support acceptance. In response, the authors argued that in the present era of machine learning, there are function classes over which we do learning (certain classes of deep networks, for example) that have finite pseudodimension. I think it would be tough to make the argument that we can regularly do learning over such classes, say, a decade ago; however, at this point in time, the argument is at least moderately convincing. Therefore, I am willing to accept this assumption.

The second core critique is that due to the assumptions, the analysis essentially boils down to using technical approaches that work under realizability. Moreover, it does not seem that these ideas can be extended to agnostic situations. However, given the history of results that are proved in realizability settings, and the new perspective offered by this work (by allowing for the weight function), I still do think that there the results herein are sufficiently interesting to have an audience within the learning theory community (and hence, to have some audience within TMLR). Even so, the critique stands that there do not appear to be new techniques in this work. However, it is not necessary to introduce new techniques. The novel balanceable Bernstein condition, combined with the new perspective and potential applications to classification with a reject option as well as distribution shift, make this paper interesting enough for acceptance.

I recommend this paper be accepted without minor revisions. Even so, for the camera-ready version, please do another careful pass to check for any remaining typos.
Decision: Accept (poster)