The weighted mean trick – optimization strategies for robustness
Authors: Valeriu Balaban
Paul Bogdan
Venue: ICLR 2022 Submitted (2022)

Abstract: We prove that minimizing a weighted mean results in optimizing the higher-order moments of the loss distribution such as the variance, skewness, and kurtosis. By optimizing the higher-order moments, one can tighten the upper bound of the loss mean deviating from the true expectation and improve the robustness against outliers. Such types of optimization problems often lead to non-convex objectives, therefore, we explore the extent to which the proposed weighted mean trick preserves convexity, albeit at times at a decrease in efficiency. Experimental results show that the weighted mean trick exhibits similar performance with other specialized robust loss functions when training on noisy datasets while providing a stronger theoretical background. The proposed weighted mean trick is a simple yet powerful optimization framework that is easy to integrate into existing works.

Under review as a conference paper at ICLR 2022
THE WEIGHTED MEAN TRICK OPTIMIZATION STRATEGIES FOR ROBUSTNESS
Anonymous authors Paper under double-bling review
ABSTRACT
We prove that minimizing weighted mean results in optimizing the higher-order moments of the loss distribution such as the variance, skewness, and kurtosis. tighten the upper bound of
By optimizing the higher-order moments, the loss mean deviating from the true expectation and improve the robustmess against outliers. Such types of optimization problems often lead non-convex objectives, therefore, we explore the extent to which the proposed weighted mean trick preserves convexity, albeit at times at decrease in efficiency. Experimental results show that the weighted mean trick exhibits similar performance with other specialized robust loss functions when training on noisy datasets while providing a stronger theoretical background. The proposed weighted mean trick is simple yet powerful optimization framework that is easy to integrate into existing works.
INTRODUCTION
The most common objective in machine learning is minimize the average loss. However, by doing this it implies that all samples are equal. The goal of this work is to convince the reader that from the perspective of the model, not all samples are equal and optimizing a weighted mean a more progressive approach. For instance, at the end of the training, few samples are left uncaptured by the model which yield large loss values. One might choose to put more weight on those samples to help the model learn (individual fairness), or consider them noise and assign them less weight (robustness). The decision will also be reflected by the loss variance; will decrease when hard samples are weighted more and increase otherwise.
Before introducing more formally the weighted mean, we first justify theoretically the impact of variance penalization. Using the empirical Bernstein bound (Maurer & Pontil 2009) for i.i.d. loss values and bounded variance we have with probability 1
1
2V 2 2n/6 + C2 7
B[Z]
Z: < CiV
(1)
3(n 1)
where C1, C2 problem depended constants. This incquality reveals two things. First, we have with high probability that the empirical mean is close to the theoretical value. This bound along with similar PAC-Bayes bounds (Seldin al.) 2012 Tolstikhin & Seldin 2013) justifies the practical success of the empirical risk minimization (ERM) which has the objective to minimize the mean loss value (Namkoong & Duchi 2017). Secondly, the difference between the two is bounded in terms of the empirical variance. This second implication led to a large and growing body of studies that investigate
variance penalization (Maurer & Pontil| (2009] Namkoong & Duchi, 2017: Duchi & Namkoong 2019 Stabb et al.) [ Lam [019]] Heinzee-Deeee -Deml & Meinshausen] 2021 Hu et al. 2018). Moreover, by penalizing the variance, the intrinsic bias-
tradeoff of the ERM can be controlled. Two studies influential for this paper are that of Duchi & Namkoong (2019) and et al. (2021). Duchil & Namkoong (2019) proposed taking the expectation with respect to a different distribution which allowed penalizing variance while preserving the convexity of the objective. Similarly, Li al. (2021) investigated optimizing a tilted empirical risk which is equivalent to penalizing all the higher-ordd moments simultaneously. In summary, previous methods either penalize only moment or all the higher-order moments but are not flexible enough to penalize any desired combination of the higher-order moments.

---

Under review as a conference paper at ICLR 2022
Inspired by the above works, propose to optimize weighted mean and prove that for certain weights, it is equivalent to optimizing any higher-order moments of the loss distribution such as variance, skewness, and kurtosis. Our approach generalizes that Duchi & Namkoong (2019); Li iet al. 2222)) while simplifying the optimization procedure and enabling separate penalization of the higher-order moments.
In particular, we will construct the weights w such that optimizing the weighted mean of the loss / is equivalent to applying variance penalization:
E [ = B[A] + X
or penalization of other central moments (e.g., skewness, kurtosis). Construction of these weights are treated by Theorems [ and 3 from section 3 which require minimal computational resources. Important note, penalizing directly the variance preserves convexity only for values of À within very narrow range (Maurer & Pontil 2009).. However, the weighted mean method yields a convex objective for any positive values of X.
In detail, our contributions are as follows:
(C1) We build upon the work Duchi Namkoong (019)) and prove that optimizing a weighted mean is equivalent to reducing amplifying both variance (Theorem and higher-order moments (Theorem 3
(C2) We derive the limits of the penalization interval which preserves convexity (Lemma 2 and also show how penalize with values outside this interval while still maintaining convexity (Lemma
(C3) We connect the variance and higher-order moments penalization using weighted mean the robustification of loss functions (Lemma S and Lemma 6).
(C4) We develop convex version of the variance penalized cross-entropy loss which provides higher accuracy in high noise scenarios with class dependent noise.
(C5) We show experimentally that negative variance penalization improves model accuracy when training with noisy labels.
The implications of the weighted mean are much broader than what we investigate in this work. We limit the scope of this paper to classification using deep neural networks trained with noisy labels. But the mathematical framework also covers the control of the birs-variance trade-off and can also be applicable to regression problems.
In the sequel, we start by introducing the notation in section 2 then in section 3 we present the moment penalization strategy and the weighted mean trick which is a computational technique to make moment penalization practical. Next, in section 4 we illustrate the application of moment penalization using the weighted mean formulation optimize for robustness when training with noisy labels.
NOTATIONS
In subsequent sections we will use the following notation. A training data set is defined as D v where x; E X' the features and y; E .k represent the class labels. A classifier [xx; 0) is mapping / X' x θ + V from the feature space to the probability simplex V parameterized by 6. A loss function l x y + oo) gives penalty f(v. y) when the model predicted the value u and label was observed. A weight function
V x y - R assigns to each sample a weight. As we are interested in optimizing the model output value that minimizes the penalty, we will focus our investigation on loss functions l(v. y) that are convex in and seek to preserve the convexity when optimizing the weighted mean. ERM, we are interested with finding the model parameters 0 that minimize the empirical risk calculated as [p[ 0), v:)l and the weighted form Ep (xx 9), y/) (xx;); 8), 34)1 To simplify the notation, we will drop the dataset D from the expectation subscript and the arguments of the loss and the weight function, (z ie., E[e] + ED[ [xx 8), y:)] and E| [ + ED[w(E(f 6), (:)) (z 8), ys ). Similarly, the notation for minimum is simplified min miD ([f(x; 0), y.).
2

---

Under review as a conference paper at ICLR 2022
A<0
3
-0.0
-13
A<0
A= 0
200
200
0.5
A >
33
0.0
2
0.
1.3
-3
0
-3
3
3
Figure I: Variance penalization for classification problems. Contour lines of the weights distribution for positive À are shown on the left and for negative on the center plot. The right plot shows the use of variance penalization for outlier suppression amplification (robustification versus generalization).
PENALIZING MOMENTS
In ERM, the impact the model parameters of sample is determined by the balance between its loss value and the average of the training batch. Especially observed in the latter stages of the training when majority of samples have small loss value except few which are left uncaptured by the model. Case in which we consider them as unlearned samples and amplify their impact or as outliers and suppress their impact
the model. The weighted mean trick consist in assigning weights each sample based on the loss value to either amplify or suppress their impact on training. This is similar to dropout (Hinton et al., 2012) but for losses. However, weighted mean trick is a deterministic process and the weights are not restricted to 0 or 1 but can take any non-negative value.
In what follows, we apply the weighted mean trick to extend the ERM framework to multi-ohjective optimization of the mean and nigher-order moments of the loss function. First, we show that for some distinct weights, optimizing the weighted mean is equivalent to a simultaneous optimization of the mean and variance.
Theorem (Variance Expansion). Let l be loss function with finite E[(] and finite Vid] and let w(v,y) v) 1 + A(((v, B[C(v, u)l), then we have:
E wfl B[A LAAK
(3)
Proof. Replacing w with the definition and using the linearity property of the expectation along with Proposition 7 we get: E[we]] = [[{]+ XE (E-E((() = E[ + AE (E E = [[(]+V[[
Thus, switching from mean weighted mean allows us control the bias- tradeoff through to improve the distributional robustness of the model (Maurer & Pontil 2009). However, the range of \ values that preserve the convexity of the objective depends on the average and the minimum penalty returned by the loss function E as shown by the next lemma.
As introduced in Theorem 7 the variance expansion of a convex loss function E(V. y) in v yields new objective that is also convex in if [0, Amax), where Xmaz 1/ min f).
Lemma 2.
Proof is provided in Appendix A This lemma precisely shows why directly penalizing the variance does not preserve convexity besides when takes values in narrow interval. Moreover, directly penalizing the variance as part of numeric optimization objective, the upper limit of the interval, Amaz, is not constant and changes with each iteration of the optimization algorithm. Note that the further the minimum value min is from the average loss value [[C]. the narrower the interval is. Conversely, when E[{] = min l results that is constant and V[ = thus the objective is convex for X > O. However, using the weighted mean trick the objective remains convex for any positive irrespective if V[4 O.
Figure I shows the weights distribution for different values of X. When À is positive (left plot) samples closer to the decision boundary (which also means with larger loss values) receive more weight. On the other hand, when À is negative (center plot) samples with larger loss values receive
3

---

Under review as a conference paper at ICLR 2022
less weight. Of note, for À < 0 the objective is not convex, however we will show later will still converge to an optimal solution. The right plot shows a binary classification problem where each class consists of two clusters, squares and circles. The parameter À controls the placement of the decision boundary with respect to these two clusters. Positive À values place more weight on the cluster of squares which is closer to the decision boundary and as result the boundary is horizontally aligned. On the other hand, negative À values place more weight on samples farther from the boundary and in this case the cluster of circles. Therefore, this aligns the decision boundary with respect the cluster of circles oriented diagonally.
Variance not the only central moment that can be penalized using the weighted mean. In fact, any combination of the moments can be penalized. The next result generalizes Theorem !
Theorem 3 (Moments Expansion). Let E be a loss function with finite first m central moments and define w(v y) Ei=1 \e(((v, v) [[((v, y)l) then we have:
m
F[we] A[e +
Σ X [()-)(4))
where A, = b2 + [4] m and Xm Am
Proof is provided in Appendix A We notice that penalizing moments higher than two incurs an additional penalization of the previous moment. For example, penalizing skewness by X incurs a variance penalization of Ag]B[e].
Theorem 3 also has an algebraic interpretation. Note that the formula for weights is nothing more than a polynomial in f(v, y) translated by E[{]. When penalizing the variance, X2 controls the slope of the linear equation, and when penalizing the skewness, λ controls the curvature of the quadratic equation. Moreover, the penalization factors A; also deñne the placement of the roots of the polynomial and the convexity of the weighted mean objective.
Lemma 4 (Convexity of Moments Expansion). Let E(v.y) v) be loss function convex in and p:R [0, oc) be a non negative and differentiable convex function and M z O, then the weighted objective w(v, f),(v, v) with w(v, v) p(((v,,))) v) MM) is convex in v if p is non decreasing.
Proof is provided in Appendix A The above result lists two requirements on the polynomial function p such that the weighted mean objective remains convex. First, the function must be non-negative, thus, the negative weights must be clipped to 0. Secondly, when the function takes positive values, p must non decreasing and convex. Of note, this result similar scope to Lemma 2 but does not generalize it as the weights are non negative. Isgure several examples of polynomials. Variance penalization implics p is an affine function, and thus, convex. However, for X polynomial is non decreasing (left plot) and for A₁ < 0 the polynomial is non increasing (center plot). Thus, only positive variance penalization will result in a convex objective. Of note, Lemma [ upper bounds λ₁ if weights not clipped to 0, however, if clipping is used A is not upper bounded. When penalizing skewness, is a quadratic function (right plot). In this case, if X > then is convex and non decreasing only on restricted interval instead of the entire real line. Thus, convexity can be preserved by appropriately adjusting the roots of the polynomial such that p is convex on [min l [[]), +oo).
the
Clipping negative weights to 0 prevents the optimization objective to switch from minimization to maximization, usually an undesirable behavior. Consequently, the samples with corresponding zero weight reach their maximum contribution in the moments penalization. Further increasing the factors A₁ will have no effect on those samples and only samples with non-zero weights will participate in the training. As a result, the efficiency of the moments penalization will slightly fall. Moreover, the samples with zero weight will be excluded from training, technique used in the past by multiple studies. Rockafellar & Uryasev (2000) used similar clipping technique to optimize only for samples that are part of the tail of the distribution.
The Moments Expansion Theorem extends the variance expansion proposed by Duchi & Namkoong (2019) to include higher-order moments. However, even when only the variance is penalized, the two methods are still slightly different. Moments Expansion Theorem computes the weights directly from the loss values, whereas the variance expansion Duchi & Namkoong (2019) solves a secondary optimization problem to find the weights. The of secondary optimization problem has the
4

---

Under review as a conference paper at ICLR 2022
3
0-
-1
-2 km
Ao
-25
Figure 2: Polynomial functions for moments penalization. Dotted lines show the complete polynomial whereas solid lines the clipped version. Left plot shows two convex and non decreasing polynomials. Center plot shows a convex but non increasing polynomial and thus will result in a non convex objective. Right plot shows convex and non decreasing polynomial on the interval [-1, +oo]. and thus will yield a convex objective when min(e) - E.e] 2 -1
advantage of penalizing the variance more consistently despite being more computationally expensive. On the contrary, when using the moments expansion, the penalized variance can be slightly lower depending on the number of weights that are However, the advantage is that the direct computation of the weights makes it easier to include the method into existing analysis frameworks. A similar method that extends the optimization objective to include penalization factors for higher-order moments proposed by Li et al. (021). The proposed method replaces the ERM objective with tilted version calculated as 7 (I) = i log B[c""] where ((t) is the sumulant-generatinn function of the loss f. The penalization of the higher order moments of the tilted ERM can be recovered from the power series expansion of K (t). The distinction between the two is that the moments penalization introduced in this paper represents generalization of the tilted ERM as it allows any combination of the higher order moments to be penalized whereas tilted ERM uses a single parameter that governs the penalization factors. In summary, the moments penalization implemented using the weighted mean trick is more flexible, however, it comes at cost as there are more parameters to tune when penalizing multiple moments compared to tilted ERM of et al. (202T).
Convergence and convergence rates. The moments penalization problem along with variance expansion of Duchi & Namkoong (2019) fall under the class of distributional robust stochastic programşSun & Xu (2016) (DRSP) which is subclass of ambiguity mogram & Wets (0117) (AP) where the general objective is:
AP: min sup SEE PED(O)
(00 P)
where θ is the set of model parameters and D(0) is the ambiguity set. In DSRP, the bivariate function 50, P)= = Ep[[] where É is the loss function and the ambiguity set D(0) is a set of probability distributions. In the case of moments penalization problem, the ambiguity set ((0) depends on the model parameters and a singleton as the weights uniquely transform the empirical distribution. As result, the optimal value of the inner maximization problem becomes suppe ((8, [[]]([]]]] Intuitively, model will converge if changes in its parameters will cause minor changes in the distribution P. and with each step the distribution will approach the optimum distribution p Formally, to quantify the changes in the distribution, we would need a distance or metric. (Sun & Xu (2016) use the total variation metric and a pseudometric to prove uniform convergence, possibly at an exponential rate, if P converges to p under total variation metric and É is uniformly bounded (see [Sun Xu 2016) Th. and Prop. 3). Royset & Wets (017)) proposed hypo-distance metric and proved lop-convergence given that the bivariate function ((0 P) satisfies some assumptions, (see Royset& & Wets (2017) Def. 4.1). Duchi & Namkoong 2019) provide guarantees for a number of stochastic risk minimization problems when only the variance is penalized and P is in the local neighborhood of the empirical distribution defined using the 2 <dvveergeeece. We refer the reader to the works of Sun & Xu 2016) and Royset & Wets (201)) and the references therein for additional guarantees if more information about the problem structure is available, or if other metrics are used. For the moments penalization problem, the moments penalization factors X for > 2 determine how
5

---

Under review as a conference paper at ICLR 2022
much the distribution P changes when the loss changes. Small values of the penalization factors will keep P in the neighborhood of the empirical distribution, whereas large values will make the weights sensitive to changes in the loss values that can cause stability convergence issues. The exact values depend on the empirical distribution of the data and the choice of the model and loss function.
Weighted mean trick in practice. To apply the method in practice, the classical batch training algorithm must be extended to include an additional step, the weights calculation. Instead of directly calculating the average loss, the user will calculate the loss value for each sample in the batch and then use the expression from Theorem 3 to compute the weights and the weighted mean. The moments penalization factors X are the
and are tuned in ascending order with respect to i. However, penalizing higher-order moments might affect the impact of the lower-order ones, and thus, might require a few iterations to find the optimal combination. The implementation of this algorithm in Pytorch (Paszke et al., 2019) is available on GitHub[ Since the gradient of the weighted mean the weighted gradient of the elements, this allows weights to control the impact of each sample on the model parameters, Of note, the theoretical results hold when switching from expectation to sample expectation, EE, [$] = Σ {(f(z, 0).
Algorithm I: Training with Moments Penalization
(x;;)
model to be trained
input: { W: 1a
batch of training data
[X]
penalization factors
f(v. v)
loss function
while stopping criteria not reached do for i + 1 to n do
23 (( 0), :)):
/* sample loss x/
w E 21(2 E., (=((=)>-11
fuw Σn W
/* weighted mean x/
0
1
Voow
/* update model parameters */
4 ROBUST CLASSIFICATION
In this section, we will explore the robustification of the cross-entropy function under label noise by bounding the loss values using negative variance penalization. In this case, the resulting objective will not be convex. However, for small penalization factors the objective remains convex on almost the entirety of the domain (Figure and does not hinder the convergence. Moreover, we also develop a convex version of the variance penalized cross-entropy though for a minor price in performance.
The objective of robust classification to learn an optimal classifier f" that minimize the average loss for both clean and noisy data. Formally, f = argmin [( 0), 9)] where á represents noisy labels. The noise considered in this paper corrupts the class labels with probability P(j, * y:) n or preserves it with probability P(A: = yi) = 1 7), where 1) [0. 1]. We investigate two scenarios: when 1) does not depend on the class label (class independent or uniform noise), and when 7) depends on the class label (class dependent or asymmetric noise). In what follows, we show that using negative variance penalization factor, A2 < O, bounds the loss function. By bounding the loss function the impact of misclassification of noisy samples the average loss will decrease, and thus reduce the noise impact on the model during training.
Ghosh et al. (2017) outlined the distribution independent sufficient conditions for a loss function to be robust under both, class dependent and class independent noise. Specifically, if the loss function satisfies the symmetry constraint:
k
((((=,0)))))
(6)
where l is the number of classes, C' is a constant, and the equality holds Vf, 48, E X. then E is noise tolerant for class independent noise when 17 < b
R- (see Theorem I of Ghosh et al. (2017))
For this phase we submit the source code as part of supplementary materials to preserve anonymity, however, the final version will contain link GitHub repository.

---

Under review as a conference paper at ICLR 2022
Moreover, if the population risk of the optimum classifier (z, 0 then is also noise tolerant for class dependent noise. Among the commonly used losses, only the mean absolute error (MAE) satisfies the symmetry constraint. On the other end, the cross-entropy (CE) loss which is widely used for classification,, not only does not satisfy the above constraint, but is also an unbounded loss making it extremely susceptible noise. Specifically, as noise makes the predicted probability of the correct class approach zero, its cross-entropy loss will approach infinity.
Following the work of Ghosh et al. (2017), many studies (Zhang & Sabuncu 2018 Feng et al. 2020 Wang et al. 2019b; Ma et al. 2020 Wang el al., (019a) found that models trained with MAE struggle to converge and proposed novel losses that rely on boundedness to achieve robustness instead of the symmetry constraint to avoid convergence problems.
In the following, we use the weighted mean trick to bound loss function to improve its noise robustness. The next lemma specify the requirements for the weights:
Lemma 5. Let f(v. y) represent an unbounded loss function and w(u, y) a corresponding non- negative and bounded weight function, then the product w(e, y)l(v, y) is bounded if there exists finite threshold Lo such that when E(v. v) 2 Lo the corresponding weights w(v. v) are 0.
Proof. Since when ((u. 2 Lo the corresponding weights w(v, y) 0 the product w(v, y)((v, y) is also 0. When {(v.8) < Lo since the weights are bounded the product w(v, (), y) is also bounded and thus 0 S ((, y). y) S B where B is a positive constant that depends on the weight and loss function.
The simplest solution that satisfies the above requirement is the negative variance penalization, A2 < O, as the next lemma shows:
Lemma 6. For weights w(v.y) y) computed using Theorem 3 with X 0. A2 < 0. A 0 and with negative weights clipped to O, penalties E(v, v) 2 Lo with Lo = R[{] X2 associated weight of
will have an
Proof. From Theorem 3 the weights are calculated as w(v, y) = A + A2(E(1,
(v. y) - E(((u, y)l) from Thus due to the non-negativity constraint
which we can determine the threshold value Co = E[(] A1. X2
f(v, y) 2 Lo will have an associated weight of O.
Of note, bounding unbounded loss function is achieved through clipping, however, the resulting loss function is not convex as shown in Figure 3 With parameters x and X along with E[ establishing which samples are considered noise and excluded from training. Since for X < 0 the threshold Lo- E[{]- As is greater than B[C]. thus all excluded samples have a loss value above average. Moreover, the higher the magnitude of X2, the closer the threshold Co is to E[E]. This seen in the left plot of Figure 3 Moreover, the magnitude of 22 dictates by how much the loss values below average are amplified and the ones above average suppressed. In practice, using larger values for λ will decrease the impact of misclassified samples and those near the separating hyperplane on the placement of the decision boundary. Lemma 6 can be extended include other higher- order moments. In this case, the roots of the weights polynomial will determine which samples participate in the optimization problem and which are considered noise. For example, if the decision boundary should be decided by samples with average loss values then a possible solution is to penalize skewness. Since in this case the weights are defined by a quadratic equation, and using a negative value for Às will make the parabola open downwards and thus assign non-zero weights only to samples around the mean.
Ag
be
Moreover, also develop convex version of the variance penalized cross-entropy by constraining the second derivative of the weighted loss function to be non-negative. The cross-entropy loss is log u[y] where f[y] represents the predicted probability of sample having the label y. To simplify the notation, we will rewrite the cross-entropy loss as fee (u) = log u where
defined as l.e (v.y)
tfy] and similarly rewrite the weight function as w(u) = >x + >2 (f(u)) E(((u))). We note that
the second derivative of the weighted loss (u))((u) is 0 for u u. with u. exp [1 + 2 and is negative for u < u, To constrain the second derivative be non-negative we linearly interpolate (()x)(()))))) for u u. using the derivative of (u)))((u) ue- The weight function for u. ue
E[
7

---

Under review as a conference paper at ICLR 2022
-00 An -30
-0.5
1515
-0.15
-0.1
3
81e - 2.5
2
Eff..|
1
R[f...] 0.5
0
Figure 3: Bounding the cross-entropy loss function using negative variance penalization. Classical cross-entropy function is shown a dotted line. The impact of the X2 parameter on the loss shape when the average loss is constant is shown on the left plot and the impact of the average loss on the shape in the center plot. Right plot shows the convex version of the variance penalized cross-entropy loss.
is w(u) = [w(u)))(ue) + d
uz)] /E(u) with the resulting convex objective shown the right plot of Figure 3] The black dot marks the transition point ue from the weighted cross-entropy on the right to the linear interpolation on the left of the dot. Of note, the resulting objective is still bounded albeit with higher upper bound.
du
In general, methods for training of neural networks in the presence of noise can be classified into two broad categories: noise model-free and noise model-based strategies. Noise model-free methods
focus on reducing the impact of outliers and the two main subcategories are: robust losses (Bartlet
et al.] 2006; Ghosh al.. 2217]] Wang al. 20T9a; Zhang & Sabuncu 2018 Wang al. 20196
Natarajan et al. [2033] Mmih & Hinton 2012 Xu et al. [2119] Patrini et al. [0066] Rooyen et al. 2015
Feng et al. 2020; Ma et al.] (2000) and learning management such as meta- meta-leerning or regularization
(Li et | 2017 Szegedy et al. 2016 Hendrycks et al., 2019 Liu et al.) 2020 Harutyunyan et al.
2020;
Lugosi Mendelson
Laforgue et al. al.] [201]] Lecué et al. [020]]L] Li (017). On the other
hand, noise model-based methods estimate the properties of the noise and use this information when
train the model. Noise model-based methods can be further divided into:
using noise transition
matrix (Patrini et al.][2117 Hendrycks et al. 2018: Chen & Guptal 2015 Bekker & Goldberger
2016, Goldberger & Ben-Reuven. 2017 [Sukhbattar et al. [015:] Xia et al.
2019; Yao et al. 2019]]
Wang et al.] [020 [ia et 2020
Yao et al.
[020 Lukasik al. (2020), noise mitigation requiring
a clean dataset (Jiang et all,|2018 Ren et al.
2018 Veit et al. 2017 Zhang et al. 2020 Yuan et al.
2018 Vahdat 2017, MirzasMizanllemmann
al.
2020), incrementally Ily estimating the distribution of clean
data [Lu et al. [017]] Zheng et al. [0200] Arazo et al. 2019: Zhang et al. 2017 Ghosh & Lan 2021
Zhang et al. 202 1 Wu et al. (0200).
Experimental results. The procedure adopted for all the experiments and elaborated in Appendix B is similar to other studies investigating robust losses (Zhang & Sabuncu 2018 Feng et al. 2020
Wang et al. 2019b; Ma et al. 2020). The source code to reproduce our results is available online
The model accuracy for clean data when training with both, class independent and class dependent noise, are summarized in Table 1. Of note, we report the accuracy results of the model at the end of the training to capture the overfitting on noise for some models. We observe that the training with moments penalization outperforms other methods in low to moderate noise level scenarios and falls behind in high noise scenarios but still above the classical cross-entropy. The performances of the convex version and non-convex version are very similar, however, the convex version shows better results in high noise scenarios under class dependent noise. Additional results when penalizing higher-order moments are provided in Appendix C
In case of CIFAR datasets, for class independent noise and for 7 = 0.2 the proposed method improves the accuracy of the classical cross-entropy by 6%, the highest improvement among the investigated methods. However, this improvement decreases as the noise ratio increases and approaches the accuracy of the classical cross entropy loss for 17 0.8. To note, the convex version of the moments method for 1/ = 0.8 underpeformed by 0.8% compared to the classical cross-entropy, however, in
For this phase we submit the source code as part of supplementary materials to preserve anonymity, however, the final version will contain a link GitHub repository.
8

---

Under review as a conference paper at ICLR 2022
Table 1: Mean accuracy (%) and standard deviation clean data 5 runs. The best result for each scenario is underlined.
Noise rate
Loss
Class independent
Class dependent
0.2
0.4
0.6
0.8
0.1
0.2
0.3
0.4
Classical
84.1
77.1
66.3 0.4
36.0
88.9
87.2
84.8
80.9
Moments
90.5
85.0 o.s
69.1 0.4
37.3 2.3
91.4 0.2
89.9 0.o
85.0
72.9 3.5
CIFAR-IN 10
Moment-convex
86.8
80.3
67.6
35.2
89.1
87.2 0.2
83.5
78.7
TERM
89.5
83.2 0.2
69.2 0.s
37.2 2.0
91.1 0.2
88.1 0.4
83.0
77.2
Taylor
90.2
86.2
73.0 o.s
10.0
91.0
88.4 0.3
83.7
78.0
Normalized
90.1
87.0 0.2
80.8 0.2
32.6
90.9
89.7 0.3
86.8
79.8
Symmetric
90.0 o.2
86.7 o.2
80.2 1.0
44.8 4.0
90.8 0.2
89.6 0.1
86.5 0.4
79.8 0.9
Generalized
90.4 0.2
86.0 0.2
67.3 2.7
10.0 0.o
91.3 0.1
89.0 0.1
83.7 0.5
74.0 3.2
Classical
40.0 o.5
31.1 1.1
20.7 0.5
11.6 0.4
46.8 o.a
42.6 0.4
37.2 o.s
31.7 0.3
Moments
46.1
39.8 o.r
27.9 0.8
13.8 1.0
47.8 b.2
43.7 0.9
37.2 u.4
30.6 0.5
CIFAR- CIFAR-100
Moment-convex
44.5
36.7 0.9
24.8 0.8
13.4 0.8
47.4 0.5
42.9 0.7
36.9 0.s
31.1 0.6
TERM
45.5
41.5 1.1
32.8 1.4
17.7 0.4
47.1 6.8
44.6 0.7
37.9 o.s
30.9 0.6
Taylor
37.3 0.8
33.0 1.5
27.2 0.8
17.4 0.2
38.7 0.6
35.9 0.7
31.0 1.0
26.6 0.5
Normalized
32.9
27.8 0.7
22.7 0.4
13.9
35.0 0.3
32.7 0.3
30.2 0.4
26.5 0.6
Symmetric
43.1
37.9 0.4
31.2
19.4
45.3 0.5
43.1 0.2
41.0
34.6
Generalized
38.2 0.4
33.7 0.9
28.1 0.6
18.5
39.6 0.9
37.9 0.7
35.8
29.9 0.s
Classical
92.0
90.9
89.0 0.2
78.4
92.9
92.5 0.a
92.0
90.6
Fashion- -NIST MNIST
Moments
92.1
91.6
89.8
80.2
87.0
82.4 6.7
79.3
62.1
Moments-convex
92.4
91.6
89.6 0.2
79.7 1.0
92.6 o.a
92.0 0.4
92.4
88.7
TERM
92.1
91.6
90.0 0.1
80.3
92.3
92.0 0.4
91.5
91.3
Taylor
90.6
89.4
86.8 0.2
75.7 0.7
91.1 0.1
90.7
89.8
85.1 3.8
Normalized
91.5 0.1
90.8 0.3
88.9 0.3
79.8 1.3
91.7 0.1
91.7 0.3
90.7 0.T
91.1 0.1
Symmetric
92.3 0.1
91.8
90.6
82.8
92.5 0.1
92.3 0.3
91.9 0.6
91.5 1.3
Generalized
91.2
90.2
88.1 0.2
73.9
91.6 6.1
91.3 0.3
90.7 0.7
90.4
all other scenarios it had a higher accuracy. High noise scenarios turned out to be challenging for all losses with the Taylor and Generalized cross-entropy losses not converging CIFAR-10 for 27 = 0.8 and the best accuracy was registered by Symmetric cross-entropy loss proposed by Wang al. (2019b). For class dependent noise, the moments penalization registered the highest accuracy for CIFAR-10 when = 0.1 and v 0.2 but had the lowest accuracy for v) = 0.4. To further investigate this behavior,
the experiments and monitored the accuracy for each individual class. The classical cross-entropy recorded similar accuracy as the robust methods on classes affected by noise and outperformed them classes and both not targeted by class dependent noise. The poor performance of robust methods can be justified by using parameters tuned for low and moderate noise, In case of Fashion-MNIST dataset, as similar behavior as with CIFAR datasets was observed.
CONCLUSION
The main goal of the current work was to investigate the optimization of weighted mean and the flexibility it provides in enforcing properties such as robustness when training with noisy labels. In addition, we extended previous variance penalization methods to include higher-order moments while eliminating some of their limitations. One of the significant findings to emerge from this study was that we can control the final distribution the loss values by penalizing higher-order moments. In particular, by enforcing the distribution of the cross-entropy to have higher variance through negative variance penalization, we improved the models' accuracy when trained with noisy labels. Although this paper centers classification problems, the framework can also control the bias-variance trade-off and can also be applicable to regression problems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, 2019.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. JASA, 2006.
Alan Joseph Bekker and Jacob Goldberger. Training deep neural-networks based on unreliable labels. In ICASSP, 2016.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In ICCV, 2015.
John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2450-255 2504, 2019.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise. In ICCAI, 2020.
Aritra Ghosh and Andrew Lan. Do we really need gold samples for sample weighting under label noise? In WACV, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
J. Goldberger and E. Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information in neural network weights. In ICML, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Christina Heinze- deinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. Machine Learning, 1002))303-348, 2021.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. NeurIPS, 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019.
Geoffrey E. Hinton, Nitish Srivastava, A. Krizhevsky, Ilya Sutskever, and Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors. NewrIPS, 2012.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2037-2037. PMLR, 2018
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, 2018.
Pierre Laforgue, Guillaume Staerman, and Stephan Clémençon. Generalization bounds in the presence of outliers: median-of-ment study. In International Conference on Machine Learning, pp. 5937-5947. PMLR, 2021.
Henry Lam. Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization. Operations Research, 67(4):1090-111 1105, 2019.
Guillaume Lecué, Matthieu Lerasle, and Timlothée Mathieu. Robust classification via mom mini- mization. Machine Learning, 10955):1635-166) 2020.
10

---

Under review as a conference paper at ICLR 2022
Jerry Li. Robust sparse estimation tasks in high dimensions. arXiv preprint
2017.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. ICLR, 2021.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, 2017.
Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez- Granda. Early-learning regularization prevents memorization of noisy labels. NeurIPS, 2020.
Xin Liu, Shaoxin Li, Meina Kan, Shiguang Shan, and Xilin Chen. Self-error-correcting convolutional neural network for learning with noisy labels. In FG, 2017.
Gabor Lugosi and Shahar Mendelson. Robust multivariate mean estimation: the optimality of trimmed mean. The Annals of Statistics, 49(1):393 410, 2021.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In ICMI., 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, 2020.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penaliza- tion. Proc. Computational Learning Theory Conference (COLT), 2009.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovee. Coresets for robust training of deep neural networks against noisy labels. In NeurlPS, 2020.
Volodymyr Mnih and Geoffrey E Hinton. Learning to label acrial images from noisy data. In ICML, 2012.
Hongseok Namkoong and John C Duchi. Variance-hased regularization with convex objectives. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NIPS, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peerfomance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In /CML, 2016.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2000.
Brendan van Rooyen, Aditya Krishna Menon, and Robert C Williamson. Learning with symmetric label noise: the importance of being unhinged. In NeurIPS, 2015.
Johannes O Royset and Roger J-B Wets. Variational theory for optimization under stochastic ambiguity. SIAM Journal on Optimization, 27 2017.
2017.
11

---

Under review as a conference paper at ICLR 2022
Yevgeny Seldin, François Laviolette, Nicolo Cesa-Bianchi, John Shawe- Taylor, and Peter Auer. Pac-bayesian inequalities for martingales. IEEE Transactions on Information Theory, 58(12): 7086-7093,
Matthew Staib, Bryan Wilder, and Stefanie Jegelka. Distributionally robust submodular maximization. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 506-516. PMLR, 2019.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In ICLR, 2015.
Hailin Sun and Huifu Xu. Convergence analysis for distributionally robust optimization and equilib- rium problems. Mathematics of Operations Research, 41(2):377-401, 2016.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Iyy Tolstikhin and Yevgeny Seldin. Pac-bayess bareobayeeeeriicllbeeeeeeee bernstein inequality. Advances in Neural Information Processing Systems 26 (NIPS 2013), pp. 1-9, 2013.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NeurIPS, 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, 2017.
X Wang, E Kodirov, Y Hua, and NM Robertson. Improved mean absolute enror for learning meaningful patterns from abnormal training data. Technical report, Technical Report, 2019a.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019b.
Zhen Wang, Guosheng Hu, and Qinghua Hu, Training noise-robust deep neural networks via meta-learning. In CVPR, 2020.
Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. In NeurlPS, 2020.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L L_d: A novel informatioon-theorriie loss function for training deep nets robust to label noise. In NeurlPS, 2019.
Jiangchao Yao, Hao Wu, Ya Zhang, Ivor W Tsang, and Jun Sun. Safeguarded dynamic label regression for noisy supervision. In AAAI, 2019.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t; Reducing estimation error for transition matrix in label-noise learning. In NewrIPS, 2020.
Bodi Yuan, Jianyu Chen, Weidong Zhang, Hung-Shuo Tai, and Sara McMains. Iterative cross learning on noisy labels. In WACV, 2018
Jing Zhang, Victor S Sheng, Tao Li, and Xindong Wu. Improving crowdsourced label quality using noise correction. IEEE Trans. Neural Netw. Learn. Syst., 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In ICLR, 2021.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018.
12

---

Under review as a conference paper at ICLR 2022
Zizhao Zhang, Han Zhang, Serean O Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. CVPR, 2020.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, 2020.
13

---

Under review as a conference paper at ICLR 2022
A PROOFS AND ADDITIONAL THEORETICAL RESULTS
First, we present a proposition used in the proof of Theorem !
Proposition 7. Let and V be two random variables, and let U and V denote their expectation, then:
E - E
To prove write V as (V- V) + V and then use the linear property of the expectation operator:
Proof.
- v]
E [(U - U) ((V - + +)]
E [(U U) (V- + [UU - V]
= =E ((U-U) U) (V-V) +V (V(BUU -0)
=[[(U- UU) (V-V) -V)
Next present two propositions used in the proofs of the main theorems of this paper.
Proposition 8. Let e R - [0, oo) and R R two convex functions, then the composition w(x) p(((x) M) where M 2 convex if p is non decreasing.
Proof. Using the convexity of for any x, y E T and [0, we have:
f(ox + (1 - a)w) (x) + (1 - a)((y)
(8)
Using the fact that p non decreasing along we the above inequality we obtain:
pll(ax + - (1-a)y)) s p(al(x) + (1-aa)(())
(9)
and using convexity of p for the term on the right we get:
p(al(x) + (1 a)e(y) s ap({(z)) (11-))((((w))
(10)
Combining the two inequalities leads the following result (f(ar + (1 -1 a)y)) S ap(((x))
(1 - =))(((()) which proves the composition p(((x)) is convex. Moreover, since p is convex and ((x) - is in the domain of p then p(((x)) MM is also convex as it a composition with an affine mapping. This proves that w(x) is convex.
Proposition 9. Let f.g: product h(x) f())g(x) is convex if (x) f(v)ll(()) ((v)) 0, Vx. y Rd
: R" + (0, oc) be two convex function taking non-negen non-negative values then their
Proof. The function h is convex if satisfies the following inequality where [0,
0 < ah(x) + (1 - (1-a)((()) h(ax + (1 a)y)
(11)
< alfg) + (1-a))))0
(fg) + (fo))
a)v)
(12)
Note that since the two functions convex the following inequalities hold:
f(ar + (1- (1- s + (1-
(13)
g(ar + a)v) S ag(z) + +1 (1-a)g(1)
(14)
14

---

Under review as a conference paper at ICLR 2022
Given that the two functions take non non-negative values we can multiply the above two inequalities.
fax (1 - aa)))g(aa + (1 - (ry)
+ (1 (1- (1 a)g(v))
(15)
(fg) + (1 - ()v) - (1- (1-x) [o()) (1-
(16)
Substituting (fg) + (1-10000 (1-a)y) from 16 in inequality
we obtain:
5 a(()) + (1 - (v) - [af(2))
+ (1 a))( [g(x)) + (1- a)g(v))
(17)
Multiplying the two square brackets and then grouping the terms we get:
5 a(1 - [f(=) f(w)
g(y)l
(18)
We complete the proof by noting that a 2 0 and thus the product of two convex functions is convex
if [f(x) f()) (9(2) - g(v)] 2 O.
Lemma 2. As introduced in Theorem 7 the variance expansion of a convex loss function {(v, y) in yields new objective that is also convex in if X E [0, Amaz], where Xmax [ min t))
Proof. Since y are constants and take finite values we can index both, loss and weight functions,
using y as (v( = E(v. y) and (v) = w(v ((..)) y). We note that wy (v) p(l,(v) [[]]) where p is
polynomial of degree 1 with p(t) = At + 1. First we use the result of Proposition 8 and get that (v)
convex if A non negative thus the lower bound of the interval. Next, we use Proposition which
states that Wg (b) (v)) is convex if [w) wy (u)] (v) - (u)) 2 0 and wy(v) is non negative.
Since p is linear function we can use the equality p(a) p(b) A(a b). with a = Ly(w) E[{]
and Ey (u) E[{] and replace Wy in the previous inequality to obtain A A(((z) {t)] > 0 which
is always true for À >> O. However, (v) is non negative only for À S 1/ //min L E(e) which
proves the upper bound of the interval.
Theorem (Moments Expansion). Let E be a loss function with finite first m central moments and define w(, = Em Ae(((v. v) - E[é(v, y)l)*= then we have:
E[we]= = A + Σ
A. [ EE4)
(4)
2=2
where d, = -As + 1et1 < m and 7m = Am
Proof. In this case, for i 2 2 we cannot apply Lemma 7 since the expression (e B[2))* is not
guaranteed to have zero mean and as result it incurs an additional penalization of the previous moment. The proof follows the same steps Theorem T
E W E
Ai(E -
E [Ce =
A +
AE ((E - E((T)
A Σ
ΛE (E E[J) le- ([- +(()
>E +
X, [(E - E(()) + EEEE [Ce - E[
2=2
m-
-1 Σ
E [(e E[D) + ATn. E [(e B[A)
15

---

Under review as a conference paper at ICLR 2022
For the last step, we combine the two sums by matching the E [(e E[e))" terms and consolidate the penalization factors as X; = A + Art1
Lemma 4 (Convexity of Moments Expansion), Let E(v, y) be a loss function convex in and p:R + [0, oc) be a non negative and differentiable convex convexfunction and M 2 0, then the weighted objective w(u, ()((v, v) with w(v y) = p(((v, M) convex in vo if p is non decreasing.
Proof. Since y are constants and take finite values we can index both, loss and weight functions, using such as (y(v) = f(v, y) and (u) = w(v,3) y). With wyy(v) = (((((v) - M) and using the result of Proposition g we obtain that wy(c) is convex as p is non decreasing. Next, from Proposition
9 and since p takes non negative values we get that the product (y(v)), is convex if (v) by (u,( (v) - [()( 2 O. Since p is convex and differentiable we use the first order
condition p(a) p(b) 2 (a b) with a = (g( M and b = Ey(u) - M to obtain:
pll(()) M) (u) M) 2 ((, ()) M - Eg(u) + M]
(19)
wy (v) wy(u) 2 (1) (o) [w)
(20)
Substituting this result in the requirement from Proposition we obtain:
- (((e((t)-M)(((())) L,(2]]] 2 0
(21)
Given that p is non decreasing implies p'(E(y) - M) 2 0 and proves that the inequality always holds.
B EXPERIMENTAL DETAILS
After corrupt the training datasets (CIFAR-10. 10, CIFAR-100, Fashion-MNIST) with noise retain 10% as a secondary validation dataset. This allows us to detect when the model is overfitting on noise by comparing the performance on clean versus noisy validation data. For the class independent noise, we flip the label to any other class with equal probability such that the ratio of noisy labels is For class dependent noise, for CIFAR- 10 and Fashion-M -NNIST we only flip the label in the source classes (9, 2, 3, 5, } to the corresponding target class {1,0, 5, 3, 7) given the noise ratio 1). And for CIFAR- 100 we flip between two randomly selected subclasses withing each superclass. We use the same modes as in Wang et al. (2019b) when training on CIFAR-10 which is an 8 layer network composed of 6 convolutional layers followed by 2 fully connected layers. For Fashion -NNIST the model contains 4 convolutional layers followed by 3 fully connected layers. For CIFAR-100 the model we use is ResNet-34 34 (He et al. 2016). We train using SGD with 0.9 momentum, 0.005 weight decay for the convolutional layers and 0.01 for the fully connected layers, and starting learning rate of 0.01 which we divide by 10 every 20 epochs for total of 60 training epochs for CIFAR datasets and every 5 epochs for total of 15 training epochs for MNNHS..-MNIST. The training batch size is 128 samples.
Fashion-MNIST
We compare our solution for robustification of the loss function through moments penalization against five other state of the art methods: i) Tilted empirical risk minimization (TERM) introduced by Li al. 2202 ). ii) Taylor expansion of cross entropy (Taylor) proposed by Feng et al. (2020). tax) Normalized cross entropy coupled with reverse cross entropy (Normalized) investigated by Ma et al. (2020). iv) Symmetric cross entropy (Symmetric) studied by Wang et al. (2019b), and v) Generalized cross entropy (Generalized) explored by Zhang & Sabuncu (2018).
Zhang Sabuncu (018)) proposed generalized cross-entropy loss parameterized by g that recovers CH loss for - 0 and MAE for 1. For intermediate values of g, the loss is bounded and trades robustness for convergence. Feng et al.. (020) found that the first two terms of the Taylor series expansion for the CE loss are the MAE and MSE, respectively. And suggested adjusting the number of terms in the Taylor expansion for the CE to balance noise robustness and convergence. [Wang et al. (2019b) proposed using two term loss, complimenting the CE loss with secondary reverse cross-entrope term that satisfies the symmetry constraint. The research by Ma et al. (2020) extended the concept of two term loss combining a robust "active" loss and robust "passive" loss.
16

---

Under review as a conference paper at ICLR 2022
For these methods, when possible used the parameters suggested by their authors, however, when the method underperfomed we used cross- coos-validation to find better ones. When training on CIFAR-10 10 and Fashion-MNIST datasets used: Å 1, A2 -0.5 for Moments and its convex for Normalized, a = 0.1, 8 = version, = -0.5 for TERM, = for Taylor, a = 10, / CIFAR-100 the parameters where: 0.7 for Generalized. When training A = L, A2 = -0.5 for Moments and its convex version, & = -0.5 for TERM, & = 6 for Taylor, 10. 0.1 for Normalized, a 6.0, 0.1 for Symmetric, and q = 0.7 for Generalized.
for Symmetric, and
C ADDITIONAL EXPERIMENTAL RESULTS
Table shows the results when penalizing the third (skewness) and fourth (kurtosis) central moments. The resulting polynomial and the corresponding weighted cross-entropy loss are illustrated in Figure 4 When penalizing the third central moment samples with loss values close the mean receive most weight and the one deviating most from the mean receive the least weight. Experimental results on CIFAR-1O dataset show that when increasing the magnitude of X3 the accuracy for class independent noise rates 0.2 and 7 0.4 improves by around 1%. Moreover, in both cases the resulting accuracy is higher than that of the classical CE. Similarly, for class dependent noise, the accuracy increased by around 1% when penalizing with A -0.5 for noise rates 7) 0.2 and r 0.3. Likewise, when increasing the magniuude X the accuracy for class independent noise rates 1j = 0.2 and 1) 0.4 improves by up to 3.5%. However, for class dependent noise the results similar.
Table 2: Mean accuracy (%) and standard deviation clean data over 5 runs. The best result for each scenario is underlined.
Noise rate
Loss
Class independent
Class dependent
0.2
0.4
0.6
0.8
0.1
0.2
0.3
0.4
Classical
84.1
77.1
66.3 0.4
36.0
88.9
87.2 o.a
84.8
80.9 o.r
CIFAR-10
A = 1, As = 0.1
87.7 0.1
78.8 0.3
67.4 1.0
35.7 2.1
89.6 0.3
87.3 0.2
84.6 0.1
80.6 a.4
A1 = 1, A3 = 0.55
88.4
80.3 0.2
67.4 o.3
35.4
89.3 0.2
88.3 0.4
85.9 0.4
76.9 3.0
A 1, = 0.05
88.4 0.1
78.5 0.2
67.6 o.s
35.6 2.1
89.7 0.1
87.3 0.2
84.9 0.4
80.9 a.s
À = 1, = -011
88.7
82.0 0.3
66.8 o.7
35.2
89.6 0.4
88.0 o.1
84.3
80.0 a.e
17

---

Under review as a conference paper at ICLR 2022
3
0.1 À
3
0.5 À
B[ece] = 2.0
0
3
0
3
3
A =
0.00 A =
3
A 0.1 A
B[ece] 2.0
0
0
3
0
3
0
Figure 4: Polynomial functions for moments penalization and the corresponding weighted cross- entropy loss. Left column shows the polynomial functions used for penalizing the third central moment, top, and forth central moment, bottom. Right column shows the weighted cross-entropy function for [[ 2. Dotted line shows the classical cross entropy for reference.
18


Reviews:
[{"review_id": "t1I5OUDdZqZ", "reviewer": "Reviewer_ndU8", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the first section, the paper motivates the moments penalization by showing that variance penalization leads to tighter confidence intervals (Maurer and Pontil, 2009). In the context of learning, the goal then becomes to find a \n$\\theta \\in \\Theta$ that minimizes the following (focusing on second moments for simplicity):\n$$ E_n [\\ell(X; \\theta)] + \\lambda V_n(\\ell(X; \\theta)), \\qquad \\qquad (1) $$\nwhere $E_n$ and $V_n$ denote empirical mean and covariance respectively of $X \\sim P_n$, the empirical distribution.\nHence the objective is to find a choice of weights $W(X;\\theta)$, that may depend on $\\theta$, so that the following holds:\n$$ E_n[W(X;\\theta) \\ell(X; \\theta)] =  E_n [\\ell(X; \\theta)] + \\lambda V_n(\\ell(X; \\theta)) \\qquad \\qquad (2).$$\n\nTheorem 1 shows that if one chooses $W(X;\\theta) = 1 + \\lambda (\\ell(X;\\theta) - E_n[\\ell(X;\\theta)] )$, then (2) above holds.\n\nAt this point, this result is not useful unless it is easy to minimize $E_n[W(X;\\theta) \\ell(X; \\theta)]$:  as Theorem 1 shows equality with (1), one could have directly tried optimizing (1) above. This is where Lemma 2 comes in (preserving convexity). \n\nHowever, the paper just drops the dependence on $\\theta$ everywhere in the paper: If $W$ were to depend on $\\theta$, which it does, then I do not think Lemma 2 is true, i.e., $E_n[W(X;\\theta) \\ell(X; \\theta)]$ might not be convex in $\\theta$. As $W$ clearly depends on $\\theta$, I do not even understand what Lemma $2$ means when $W$ does not depend on $\\theta$ --- convex in which parameter? I would appreciate if authors can clarify if my arguments are incorrect.\n\nAfter hiding this dependence on $\\theta$, the theoretical results in the paper ---Theorem 1, Lemma 2, Theorem 3 --- become straight-forward.\nEven in the experiments section, as noted on Page 4, Section 2, the operations for computing weights are not part of the computation graph, and it is not clear what Algorithm 1 is minimizing.\n\nIt is possible that Algorithm 1, by detaching weights from the computation graph, is approximately optimizing the desired objective of (1), but I don't see any immediate connection at this moment. Investigating this connection might be a possible research direction for future but it is beyond the scope of the submitted paper. \n\nThough paper claims that it builds and extends the works of Duchi and Namkoong (2019) and Li et al. (2021), this limitation of the present paper is not present there.\n\nI would be happy to increase my scores if my understanding of the results in the paper is flawed, and authors can clarify their contributions.  \n\n\n## Minor Comments\n\n1. Since the goal is to penalize by empirical variance, first section should cite the results from (Maurer and Pontil, 2009) who showed that penalizing from empirical variance also works (instead of true variance as is cited in Equation (1) from Hoeffding's inequality). \n3. Proposition 7 is a basic fact in probability and certainly not attributed to Duchi and Namkoong (2019).\n\n", "summary_of_the_paper": "The paper studies the role and implications of weighted empirical risk minimization, where one can also choose the weight of each sample instead of fixing it to be equal.\nThe paper then shows that specific choices of weights lead to variance penalization and higher-order moments penalization (Theorems 1 and 3). Lemma 2 and 4 study the regimes for which these chosen weights also preserve the convexity of the original loss functions.\nThis framework is then empirically validated through multiple experiments.", "summary_of_the_review": "In my current understanding, the paper seems to study a simplistic setting (and hence the theoretical results of the paper are immediate) and it is not clear to me how the results/framework extends to the general setting. Hence, I do not think the paper is fit for ICLR in its current form. I will be happy to increase my score if my understanding is incorrect.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ndIhudB5gp", "reviewer": "Reviewer_i96S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied is interesting, but the provided answer seems very limited to me, especially at the theoretical level. In particular,\n\n**I am a bit puzzled by the negative $\\lambda$**:\n- it seems to me that it allows to do pretty much everything. For instance, the initial motivation is provided by eq. 1 (on another note, the link between eq. 1 and minimizing eq. 3 could be better explained, as it is not immediate to me). However, eq. 3 only makes sense if $\\lambda$ is positive, but Section 3 is all about negative $\\lambda$.\n- the sentence \"Case in which we consider them as unlearned samples and amplify their impact or as outliers and\nsuppress their impact on the model\" is very revealing to me, as it shows you can do one thing and its opposite, but the most difficult question remains: is the data point an outlier or unlearned? How can you answer this?\n- what is the motivation/intuition/incentive for minimizing eq. 4 instead of the standard sum of losses?\n- it is a bit disappointing that so much effort is put in Section 2 to show convexity properties, that are totally omitted in Section 3. The weights used are actually completely different from the family studied in Section 2, breaking the coherence of the paper\n- overall, my feeling is that we have a motivation for variance penalization (eq. 1), but the approach proposed is the complete opposite as a negative $\\lambda$ is used. As for higher order moments, I struggle to see any similar justification...\n\n**About practical aspects**:\n- the weights depend on expectations that are unknown. Although it is specified that the results stated still hold with the empirical distribution, then one has to relate the minimized empirical quantity to the true risk, which seems nontrivial.\n- as highlighted by the authors, considering higher order moments also multiply the number of hyperparameters. I wonder to what extent it is not always possible to find a combination of (positive or negative) $\\lambda_i$ such that there is an improvement\n- do authors have any convergence guarantee about Algorithm 1?\n- about Algorithm 1: why not defining $\\tilde{\\ell}_i = w_i \\cdot \\ell_i$ and perform Gradient Descent on $\\tilde{\\ell}$?\n\n**Two relevant papers/ideas**:\n- \"Robust multivariate mean estimation: the optimality of trimmed mean\" by Lugosi & Mendelson (2021). One could imagine a weighted ERM procedure with 0/1 weights determined by the Trimmed Mean approach on the loss values. How would this approach compare to the one developed in this paper? Some thresholding ideas seem to be shared\n- \"Robust classification via MOM minimization\" by Lecué et al. (2020). MoM-minimization can be seen a weighted ERM, with nontrivial 0/1 weights that depend on the median operator, see in particular Algorithm 1 therein. MoM-minimization has been recently shown to be robust to the corruption model presented at the beginning of Section 3 (\"Generalization Bounds in the Presence of Outliers: a Median-of-Means Study\" by Laforgue et al. 2021). How can it be related to the weighted approach proposed?\n\n**Minor comments**:\n- notation inconsistency: $n$ refers both to the number of samples (eq. 1, Algorithm 1, empirical expectation above Alg. 1), the highest order for the moments (Theorem 3), and the dimension of the input set (Lemma 4)\n- \"is bounded by the variance\" --> \"is bounded in terms of the variance\"\n- *with respect to* (2nd paragraph)\n- $W$ is not defined in eq. 2\n- the second sentence of Section 2 is not understandable\n- $\\lambda_{min}$ in Figure 1 is never defined\n- I find the $W$ notation a bit confusing (in Theorem 1 for instance), as capital letters usually refer to matrices. Why not using $w = 1 + \\lambda(\\ell - \\mathbb{E}[\\ell])$, $w\\ell$ (or $w \\cdot \\ell$) and $w(x)\\ell(x)$?\n- in Theorems 1/3, replace \"taking the weighted mean of $\\ell$ with weights $W$ is equivalent to\" by \"we have\"\n- Theorem 1 should be capitalized, i.e., not theorem 1 (Lemma 2, above Theorem 3)\n- \"changes with each iteration of the optimization algorithm\" is not clear at all (at least at this point where Alg. 1 is not yet introduced)\n- about the comments on the interval in Lemma 2, you could add that when $\\mathbb{E}[\\ell] = \\min \\ell$, you actually have $\\ell$ constant, i.e., $\\mathbb{V}(\\ell) = 0$, and the admissible interval is $\\mathbb{R}^+$, which makes sense since $\\mathbb{V}(\\ell) = 0$ and the \"penalized\" criterion is actually always equal to the original one, which is convex\n- \"the three central points aligned across a horizontal line\" is not clear: the other cluster seems to have much more that 3 points, which ones are they? why are other points present?\n- Figure 2: $\\lambda_0$ is not defined, it starts at $\\lambda_1$ in the given definition\n- in Lemma 4, I would definitely use $w(x)\\ell(x)$, to show that $W$ depends on $x$\n- could be nice to highlight that Lemma 4 does not generalize Lemma 2, as you assume here $p$ to have positive values\n- ~When~ clipping negative weights to 0 ~it~ prevents...\n- Agorithm 1: $\\mathfrak{L}$ is not defined, nor $k$ and $\\gamma$\n- Eq. 5 is pretty unclear: what is $k$ here? the loss is not taken w.r.t. to $i$, not $y_i$? Also there is always a $C$ satisfying the equation (the one equal to the left hand side), so what is the condition upon? Should $C$ be independent of some parameters?\n- Lemma 5: Let $\\ell, W$ **be**. Also $\\ell$ must be positive\n- Proof of Lemma 2: using $p(t)$ instead of $p(x)$ would be less confusing. Also, you provide a proof for a general $p$ while we have $p(t) = 1 + \\lambda t$ at the beginning, this is also confusing. In particular the inequality $p(a) - p(b) \\ge \\lambda (a- b)$ holds with equality for instance. Next, we use Proposition **9**\n- Proof of Lemma 4: it seems that you need $p$ to be (sub)differentiable", "summary_of_the_paper": "This paper studies weighted Empirical Risk Minimization (ERM), where the weight at data point $(x_i, y_i)$ is a (polynomial) function of the loss value $\\ell(f(x_i), y_i)$. For affine functions, the authors show that it is equivalent to perform variance penalization (Theorem 1 and Lemma 2). Polynomials with higher degrees involve higher order moments of the losses (Theorem 3 and Lemma 4). The authors also propose an iterative algorithm to perform the weighted ERM (Algorithm 1). In Section 3, a different choice of weights is proposed, that ensure the convexity of the cross-entropy criterion despite a negative $\\lambda_1$, and experiments are presented.", "summary_of_the_review": "Interesting problem but insufficient theoretical contribution. The provided insights are made confused by the fact that both positive and negative $\\lambda$ are considered, with completely opposite effects. The technical derivations are basic computations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IbX6A5OBItF", "reviewer": "Reviewer_Mbr3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are my comments on the paper:\n\n(1) Theorem 1 assumes that the first moment of $\\ell$ exists. What if $\\ell$ has heavy-tailed distribution and its mean may not even exist?  \n\n(2) The statement of Lemma 2 is quite weird. I think $\\ell$ is a random variable. What do the authors mean by \"convex objective $\\ell$\"? What does $\\min(\\ell)$ mean? Does that notation stand for the minimum coordinate of $\\ell$?\n\n(3) \" When using the weighted mean trick to penalize the variance and clip the negative weights to 0 then the objective remains convex for any positive\": Why is it true?\n\n(4) When using the weighted mean trick, what is the concentration behavior of the samples from $W \\ell$ around their expectation $E(W \\ell)$? I also have a similar question for moments expansion in Theorem 3.\n\n(5) When the weights are clipped to zero due to their negative values, will the expansions in Theorems 1 and 3 still hold?\n\n(6) \"However, for moderately small penalization factors the objective remains convex on almost the entirety of the domain and does not hinder the convergence\": I am not so sure about how to quantify \"moderately small\". The authors need to explain this point further.\n\n(7) \"In what follows, we show that using a negative variance penalization factor,  $\\lambda_{2} < 0$, bounds the loss function\": I wonder the relation of $\\lambda_{2}$ and $\\eta$, which is class label misspecification probability. Intuitively, $\\lambda_{2}$ should depend on $\\eta$.\n\n(8) I do not understand the writing in Page 7. Several parts are written without proper explanation. The authors may consider rewriting the writing in Page 7 to improve the readability of the paper.", "summary_of_the_paper": "In the paper, the authors proposed minimizing a simple weighted mean, which leads to an optimization problem of the high-order moments of the loss distribution. Experiment results show the weighted mean trick has similar performance as other robust loss functions.", "summary_of_the_review": "In my opinion, the contribution of the paper is quite marginal and the paper is quite hard to understand, which causes by the poor writing and lacks of proper explanation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DzkBXh-vZm9", "reviewer": "Reviewer_fmnG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nI think this is an interesting observation and could potentially have consequences in several robust learning algorithms. \n\nWeaknesses:\nI would have liked to see some applications to existing results that optimize the weights to solve estimation problems in outlier-robust settings -- for instance [1]. But there are several other works that also do such things. \n\n[1] \"Robust Sparse Estimation Tasks in High Dimensions\" Li'17 ", "summary_of_the_paper": "In this paper, the authors demonstrate that minimizing the weighted mean results in higher oder moments of the loss distribution. \n\nThey do this by explicitly demonstrating specific choices for the weights such that the expected weighted loss actually corresponds to minimizing the original loss regularized by a linear combination of the higher moments. They also note the ranges of the regularization parameter that preserve convexity. \n\nThe authors state their work in the context of two recent papers [1], [2] which attempt. to control the bias-variance tradeoff in different ways, effectively controlling either just the variance, or all the higher moments. In contrast, this technique allows them to control any specific combination of higher moments. \n\n[1] \"Variance-based regularization with convex objectives\" Duchi, Namkoong'19\n\n[2] \"Tilted Empirical Risk Minimization\" Li, Beirami, Sanjabi, Smith'20", "summary_of_the_review": "I think this is an interesting paper and vote to accept it. I think the observation carries value and might have several applications down the line. \n\nThe only detracting aspect is that the theorems are relatively straightforward, and so I would have liked to see more consequences of this observation in the context of other algorithms that exist in the literature. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates weighted empirical risk minimization where the weights on an example in the training set is given by a polynomial function evaluated on the loss on the given example. Authors show that the choice of the weighting function induces a data-dependent variance penalization in the training objective. Authors present an algorithm for weighted ERM and empirical results to support their claims. While the problem setting is broadly relevant and the approach the authors take in this paper is interesting, several questions remain unanswered. First, the authors argue that variance penalization helps but do not compare with other regularized ERM approaches. Second, it is not clear if the proposed algorithm is indeed gradient descent on the weighted ERM objective as pointed out by one of the reviewers. Finally, the writing can be improved with more emphasis on the novelty and significance of the contributions. I believe the initial comments from the reviewers has already helped improve the quality of the paper. I encourage the authors to further incorporate the feedback and work towards a stronger submission.
Decision: Reject