iFlood: A Stable and Effective Regularizer
Authors: Yuexiang Xie
Zhen WANG
Yaliang Li
Ce Zhang
Jingren Zhou
Bolin Ding
Venue: ICLR 2022 Poster (2022)
Keywords: overfitting
regularizer

Abstract: Various regularization methods have been designed to prevent overfitting of machine learning models. Among them, a surprisingly simple yet effective one, called Flooding, is proposed recently, which directly constrains the training loss on average to stay at a given level. However, our further studies uncover that the design of the loss function of Flooding can lead to a discrepancy between its objective and implementation, and cause the instability issue. To resolve these issues, in this paper, we propose a new regularizer, called individual Flood (denoted as iFlood). With instance-level constraints on training loss, iFlood encourages the trained models to better fit the under-fitted instances while suppressing the confidence on over-fitted ones. We theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in training data, which makes it suitable for a variety of applications to improve the generalization performances of learned models. We also theoretically link iFlood to some other regularizers by comparing the inductive biases they introduce. Our experimental results on both image classification and language understanding tasks confirm that models learned with iFlood can stably converge to solutions with better generalization ability, and behave consistently at instance-level.

Published as a conference paper at ICLR 2022
IFLOOD: A STABLE AND EFFECTIVE REGULARIZER
Yuexiang Xie', Zhen Wang', Yaliang Li', Ce Zhang", Jingren Zhou', Bolin Ding'
Alibaba Group. 'ETH Zürich
"uuexinn-xyx,, jones.wz wz, yaliang.li, li,
jinglann jingren.zhou, bolin.ding) @alibaba-inc.com ce.zhang @in..ethz.cch
ABSTRACT
Various regularization methods have been designed prevent overfitting of ma- chine learning models. Among them, surprisingly simple yet effective one, called Flooding, is proposed recently, which directly constrains the training loss on aver- age to stay at a given level. However, our further studies uncover that the design of the loss function of Flooding can lead to a discrepancy between its objective and implementation, and cause the instability issue. To resolve these issues, in this paper, we propose a new regularizer, called individual Flood (denoted as iFlood). With Instance-level constraints on training loss, iFlood encourages the trained models to better fit the under-fitted instances while suppressing the confidence on over-fitted ones. We theoretically show that the design of iFlood can be intrinsi- cally connected with removing the noise or bias in training data, which makes it suitable for variety of applications to improve the generalization performances of learned models. We also theoretically link iFlood to some other regularizers by comparing the inductive biases they introduce. Our experimental results both image classification and language understanding tasks confirm that models learned with iFlood can stably converge solutions with better generalization ability, and behave consistently at instance-level.
INTRODUCTION
Though overparameterized neural networks have achieved success on a wide range of tasks and applications. it is worth noting that their capacities are sufficient to memorize the entire training data (Zhang et al. 2017a Arpit et al. 2017). which often leads to an intolerable generalization gap, or in other words, overfitting. To prevent overfitting, many methods have been proposed to regularize how machine learning models fit the training data via introducing additional constraints to control the capacity in effect. For example, L1- or L2- regularizer (i.c., Weight Decay) (Hanson & Pratt 1989), ) Early Stopping (Yao et al. 2007), Dropout (Srivastava et al. 2014), Label Smoothing (Szegedy et al.
2016). Confident Penalty (Pereyra et af. (017)) etc (Zhang et al 2017b Izmailov et al. 2018) Zheng
al. 2021 Foret et al. 2021 Yang al. (0220).
Recently, a new method named Flooding [shidaa et al., 2020) is proposed, which controls the extent
which machine learning models fit the training data via directly encouraging the averaged training
loss to stay at given level rather than achieving (near-)zere loss. Formally, Flooding suggests the following loss function Flooding be minimized:
Laoodding = C b| 1//+ +
where L denotes the averaged training loss that is defined over total of N training instances as L= N Σi= Li, and b 2 0 is hyper-parameter called "the flood level" to control the training loss. Designed such a way, whenever L is below h, the gradients will negated increase it, so that L will stay around b and avoid being (zcro. Ishida et al. (2020) suggests to implement Flooding by minimizing FFooodin with mini-batch SGD, where L is estimated over a mini-batch of instances rather than the full data. Although they have pointed out that the objective being optimized by SGD is an upper bound of its desired one, we notice that this gap increases as the batch size decreases, which makes the discrepancy between the objective and implementation of Flooding more serious.
By investigating how machine learning models learned with Flooding behave on the training instances and generalize on the testing ones, we uncover the instability issue of Flooding, where it can lead

---

Published as a conference paper at ICLR 2022
to different solutions, and the solutions inconsistent in their generalization abilities and their behaviors over individuals. Further, we point out that the reason for the instability issue is that Flooding can only guarantee "global convergenee" Flooding encourages the averaged training loss to be sufficiently close to b, while having no requirement on the individual losses.
Both the discrepancy and instability issues can be attributed to the design of its loss function, which motivates us to propose a new regularizer in this paper, called individual Flood (denoted as iFlood). The proposed regularizer iFlood defines loss function LiFlood as:
1 N [ E1 N
((c-
LiFlond
where b 2 0 is the flood level, N is the size of training sample and L; is the loss function defined upon the i-th instance. When b= = 0, L. Lriooding. and Cioood are equivalent. With 6 > 0, iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones.
Although the modification is simple, the proposed new regularizer has the following merits: (1) The design of the loss function of iFlood ensures that it can be optimized by SGD without discrepancy between its objective and implementation. (2) Compared with Flooding, the models learned with iFlood can achieve "local convergence", that is, the individual losses (i.e., L,S) rather than the averaged loss (i.e., C) are encouraged to be sufficiently close to the specified level 6, which ensures that the learned models behave consistently over individual instances and produce stable generalization performance. (3) Mcanwhile, we theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in the training data, making iFlood suitable for a variety of applications to improve the generalization abilities. (4) Moreover, we theoretically compare iFlood with some related works (Szegedy et al.) 2016 Pereyra et al., 2017). showing that iFlood discounts the over-confident predictions with less inductive bias.
We conduct extensive experiments on both image classification and language understanding tasks to compare the performance improvements gained by different regularizers. demonstrating the effectiveness of the proposed iFlood. Further, we evaluate the stability of iFlood from several measurements, such as total variation distance and gradient norm. All the experimental results show that, with the "local convergence" suggested by iFlood, the learned models stably converge to solutions with better generalization ability.
PRELIMINARY
For ease of discussion, introduce some notations first. Without loss of generality, we consider a typical classification problem: Given a training dataset D = {(x;, 11) e X, K,y&V, VV 1,... N) where X' stands for the instance domain, y stands for the set of labels, and each instance (x., ys) is independently drawn from an underlying joint distribution Pr(X, we aim to learn a function f X - A()) (i.e., mapping from the instance domain to the space of probability distributions over the labels) to minimize the generalization error (xxx))))P((xx)) yearg y'
max max f(x) (x)).. where f(())
denotes the probability of taking the class y/.
The function of is often learned by minimizing certain loss function Uyy fx)) (c.g., Cross-Entropy loss). For simplicity, training dataset as L . Σ L.- Note that, besides the classification problem, the regularization methods discussed this paper are applicable to other machine learning tasks, e.g., regression.
denote the loss over the i-th training instance as C. and that over the whole
3 INDIVIDUAL FLOOD (IFLOOD)
In this section, we first compare iFlood with Flooding from various aspects to demonstrate the advantages of the proposed new regularizer iFlood. Then provide theoretical analysis about the effect of iFlood for removing the noise or bias in the training data, and connect iFlood with existing regularization methods, such Label Smoothing (Szegedy al. 2016 and Confident Penalty (Pereyra et al. 2017).
In this paper, "stability" has different meaning from that used in learning theory Mohri et al. 2018
2

---

Published as a conference paper at ICLR 2022
3.1 LOCAL CONVERGENCE
Let us revisit how Flooding works first. By applying Flooding, once the original averaged loss L has approached the flood level, it goes below and above b repeatedly until its convergence. Such process is called "flooding" in Ishida et al. (2020). The loss function of Flooding encourages the averaged loss L to approach to b, thus Flooding pursues the "global convergence" (i.e., |C model parameters A by minimizing Lionoinng defined in [.. the gradient of Litooding w.r.t. 0 is in the same direction as that of when L 2 b, and in the opposite direction when C b. To be specific,
0). To learn the
N VoL= =
Σ
VoLis
if b.
N
V Flooiinnn
1 N (-1) VoLi, <
VoL
N
From 3). we know that whether the original gradient of an instance (i.e., VeC;) should be negated is determined by the fact that whether the averaged loss L is below or above the flood level b However, in most practical cases and meanwhile the experiments of Flooding, deep neural networks are trained with mini-batch SGD where L is estimated from a sampled mini-bater of instances at each step. Thus, whether the gradients should be negated is determined by the averaged training loss estimated over mini-bateh of instances rather than the full data. In this way, the objective being actually optimized by mini-batch SGD an upper bound of Flooding as pointed out by Ishida et al. (2020). This leads to discrepancy between the loss function of Flooding and its implementation, which introduces randomness to the choice of whether to negate the gradients and makes the flooding process become random. In this study, further notice that, due to the property of absolute operator, such discrepancy increases along with the decreasing of batch size.
For iFlood, according Eq.22 in training dataset consisting of N instances, the gradient of Latiood w.r.t. the model parameters can be given VoLirood N ΣN Volc. - b/. Therefore, for each individual, or to contribute the negation of that, when L. b. By applying iFlood, the model parameters are encouraged to walk along the contour of [ b like what Flooding does, but, in contrast to Flooding, this is achieved by demanding each individual loss L,, Vi (I.....N) N } to be close to b. Intuitively speaking, iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones, according to the given flood level b. In word, iFlood pursues the "local convergence" (i.e., [C, - bj x O, Vi E (1....,N), We will analyze the advantage of "local convergence" over "global convergence" from the aspect of stability in Section 3.2 For now, the additive property of (q. ensures the gradients calculated in each SGD an unbiased estimation, which seemingly eliminates the discrepancy between the design and implementation.
guaranteed to contribute N VaLi to the aggregated gradient, when L,
Note that when optimized by SGD with batch size of 1, Flooding coincides with iFlood. However, is unreasonable is empirically confirmed in Appendix C For practical batch sizes, the designs of the loss function of iFlood and Flooding are different, as aforementioned, and we will empirically demonstrate those differences in Section 4
batch size to for training machine learning models in most cases, which
3.2 STABILITY
To better understand the differences between Flooding and iFlood, we plot the distributions of individual training losses with Flooding and iFlood in Figure [I, which is produced by training ResNet18 8 model on C10 Compared with the distribution corresponding to Flooding (see Figure Tap. we can observe that the distribution corresponding to iFlood (i.e.. Figure Tb) much more concentrated, and almost all the instances can be regularized by iFlood to achieve "local convergence" (i.c., C. 6/ x 0, Vi NJ).
Figure la confirms that the models learned with Flooding can achieve "global convergence" (i.e., B x 0). The inconsistent in behaviors over individuals achieved by Flooding can causes the instability of Flooding, namely that Flooding leads to various solutions with different generalization abilities. Let's consider the following cases: One learned model achieves L, = b for every training instance, and the other learned model achieves L, 0 on half of the whole training instances while
3

---

Published as a conference paper at ICLR 2022
0.04
Prohubility
b-0.03
Probabiltty
0-0.3
0.1
0.02
0.00
88.00
0.0
0.1
0.2
0.3
0.02
0.04
0.06
(a) Individual losses of Flooding
(b) Individual losses of iFlood
Figure 1: The distributions of individual losses of Flooding (a) and iFlood (b). The read dotted lines
represent the flood level ò 0.03. Best viewed with color.
C; 26 on the rest ones. Obviously, these two learned models have achieved the same "global convergence" that C = 0, but they behave inconsistently over the (training) individuals. Moreover, when b is taken to be level where L; b corresponds to being correctly classified and L; 2b corresponds to being mis-classified, the generalization ability of the former surpasses that of the latter in most cases. The analysis of these cases sheds light on the reason for Flooding's instability. Albeit the learning dynamics would not lead to these extreme cases in practice, we experimentally observe that, over a considerable portion of instances, the individual losses produced by model learned with Flooding are far away from b, as Figure Ia shows.
To address this issue, iFlood is introduced achieve "local convergence", which ensures the learned models behave more consistently than those models learned with Flooding. In this way, iFlood can achieve stable boost of generalization ability. We will provide quantitative analysis to demonstrate the advantages of iFlood on stability in Section 4
3.3 THEORETICAL ANALYSIS
has been noticed by Ishida et al. (2000) that a larger b is in favor, when the training data contain noisy labels. Intuitively, b controls how confident the model should trust the training data. Inspired by this observation, is worth discussing (1) what the meaning of in iFlood; (2) when and why do the models learned with iFlood benefit from the regularization. In the rest of this section, look into
these questions under two cattings-noisy labels and biased sample, which are commonly considered in both academia and industry.
3.3. NOISY LABELS
For many real-world applications, the observed labels are polluted due to certain reasons (Angluin & Laird 1988; Patrini et al., 2016 Thulasidasan et al.) (019), e.g., the mistakes made by annotators, the perturbation of some attackers, etc. Here, consider a widely adopted setting where noisy training sample Duwiay ((I,, 2x) E Xx 2; E V I,...,N] _ is given, with each (z,, 2,) being independently drawn from distribution Pr(X, Z) in the following process: first, instance (x.y) is drawn from the underlying joint distribution Pr(X.Y); then, y is perturbed to 2, taking incorrect label s/ with probability a, 0 S 1, keeping the correct label with probability (1 - a),, where the incorrect label drawn from a uniform distribution over M{y) This generation process implies Z X|Y and a conditional probability distribution Pr(Z|Y) with Vy E )) Pr = yly) = and Vy' # y, Pr( 2 y'ly)= DT-T
Given D Dinisy. suppose the data generation process is known, we can learn a model parameterized with 0 by minimizing its negative lig-liici:hoo::
N Elog
log Pr(ZIN|EEN, 8) log II
Pr(z
--===-oo log Σ (1
1-1
1-1 yF
N Σ
=
log
Σ Pr(u)=,;))
log a)) 1--)))P(p Pr(y = 0) + a D
Σ
Pr(ylo,,
1=1
y
1
N
Σ 1=1
(1 o) Pr(y 2/2,, 0) D - P1-Pr(Y 2//0;, 0))}
(4)
where we denote the cardinality of y as yl However, when we have no idea about the underlying data generation process, the RHS of Eq. 4 unknown
4

---

Published as a conference paper at ICLR 2022
If Dusisy is directly adopted as training data without denoising. 0 will be learned by minimizing the Cross- posss-Entropy loss: C = 인े. Ei Eyey 1g Pr(Y yy|,, 0). In this case, the learned model tends to generalize poorly, as it has been misled by the noisy sample and learns a distribution different from the actual joint distribution Pr(X.Y).
To explore the usage of iFlood in denoising, we first analyze Eq. 4. and present the following proposition about it.
Proposition 1. The negative log- liliikeeihodd (i.e., (q.44) ) is upper bounded by:
N N - E((1- a) log Pr(Y 2, 21 0) + a log(1 Pr(Y 21/2,,
(5)
which is minimized when Pr(Y 21 0) a; I.
Proof. Since - log(.) monotonically decreasing and |>] 2 2, Eq.4 upper bounded by: I-a -E log( 1- Pr(Y = 2)|æ,,0) + - - Pr(Y 2; z, 0)) within its domain. This func-
tion is further upper bounded by Eq.(5) with Jensen's 's inequality. By checking the derivatives of Eq. we know that it is minimized when Pr(Y 2| 0)
When we regularize the Cooss-Entropy loss defined over D Puoisy with iFlood, it becomes:
Litoodd
Σ l1 F1
12 yEy
log Pr(Y = y/ 0) - b1 b}.
(6)
N
Once we specify b= log(1--))) Fiiood encourages every Pr(Y = 22|x,, 0) I,..,, to take exp( (ob(-b) 11 a, which minimizes an upper bound (5q of the actual negative likeliielihood (Eq. 4 In this way, without knowing how the observed sample becomes noisy beforehand, Cirood can still serve surrogate function for the unknown negative oiglliielhooo (Eq.(4) of interest, extending the usage of iFlood to denoising.
3.3.2 BIASED SAMPLE
The second setting to explore iFlood when the collected training data does follow the probability
distribution of interest. Such data sample is often named biased sample, which is ubiquitously
observed in real-world applications (Abdollahpouri et al. 2019 Kowald et al.) (020). For instance, in
recommendation systems, there exists popularity bias (Abdollahpouri et al. 2019 Kowald al. 2020), and in face recognition and object recognition, data samples are biased regarding the backgrounds or the illumination conditions (Kortylewski et al. 2019: Barbu et al. 2019). Again, represent an observed label by Z and denote such a biased sample as D biased {I,, 23 |22 é X, e V,i 1.....Ny N). The biased sample setting can be formally given as: The observed distribution of input, denoted as Pr' Pr')), is different from the ground-truth one Pr( Pr(X), which leads to the joint distribution Pr(X, 7) Pr(X, Y) and thus model tends to learn biased conditional distribution Pr ( rather than the underlying one Pr(Y
Models learned from biased sample cannot generalize well due the distribution drift. Luckily, we can apply iFlood to debiase. In most cases, although we have no idea about the analytic form of Pr(Z]X, , it is reasonable to assume and validate some properties about Z|X that Pr(Z|X, Y) implies. With such properties, iFlood is able to recover the unbiased label. Formally, let's 's assume that according to distributions Pr(Z|X, Y) and Pr(X.Y which generate Drined, the difference between observed labels X and underlying labels Y|X follows some specific distribution:
1(2|X,
Lap(-lp,
where / denotes the adopted loss function, and Lap(-/p, >) denotes the Laplace distribution with parameters f1 2 0 O. With such knowledge, the negative log- legelikllhhodd of observing the biased sample Dnoisy is as follows:
log Pog.
1 N
Σ log
Σ ((.... ml| N [=
1
(8)
2X
This happens to be in the same form as Lirood- Thus, learning a model from the biased sample D Diiased via iFlood with i p leads to the same solution learning from the corresponding unbiased sample.
5

---

Published as a conference paper at ICLR 2022
3.4 COMPARING IFLOOD WITH OTHER RELATED WORKS
There exist regularization methods that prevent model from overfitting by suppressing its over-confident predictions, such as Label Smoothing (denoted as L.) (Szegedy et al. 2016) and Confident Penalty (denoted as CP) (Pereyra et al., 2017). Following Meister al. (2020), these methods can be summarized as follows:
--s L+ (ulk)()l)))
Lep - L+ (D3xk))pl))
where 3 is the regularization strength, (-I(- represents the KL divergence, is the predicted probability distribution over labels, and is a prior of the label distribution. From Eq. we can see that Label Smoothing and Confident Penalty add a regularization term to the loss function L to encourage the distribution of predicted probability to be close to the prior. Without additional domain knowledge, the uniform distribution is often adopted as the prior, which is expected to prevent the peaked distributions (i.e., the over-confident predictions) and lead to a better generalization.
Compared with Label Smoothing and Confident Penalty, iFlood performs similarly but has less assumption on the prior of the label distribution. iFlood prevents the model from becoming over- confident on the observed training sample via discounting its prediction probability over the observed label, where the extent of discount is controlled by the flood level b. On the other hand, iFlood does not make any assumption on how to distribute the reserved confidence.
Formally, we compare iFlood with Label Smoothing under the setting where Cross-Entrop) loss is considered and a uniform prior is adopted by Label Smoothing. In this case, the predicted probability distribution p (over labels) that minimizes the objective of Label Smoothing (Eq. ( is exactly one of the distributions that minimize the loss function of iFlood. As the regularization is posed at instance-leve in both methods, we show this relationship by checking the minimizer of the objective of Label Smoothing over any (I. E D:
3K (u]f(=)) ((v.p) (ulp) log(p,) 3 D
B
log(P,)
Σ
log(Ps')
[
V'
(10)
2 - 8 D
log(p,) + (1)1-13 1 Py
|/] D3 - |
(191-1)3
log(
log (pg()))
log(1 - Ppy + c).
D
|
(11)
where denotes the probability distribution over labels predicted by f, is constant, and | denotes
the cardinality of y. On one hand, II) tight lower bound of [QI( where equality is
established when p satisfies that Vy + y.Py D--1 On another hand, derivative of Eq. 11 shows
1-Py.
that it (as well as Eq. 100) can be minimized at Py [1+(+ Thus, the objective of Label Smoothing
encourages the learnedTunctionn f to predict distribution that satisfies both of the above conditions. Meanwhile, when we specify iFlood with b log( (2((
1 ). according to Eq. ) it encourages the
learned function f to predict a distribution that satisfies P, = exp( b) = [1+(1+ Based on the
above analysis, we see that iFlood discounts the prediction confidence over the observed label to the same level as that of Label Smoothing, without encouraging the reserved confidence to equally distributed to other labels. When it is hard to obtain some prior knowledge about the label distribution, iFlood can provide a less-biased and more flexible regularization, compared with Label Smoothing. Similar analysis and conclusion can be applied to Confident Penalty.
EXPERIMENTS
In this section, we conduct series of experiments to demonstrate the effectiveness of iFlood, with the aim to answer the following questions: Q1: Does iFlood provide larger boost of generalization ability, compared with existing regularization methods benchmark datasets? Q2: Compared to Flooding, can iFlood stably converge to solutions with better generalization ability? Q3: Can iFlood address the noisy label issue well in practice?

---

Published as a conference paper at ICLR 2022
Table 1: Accuracy (%) comparison on benchmark datasets.
Regularizer
CIFAR-10
CIFAR-100
SVHN
ImageNet
SST-2
QQP
QNLI
Unregularized
94.59
78.24
96.94
77.24
91.88
90.40
90.79
Label Smoothing
94.78
77.32
97.06
77.44
91.63
91.06
91.35
Confident Penalty
94.61
78.28
97.01
77.28
91.88
91.10
91.53
Flooding
94.58
78.63
96.98
77.25
91.86
91.14
91.43
iFlood (ours)
94.95
79.06
97.16
77.58
92.09
91.22
91.64
4.1
SETTINGS
Datasets. We consider both image classification and language understanding tasks. For image classi- fication, we use CIFAR-10, CIFAR- (Krizhevsky al. 2009), SVHN (Netzer et al. 201T), and ImageNet (Russakovsky et al. 2015). For language understanding, Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), and report the experimental results SST-2, QQP, and QNLI The details of these datasets and experimental results GLUE benchmark can be found in Appendix A 1.2 respectively
adopt the General Language
Baselines. We compare iFlood with the following baselines: (1) Flooding (Ishida al. 2020 (2) La-
bel Smoothing (Szegedy et al. 2016); (3) Confident Penalty (Pereyra et al. (017): ; (4) Unregularized, denotes that the models are learned without Flooding, iFlood, Label Smoothing and Confident Penalty. Data augmentation (e.g.. random crop (Krizhessky ct al. 2012) and horizontal flip (Simonyan & Zisserman 2015)) have been necessary plug-ins of the learning procedure on image classification datasets, we do not remove them in our experiments. As the basic regularizers, Lg-regularization used in both image classification and language understanding tasks, and Dropout adopted in language understanding task.
Implementation details. On the image classification datasets, we consider training ResNet18 (He et al.l.,2000 2016) for CIFAR-10, CIFAR-100 100 and SVHN, and training ResNeXt50 (Xie et al.) (017) for ImageNet, adopting momentum SGD as the learning procedure to be regularized. We train R8 for 300 epochs with 128 as the batch size. The learning rate is initialized as 0.1 and decays (multiplicd by 0.2) at the 80-th, 160-th and 200-th epochs. As for ResNeXt50, we train for 90 epochs with 256 as the batch size. The learning rate is initialized as 0.1 and decays (multiplied by 0.1) at the 30-th, and 60-th. On the language understanding datasets, we consider fine-tuning BERT (Devlin et al. 2019) via Adam (Kingma & Ba 2015) as the learning procedure to be regularized. We adopt the pre-trained BERT model provided by huggingface (Wolf et al., 2020) and fine-tune it on the target datasets. The number of epochs tuned among (3, 4, 4, the batch size is 16, the learning rate is tuned among (2e 5, 5e - 5). and the dropout rate is 0. For Flooding and iFlood, the flood level b is tuned in the range of [0. [0.10, 0.50] via grid seach with 0.05 as the step size for ImageNet, and tuned in the range of [0.01, 0.10] via grid search with 0.01 as the step size for other datasets.
Syyper-parameter spaces considered for baseline methods can be found in the Appendix B All models are implemented using PyTorch (Paszke et al. (019) and trained on NVIDIA GeForce GTX 1080 Ti or Tesla V100 GPUs For fair comparison, we search for the optimal configuration of hyper- parameters for each method. Then we run each method for 5 times using its optimal configuration and report the averaged results, reducing the randomness in the comparison.
4.2 IMPROVEMENTS ON GENERALIZATION ABILITY (Q1)
We compare iFlood with the baseline methods on benchmark datasets to demonstrate its effectiveness. The experimental results are summarized in Table Obviously, with appropriate configurations (i.e., the strength of regularization), all these regularization methods can improve the performance of the learned model compared to unregularized case, on most of the datasets. This phenomenon can be explained by the fact that, in this experiment, the number of parameters to be estimated is larger than the size of training sample, which makes regularization indispensable. We can also observe that, on both the image classification and language understanding datasets, iFlood provides larger improvements of the performance than those of the baseline methods. This confirms the effectiveness of iFlood in improving generalization ability of learned models, which is brought by the design of iFlood, encouraging the learned model to better fit under-fitte instances while suppressing the confidence on overfitted ones. We also conduct experiments on large-scale dataset Criteo and report the results in Appendix D.I. which confirms the effectiveness of iFlood.
7

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of performance variances. The relative changes in terms of "Unregularized" cases are also reported.
Dataset
Regularizer
The std. of test acc.
max Drv
avg Drv
Unregularized
0.16%
0,0004
0.0003
CIFAR-10
Flooding
0.29% (+0.13%)
0.0237 (x59.25)
0.0221 (x73.67) 0.0053 (x (x1..77)
iFlood
0.19% (+0.03%)
0.0068 (x17.00)
Unregularized
0.29%
0.0029
0,0028
CIFAR-100
Flooding
0.36% (+0.07%)
0.0241 (x8.31)
0.0236 (x8.43)
iFlood
0.22% (-0.07%)
0.0167 (x5.76)
0.0165 (x5.89)
Unregularized
0.29%
0.0089
0.0084
SST-2
Flooding iFlood
0.37% % (+0.08%)
0.0205 (x2.30)
0.0193 (x2.30)
0.21% (-0.08%)
0.0103 (x1.16)
0.0099 (x1.18)
Unregularized
0.12%
0,0220
0,0173
QQP
Flooding
0.14% (+0.02%)
0.0376 (x1.71)
0.0321 (x1.86)
iFlood
0.12% (+0.00%)
0.0223 (x1.01)
0.0175 (x1.01)
worth pointing out that, iFlood can work well with some other regularization methods, such as weight decay. Under the scenario of peee-parameterized neural networks, near-7ero training loss is achievable regardless of the regularization constraints posed upon the model parameters. Therefore, when cooperated with weight decay, iFlood controls the ultimate extent which such models fit the training data, while weight decay affects learning dynamics (Golatkar et al. 2019).
4.3 STABILITY OF IFLOOD (Q2)
provide analysis about the instability of Flooding, which further motivates to propose iFlood. In this section, we conduct experiments to show that, compared to Flooding, iFlood can stably converge to solutions with better generalization ability.
In Section 3
Variances. We use the standard deviation (denoted as "std.") of test accuracy to measure the differences of the generalization abilities among the learned models. Beseides, we adopt the total variation distance (denoted as Drv) to measure the difference between any two learned models, which can be estimated on the training sample as:
1 2N =l
N f((x)) (x())
DP((f.))
(12)
where and denote two learned models. We estimate the distance for every pair of the ten models and report both the maximum and the averaged values of them. The experimental results are summarized in Table 2
The results show that the test accuracy of models learned with iFlood vary less than that of models learned with Flooding, with the largest difference between them could be 0.16% The less variance in
generalization ability demonstrates the stability of iFlood. Mcanwhile, from Table 2 we can observe that, the variance of iFlood is much smaller than that of Flooding (c.g., nearly 24% 70% of Flooding from the aspect of avg Drv). Further, the comparison of total variation distance Drv between Flooding and iFlood is consistent with the individual losses shown Figure T which supports our idea: Under the individual-leve constraint, iFlood can achieve "local convergence" and converge to solutions with better generalization ability, a stabler manner than Flooding.
The norm of gradients. To further compare the stability between Flooding and iFlood, we train ResNet18 on CIFAR-10 (Figure 2ab and CIFAR- (Figure 2 and monitor the T., norm of the gradients at each epoch. From the figure, the norm of gradients is larger than that of iFlood by a noticeable margin, and the gap has not been filled until the end of training. It implies that even at the end of training, the model parameters learned with Flooding are changed more significantly than those of iFlood, which could cause the instability of learned models.
observe that, during the training course of Flooding,
The effect 6. We further study the effect of flood level via training a ResNet18 on CIFAR-10, and report the training accuracies of both Flooding and iFlood with varied in Figure 2c From the
8

---

Published as a conference paper at ICLR 2022
Flooding iRludl
3 Nomo Gradions
Flooding i'lood
Accumey
100%
of
95%
Flooding
illood
90%
250
230
Epoch
Epoch
Flood
(a)
(b)
(c)
Figure 2: The comparisons between Flooding and iFlood. (a) The L1 norm of gradients on CIFAR- 10;
(b) The L, norm of gradients on CIFAR-1001 100; (c) The training accuracy on CIFAR-I0 w.r.t. various flood level b. Best viewed with color.
figure we can observe that, as the flood level b increases, the training accuracy of Flooding drops drastically, while that of iFlood stays the same level until 0.60. This result implies that iFlood is much more robust w.r.t the hyper-parameter compared to Flooding. Since Flooding just encourages the averaged loss instances stay under-fitted regrading the flood level b. When is taken to be relatively large value, the training instances whose individual losses are larger than might become under-fitted or even mis-classified. Such instances might degrade the model's generalization ability (Belkin et al. 2019). As iFlood encourages the model to fit every individual instance to the same extent (i.e., [2; 6 2 0), it can overcome such under-fitted instance issues via achieving "local convergence".
close enough to but has no requirement on the individual losses, some
4.4 EFFECTIVENESS IN DENOISING (Q3)
In this section, we instantiate the noise label setting to evaluate the effectiveness of iFlood in denoising. Following the setting discussed Section. [33.I] the polluted version of CIFAR-IO and SST-2 training datasets are constructed. We train ResNet model on the polluted CIFAR-10 dataset and train BERT model on the polluted SST-2 dataset, with Flooding or iFlood applied as the regularizer for denoising. More implementation details can be found in the Appendix
90%
90%
Accuracy
85%
Accuracy
8.5%
80%
Unreg.
Unreg.
75%
Flooding
Flooding
75%
70%
iFlood
iFlood
70%
10%
30% Noisy Instance Raito a
20%
40%
10%
30% Noisy Instance Raito a
20%
40%
polluted SST-2 Figure 3: Performance comparison on datasets polluted with noisy labels.
polluted CIFAR-
Results are plotted in Figure 3 It can be observed that, on both polluted CIFAR-10 and polluted SST-2 datasets, models learned with iFlood generalize much better than those with Flooding and those without any regularization. Further, the advantages of iFlood become more significant the ratio of noisy instance a increases. These experimental results confirm our theoretical analysis in Section 3.3.I that iFlood, regularizer, has the capability to denoise data, that the regularized models can achieve better generalization performances, even when they are trained on noisy datasets.
CONCLUSIONS
In this paper, we uncover the instability issue of a recently proposed regularization method Flooding. Our analysis on its causes motivates us propose new regularizer iFlood, which encourages the individual losses to approach a specified level. Experimental results on a variety tasks show that the models learned with iFlood produce stabler generalization improvement. We also theoretically show that, the objective of iFlood over a noisy or biased training sample can serve as a surrogate or match the maximum likelihood estimator applied to clean sample, which is also confirmed by our experimental results. Further, we theoretically compare iFlood with some other regularization methods and find that iFlood discounts the over-confident predictions with less inductive bias injected.

---

Published as a conference paper at ICLR 2022
REFERENCES
Himan Abdollahpouri. Masoud Mansoury. Robin Burke, and Bamshad Mobasher. The unfairness of popularity bias in recommendation. In the RMSE workshop held in conjunction with the 13th ACM Conference on Recommender Systems, 2019.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4))343-370 1988.
Devansh Arpit. Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242, 2017.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in neural information processing systems, pp. 9453-9463, 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern leaaing- practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, (16632):1584--15854 2019.
Jacob Devlin, Ming-Woit Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4171-4186, 2019.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems, pp. 10678-10688, 2019.
Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with back-propagation. In Advances in neural information processing systems, pp. 177-185, 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Do we need zero training loss after achieving zero training error? In Proceedings of the 37th International Conference on Machine Learning. 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the ThhirtyFFourhh Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 876-885, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. 2015.
Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, and Thomas Vetter. Analyzing and reducing the damage of dataset bias to face recognition with synthetic data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Dominik Kowald, Markus Schedl, and Elisabeth Lex. The unfaimess of popularity bias in music recommendation: A reproducibility study. In European Conference on Information Retrieval, pp. 42, 2020.
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. In Proceedings of the 58th Annual Meeting of the Associa- fion for Computational Linguistics, pp- 6870-6886, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high sesh-eeffrrmncce deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035. 2019.
Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In International conference on machine learning. pp. 708-717, 2016.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR, Workshop Track Proceedings, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 2522, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2822 2016.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff A. Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In Proceedings of the 36th International Conference Machine Learning, pp. 6234 6243, 2019.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations, 2019.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684-8694, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Sfate-of-the--u natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38,45, October 2020.

---

Published as a conference paper at ICLR 2022
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Taojiannan Yang, Sijie Zhu, and Chen Chen. Gradaug: A new regularization method for deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14207-14218. 2020.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning.
Constructive Approximation, 26(2):289-3155 2007.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of the International Conference Learning Representations, 2017a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations, 2017b.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
12

---

Published as a conference paper at ICLR 2022
A DATASETS
We conside both image classification and language understanding tasks, and adopt 7 benchmark datasets for evaluation, including CIFAR-IQ" CIFAR-100 SVHN ImageNe SST-2 QQP/ and QNL The statistics of the datasets are summarized Table 3
Table 3: The statistics of benchmark datasets.
Dataset
# Train
# Test
CIFAR-10
50,000
10,000
Image Classification
CIFAR-100 100
50,000
10,000
SVHN
73,257
26,032
ImageNet
1,281,167
50,000
SST-2
67,349
872
Language Understanding
QQP
363,846
40,430
QNLI
104,743
5,463
B DETAILS OF HYPERPARAMETER OPTIMIZATION
randomly split the training data into training and validation sets with the proportion of 9:1, and apply grid search the validation dataset for hyperparameter optimization (HPO). We adopt the optimal configuration provided by HPO to train and evaluate each method times, alleviating the impact of randomness.
Specifically, for Label Smoothing (Szegedy et al., 2016), we tune the smoothing parameter value among (0.05, 0.1, 0.2), and adopt uniform distribution the prior of label distribution for interpolation. For Confident Penalty (Pereyra et al. 2017). following the original paper, the strength of the confidence penalty 3 is tuned among (0.1, 0.5, 1.0...0,
C SGD WITH VARIOUS BATCH SIZE
When optimized by SGD with batch size of 1, Flooding coincides with iFlood. However, it is unreasonable batch size I for training practical machine learning models, according the literature and our experimental results shown in Figure
Test Error Rate
0.8
Test Loss
0.6
batc sice 16
butch 124
50
150
200
50
Epoch
Epoch
Figure 4: ResNet18 with Flooding on CIFAR-10.
As mentioned in (Ishida et al. 2020 the objective being optimized by SGD an upper bound of its desired objective, where notice that the gap between these two objectives increases w.r.t. the
"hups//wwwwes.ce toronto.edu. krizlcifar.html "hups://www.cs.. www.:cs..tooooood.... edu/ kriz/cifar.hmml
sta.ffod edd/houseenumerrs//
13

---

Published as a conference paper at ICLR 2022
decreasing of batch size. As for the practical batch sizes, the design of the loss function of iFlood and Flooding are different, and we have shown iFlood outperforms Flooding in Section
D EXPERIMENTS ON LARGE-SCALA DATASET AND GLUE BENCHMARK
D.1 LARGE-SCALE DATASET CRITEO
To demonstrate the effectiveness of the proposed iFlood, we train an MLP for click-through rate prediction Criteo, real-world advertisement dataset including around 45 million instances. The AUC of "Unregurized" v.s. Flooding v.s. iFlood are 78.08% v.s. 78.41 % v.s. 19.14%. According to literature, 0.001-level improvement in offline AUC evaluation makes significant difference. Thus these results confirm the effectiveness of iFlood on large-scale real-world datasets.
D.2 GLUE BENCIIARR
Table 4: Accuracy (%) comparison on GLUE benchmark.
Dataset
Metric
Unregularized Label Smoothing Confident Penalty
Flooding
iFlood
CoLA
Matthews Corr.
56.82
56.72
56,85
56.53
$7.87
SST-2
Accuracy
91.88
91.63
91.88
91.86
92.09
MRPC
Accuracy
84.46
84.36
84.90
83.64
85.25
STS-B
Pearson Corr.
89.00
89.37
89.46
QQP
Accuracy
90.40
91.06
91.10
91.14
91.22
MNLI
Accuracy
83,22
83.43
83.12
83.22
83.22
QNLI
Accuracy
90.79
91.35
91.53
91.43
91.64
RTE
Accuracy
65.49
66.28
66.50
65.05
67.29
WNLI
Accuracy
56.34
56.34
56.34
56.34
56.34
The experimental results on GLUE benchmark are shown in Table from which we can observe that iFlood outperforms other baseline methods by a noticeable margin. These experimental results are consistency with those reported in Table which confirm the effectiveness iFlood in improving generalization ability of learned models.
E DATASETS CONSTRUCTION IN NOISY LABEL
Following the setting discussed noisy label (Section 3.3. I). the polluted version CIFAR- and SST-2 datasets are constructed as follow: We randomly choose proportion of the original training instances according to a(0 < a < 1), and then pollute the label of each chosen instance by uniformly picking one from its corresponding incorrect classes. We enumerated the noise instance ratio with 0.1, 0.2, 0.3 and 0.4. Note that the test instances are kept clean to reflect the real-world scenarios. We search for the optimal flood level b in the range of 10.10, 0.50] via grid search with 0.05 the step size for both Flooding and iFlood. The experimental results can be found in Section 4.4
F ADDITIONAL EXPERIMENTS
F.1 LOW-FREQUENCH COMPONENT V.S. HIGII-FREQUENCY COMPONENT
Inspired by previous study (Wang et al. 2020), we conduct experiments to show how model learned with different regularizers reacts different levels of details of the data (e.g., the low-frequency component and figh-freuenncy component of images). To be specific, for each instance in the training data, we decompose the data into www-freuuency component and high-freeuency component w.r.t. different radius thresholds via applying Fourier transform and inverse Fourier transform. Then we train a ResNet-18 on CIFAR-10 using the raw training data, and evaluate the model on both
Labeel Smoothing and Confident Penalty not suitable to be adopted on STS-B, since the output dimension
1.
14

---

Published as a conference paper at ICLR 2022
cow-freueeney component and high-freg freuueryy component. More details can be referred Section 3.1 Wang et al. (2020).
high Flooding
Vanilla
100%
100%
1007%
5076-
80%
5417
Accuncyy
40%
4076-
40%
207
Epoch
Epoch
Epoch
Figure 5: How models learn with low-frequency component and freuunfry component.
The experimental results illustrated in Figure 5 where 4/8/12/16 low denotes the low- frequency component and r //8/12/16 high denotes the high-frequency component. From these experimental results we can conclude that model performs better on low-frequency component than high-frequency component when r 8/12/16, but worse when r = 4, which are consistency with the results Wang et al. (2020) (note that BatchNorm is adopted). Compared Flooding and Vanilla, we can observe that model learned with iFlood catches more low-frequency component (e.g.. 4/8 low) and less high-frequency component (e.g., r 8/16 high), which confirms the effectiveness iFlood in improving the generalization ability of model since low- owwfrreueency component is much more generalizable than high-frequency component (Wang et al. 2020).
F.2 THE NORM OF GRADIENTS
We train ResNet18 CIFAR-10 10 and CIFAR-100, and monitor the L, norm of the gradients each epoch. The experimental results shown in Figure (6a and [b]coofirm that: (1) The design of the loss function of Flooding leads the instability issue, which is supported by the phenomenon that the norm of gradients is larger than other methods by noticeable margin; (2) The design of the loss function of iFlood brings the merit of significantly reducing the gap of the norm of gradients between Flooding and other methods.
Gradlett
Unxey
Nora Grodienis
Unay. Flooding iFlodd
Flooding iFlodd
85%
Unng
Flooding
LS
LS
illoed
25%
CP
100
150
200
250
300
50
100
150
200
250
Upoch
Epoch
(a)
(b)
(c)
Figure 6: The comparison between iFlood and baselines. (a) The norm of gradients on CIFAR-I0:
(b) The L1 norm of gradients on CIFAR-100 (c) Performance comparison on polluted CIFAR-10.
F.3 EFFECTIVENESS DENOISING
We
train ResNet-18 on polluted CIFAR-10 to evaluate the effectiveness iFlood in denoising. The generation process of polluted CIFAR-10 can be referred to Appendix D and the experimental results are shown in Figure 6c From the figure we can observe that, models learned with iFlood outperform those with other regularizers by noticeable margin, and the advantages of iFlood become more significant as the ratio noisy instance a increases. These results confirm the effectiveness iFlood in denoising.
15

---

Published as a conference paper at ICLR 2022
F.4 CONFIDENCE DISTRIBUTION
To further confirm that "IFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones" from the perspective of model confidence, demonstrate the distribution of model confidence in Figure 7 We adopt the same experimental settings as those used Figure ResNet-18 trained CIFAR-10, and the flooding level is set to 0.03 for both Flooding and iFlood. From the figure we can observe that, compared with Flooding, iFlood encourages the model to continue to fit the under-fitted instances w.r.t. b, (i.e., the instance with confidence less than e ~ 0.97), while suppressing the over-fitte fitted ones (i.e., the instance with confidence large than -b 0.97). These results are consistent with those Figure
Flooding
iFlond
Probability
Probability
0.02
0.1
0.01
0.00.73
0.0
0.80
0.85
0.90
0.95
1.00
0.90
0.92
0.94
0.96
0.98
1.00
Confidence
Confidence
Figure 7: The distributions of model confidence.
16


Reviews:
[{"review_id": "eH-A_LKKREu", "reviewer": "Reviewer_bQVY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper is well written and it is enjoyable to read.\n+ The idea of iFlood is neat. \n+ It is theoretically showed that the design of iFlood can be intrinsically connected with removing the noise or bias in training data. \n\nMy main concerns are on the effectiveness of iFlood:\n+ There are many regularizations which can prevent overfitting. I think more powerful baselines should be added (e.g. mix up).\n+ The improvements are very marginal. This makes me very doubt whether iFlood is really effective. \n+ The performance of baseline in GLUE is not convincing (SST-2 and MNLI). It is shown in https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/README.md that sst-2 can achieve 93.2 and mnli can achieve 83.9. \n", "summary_of_the_paper": "This paper proposed a stable and effective regularizer to prevent overfitting. Specifically, this paper proposes individual Flood. Different from Flood which constrains training loss on mini-batch level, iFlood gives instance-level constraints on training loss. This paper also theoretically shows that the design of iFlood can be intrinsically connected with removing the noise or bias in training data. ", "summary_of_the_review": "I think the idea of iFlood is neat and simple. Theoretical analysis is good enough. However, the effectiveness of iFlood can not be guaranteed in experiments. So I think this paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dQdDnVK9eFA", "reviewer": "Reviewer_fRos", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\nThe paper poses several interesting contributions in terms of training of models while limiting the models' overly attention to a subset of samples. \n - The idea of extending Flooding to iFlooding is very intuitive, thus it's fairly clear that such extension is needed. \n - Some additional discussions on the properties iFlooding in terms of robustness to noise labels and biased samples are offered. \n\n**Weakness**\n\nOn the other hand, I also have several concerns about the presentation of the idea, especially on the empirical end\n - It's not clear that in the experiments, whether each data are shuffled when every iteration ends. Intuitively speaking, SGD (and its variants) has a natural counter to overly attending specific samples as the gradient are calculated every batch and shuffling samples at every iteration will avoid the batch to stay the same, thus the gradient will always be different (even if the weights are the same), avoiding some overly attending to specific samples. Therefore, the lack specification of this setup posts a challenge to appreciate the effects of the empirical results. \n\n - Related, it has been shown that when the model is being trained, the model pays attention to different levels of details of the data as the iteration goes (Figures 4, 5 in [1]), therefore, instead of presenting a fixed table at the end of all the iterations, showing the curves of accuracies along the training process will help better appreciate the idea of iFlooding. It might also be interesting to present accuries on different test copies as in (Figures 4, 5 in [1]). \n\n - If I understand correctly, Table 1 is a comparison of generalization (accuracy) improvement ability of methods, instead of a comparison of the methods devoted to calibrating the overly attention, thus conventional generalization boosting methods such as dropout, batchnorm, or even mixup should probably also been compared. I don't think the authors need to worry whether BatchNorm will deliver a higher accuracy than what their method gets, since these methods are motivated by different problems. The authors could always discuss these, or present the numbers in different cells. However, my point is that for a table that serves as a comparison of accuracy-boosting techniques, it's probably not a great idea that these conventional methods are not compared. \n\n - I do not fully follow why Sections 4.3 and 4.4 are devoted only to iFlooding and Flooding, why it is not interesting to compare other methods? Even if these methods do not explicitly account for stability, it's still interesting to see the curves, especially since it seems Figure 2(a)(b) can be computed simultaneously for the experiments used to generate Table 1 with no additional computing resources needed. \n\nI might have asked for too many empirical results, which may not be doable in the rebuttal phase, but I will suggest the authors at least touch each of these with the CIFAR10 experiment. \n\n[1] High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks ", "summary_of_the_paper": "This paper extends the idea of Flooding to a sample-specific level and call it iFlooding. The extension is intuitively important and the authors also offered several analytical discussions to show its importance beyond the intuition. The empirical results are fairly relevant and strong. ", "summary_of_the_review": "I think the paper is very interesting, intuitive, and a natural extension of the Flooding idea. The empirical section can be improved though. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X6QMw0M_e_", "reviewer": "Reviewer_asHY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well organized and easy to read. Although the method is simple, it has a specific theoretical basis. \nMy primary concerns are listed as follows:\n\n1. In Section 3.1, the authors gave some analyses and concluded that ''iFlood encourages the model to better ﬁt the under-ﬁtted instances while suppressing the conﬁdence of over-ﬁtted ones.'' However, according to the analysis, it is difficult to determine how this conclusion was reached. Since, sometimes, the individual loss cannot reflect the model confidence. I think more experiments about model calibration are needed to support this claim. \n\n2.  In Eq.7, why do the authors use the Laplace distribution to simulate the loss function? \n\n3. For Table 2, the authors provided the results of variances. I understand that the authors evaluate ten models trained with different methods. I would like to know whether these results were obtained under fixed random seeds? Or is the random seed selected randomly for each experiment evaluation?\n\n4. In Figures 2 (a) and (b), I noticed some abrupt changes in the learning curve, especially near the second drop of the learning rate. What is the reason for this phenomenon?\n\n5. I think the experiments of label noise are not very good compared to current methods (like self-adaptive training [1]). \n\n[1] Self-Adaptive Training: beyond Empirical Risk Minimization, NeurIPS 2020. \n\n\n\n", "summary_of_the_paper": "This paper introduced a simple but effective method to mitigate overfitting, which modifies the existing Flooding scheme. The proposed iFlood can solve the potential problems of the Flooding algorithm and improve the stability. Furthermore, the authors gave some theoretical analyses of the proposed method. The experiments indicated the proposed method is better than the baselines.\n", "summary_of_the_review": "I think the proposed iFlood is simple but effective. The theoretical analysis is straightforward. However, there are still some problems, and the experimental results are not good. So, I recommend to ''marginally below the acceptance threshold''.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6oq9N0zcTI", "reviewer": "Reviewer_WRr1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Idea is simple and novel.\n- The empirical improvement compared to standard flooding is encouraging.\n- Experiments on noisy labels show significant improvement.\n\nWeaknesses:\n- The formulation of noisy labels and biased sample distribution is unclear. The current formulation of biased sample seems like another form of incorrect labels, where P(z|x) and P(y|x) are different. These two forms of noisy labels could potentially be merged in a unified setting. On the other hand, I am wondering whether distribution shift in the input distribution P(x) could be covered as well?\n- I believe the benefit of noisy labels and biased samples should be further investigated. Currently only 2 out of the 7 datasets ran a simple noisy label experiment. There is no empirical evaluation of biased samples since the noisy labels are just uniform (based on the description in Appendix E).\n- It’s unclear whether the stated theoretical properties applies only to iFlood or both Flood and iFlood and I would appreciate it if the authors can clarify the relations here.\n- On standard benchmarks the improvement seems marginal. This wouldn’t be a major problem if the other properties such as stability and noisy labels can be highlighted more in the experiments.\n- Regarding to the claims around stability, the paper uses several measures to look at the loss distribution and the gradient norm. However, there isn't strong and direct evidence that Flooding leads to \"unstable\" training, i.e. the training loss diverges. This argument could be strengthened more.", "summary_of_the_paper": "The paper proposes a flooding loss function that encourages the training loss for each example to be a positive bias instead of zero. The paper has found that such training objective stabilizes the training compared to the regular Flooding, which regularizes the average loss instead. The paper also provides reasoning why such a loss function provides more robustness for training with noisy labels and biased label distributions. The empirical benefit on standard datasets seems marginal, but the benefit on noisy labels is significant.", "summary_of_the_review": "This paper presents an interesting loss function that provides more robustness and also performs slightly better in regular settings. I found the paper interesting and potentially have a strong impact. I think the experimental design and theoretical sections could be strengthened more and highlight more benefits dealing with noisy labels and dataset biases. My overall rating is “weak accept”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the original work on flooding to individual instance level to prevent overfitting. Even though the technique is a intuitive extension, the reviewers appreciate its simplicity and effectiveness, and consider the extension necessary. Most reviewers' concerns were addressed through rebuttal.
Decision: Accept (Poster)