AdaFlood: Adaptive Flood Regularization
Authors: Wonho Bae
Yi Ren
Mohamed Osama Ahmed
Frederick Tung
Danica J. Sutherland
Gabriel L. Oliveira
Venue: Accepted by TMLR ()

Abstract: Although neural networks are conventionally optimized towards zero training loss, it has been recently learned that targeting a non-zero training loss threshold, referred to as a flood level, often enables better test time generalization. Current approaches, however, apply the same constant flood level to all training samples, which inherently assumes all the samples have the same difficulty. We present AdaFlood, a novel flood regularization method that adapts the flood level of each training sample according to the difficulty of the sample. Intuitively, since training samples are not equal in difficulty, the target training loss should be conditioned on the instance. Experiments on datasets covering four diverse input modalities -- text, images, asynchronous event sequences, and tabular -- demonstrate the versatility of AdaFlood across data domains and noise levels.

Published in Transactions on Machine Learning Research (08/2024)
AdaFlood: Adaptive Flood Regularization
Wonho Bac
whbae@ass ubc.cn
University of British Columbia
Yi Ren
rengi.joslcuo.Bgaiik.cmmm
University of British Columbia
Mohamed Osama Ahmed Borealis Al
mohamed o. www.med@@orraaliicctc
Frederick Tung
frederick wung@brreaiiiii.ccomm
Borealis Al
Danica J. Sutherland
dsuth ubc.cr
University of British Columbia & Amii
Gabriel L. Oliveira
gabriel. owwiveiirrbooeelsiiiiiiii. com
Borealis Al
Reviewed OpenReview: https: openreview. httts:://penrevveeu net/ forum? id= 2s5YU6CSEz
Abstract
Although neural networks are conventionally optimized towards zero training loss, it has been recently learned that targeting non-zero training loss threshold, referred to flood level, often enables better test time generalization. Current approaches, however, apply the same constant flood level to all training samples, which inherently assumes all the samples have the same difficulty. We present AdaFlood, a novel flood regularization mothod that adapts the flood level of cach training sample according to the difficulty of the sample. Intuitively, since training samples are not. equal difficulty, the target training loss should be conditioned on the instance. Experiments on datasets covering four diverse input modalities text, images, asynchronous event sequences, and tabular demonstrate the versatility of AdaFlood across data domains and noise levels.
1 Introduction
Preventing overfitting is important problem of great practical interest in training deep neural networks, which often have the capacity to memorize entire training sets, even ones with incorrect labels (Neyshabur
et al.
2015 Zhang et al. 2021). Common strategies to reduce overfitting and improve generalization per-
formance include weight regularization (Krogh & Hertz
1991
Tibshirani
1996 Liu & Ye 2010 dropout
(Wager et al. 2013 Srivastava et al. 2014 Liang et al. 2021 1. label smoothing (Yuan et al. 2020 and data
augmentation (Balestriero et al.] 2022)
Although neural networks are conventionally optimized towards zero training loss, it has recently been shown that targeting non-zero training loss threshold, referred to flood level, provides surprisingly simple yet effective strategy to reduce overfitting (Ishida et al. 2020 Xie et al. 2022). The original Flood regularizer (Ishida et al.] 2020) drives the mean training loss towards a constant, non-zero flood level, while the stale-of-the-art iFlood regularizer (Xie et. al.) 2022 applies a constant, non-zero flood level to each. training instance.
Training samples are, however, not uniformly difficult: some instances have more irreducible uncertainty than others (i.e. heteroskedlastic noise), while some instances simply easier to fit than others. It. may

---

Published in Transactions on Machine Learning Research (08/2024)
not be beneficial to aggressively drive down the training loss for training samples that are outliers, noisy, mislabeled. We explore this difference in the difficulty of training samples further in Section 3.1 To address this issue, we present Adaptive Flooding (AdaFlood), a novel flood regularizer that adapts the flood level of each training sample according to the difficulty of the sample (Section 3.2) We present theoretical support for AdaFlood in Section 3.4
Like previous flood regularizers, AdaFlood is simple to implement and compatible with any optimizer. AdaFlood determines the appropriate flood level for each sample using auxiliary network that is trained subset of the training dataset. Adaptive flood levels need to be computed for each instance only once, in a pre-processing step prior to training the main network. The results of this pre-processing step are not specific to the main network, and so can be shared across multiple hyper-parameter tuning runs. Furthermore, we propose a significantly more efficient way to train an auxiliary model based on fine-tuning, which saves substantially memory and computation, especially for overparameterized neural networks (Soctions 3.3 and 4.6).
Our experiments (Section 4 demonstrate that AdaFlood generally outperforms previous flood methods variety of tasks, including image and text classification, probability density estimation for asynchronous event sequences, and regression for tabular datasets. Models trained with AdaFlood are also more robust to noise (Section 4.3 and belter-calibrated (Section 4.4| than those trained with other flood regularizers.
2 Related Work
Regularization techniques have been broadly explored in the machine learning community to improve the generalization ability of neural networks. Regularizers augment or modify the training objective and are typically compatible with different model architectures, base loss functions, and optimizers. They can be
used to achieve diverse purposes including reducing overfitting (Hanson & Pratt 1988 Ioffe & Szegedy 2015
Krogh & Hertz 1991
Liang et al.] 2021 Lim et al.) 2022) Srivastava et al.) 2014 Szegedy et al. 2016 Verma
et al. 2019 Yuan et al., 2020 Zhang et al. 2018), addressing data imbalance [Cao et al. 2019
Gong et al.
2022 and compressing models (Ding et al. 2019 Li ct al. 2020 Zhuang et al. 2020).
AdaFlood is a regularization technique for reducing overfitting. Commonly adopted techniques for reducing overfitting include weight decay (Hanson & Pratt 1988 Krogh & Hertz 1991), dropout (Liang et al.
2021
Srivastava al. 2014), batch normalization (Toffe & Szegedy 2015), label smoothing (Szegedy et al.
2016
Yuan el al.) 2020 and data augnentation (Lim et al. 2022 Verma el al. 2019 Zhang el al. 201 8). Inspired
by work on double descent (Belkin et al. 2019 Nakkiran et al. 2021), Ishida et al. (2020): Xie et al. (2022)
proposed Flood and iFlood, respectively, to prevent the training loss from reaching zero by maintaining a
small constant value. In contrast to the original flood regularizer, which encourages the overall training loss towards a constant target, iFlood drives each training sample's ' loss towards some constant b.
AdaFlood instead uses an auxiliary model trained on a heldout dataset to assign an adaptive flood level to each training sample. Using a heldout dataset to condition the training of the primary model is an effective strategy in machine learning, and is regularly seen in meta-learning (Bertinetto et al. 2019 Franceschi et al.
2018), batch or data selection [ann et al.] 2018 Mindermann et al. 2022 and neural architecture search among of her areas.
[Liu et al. 2019 Wang el al. 2021
3
Adaptive Flooding
Adaptive Flooding (AdaFlood) is general regularization method for training noural networks; can ac- commodate any typical loss function and optimizer.
3.1 Problem Statement
Background Given labeled training dataset D {(x,, W) where a, X are data samples and
V4 E y are labels, we train a neural network f + y by minimizing training loss é y x 5 - R. In supervised learning we usually have e 2 0, but in settings such density estimation it may be negative.
2

---

Published in Transactions on Machine Learning Research (08/2024)
1.
CIFARIO
095
Noiee
Whrn
1.2
CLEAR1O
CIEAR1O
2055
Nosse
Noise
109
Hand
10
CIEARIOI USS Noise
Training Loss
Density
0.5
CIEARION
20%6 Noise
10-1
CIFARION
40% Noise
0.6
10-2
0.4
10-3
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
50
100
150
200
250
Difficulty (Test Loss)
Epoch
(a) Dispersion of difficulty
(b) Training dynamics by difficulty
Figure (a) Illustration of how difficulties of examples are dispersed with and without label noise (where the relevant porlion of examples have their label switched to random other label). (b) Comparison of training dynamics some examples between iFlood and AdaFlood. The "Hard" example is labeled horse, but models usually predict cow; the "Wrong" example is incorrectly labeled in the dataset as cat (there is no rat class).
While conventional training procedures attempt to minimize the average training loss, this can lead to overfitting on training samples.
The original flood regularizer [shida et al. 2020) defines a global flood level for the average training loss, attempting to reduce the "incentive" to overfit. Denote the average training loss by C(f.B) = B EE f(;)), where f(z;) denotes the model prediction and B = {(0, yi) is a mini-batee with size of B. Instead of minimizing L, Flood (Ishida et al., 2020) regularizes the training by minimizing
Leoood (J, (C( - bb
(1)
where the hyperparameter b is a fixed flood level. Individual Flood (iFlood) instead assigns "local" flood level, trying to avoid instability observed with Flood (Xie et al. 2022):
1
FiFlood (f,
Σ
(I(()))(a))) - b) +
(2)
B
Motivation Training samples are, however, not uniformly difficult: some are inherently casier to fit than others. Figure Ta shows the dispersion of difficulty on CIFAR10 and 100 with various levels of added label noise, as measured by the heldout cross-entropy loss from cross-validated models. Although difficulties on CIFAR10 without noise are concentrated around difficulty S 0.5, as the noise increases, they vastly spread out. CIFAR100 has wide spread in difficulty, even without noise. A constant flood level as used in iFlood may be reasonable for un-noised CIFAR10, but it seems less appropriate for CIFAR100 noisy-label cases.
Moreover, it may not be beneficial to aggressively drive the training loss for training samples that are outliers, noisy, or mislabeled. In Figure Ib we show training dynamics on an easy, wrong, and hard example from the training set of CIFAR10. With iFlood, cach example's ' loss converges to the pre-determined flood level (0.03); with AdaFlood, the easy example converges towards zero loss, while the wrong and hard examples maintain higher loss.
3.2 Proposed Method: AdaFlood
Many advances in efficient neural network training and inference, such as batch or data selection (Coleman ct al. 2020 Fan al. 2018 Mindermann al. (002) and dynamic neural networks (Li al. 2021 Verelst & Tuytelaars] 2020)), stem from efforts address the differences in per-sample difficulty. AdaFlod connects this observation to flooding. Intuitively, easy training samples (e.g. correctly-labeled image of cal in
3

---

Published in Transactions on Machine Learning Research (08/2024)
Train auxiliury models
2) Train main models
raux2
Pre-compute flood levels via Eq (4)
Optimize
Eq.
Pre-computc Bood levels via Eq. (4)
Truining dutaset
Training dulasel
Figure 2: AdaFlooo for seltings where training data is limiled and acquiring additional data is impractical. In the first stage, we partition the training set into two halves and train two auxiliary networks faux,1 and faux,2; one on each half. In the second stage, we use each auxiliary network to set the adaptive flood level of training samples from the half has not seen, via equation !| The main network f then trained on the entire training set, minimizing the AdaFlood-regulariiee loss, equation
B Note that the flood levels are fixed over the course of training f and need to be pre-computed once only.
typical pose) can be driven more aggressively to zero training loss without overfitting the model, while doing so for noisy, outlier, incorreetlyllbbeld training samples may cause overfitting. These types of data points behave differently during training (Ren et al. 2022 and so should probably not be treated the same. AdaFlood differentiates training samples by setting sample-specific flood level 0 [0 in ils objective:
Addafloo (B,
1 B
Σ
- - [ 0.).
Here the sample-specific parameters should be according to the individual sample's difficulty. AdaFlood estimates this quantity according to
0, = Ely,, (faux. (x.), ys)),
(4)
where faux,s is an auxiliary model trained with cross-validation such that æ; is in its heldout set, and (,(.) is a "correction" function explained in a moment. Figure illustrates the training process using equation 3 Section 3.5 gives further motivation, and Section 3.4 provides further theoretical support.
The flood targets are fixed over the course of training the main network f, and can be pre-computed for each training sample prior to the first epoch of training f. We typically use five-fold cross-validation a reasonable trade-off between computational expense and good-enough models to estimate 6,, but see further discussion in Section 3.3 The cost of this pre-processing step can be further amortized over many training runs of the main network f since different variations and configurations of f can reuse the adaptive flood levels.
Correction function. Unfortunately, the predictions from auxiliary models are not always correct even when trained on most of the training set-if they were, our model would be perfect already. In icuticular, the adaptive flood levels 0 can be arbitrarily large for any difficult examples where the auxiliary model is incorroct; this could lead to strange behavior when we encourage the primary model to be very incorrect. We thus "correct." the predictions with the correction function Φyy, which mixes between the dataset label and the heldout model's signal.
For regression tasks, the predictions f(x;) E R should simply be close to the labels E R. The correction function linearly interpolates the predictions and labels as,
(faux (x,), vi) (1 faux Yis
(5)
4

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm Training of Auxiliary Network(s) and AdaFlood
Train single auxiliary network faux on the entire training set D
Fine-tuning method only
2: for Daux, in {Daux.ijy J==1 Ji=1 do
3: Train faux.;, either from scratch or by fine-tuning faux on D Daux,
4: Save the adaptive flood level 0g for each 2, D using faux, on a E Daux,
5: end for
6: Train the main model f using Equation 3 and adaptive flood levels computed above
Here 0 fully trusts the auxiliary models (no "correction") ") while y = 1 disables flooding.
For K-way classification tasks, f(x))) E RRK is a vector of output probabilities (following a softmax layer) and the label is y, [0, 11 usually considered as a one-hot vector. Cross-entropy loss is then computed as: Elys, f(x,)) - Σm= k log f())). Similar to the regression tasks, we define the correction function Φr (faux (1.u (x.). y:) for classification tasks as a lincar interpolation between the predictions and labels as:
Φr (J (Jnux (xx)) v/) (1 - >-/ m,xx))((((;))))) + 7
(6)
Again, for 7 = there is no "correction," " and for y = flooding is disabled, as 02 = E Yo,k log y is the lower bound of the cross-entropy loss.
The hyperparameter 7 E [0, 1] is perhaps simpler interpret and search for than directly identifying a flood level as in Flood or iFlood; in those cases, the level is unbounded (in [0, oo) for supervised tasks and all of R for density estimation) and the choice is quite sensitive to the particular task.
(0)
3.3 Efficiently Training Auxiliary Networks
Although the losses from auxiliary networks can often be good measures for the difficulties of samples, this is only true when the number of folds n is reasonably large; otherwise the training set of size about -=1 may be too much smaller than D for the model to have comparable performance. The computational cost scales roughly linearly with n, however, since we must train auxiliary networks: if we do this in parallel it requires n times the computational resources, or if we do it sequentially it takes n times as long as training single model.
To alleviate the computational overhead for training auxiliary networks, we sometimes instead approximate the process by fine-tuning a single auxiliary network. More specifically, we first train a single base model faux on the entire training set D. We then train each of the n auxiliary models by randomly re-initializing the last few layers, then re-training with the relevant fold held out. The overall process is illustrated in Algorithm I and n case is described in Figure 3
Although this means that æ; does slightly influence the final prediction fx.) ("training the test set"), it is worth remembering that we use O only as parameter in our model, not to evaluate its performance: a; is in fact a training data point for the overall model f being trained. This procedure is justified by recent understanding in the field that in typical settings, a single data point only loosely influence the early layers of a network. In highly over wee-parameterized settings (the "kernel regime") where neural tangent kernel theory is a good approximation to the training of faux (Jacot et al., 2018), re-initializing the last. layer would completely remove the effect of x; on the model. Even more realistic settings, although the mechanism is not yet fully understood, last layer re-training seems to do an excellent job at retaining "core" features and
removing "spurious" ones that are more specific to individual data points (Kirichenko et al. 2023 ct al. 2023).
LaBonte
For smaller models with fewer than a million parameters, we use 2- or 5-fold cross-validation, since training multiple auxiliary models is not much of a computational burden. For larger models such as ResNet18, however, we use the fine-tuning method. This substantially reduces training time, since each fino-tuning gradient step less expensive and the models converge much faster given strong features from lower levels than they do starting from scratch; Section 4.6 gives comparison.
5

---

Published in Transactions on Machine Learning Research (08/2024)
Fine-tune
paus
faax
Fine-tine
Randomly initialize layers
Figure 3: Efficient fine-tuning method for training a auxiliary network when held-out split 2. First, a single model faux trained on the entire training D. Then, the last few layers of each of the auxiliary models are randomly re-initialized and re-trained with the relevant fold held out.
To validate the quality of the flood levels from the fine-tuned auxiliary network, we compare them to the flood levels from n 50 auxiliary models using ResNet18 (He al.) 2016) on CIFAR10 (Krizhevsky et al., 2009); with
50, each model is being trained on 98% of the full dalaset, and thus should be a good approximation to the best that this kind of method can achieve. The Spearman rank correlation between the flood levels 0g from the fine-tuned method and the full cross-validation is 0.63, a healthy indication that this method provides substantial signal for the "correct" 8, Our experimental results also reinforee that this procedure chooses a reasonable set of parameters.
3.4 Theoretical Intuition
For a deeper understanding of AdaFlood's advantages, we now examine a somewhat stylized supervised learning setting: an overparameterized regime where the 0; are nonetheless optimal.
Proposition 1. Let be set of condidate models, and suppose there erists an optimal model fopt E arg mi Ey, (x)), where is a monnegative loss function. Given a dataset D (e), let femp denote a mènèmizer of the empirical loss L(f,D) = + Ei= fyy,, f(x))); suppose that, in an over- parameterized setting, [(femp. DD- = O. Also, let fada be a ménömizer of the AdaFlood loss equation 3 using "perfect" flood levels = {0:),r where 0; = E(y,, Jope(at)). Then we have that.
C(Jemp: D) = 0 < L(S D) [(Jadai D).
(7)
Furthermore, we have that
LAdeFloodlfempp D,0) 2C(sops, D) 2 C(fopts D) = AddeFlood D,0) CAdaPlood D, D,
(8)
Proof. We know that [(Jopt, D) will be approximately the Bayes risk, the irreducible distributtinna error achieved by fopti this holds for instance by the law of large numbers, since fopt is independent of the random sample D. Thus, the Bayes risk is nonzero and the are optimal, we can see that empirical risk minimization of overparametrized models will find femp. and disallow fopti minimizing LAdaFlood, the other hand, will allow the solution fopt and disallow the empirical risk minimizer Semp-
With this choice of 0,, we have that
AddaFlood (..dFloodU/DD00)) N
Σ
((u.. - Sopt( + E(vr, Sopa (xx))) 1=1
Since |-| is nonnegative, have LAdaFhooa(,DD > [(Sopt, D) for any J, and LAdaFlood (Sot: D,0) C(fope, D); this establishes that fopt minimizes LAdaFlood, and that any minimizer fada must achieve E(yi, fada (x;)) = 0, for each so L(fada: D) L(sopt,, D). Using that E(y,, Semp (2:)():)) = 0 for each is necessary for é 2 when C(femp, D) 0, shows [AdaFlood(femp. D,0) 7 E." 20, 2C(fppt.
In settings where 0, is not perfect (and we would not expect the auxiliary models to obtain perfect estimates of the loss) the comparison will still approximately hold. If 0, consistently overestimates the fopt loss, fape will

---

Published in Transactions on Machine Learning Research (08/2024)
TSNE of the toy data
Loss of training samples
Loss of valid samples
1.0
Regular
Regular
Iregullrr
0.8
Irregular
Wrong Label
Wrong Label
0.4
0.4
0.2
0.2
0.0
0.0
0
50
100
150
200
0
50
100
150
200
Figure 4: Left: the t-SNE Van der Maaten & Hinton 2008 of toy Gaussian example; middle: loss of different samples in the training set; right: loss of different samples in the validation set.
still be preferred to femp: for instance, if 0; 20(ys, fope (xx)), then CAdaFlood(eemp D, = 4c(fopt: D) > {.ppp D) = L AdoFoood (oop: D, 0). On the other hand, if 0; = [VVi (x;;) nof-unreasonable situation when using a correction function thenn (AdaFlood(femp D, 0) = [(Sopt. D) = L AddaFlood (fopt. D, D) When 0, random, the situation more complex, but we can expect that noisy 0, which somewhat overestimate the loss of fopt will still prefer fopt to femp-
3.5 Discussion: Why We Calculate Using Held-out Data
we estimate 0, for each training sample using the output of an auxiliary network (xu((z;) that is trained on a held-out dataset. In fact, this adaptive flood level 6g can be considered the sample difficulty when training the main network. Hence, it is reasonable to consider existing difficulty measurements based learning dynamics, like C-score (Jiang al., 2021 forgetting score (Maini al. 2022). However, we find these methods are not robust when wrong labels exist in the training data, because the network will learn to remember the wrong label of x;, and hence provide a low 0, for the wrong sample, which is harmful to our method. That why we propose to split. the whole training set into n parts and train yaux(r;) for n times (cach with different n parts).
In Section 3.2
Dataset and implementation To verify this, we conduct experiments on a toy Gaussian dataset, as illustrated in the first panel Figure Assume we have N samples, each sample 2-tuple (x, y). To draw sample, we first select the label y following a uniform distribution over all K classes. After that, we sample the input signal (y k) ([u,,02) where is the noise level for all the samples. μ4 is the mcan vector for all the samples in class k. Each μæ is 10-dim vector, in which each dimension is randomly selected from {-6 0, 8, Such process is similar to selecting 10 different features for each class. We consider 3 types of samples for each class: regular samples, the typical or easy samples in our training set, have a small σ; irregular samples have a larger σ; mislabeled samples have a small Ø, but with a flipped label. We generate two datasets following this same procedure (call them datasets A and B). Then, we randomly initialize 2-layer MLP with ReLU layers and train it on dataset. A. At the end of every epoch, record the loss of cach sample in datasct A.
The learning paths are illustrated in the second panel in Figure The model is clearly able remember all the wrong labels, as all the curves converge to a small value. If we calculate 0, in this way, all 0, would have similar values. However, if we instead train the model using dataset B, which comes from the dataset A will behave
Result
same distribution but is different from dataset A, the learning curves samples like the last panel in Figure g The mislabeled and some irregular samples can be clearly identified from the figure. Calculating 8 in this way gives different samples more distinct flood values, which makes our method more robust to sample noise, as our experiments on various scenarios show.
7

---

Published in Transactions on Machine Learning Research (08/2024)
Uber
Reddit
Stack Overflow
NTPP
Method
RMSE
NLL
RMSE
NLL
ACC
RMSE
NLL
ACC
75.83
3.86
0.25
1.28
55.26
6.69
3.66
45.52
Unreg.
(6.12)
(0.05)
(0.01)
(0.07)
(0.57)
(0.98)
(0.12)
(0.07)
64.34
4.01
0.25
1.17
57.46
4.12
3.46
45.76
Flood
(3.85)
(0.02)
(0.01)
(0.06)
(0.84)
(0.23)
(0.03)
(0.03)
Intensity-free
67.07
3.97
0.23
1.11
56.59
4.12
3.46
45.76
iFlood
(3.12)
(0.06)
(0.01)
(0.12)
(0.92)
(0.23)
(0.03)
(0.03)
59.69
3.75
0.26
1.09
59.02
3.26
3.45
45.67
AdaFlood
(1.49)
(0.01)
(0.02)
(0.13)
(0.91)
(0.25)
(0.04)
(0.03)
71.01
3.73
0.28
0.82
58.63
1.46
2.82
46.24
Unreg.
(6.12)
(0.05)
(0.01)
(0.07)
(0.57)
(0.98)
(0.12)
(0.07)
68.61
3.70
0.26
1.02
58.05
1.39
2.79
46.31
Flood
(3.85)
(0.02)
(0.01)
(0.06)
(0.84)
(0.23)
(0.03)
(0.03)
THP+
68.61
3.70
0.25
0.92
58.93
1.46
2.82
46.24
iFlood
(4.76)
(0.17)
(0.01)
(0.28)
(1.26)
(0.06)
(0.04)
(0.08)
54.85
3.55
0.25
0.80
61.34
1.38
2.77
46.41
AdaFlood
(1.49)
(0.01)
(0.02)
(0.13)
(0.91)
(0.25)
(0.04)
(0.03)
Comparison of flooding methods on asynchronous event sequence datasets. The numbers are the means and standard errors (in parentheses) over three runs.
Table 1:
4 Experiments
We demonstrate the effectiveness of AdaFlood three tasks (probability density estimation, classification and regression) in four domains (asynchronous event sequences, image, text and tabular). We compare flooding methods on asynchronous event time in Section 4.1 and image classification tasks in Section 4.2 We also demonstrate that AdaFlood more robust to various noisy settings in Section 4.3 and that it yields better-calibrated models Section 4.4. Some ablation studies are provided Sections and 4.6
4.1 Results on Asynchronous Event Sequences
We compare flooding methods on asynchronous event sequence datasets of which goal is to estimate the probability distribution of the next event time given the previous event times. Each event may have a class label. Asynchronous event sequences are often modeled as temporal point processes (TPPs).
Datasets We use two popular benchmark datasets, Stack Overflow (predicting the times at which users receive badges) and Reddit. (predicting posting times). Following [ae al. (2023), also benchmark method on dataset with stronger periodic patterns: Uber (predicting pick-up times). We split each training
dataset into train (80%) and validation (20%) sets. Details are provided in Appendix A
Following the literature in TPPs, we use two metrics to evaluate models: root mean squared error (RMSE) and negative logliiket (NLL). As NLL can be mislcadingly low the probability density mostly focused on the correct event time, RMSE is also considered complementary metric. However, RMSE has own limitation: if baseline is directly trained on the ground truth event times as point estimation, the stochastic components of TPPs are ignored. Therefore, to ensure that we do not rely too heavily on RMSE scores and account for the stochastic nature of TPPs. When class labels for events are available, we also report the accuracy of class predictions.
train TPP models on NLL and use RMSE at test time
Implementation For TPP models to predict the asynchronous event times, we employ Intensity-free mod- els (Shhhu et al. 2020) based on GRU (Chung et al. 2014), and Transformer Hawkes Processes (THP) ZZo et al. 2020) based on Transformer (Vaswani al.) 2017). THP predicts intensitics to compute log-likelihood and expected event times, but this approach can be computationally expensive due the need compute integrals, particularly double integrals to caleulate the expected event times. To overcome this challenge

---

Published in Transactions on Machine Learning Research (08/2024)
SVHN
CIFARIO
CIFAR100
Method
w/o L2 reg.
W/ Lu reg.
w/o Lg reg.
w/ L2 reg.
w/o L2 reg.
W/ L2 reg.
Unreg-
95.65 + 0.05
96.07 + 0.01
87.80 + 0.31
90.35 + 0.21
56.59 + 0.32
61.49 + 0.16
Flood
95.63 + 0.02
96.13 + 0.02
87.57 + 0.16
90.09 + 0.20
55.88 + 0.18
60.96 + 0.03
iFlood
95.63 + 0.08
96.05 0.02
87.96 = 0.07
90.57 = 0.12
56.32 = 0.05
61.63 = 0.12
KD
95.69 = 0.02
96.08 + 0.10
88.06 + 0.23
90.65 = 0.03
56.67 = 0.15
61.29 + 0.03
AdaFlood
95.72 +: 0.01
96.16 + 0.02
88.38 + 0.18
90.82 + 0.08
57.25 + 0.14
62.31 + 0.14
Table 2: Comparison of flooding methods image classification datasets with and without Lą regularization. The numbers are the means and standard errors over three runs.
while maintaining performance, we follow [ae et. al. (2023) in using a mixture of log- distributions, proposed in Shehur et al. (0200). for the decoder; call this THP
For cach dataset, we conduct hyper- pperrpparaneter tuning for learning rate and the weight for L2 regularization with the unregularized baseline (we still apply early stopping and L2 regularization by default). Once learning rate and weight decay parameters are fixed, we search for the optimal flood levels. The optimal flood levels are selected via grid search on (-50, -45,- -40-440... ( -..... 4) for Flood and iFlood, and optimal 7 on 00.0, 0.9) for AdaFlood using the validation set. We use five auxiliary models.
Results In order to evaluate the effectiveness of various regularization methods, present the results of our experiments in Table [ (showing means and standard errors from three runs). This is the first time we know of where flooding methods have been applied in this domain; we see that all flooding methods improve the generalization performance here, sometimes substantially. Furthermore, AdaFlood often outperforms other flooding methods on various datasets, suggesting that instance-wise flooding level adaptation using auxiliary models can effectively enhance the generalization capabilities of TPP models. However, there are instances where AdaFlood's performance is comparable to slightly worse than other methods, indicating that its effectiveness may vary depending on the specific context. Despite this variability, AdaFlood generally appears to be the best choice for training TPP models.
4.2 Results Image Classification
Datasets We use SVHN (Ntter el al. 2011
CIFAR-10, and 100 (Krizhevsky el. al. 2009 for image classification with random crop and horizontal flip as augmentation. Unlike Xie et al. (2022), we split each training dataset into train (80%) and validation (20%) sets for hyperparameter search; thus our numbers are generally somewhat worse than what they reported, as we do not directly tune on the test set.
Implementation Following Ishida et al. (2020) and similar to Xie al. (2022), we consider training R8sNet18 (He et al.) 2016) with and without Le regularization (with a weight of 10-4). All methods are trained with SGD for 300 epochs, with early stopping. We use a multi-step learning rate scheduler with an initial learning rate of 0.1 and decay coefficient of 0.2, applied at every 60 epochs. The optimal flood levels are selected based on validation performance with a grid search on (0.01, 0.02.... 0.1, 0.15, 0.2.... 122..,,1.0.... 1.0} for a.1......95) Flood and iFlood, and (0.05, O....., 0.95] for AdaFlood. We use a single ResNet18 auxiliary network where its layer 3 and 4 are randomly initialized and fine-tuned on held-out sets with n = 10 splits.
Furthermore, we compare with knowledge distillation (KD) baselines following Hinton et al. (0144) for the implementation of its loss. For a mini-batch B, the KD loss is defined as:
(RD B, ax) cv) = aLce(Js B) + -1- (1 Cr St. B)
(9)
where J denotes a temperature scale, which is an additional input to student model fa and teacher model Jt. Also, Lee and Liittlll are defined as:
Eoman(se, fú, B) B Σ Uy, ((x)), ())(S B ==1
1 Σ (), r)).
(10)

---

Published in Transactions on Machine Learning Research (08/2024)
We set = 0.5 following one of the experiments in Hinton et al. (2014) that all the methods have only one hyperparameter to tune. We tune the temperature scale T with a grid search on 2, ,9, .9,10) 10).
Results The results are presented in Table We report the means and standard errors of accuracies over three runs. We can observe that KD and flooding methods, including AdaFlood, are not significantly better than the unregularized baseline on SVHN. However, AdaFlood noticeably improves the performance over the other methods on harder datasets like CIFAR1O and CIFAR100, whereas iFlood is not obviously better than the baseline and Flood is worse than the baseline on C1FAR100. The gap between AdaFlood and KD is more noticcable on CIFAR100, particularly with L2 regularization.
Discussion While iFlood is closely related to label smoothing, AdaFlood shares similarities with KD both utilize auxiliary networks. However, motivation behind two algorithms are fundamentally different. KD relies on predictions made on already-seen training examples, whereas AdaFlood leverages predictions on intentionally forgotten (or unseen) examples. Since the predictions of teacher networks in KD are based on already-seen examples, they do not serve as meaningful measures of uncertainty. In contrast, the predictions from an auxiliary network in AdaFlood can effectively measure uncertainty, and flood levels computed from these predictions can function uncertainty regularizations. A disadvantage of AdaFlood, however, is the additional fine-tuning step required to forget already-seen examples, which is not necessary in KD.
4.3 Noisy Labels
Datasets In addition to CIFAR10 for image classification, we also use the tabular datasets Brazilian Houses and Wine Quality from OpenML (Vanschoren et al. 2013), following Grinsztaju et al. (2022). for regression tasks. We further employ Stanford Sentiment Treebank (SST-2) for the text classification task, following Xie et al. (2022). Details of datasets are provided in Appendix A
We report accuracy for classification tasks. For regression tasks, we report mean squared error (MSE) in the main body, as well mean absolute error (MAE) and R² score in Figure R (Appendix
Implementation We inject noise for both image and text classification by changing the label to a uniformly randomly selected wrong class, following Xie et al. (2022). More specifically, for a% of the training data, change the label to uniformly random class other than the original label. For the regression tasks, we add errors sampled from a skewed normal distribution, with skewness parameter ranging from 0.0 to 3.0. Similar to the previous experiments, we tune learning rate and the weight for I-2 regularization with the unregularized baseline (with early stopping and L2 regularization by default except for Figure 5c). Then, we tune the flood levels with the fixed learning rate and L2 regularization.
Results Figure 5 compares the flooding methods for noisy settings. We report the mean and slandard error over three runs for CIFAR1O, and five and seven runs for tabular datasets and SST-2, respectively. We provide AAce (%) for CIFARIO and SST-2 compared to the unregularized model: that is, we plot the accuracy of each method minus the accuracy of the unregularized method, to display the gaps between methods more clearly. The mean accuracies of the unregularized method are displayed below the zero line.
Wine Quality, Figure 5a AdaFlood slightly outperforms the other methods at first, but the gap significantly increases as the noise level increases.
Brazilian Houses, Figure 5b There is no significant. difference between the methods for small noise level, e.g. noise < 1.5, but MSE for AdaFlood becomes significantly lower as the noise level increases.
CIFAR10, Figure 5cl iFlood and AdaFlood significantly outperform Flood and unregularized. AdaFlood also outperforms iFlood when the noise level is high (e.g. 50%).
SST-2, Figure 5d Flooding methods outperform the unregularized method. AdaFlood is comparable to iFlood up to the noise level of 30,, but noticeably outperforms it the noise level increases.
10

---

Published in Transactions on Machine Learning Research (08/2024)
0.9
Pood
Alaflod
Fbed
iPodd
0.8
MSE
1.0
A Ace (%)
Mool
0.3
Flod
0.8
iFlod
0.4
iFlod
Addiood
0.7
0.3
0.5 1.0
1.5 20 2.5 3.0 Noise Lovel
0.0
0.5 1.0 1.5
2.0 Noise Lovel
2.5 3.0
10
20
30
40
50
60
10
20
30
40
50
Noise Level (%6
Noise Level (%6
(a) Wine Quality
(b) Brazilian Houses
(c) CIFAR1O
(d) SST-2
Figure 5: Comparison of flooding methods tabular and image datasets with noise and bias.
ECE: 18.31 + 0.24
ECE: 13.96 + 1.6
ECE: 19.82 + 2.27
ECE: 5.49 + 0.33
0.2
0.0
0.1 0.2.3.004.333 0. 0.809
0.1020.:040. 0.80700020
0.2 0.3
0.4 0.5 0.600.0000
0.2 0.3 0.4 0.500000.0003 0800
(a) Unregularized
(b) Flood
(c) iFlood
(d) AdaFlood
Figure 6: Calibration results of flooding methods with 10 bins on CIFARI00. The bars and errors are the means and standard errors over three runs, respectively.
Overall, AdaFlood is more robust noise than other flooding methods, since the model pays less attention to samples with high losses.
4.4 Calibration
Datasets and implementation Miscalibration-nen neural networks being over or under-confident- has been a well-known issue in deep learning. We thus evaluate the quality of calibration with different flooding methods CIFAR100, measured by the Expected Calibration Error (ECE) metric. (Figure does the same for CIFAR10, but since model predictions are usually quite confident, this becomes difficult to mea- surc.) We use ResNet18 with L2 regularization with the optimal hyperparameters for the bascline and flooding methods. The optimal hyperparameter varies by seed for each run.
Result Figure provides the calibration quality in ECE metric as well as a visualization over three runs, compared to perfect calibration (dotted red lines). We can observe that AdaFlood significantly improves the calibration, both in ECE and visually. Note that iFlood significantly miscalibrates at the bins corresponding to high probability e.g. bin 2 0.7, compared to the other methods, and also has high standard errors. This behavior is expected, since iFlood encourages the model not to predict higher than probability of exp(-b), where b denotes the flood level used in iFlood.
4.5 Ablation study: Relationship with Other Regularization
In this ablation study, we design experiment that shows how different regularization methods interact with flooding methods. We conduct the experiment on CIFAR100 with ResNet18, gradually adding regularization methods in the order of early stopping, L2 regularization, dropout and CutMix (Yun et al. 2019 popular data augmentation method, as shown in Table 3 Please note that the second row with early stopping and the third row with early stopping + L2 regularization are the same as what we report in Table 2
Similar the results Table 2 Flood is comparable to or slightly worse than the unregularized bascline for the case with dropout and with both dropout and CutMix. Although iFlood is generally better than the
11

---

Published in Transactions on Machine Learning Research (08/2024)
Regularization
Flooding
Early Stopping
L2
Dropout
CutMix
Unreg-
Flood
iFlood
Ada Flood
56.39 + 0.25
56.07 + 0.19
56.07 0.06
56.89 + 0.19
56.59 + 0.32
55.88
0.18
56.32 0.05
57.25 0.14
61.49 + 0.16
60.96 + 0.03
61.63 0.12
62.31 + 0.14
62.04 = 0.17
61.73 + 0.13
62.19 = 0.27
63.15 + 0.10
67 00.08 + 0.28
67.11 + 0.26
67.09 + 0.08
67.50 + 0.16
Table 3: Comparison between flooding methods with and without various regularization methods CI- FAR100. We report the means and standard errors of three runs.
unregularized baseline as shown in the third to fifth rows, the gap is limited. Compared to Flood or iFlood, AdaFlood shows more consistently larger improvement.
4.6 Ablation study: Fine-tuning vs. Multiple Auxiliaries
Figure 7 compares training of ten ResNet18 auxiliary net- works (original proposal) to the single fine-tuned auxiliary network (efficient variant) in terms of wall-clock time for training the auxiliary network(s), and performance of the corresponding main model, on the test set of CIFARIO. For the efficient variant, we fine-tune different layers to show insensitivity to the choice of layers: Layer3, 4 + FC,
90881-0077
wail-elock time (bouss)
90.82100,085
901921000%
00003L0,085
Layer3 + FC, and FC, where Layer3 and 4 are the 3rd, 4th layers in ResNet18 and FC' denotes the last fully connected layer. For example, Layer4 + FC' means we only fine-tune
11 J.eediaiio Loys 3-4 FC 4
Figure Comparison aux. training Layer4 and FC layers, froczing all the previous layers. Results show that training multiple auxiliary net- works yields the same-quality model as fine-tuning, though training time is 3 to times longer. There also little difference in performance between different fine-tuning methods: it seems that fine-tuning only the FC layer is sufficient to forget the samples, with early-stopping regularizing well enough for similar generalization ability. We also compare AdaFlood with varions architectures for auxiliary networks in Appendix
5 Conclusion
In this paper, introduced the Adaptive Flooding (AdaFlood) regularizer, novel reguralization technique that adaptively regularizes a loss for each sample based on the difficulty of the sample. Each flood level is computed only once through an auxiliary training procedure with held-out splitting, which we can make more efficient by finc-tuning the last few layers on held-out sets. Experimental results on varions domains and lasks: density estimation for asynchronous event sequences, image and text classification tasks as well as regression tasks on tabular datasets, with and without noise, demonstrated that our approach is more robustly applicable to varied range of tasks including calibration.
Limitation Although AdaFlood is a robust and effective regularizer on many different tasks, particularly in high-noise settings, an open question that we leave for future work is how to best apply AdaFlood in long-tailed learning. For long-tailed data, it expected that samples from the rare classes will tend to have higher losses. During the training of the main model, AdaFlood will direct the model keep the higher losses for rare classes and lower losses for common classes, which may not be desirable. One potential solution could be to adaptively adjust for different classes. Alternatively, imbalanced learning techniques such resampling, reweighting, two-stage training could be adopted.
Reproducibility For each experiment, we listed implementation details such as model, regularization, and search space for hyperparameters. We also specified datasets we used for each experiment, and how they were split and augmented, along with the description of metrics. The code is released with the final version.
12

---

Published in Transactions on Machine Learning Research (08/2024)
References
Wonho Bae, Mohamed Osama Ahmed, Frederick Tung, and Gabriel L Oliveira. Meta temporal point pro- cesses, In ICLR, 2023.
Randall Balcstricro, Lcon Bottou, and Yann LeCun. The effects of regularization and data augmentation are class dependent. In NeurIPS, 2022.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variane trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019.
Luca Bertinetto, João Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron den Oord. Are done with imagenet? Xiv preprint orXiv:2006.0m 00t:2006.07199, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with disooldisttibutionaaaaaa margin loss. In NewrIPS, 2019.
Junyoung Chung, Caglar Gulcchre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NewrIPS, 2014.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, In ICLR, 2020.
Ruizhou Ding, Ting-Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In CVPR, 2019.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigue,, and Le Song. Recur- rent marked temporal point processes: Embedding event history to vector. In KDD, 2016.
Yang Fan, Fei Tian, Tao Qin, Xiang- Yang Li, and TinYYan Liu. Learning teach. In ICLR, 2018.
Luca Franceschi, Paolo Frasconi, Saverio Salvo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel program- ming for hyperparameter optimization and meta-learning. In ICML, 2018
Yu Gong, Greg Mori, and Frederick Tung, RankSim: Ranking similarity regularization for deep imbalanced regression. In ICML, 2022.
Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In NewrIPS, 2022.
Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. In NewrIPS, 1988.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NIPSW, 2014.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Do need zero training loss after achieving zero training error? In ICML, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NewrIPS, 2018.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. In ICML, 2021.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson Last layer re-training sufficient for robustness spurious correlations. In ICLR, 2023.
Alex Krishevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John A. Hertz. simple weight decay can improve generalization. NeurIPS, 1991.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. In NewrIPS, 2023.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Jeffrey Pennington. Wide neural networks of any depth evolve as lincar models under gradient descent. NewrIPS, 2019.
and
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In CVPR, 2021.
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and dccomposition for network compression. In CVPR, 2020.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie- Yan Liu. R-Drop: Regularized dropout for neural networks. In NewrIPS, 2021.
Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnic Xu, and Michael W. Mahoney. Noisy feature mixup. In ICLR, 2022.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Jun Liu and Jieping Ye. Efficient 11/lq norm regularization. ar preprint arXiv:1009.4766, 2010.
Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter. Characteriving datapoints via second- split forgetting. In NewrIPS, volume 35, pp. 30044 30057, 2022.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In ICML, 2022.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. A fast, well-founded approximation the empirical neural tangent kernel. In ICML, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 2021 (22):12400,, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning, 2011.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In ICLR Workshop, 2015.
Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing lcarning paths. In ICLR, 2022.
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity- free learning of temporal point processes. Tn ICLR, 2020.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Lya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, pp. 1929-1958, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In CVPR, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B B (Methodological), 58(1):267-22) 1996.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. SIGKDD Eaplorations, pp. 49 60, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In CVPR, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. Tn ICML, 2019.
Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In NeurIPS, 2013.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architec- ture selection in differentiable NAS. In ICLR, 2021.
Yuexiang Xie, WANG Zhen, Yaliang Li, Ce Zhang, Jingren Zhou, and Bolin Ding. iFlood: A stable and effective regularizer. In ICLR, 2022.
Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization stralegy to train strong classifiers with localizable features. In ICCV, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Rocht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, kai Shuang, and Xiang Li. Neuron-level struc- tured pruning using polarization regularizer, In NewrIPS, 2020.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In ICML, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Details about Datasets
Stack Overflow It contains 6,633 sequences with 480,414 events where an event is the acquisition of badges received by users. The maximum number of sequence length is 736 and the number of marks is 22. The dataset is provided by Du et al. (2016); we use the first folder, following Shehur et al. (2020) and Bae
et al. 2023
Reddit It contains 10,000 sequences with 532,026 events where an event is posting in Reddit. The maxi- mum number of sequence length 736 and the number of marks 22. Marks represent sub-roddit categorics.
Uber contains 791 sequences with 701,579 events where an event is pick-up of customers. The maximum number of sequence length is 2,977 and there is marks. It is processed and provided by Bac al. (0023
Brazilian Houses It contains information of 10,962 houses to rent in Brazil in 2020 with 13 features. The target is the rent price for each house in Brazilian Real. According to OpenML (Vanschoren et al. 2013) where we obtained this dataset, since the data is web-scrapped, there are some values in the dataset that can be considered outliers.
Wine Quality Tt contains 6,497 samples with 11 features and the quality of wine is numerically labeled as targets. This dataset also obtained from OpenML (Vanschoren et al. 2013
SST-2 The Stanford Sentiment Treebank (SST-2) a dataset containing fully annotated parse trees, enabling comprchensive exploration of how sentiment influences language composition. Comprising 11,855 individual sentences extracted from film reviews, this dataset underwent parsing using the Stanford parser, resulting in collection of 215,154 distinct phrases.
B Additional Results on Image Classification
Datasets We use ImageNet100 100 (Tian al.) 2020) for image classification with random crop, horizontal flip, and color jitter as augmentation. We also add 30% of label noise as done in Section 4.3
Implementation We train ResNet34 (He et al. 2016) the dataset with L2 regularization (with a weight of 0.0001). All methods are trained for 200 epochs with early stopping using SGD. We use multi- step learning rate scheduler with an initial learning rate of 0.1 and decay coefficient of 0.5, applied at every 25 epochs. The optimal flood levels are selected based on validation performance with a grid search on {0.01, 0.02..., 0.1, 0.1.,. 0.3) for Flood and iFlood, and {0.05, 0.1..., 0.95} for AdaFlood. We use a single ResNet34 auxiliary network where its last FC layer is randomly initialized and fine-tuned on held-out sets with n 10 splits.
Results Table 4 (Left) compares flooding methods ImageNet dataset with and without 30% of label noise. We report test accuracies along with expected calibration error (ECE) on the right. Although Flood and iFlood do not improve the performance over the unregularized model, AdaFlood improves the performance by about 0.80% over the unregularized baseline. Given the size of the dataset, the gap is not marginal. This gap is even larger than that we observed in SVHN and CIFAR datasets Table 2 We conjecture it is because ImageNet contains more noisy samples. It is well-known that there are many ImageNet images containing multiple objects although the label says there is only one object (Beyer et al. 2020
C Additional Results on Tabular Regression
Datasets We use NYC Taxi Tip dataset from OpenML (Vanschoren et al. 2013), one of the largest tabular datasct used in Grinsztajn et al. (2022), for regression tasks. NYC Taxi Tip datasct contains 581, 835 rows and 9 features. As the name of the dataset implics the target variable is "tip amount'. To increase the importance of other features, the creator of the dataset deliberately ignores "fare amount' "trip distance'.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Mislabled Sample Rate
Noise Level
Method
Method
0%
30%
0.0
1.5
3.0
Unreg.
81.00 6.64
68.12 (17.23)
Unreg.
0.2373(0,3335)
0.3707 (-0.0409)
0.3910 (-0.0978)
Flood
81.18 (6.44)
68.19 (17.71)
Flood
0.2373(0.3335)
0.3707 (-0.0409)
0.3904 (-0.0980)
iFlood
81.04 (6.86)
68.24 (20.67)
iFlood
0.2370(0.3374)
0.3652 (-0.0255)
0.3902 (-0.0986)
AdaFlood
(4.89
69. (19.24117.55)
AdaFlood
0.2369(0.3348)
0.3520(0.0250)
0.3465(0.0119)
Table 4: Comparison of flooding methods ImageNet100 (Left) and NYC Taxi Tip (Right) datasets with and without label noise. The numbers the right on ImageNet100 represents ECE metric whereas on NYC Taxi Tip, they are R² scores.
0.70
0.75-
0.4
0.85
0.0
MAE
0.63-
0.2-
ifloed
AdaFFood
MAB.
0.80
0.2
iPood
AdaPlood
0.00-
&
0.0
0.50-
0.53
Useer
Flod
0.2
0.75
Lona,
Flod
0.3
0.4
0.45-
AddFood
0.4
0.70
AddFloo
0.5
1.5 2.0
1.0
1.5 2.0
Noise Level
Noise Level
Noise Level
Noise Level
(a) Brazilian House (MAE)
(b) Brazilian House (R²)
(c) Wine Quality (MAE)
(d) Wine Quality (R²)
Figure 8: Additional results in various metrics on tabular datasets with noise and bias
Implementation As with Section 4.3 we use model tailored for tabular dataset proposed by (Grinsztajn et al. 2022) and add errors sampled from a skewed normal distribution, with skewness parameter ranging from 0.0 to 3.0.
Results Table (Right) compares flooding methods on NYC Taxi Tip dataset GGrinslajn et al. 2022 with and without noises. We report mean square error (MSE) and R2 score on the right. Note that R2 score is usually between 0 and 1 but when predictions are bad, it can go below 0.
From the table, we can observe that all flooding methods perform similar to the unregularized baseline when there is no noise. Although it continues for Flood and iFlood even under noisy settings, AdaFlood significantly outperforms (lower MSE and higher R² scores) the other methods when noise level is 1.5 and 3.0. In particular, while R2 scores of other methods go below 0, it does not happen with AdaFlood, which demonstrates the robustness of AdaFlood even for the large-scale dataset like NYC Taxi Tip. It is consistent with the results we provided in Section 4.3
For additional information, in Figure g we also provide experiment results in MAE and R² on Brazilian House and Wine Quality datasets of which results in MSE is provided in Figures 5a and 5b
Additional Results on Calibration
We provide calibration results on CIFAR10 in Figure g As briefly mentioned in Section 4.4 since model pre- dictions are generally accurate and quite confident, the gap between different methods are not as significant as on CIFAR100 shown in Figure
E
Initialization of the Main Model using an Auxiliary Network
Even though we efficiently fine-tune an auxiliary network to compute flood levels, it may still too expensive to train both the auxiliary network faux and main model f. To reduce computation further, we may utilize a pro-trained auxiliary network when we use the same architecture for the auxiliary and main model. Instcad of randomly initialize the main model, we can initialize the parameters of the main model using the parameters from the pre-trained auxiliary network (before fine-tuning).
17

---

Published in Transactions on Machine Learning Research (08/2024)
BCE: 5.9 + 1.06
BCE: 4,06 + 1.3
ECE: 4.3 + 0.37
DCE: 4.23 + 0.85
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.1 0.20030 0.80.911 1.0
0.0 0.200004 0.09
0.10..0.300. 0.0700000
0.1
(a) Unregularized
(b) Flood
(c) iFlood
(d) Flooo
Figure 9: Calibration results of flooding methods with 10 bins on CIFAR10.
Freezing first t layers
Random Init
3
90.83 / 3.52
91.30 2.50
90.90
3.44
90.68 3.96
89.99 / 4.69
Table 5: Comparison between random initialization and initialization with the pro-trained auxiliary network. ECE metrics are reported on the right.
We conduct an experiment to validate if it actually saves computation without hurting performance on CIFARIO with ResNet18. We first trained an auxiliary network on the whole training set. The pre-trained auxiliary network is then used for both computing 0 through the fine-tuning step described in Section 3.3 and initializing the main model. When we "fino-tunc" the main model, we freeze the first layers (among four layers of ResNet18) save computation, and randomly initialize the last fully connected layer to have the model forget some information.
Table shows that if we initialize the parameters of the main model with those of the pre-trained auxiliary model without freezing any layers, it performs better than random initialization. As we freeze more layers to save computation, the performance gradually goes down compared to that of random initialization but it is still comparable up to k = 2 case.
F Ablation study: Various Architectures for Auxiliary Networks
VGG11
VGG19
ResNet18 Small
ResNet18
# of params
9.8 M
21.6 M
2.8 M
11.2 M
AdaFlood
90.74 1 0.13
91.09 + 0.06
90.73 0.12
90.82 + 0.08
Table 6: Comparison of various architectures for auxiliary networks in AdaFlood.
To investigate the robustness of AdaFlood in terms the choice of architectures for auxiliary networks, we conduct an ablation study on AdaFlood with various architectures for auxiliary networks: VGG11, VGG19, ResNet18 small and 18sNet18. Here, we use a ResNet18 for the main model, and utilize the efficient finc- tuning method to train auxiliary networks. We report the mean and standard error of three runs in Table B
With VGGI1 where its number of parameters is slightly less than ResNet18, the mean test accuracy of the main model is lower but the gap is marginal. With VGG19 which is larger than ResNet18, the performance slightly improves. We also try with a smaller variant of ResNet18 for the auxiliary network where its number of parameters a quarter of the original ResNet18. Even with this significantly smaller architecture, the mean test accuracy of the main model only slightly worse than much larger models c.g. VGG11 and ResNet18, which implies that what is important from the auxiliary network is the relative magnitude of losses (or flood levels) not the absolute values of losses.
18

---

Similarly, the Published in Transactions on Machine Learning Research (08/2024)
NTPP
Method
Uber
Reddit
Stack Overflow
Flood
{1.0, 4.0, 3.0]
{-1.0, -5.0, -3.0)
[0.0, 3.0, 1.0]
Intensity-free
iFlood
{4.0, 0.0, 1.0]
{-2.0, -3.0, -4.0}
{2.0, 2.0, 3.0]
AdaFlood
{0.2, 0.0, 0.3)
{0.0, 0.1, 0.13
{0.1, 0.4, 0.2]
Flood
-5.0, 0.0)
-2.0,,
(0.0, 2.0, 1.0]
THP
iFlood
{-1.0, 4.0, 3.0)
{-4.0, -4.0, -5.0)
{0.0, 0.0, -1.0)
AdaFlood
{0.0, 0.1, 0.33
{0.0, 0.0, 0.13
(0.3, 0.3, 0.3)
Table 7: Choice of flood levels for (i)Flood, and y for AdaFlood TPP datasets with three seeds.
SVHN
CIFARIO
CIFAR100
Method
w/o La reg.
w/o Ta reg.
Tig
w/o Ta reg.
Lig
Flood
(0.04, 0.02, 0.03)
(0.02, 0.03, 0.04) (0.05, 0.06, 0.06)
(0.02, 0.04, 0.05) {0.07, 0.04, 0.07) {0.95, 0.70, 0.65)
(0.07, 0.02, 0.01) (0.07, 0.03, 0.01]
[0.01, 0.01, 0.01) {0.45, 0.35, 0.30) {0.40, 0.50, 0.45)
(0.01. 0.01, 0.05) {0.30, 0.40, 0.65] {0.50, 0.50, 0.55)
(Flood
(0.05, 0.09, 0.02) {0.75, 0.85, 0.70)
AdaFlood
{0.35, 0.50, 0.65)
(0.75, 0.95, 0.65)
Table 8: Choice of flood levels for (i)Flood, and y for AdaFlood image classification with three seeds.
G Theoretic Intuition for Efficient Training of Auxiliary Networks
In this section, we provide thcoretic intuition for efficient training of auxiliary networks. Following LLe et al. 2019), we approximate the predictions neural network f on a test sample 2;
f(z,) = Σ (2)
(11)
where eNTK stands for empirical neural tangent kernel (NTK) following (Mohamadi et al. 2023 and
{ denotes data from training set. Equation says can approximate a prediction 2; as interpolation of eNTK(T,,,) with some weights
Suppose f(x) VV Vι(() where <(x) E R denotes a feature from the penultimate layer and V E R denotes the weights of the last fully connected layer (k being the number classes), consisting of E R for j-th row. II U95; i-th entry of Uj; is from NO0, o²), then Mohamadi al. (2023) haven shown that,
(Z1, x2) = vz eNTK &
(1,, 22)02 + 1 = (')((z)) ((22)
(12)
where denotes set of all the model parameters and w means set of all the paramcters except the last fully connected layer.
With this frame, we can approximate the predictions from the efficiently trained auxiliary network denoted as feune, as follows,
(x)) Eajuno suno (u (UTeNTK
d. d.trained W (2,, o trained (x,)) trained (zi)
(13)
Here, superscript "trained" means the model parameters are pre-trained on the whole training set.
On the other hand, the original "direct" training algorithm of auxiliary networks trains an auxiliary network from scratch, the training set excluding the samples that we measure the difficulty predictions of single auxiliary network denoted as fairect is approximated as,
fdirect (x:) Rv Σ direct (1FeNTKe
aa
untrained 2:)0 + unttrained (xy) q cuntrained (a:)) Wil
(14)
where superscript "untrained" means the model parameters are randomly initialized.
In NTK regime where a neural network has infinite-width, the terms in the parentheses (the term except a's) are the same for Equation (13) and Equation (14). Therefore, the difficulty measures from the efficient training of a single auxiliary network and direct training of multiple auxiliary networks are equivalent in higlly-overprrameeeriee regime.
19

---

Published in Transactions on Machine Learning Research (08/2024)
H Choice of Flood Levels
In Table 7 and Table g we report the choice of flood levels for Flood and iFlood, and for AdaFlood (recall that y is the hyperparameter for correct function adjusting interpolation level) on TPP and image classification tasks with three different random seeds.
One interesting obscrvation is that flood levels for iFlood on CTFAR100 significantly larger than on the other datasets. is because CIFAR100 particularly harder than the other two datasets. However, even though flood level is sometimes high e-g- 0.65, it is still reasonable choice because 0.65 flood level implies
that the model's highest predicted probabilities do not deviate much from 0.52 exp(-0.65). With this high regularization, the model is not too overconfident for its predictions.
20


Reviews:
[{"review_id": "Ec9JfS7tmN", "reviewer": "Reviewer_eBmj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors' idea of adapting the existing flood regularisation technique seems clear, simple and very reasonable. It is plausible that in many datasets, training examples will exhibit varied difficulty that may contribute to overfitting. This clarity of idea is a great strength of the paper. \n\nA few other strengths I've noted:\n1. I like the experimental setup for section 4.2. It's a really nice addition to have the flood techniques compared with and without $L_2$ regularization.\n2. The paper is quite clear in its setup and execution. In particular, I quite like the illustration provided in figure 1.\n3. I like the use of the calibration metric in section 4.4.   \n\nThere are, however, a few key weaknesses that I have identified:\n1. The content of proposition 1 does not seem worthy of a theorem statement. The proposition seems to amount to the statement that if one has irreducible uncertainty then to find the \"best model\" one should not try and obtain perfect performance on the training set.\n2. The paragraph following the proof of proposition 1 seems somewhat informal for a section entitled *theoretical intuition*.\n3. The comparison with $L_2$ regularization and inclusion of $L_2$ with the flood methods (which I really like in table 2) does not seem to be present in sections 4.1 and 4.3. I think this inclusion is important as I would argue $L_2$ regularization is a kind of \"default\" for the community. The paper would be much stronger if we knew $L_2$ and AdaFlood combined to produce even better models on all of the datasets.\n4. Following up on the point above, I feel comparisons with other simpler regularisation methods are lacking. If an easier-to-implement and (possibly) more computationally efficient technique like dropout works better, then there would be no need to use flooding. Obviously, I am not expecting the authors to try every technique in the literature. However, a few more comparisons would be nice. \n\nAlso a general note on the current writing. I find the current manuscript to be repetitive in several areas. I'd rather a shorter paper that gets straight to the point rather than a paper which is unnecessarily verbose. Here are some examples:\n1. The sentence \"The cost of pre-computation can be further amortized over many training runs of f with different configurations\" is repeated in the figure 2 caption and the sentence before the paragraph starting with \"correction function.\"\n2. I feel currently that when I read it, flood regularisation is explained many more times than is necessary. By the time I get to equations (1), (2) and (3) I already know what each method does.\n\nIn review, I believe the minimal burden of evidence has been surpassed to demonstrate that AdaFlood is likely a better alternative in several cases when compared to other flood techniques. This seems to be the central claim of the paper and is of interest to some members of the TMLR audience. Therefore I marking yes for both \"Claims And Evidence\" and \"Audience\". However, I would encourage the authors to consider the weaknesses identified and changes requested especially if they are looking to obtain a certification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors extend an existing regularisation technique known as *flood regularisation* in which neural networks are trained toward a non-zero training loss threshold. They extend this technique by adapting the threshold (flood level) to each training sample according to the difficulty of the sample.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are a few changes that I would like to see which relate specifically to the weaknesses I have identified above:\n1. I think section 3.4 should be moved to the appendix or further analysis added which provides more interesting conclusions. At the moment, I feel the section adds unnecessary complexity without much additional information.\n2. I would like to see the comparisons and combinations with $L_2$ regularisation reported in table 2 applied to the other datasets.\n3. I would like to see comparison with dropout added at some point in the paper if possible.\n\nThere are also a few minor changes to aspects of the text I would like to see addressed:\n1. The sentence \"Differences in per-sample difficulty are the basis of many advances in efficient neural network...\" should be rewritten. The innovations come from trying to address this fact not directly from it.\n2. If proposition 1 is kept, can it be made more readable? It currently takes some time to understand. Perhaps change each $f$ to have a subscript describing it e.g. $f_\\text{optimal}$ or $f_\\text{adaflood}$.\n3. Can the language regarding some of the results be toned down? For example the sentence \"AdaFlood significantly outperforms the other methods for most models on most datasets\" seems a touch too far given that there are cases where AdaFlood performs worse and several cases where its performance seems to be within the margin of error of other methods.\n4. The sentence \"Overall, AdaFlood is more robust to noise than the other flooding methods, since the model pays less attention to those samples with wrong or noisy labels\" seems a little too ambitious. Has it been demonstrated that it is paying \"less attention\" to these examples?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OndMVgNmTE", "reviewer": "Reviewer_UHtR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The approach is relatively straightforward and does not appear difficult to implement.\n\n- The paper provides some analysis into the effect of this approach on the resulting classifier in an idealized setting, providing intuition into the method.\n\n- A variety of datasets and modalities are used to evaluate the method, including image classification and tabular data.\n\n- Methods trained with AdaFlood are much better-calibrated than unregularized variants or uniform flooding strategies.\n\n- Training only the final layers of the auxiliary model appears to significantly improve computational efficeincy without limiting performance.\n\n- The paper is reasonably clear and easy to understand.\n\nWeaknesses:\n\n- The only baselines are two other flooding methods. While it is encouraging to see improvements over these, it is difficult to place this approach in the broader space of regularization approaches. I would be particularly interested in comparing the flooding approach with early stopping, as the strategy of not driving down the loss of hard/misaligned examples intuitively should have a similar effect to stopping training at a point where the easy samples have a low loss but the optimization process has not yet driven down the loss on hard examples.\n\n- Additionally, it would be useful to compare the performance of this approach with other regularization strategies given the same compute budget, since AdaFlood requires training the auxiliary models and it is worth comparing whether this additional cost is better than training with stronger regularization for longer. \n\n- AdaFlood has the potential to `waste’ a lot of compute training on samples whose target loss is high.\n\n- How does AdafFood perform in long-tailed distributions or class imbalances? I could imagine that samples from minority classes might be assigned a higher flood value, leading to worse performance on these classes.\n\n- The gains in performnace are quite small and AdaFlood’s standard errors often overlap with those of other methods (e.g. SVHN in Table 2), and the gains from AdaFlood are much smaller than the gains from adding L2 regularization.\n\n- AdaFlood only seems to provide noticeable benefits when used on somewhat contrived settings with large amounts of added label noise — the noise that naturally occurs in the datasets evaluated doesn’t seem to be significant enough to elicit a practically significant improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a regularization method based on flooding, where the flood level of each training example is determined based on the loss of an auxiliary model on this data point, where the auxiliary model is trained on a subset of the data which does not include the sample of interest. To improve the accuracy and efficiency of this method, the paper proposes training an ensemble of auxiliary models which share a feature encoding. The method is demonstrated to improve performance over uniform flooding approaches, and networks trained with this method exhibit improved calibration.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major: \n\n- Please include at minimum an early-stopping baseline and a distillation baseline, as these two methods seem like they should have similar effects to AdaFlood while also being cheaper/comparable to implement.  As a stretch request, I would be really curious to see if this style of approach could be used to prioritize training examples, so that the network doesn’t waste gradient passes on data points that it has already converged to its final loss on.\n\n- Figure 6 seems to be answering the wrong question. Instead of looking at the accuracy on the dataset of the trained auxiliary model, what is of interest is how useful the per-example losses from each approach are for setting the flood value. Thus the quantity of interest in Figure 6 is actually the final test accuracy of the final model trained with AdaFlood using the flooding values from each of the finetuning approaches. Please update the figure to reflect this.\n\n- Rather than emphasizing AdaFlood’s performance improvements with the potential risk of over-exaggeration, a discussion of how AdaFlood interacts with other regularization methods and guidance on when one should use AdaFlood vs another approach \n\n- The authors should discuss why the flooding baselines they use frequently underperform the unregularized baseline. As prior works have found a benefit to flooding, it is surprising to me that in Table 2 they are often the worst performers. Some more details explaining this discrepancy, and perhaps a finer-grained hyperparameter sweep near the optimum, would help with this concern that the improved performance of AdaFlood is in fact due to poor hyperparameter selection for the other baselines.\n\n\n- How significant is the choice of auxiliary model on performance? Do smaller models of the same basic architecture provide sufficient signal? What about different architectures (e.g. using a VGGNet as an auxiliary model for a ResNet, or a ViT for a ResNet)? Demonstrating that a smaller version of an architecture can provide useful signal would help to improve the potential applicability of the method.\n\n\nMinor: \n\n- With respect to what quantity is the Spearman rank correlation being computed in Section 3.3?\n\n- In eq. 3, L flood is defined for some batch of data but it is not gtiven any arguments. While possible to infer, it would be more precise to include the function arguments in the definition.\n\n- The discussion around equation 6 is somewhat confusing, for example using y_i as both an argument and an index of a function’s output, and falsely claiming that the cross-entropy only considers the probability of the true class y_i — this is only the case for one-hot distributions, and while this is a fair assumption for many datasets it will not hold if e.g. label smoothing or mixup is applied. This point should be clarified.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ilE24mWJvb", "reviewer": "Reviewer_99tH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stregths\n- The motivation is clear: it makes sense to use a different flood level for each instances.\n- The difficulty is obtaining a flood level for $n$ instances since we cannot treat them as $n$ hyper-parameters. The paper's idea to use an auxiliary network for this is a nice solution. It is simple to understand and easy to implement.\n- Various experiments with many different tasks.\n\nWeaknesses\n- There may be many other design choices of how to compute the instance-level flood levels and it is not clear why the paper chose this specific direction. The discussions in Appendix A seem to be important from this perspective and will further motivate the proposed method's approach and I recommend to move this to the main paper.\n- We will need to train neural nets many times for preparing to learn our final neural net. This may be too expensive for very large models, but still practical for many other applications. (Can we fine-tune the final net? See my comments in the next part)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an adaptive flood level for the flood regularizer called AdaFlood. It uses a instance-specific flood level, while the previous Flood and iFlood method used the same flood level for all instances. The paper further proposes to train an auxiliary neural network that can be used for computing the instance-specific flood level. The idea is to use the loss value for an held-out instance, but with some modifications with a hyper-parameter that corrects the predictions. Experiments show the effectiveness of the proposed method. Shows it is helpful for generalization but also for calibration and robustness to label noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Appendix A: the motivation explained here seems to be very important to understand the design choice of the algorithm of the paper. If there is still room, why not include this in the main paper? (it seems the main part only uses 10 pages currently.)\n- In experiments: it would be very informative if the paper can report the chosen flood levels and chosen $\\gamma$ based on validation performance for Flood, iFlood, and AdaFlood. \n- My understanding is that the fine-tuning strategy is only applied to the auxiliary nets, and the final training starts from scratch. I think we often use the same (final) net (same architecture compared with auxiliary net). Therefore, can we finetune the final net based on the trained auxiliary net?\n\nMore minor comments:\n\n- page 4, the double quotation mark is strange in the sentence \"We thus correct the predictions...\"\n- page 6, Section 3.4: the paper mentions that \"$\\mathcal{L}(f^\\star)$ will be approximately the Bayes error\" but I am wondering if this should be the Bayes risk because $\\ell$ is a general non-negative loss function (rather than the 0-1 loss function that is often used for the definition of the Bayes error)? \n- page 6: Using $\\mathcal{L}_{\\text{Adaflood}}$ here is confusing because this notation is already defined with a mini-batch in page 3, equation 3. Should we use a different notation for the full training set version?\n- page 6: comma is missing in the final term ($\\ell(y_i, f^\\star(x_i)$) in the first equation in the proof.\n- page 7, table 1: Unrge -> Unreg.\n- page 9 fig 4: can we also include the legend in Figure4(c) CIFAR10? I am guessing that this one only shows 3 colors (like Figure 4d).", "broader_impact_concerns": "I don't have any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed method is simple, intuitive, and novel. It is an effective regularization method with reasonable computational cost, as shown in experiments. The experiments study several important aspects of the method. The method is compared with previous ones on various tasks and datasets. The paper is clearly written and mostly easy to understand. One weakness mentioned by a reviewer is lack of theory. Providing theoretical understandings of the method or more experiments such as comparisons with other popular methods like dropout would make the paper stronger. In total, the reviewers all recommend accepting this paper, and I agree with them on this decision. However, we do not recommend a certification.

I request the authors to base it on the last version uploaded during the discussion. Please release the code as promised in the paper. Double-check the writing in particular on the newly added parts. Here are a few typos I have noticed: Eq. (9) is missing a comma, and there is a typo in the sentence "The, we randomly initialize a 2-layer MLP with ReLU layers (...)." In the sentence "(...) is about 3 - 4 times longer (...)", "3 - 4" looks like three minus four. Consider removing the surrounding spaces.

Also, I find these sentences difficult to understand: "While NLL can be misleadingly low if the probability density is mostly focused on the correct event time, RMSE is not a good metric if stochastic components of TPPs are ignored and a baseline is directly trained on the ground truth event times. Therefore, we train our TPP models on NLL and use RMSE at test time to ensure that we do not rely too heavily on RMSE scores and account for the stochastic nature of TPPs." I encourage the authors to improve the text.
Decision: Accept (poster)