Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks
Authors: Martin Ferianc
Ondrej Bohdal
Timothy Hospedales
Miguel R. D. Rodrigues
Venue: Accepted by TMLR ()

Abstract: Enhancing the generalisation abilities of neural networks (NNs) through integrating noise such as MixUp or Dropout during training has emerged as a powerful and adaptable technique. Despite the proven efficacy of noise in NN training, there is no consensus regarding which noise sources, types and placements yield maximal benefits in generalisation and confidence calibration. This study thoroughly explores diverse noise modalities to evaluate their impacts on NN's generalisation and calibration under in-distribution or out-of-distribution settings, paired with experiments investigating the metric landscapes of the learnt representations, across a spectrum of NN architectures, tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit cross-task effectiveness in computer vision, emphasising the need to tailor noise to specific domains. Our findings emphasise the efficacy of combining noises and successful hyperparameter transfer within a single domain but the difficulties in transferring the benefits to other domains. Furthermore, the study underscores the complexity of simultaneously optimising for both generalisation and calibration, emphasising the need for practitioners to carefully consider noise combinations and hyperparameter tuning for optimal performance in specific tasks and datasets.

Published in Transactions on Machine Learning Research (04/2024)
Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks
Martin Ferianc*
Martin.ferianc.190ua
Deportment of Electronic and Electrionl Engineering
University College London
Ondrej Bohdal
ondrej. bettre..bohda@@edd.coo
School Informalics
University of Edinburgh
Timothy Hospedales
t.hospedales@ade wed.d uk
School of Informatics
University of Edinburgh
Samsung AT Center Cambridge
Miguel Rodrigues
.rodriiuss@uccl..c. uk.uk
Departmend of Electronic and Electricul Engineering Universily College London
Reviewed on OpenReview:
Abstract
Enhancing the generalisation abilities of neural networks (NNs) through integrating noise such as MixUp or Dropout during training has emerged as a powerful and adaptable tech- nique. Despite the proven efficacy of noise NN training, there is no consensus regarding which noise sources, types and placements yield maximal benefits in generalisation and con- fidence calibration. This study thoroughly explores diverse noise modalities evaluate their impacts NN's generalisation and calibration under in-distribution or out-of-distribution settings, paired with experiments investigating the metric landscapes of the learnt repre- sentations across a spectrum of NN architectures, tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit cross-task effectiveness in computer vision, emphasising the need to tailor noise to specific domains. Our findings emphasise the fieacy of combining noises and successful hyperparameter transfer within single domain but the difficulties in transferring the benefits to other domains. Furthermore, the study underscores the complexity of simultaneously optimising for both generalisation and cali- bration, emphasising the need for practitioners to carefully consider noise combinations and hyperparameter tuning for optimal performance in specific tasks and datasets.
1 Introduction
Neural networks (NNs) have demonstrated remarkable capabilities across various tasks, yet they often grapple with overfitting to training data, resulting in suboptimal generalisation performance unseen samples (Sri- Sietsma & Dow 1991). Addressing this issue, conventional techniques such as weight decay (Krogh & Hertz 1991) and early stopping (Prechelt) 2002 have been employed to regularise NN training. Alongside these methods, the introduction of noise during the NN's training has
vastava et. al. 2014 Bishop
1995
"Joint first authors.

---

Published in Transactions on Machine Learning Research (04/2024)
emerged potent strategy to enhance generalisation (Sietsma & Dow 1991 Neelakantan et al 2017
Camuto 2021 Kukačka et. al.) 2017]. The concept of noise injections refers to the deliberate introduction of artificial perturbations into different aspects of NN training. Note that this is distinct from the concept of noise in the data itself which originates from the data collection process (Song ct al. 2022 Diverging from weight decay and early slopping that. modulate the model's ' search within the hypothesis space; noise injec- tions embrace randomness during training, fostering exploration of a broader array of representations (He ett al. 2019). The appeal of noise injections extends further due to their versatile applicability across diverse tasks, datasets, and NN architectures. These attributes establish noise injections as convenient approach for enhancing NN's generalisation.
In addition to generalisation, confidence calibration is a desirable model property, especially in safety-critical applications where confidence scores must be aligned with the model's accuracy to make informed deci- sions (Guo et al.] 2017j. Empirically, noise injections have been shown to improve confidence calibration by improving the gencralisation of the NNs in previously unscen circumstances and inherently reducing overconfidence in predictions (Guo et al.] 2017 Müller et. al., 2019 Hendrycks al.; 2020 Zhang et. al., 2018 Gal & Ghahramani 2016). However, the relationship between generalisation and calibration is not straightforward, and the two propertics are often at odds with each other (Guo et al. 2017
Various noise injection methodologies have been proposed, encompassing activation techniques such as Dropout (Srivastava et al. 2014 Gal & Ghahramani 2016) and Gaussian Dropout (Kingma et al. 2015
weight noises such as DropConnect (Wan et al.] 2013)) or additive Gaussian noise (Blundell et al. 2015
target methods such as label smoothing (Szegedy et al., 2016), input-target strategies exemplified by
MixUp (Zhang et al., 2 input modifications such as AugMix (Hendrycks et al.] 2020) the standard
horizontal flipping and center cropping (Krizhevsky et al. 2009), model approaches including weight per-
turbation (Ash & Adams 2020), and gradient perturbations involving Gaussian noise (Neelakantan et al. 2017). Despite the diversity of these techniques, comprehensive and fair comparisons are scarce, leaving a gap in understanding which approach is helpful for specific datascts, tasks and models in conjunction with generalisation and calibration.
This study aims to systematically and comprehensively investigate the effects of widely used noise injection methods on NN generalisation and calibration across multiple datasets, tasks, and architectures. This explo- ration is predicated on the premise that while generalisation focuses on reducing overfitting and improving the model's predictive accuracy, calibration deals with aligning the model's confidence with its actual perfor- mance. Rather than focusing on improving state-of-thee-at performance, we aim to provide holistic view of the effects of noise injections on NNs' generalisation and calibration for the benefit practitioners. To this end, we present the following contributions:
1. The first systematic empirical investigation into the impact of noise injections on NN generalisation and calibration across diverse datasets, tasks and NN architectures. Our exploration extends to evaluation under in-distribution (ID) and out-of-distribution (OOD) scenarios and their transferability across archi- lectures and datasets.
2. A methodological framework for simultancously combining various noise injection approaches.
3. Visualisation of the learnt representation landscape across noises, jointly comparing calibration and gen- eralisation performance.
Our investigation reveals that certain types of noise aid in generalisation by introducing robustness against overfitting and variability in data and potentially improve calibration by mitigating overconfidence in predic- tions. The findings show that AugMix, weak augmentation and Dropout prove effective across diverse tasks, emphasising their versatility. Task-specific nuances in noise effectiveness, such as AugMix's superiorily in computer vision (CV), Dropout in natural language processing (NLP) and Gaussian noise in tabular data regression, highlight the need for tailored approaches. Combining noiscs, careful hyperparamcter tuning, and task-specific considerations are crucial for optimising NN's performance. Our code publicly available at htteae///gstaaa.ccc///rtttttttrr
2

---

Published in Transactions on Machine Learning Research (04/2024)
2 Related Work
In this study, we consider artíficial addition of noise into various facets of NN training including input, target, input-target, activations, weights, gradients, and model parameters. The noise application is denoted by ('<plane>(::>)) (::6), where Oxeplace>> is the noise application methodology which can be executed at different places, e.g. Cinput for input noise, Otarget for target noise along with arbitrary arguments, depending on the noise injection methodology. For example, under this definition, additive input Gaussian noise samples a Gaussian and adds it to the input x as Cimput (x, 8) = x + eje ~ [NO0, where o² is the hyperparameter in 8. Note that this study focuses on artificial noise injections, which are purposely introduced during training, and not on natural noise, inherent in the data, e.g. label noise in the targets where the classifying label is incorrect. The natural noise needs to be addressed separately and we refer the reader to Song et al. (2022) for review of stralegies for learning with noisy labels. Under different noise placements, we review several noise injection strategies. The review focused on the most fundamental noise injection methodologies, which constitute the building blocks of more complex approaches and represent the noise injection category.
Pioncering work by Sictsma & Dow (1991) demonstrated the benefits of training with added input Gaussian noise, while [Bihhop| (1995) established its linkage to regularisation in the least. squares problems. In CV, weak augmentation, such as random cropping and horizontal flipping, has improved gen- cralisation (Krizhevsky et al. 2009). AugMix, domain-specific to CV, applics a sequence of image processing operations to the input, bolstering robustness in OOD settings. From the adversarial robustness domain, ODS augments inputs conditioned on the prediction, aiming to diversify the inputs TTashiro et al.) 2020). Target Noise: Label smoothing (Pereyra et al.) 2017) softens the one-hot classification targets by replac- ing the targets with a categorical distribution with the most mass on the correct class and the rest spread across the other classes through the addition of constant uniform noise, effectively improving NN's robust- ness (Müller et al. 2019). This differs from label noise, where the entire probability mass is an incorrect class, and the NN must learn to ignore the errors (Song et al. 2022). Input-Target Noise: Variants of MixUp have exhibited efficacy in augmenting both generalisation and calibration (Zhang et al. 2018 Müller
Input Noisc:
et al. 2019: Guo et al. 2019 Yao et al. 2022 Guo et al. 2017). MixUp adds noise to both the input and the target via linear interpolation between two samples and their targets, while CMixUp expands this approach to regression problems. Activation Noise: Widespread activation noise includes Dropout or Gaussian noise injections. Dropout (Srivastava et al. 2014 Noh et al., 2017) randomly deactivates activations through 0-1 noise, while Gaussian noise injections add noise to activations (Kingma et al. 2015: DeVries & Taylor 2017). Bayesian NNs (Gal & Ghahramani 2016) incorporate these injections during training and evaluation, in contrast to our work's focus solely on their application in training. Weight Noise: Unlike Dropout, Drop- Connect (Wan et al. 2013) randomly deactivates weights or connections between neurons, while Gaussian noise injections add noise to weights (Blundll et al.) 2013). Note that we do not model the variance of the Gaussian noise through learnable parameters, as in (Blundell et al. 2015),, but rather fix it through a searchable hyperparameter. We do this ensure a fair comparison with other noise injection approaches, such as Dropout, which do not have learnable parameters and would require changing the model architecture to accommodate them. Gradient Noise: Annealed Gaussian noise added to gradients during training has demonstrated its efficacy in improving NN generalisation Neelakantan et al. (2017:: WWlling & Teh (001)
Zhou et al.] (2119); Chaaddaaii & Soatto (2015); Wu et al. (2020). Model Noise: A recent contribution, Gaussian noise injection through periodic weight shrinking and perturbation Ash & Adams 2020). improves retraining generalisation.
In previous work, the impact of noise per injection type was studied. Poole et al. (2014) show that inject- ing noise at different layers of autoencoders implements various regularisation techniques and can improve feature learning and classification performance. Cohen et al. (2019) show that smoothing classifiers with Gaussian noise naturally induces robustness in the L2 norm. Wei et al. (2020) discntangle and analyti- cally characterise the explicit regularisation effect from modifying the expected training objective and the implicit regularisation effect from the stochasticity of Dropout noise in NNs. [Camuto| (202I): Camuto et al. (2020) show that training NNs with Gaussian noise injections on inputs and activations regularises them to learn lower frequency functions, improves generalisation and calibration on unseen data but also confers robustness to perturbation. On one hand, Jang et al. (2021) show that training NNs on noisy images can
3

---

Published in Transactions on Machine Learning Research (04/2024)
improve their robustness and match human behavioural and neural responses. On the other hand, Geirhos et al. (2018) demonstrate that adding specific noise to the input can surpass humans in generalisation on that specific noise, but not to other types of noise, while human vision is robust to a wide range of noise types. The results of Geirhos et al. (2018) are confirmed by the results of Kang et al. (2019), who show that robustness against one type of noise does not. necessarily transfer to robustness against other types of noise. Furthermore, Kang et al. (019)) consider adversarial training, where model is trained to be robust against noise-based adversarial attacks [Goodfellow et al. 2015). An adversarial attack is specific type of noise injection during evaluation, where the noise is designed to fool the model. In comparison, our work focuses on the generalisation and confidence calibration performance of NNs with respect to domain shift in the data distribution, rather than adversarial attacks. We consider enhancing robustness to adversarial attacks through artificial noise injections as future work. Moreover, (Kukačka et al. 2017 provided a laxonomy of regularisation in NNs, covering multiple noise-based approaches.
The closest work to ours is (Chun et al., 2020), which considered regularisation commonly used during training and its impact on generalisation, confidence calibration and cal-ooofdisrrbbuut ion detection in computer vision. While their focus was not noise-specific, as in our work, they overlap with our work by considering input noise: weak augmentation and Gaussian noise, target noise: label smoothing (Müller et al. 2019
input-target noise: MixUp (Zhang et al., 2018 They show that common regularisation techniques prove generalisation, confidence calibration and ont-of-distribution detection. In comparison to (Chun et al. 2020
our work focuses on a broader set of noise injections, network architectures, datasets and tasks, evaluation of the weight landscapes, and in-depth noise combinations paired with comprehensive hyperparameter search.
Past work has studied noise injection techniques in isolation, mainly focused on generalisation alone, lacked comprehensive hyperparameter optimisation, and rarely evaluated the robustness of distribution shift. For example, only MixUp, AugMix and label smoothing have been studied in calibration (Guo et al.) 2017: Müller et al. 2019 Guo et al., 2019; Yao et al. 2022 Chun et al. 2020). An exception to this is Chun et al. (2020), who studied generalisation, calibration and out-of-distribution detection for some noise injections. While promising, these methods require further unified analysis to determine their relationships across datasets, tasks, architectures and across a broader set of noise injections. Our work addresses these gaps by 1.) studying the impact across datasets, tasks and architectures; 2.) benchmarking the impact of noise injections' hyperparameters
transferability between datasets and architectures; 3.) studying confidence- calibration in addition to generalisation; 4.) performing a comprehensive hyperparameter search with fair comparisons; 5.) evaluating robustness to distribution shift; 6.) providing a methodological framework for combining and tuning various noise injection approaches across categories; and lastly 7.) visualising the learnt representation or learning landscape across noise injections in 1D or 2D (Goodfellow et al. 2014 ct al. 2018 across both generalisation and calibration.
3 Methodology
We establish a structured methodology to investigate noise injections' effects on NNs. The noise types are divided into input, input-target, target, activation, weight, gradient and model, and we enable their conditional deployment through probabilities ( number of noises.
18 in the range $ Phoise < 1, where S denotes the
The training allows simultaneous consideration of S noise types, each associated with specific hyperpa- rameters {5'}__ and an application function (a'placee>(; (,))) where of d'splace> is the noise application methodology which can be executed at different places, e.g. Oiuput for input noise, Otarget for target noise along with- arbitrary arguments, depending on the noise injection methodology. The different noise types implement only the relevant d'<paace> () function, while others are ignored. We encourage the reader to refer to the code for the implementation details for each noisc type. The probabilities 15 allow us to tune the frequeney of applying each noise lype, while the hyperparameters {8+);2 enable us to adjust the magnitude of each noise type. This enables us to tune both the magnitude and frequency of noise injections, unlike, for examplc, Dropout (Srivastava et al., 2014), which only allows the tuning of the magnitude, and it is applied every batch. The tuning of the frequency allows us avoid conflicts between noises, as it can be set to O if the noise is conflicting with other noises.
4

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm I provides comprehensive overview of the training process, executed throughout E epochs with I. batches processed per epoch. For every batch, input and larget data (15, are randomly drawn from the training dataset D = {( 36)) For each noise in S, we sample uniform random variable ~u (O0,,) 1), and if c < Phoises we enable noise i with hyperparamcters 8' for the current batch b.
For each noise in S, we sample a uni- form random variable e ~ U(0, 1), and if < Proise we enable noise through setting the toggle t' to 1 for the current batch b. The enabled noises are applied in the order: 1.) input, larget, input- target, 2.) weights, 3.) activations, 4.) gradients and 5.) model through the apply p (-,t,)) procedure. The procedure sequentially iterates over the noise types in S and applies the noise if the noise is enabled and the applica- tion function a d<<place> exists. The user specifies the order of the noises in S.
Algorithm 1 Training of a Neural Network with Noisc
Require:
Training dataset D {<2, 36) 35) batches, number of epochs E, network depth D, weights W (Wd) hid- den states 2 [diJd=1> {=
activations & ()) weighted
S noise types, probabilities of applying noise to a batch Proise = (Pioiscc Noise hyperpa-
operations f f {f W (-
rameters 5 = {60) Noise application functions axelace>> =
{a (a'<plare> (; 6))
1: Initialise W randomly
2: for e = 1 to E do
3: for b = 1 to L do
4: Randomly select a batch (I+, Ub) from DD
5: Sample e = {e¹ ~ (0,1)))
Our approach accounts for networks of depth D, denoted by LF),,
6:
Set toggles t= (t ====c" < &P 15
i==
7:
t, Target noise: apply Otarget (Vo. 1,
Input noise: apply_
involving weights together with bi- ases W = [W and activations { to produce hidden states {=g zf corresponds to the input
8:
9:
Input-target noise: apply Cinputt-targe (2b, yb t)
10:
of
= 2b
11:
for d= to D do
na, while of represents the output pro-
12:
Weight noisc: apply Oweight (Wd t,, 5)
Compute hidden state z = (2 d-1 ,Wd)
diction fb-
13:
14:
Activation noise: 2 = +=(((())
cd 0, if
For input noise, we explore Aug- Mix, ODS, weak augmentation: ran-
15:
end for
dom cropping and horizontal flipping,
Assign predictions On = =
and additive Gaussian noise injec-
18:
Compute loss L(O', ,y) and gradients WWC
tions (Hendrycks et al.] 22020: Tashiro
19:
Gradient noise: apply_ -gradiet (V Ww t,
et al. 1020||Sietmaa & Dow 1991). For
20:
Update weights W
input-target we explore MixUp and
21:
end for
CMixUp (Zhang et al. 2018 Yao al. 2022 For target noise, consider label smoothing, and the target noise also inherently involves MixUp and
22: Model noise: apply Omodel (W,
23: end for
Procedure: 25: for i = to S do
Axcpace_> (,
CMixUp (Zhang et al. 2018 Yao et al. 2022 Müller et al. 2019). The activa-
26: if l' and a d'<place> exists then
27: a axplace> (+8*)
tion noise examines Dropout and ad- ditive Gaussian noise (Srivastava et al.] 2014 Kingma et al. 2015) prior ac- tivations for all linear or convolutional
28: end if
29: end for
layers, except the last layer. For weight noise we consider Gaussian noise added to the weights Blundell et al. 2015) or DropConnect (Wan et al. 2013) for all linear or convolutional layers, except the last layer. We consider gradient Gaussian noise added to all gradients of the loss function (Neelakantan et al. 201 After the update of the weights, the model noise is applied to the weights, for which we consider shrink- ing the weights and adding Gaussian noise (Ash & Adams 2020), but not in the last 25% of the training epochs. Out of these noises, label smoothing, MixUp and ODS are exclusive to classification, and CMixUp is applicable only in regrossion. AugMix and weak augmentation are exclusive to the CV data. The other noises are broadly applicable across tasks.
5

---

Published in Transactions on Machine Learning Research (04/2024)
4 Experiments
Next, in Section 4.1 we present the concrete datasets, tasks and architectures used in our experiments,
followed by experiments on ID data in Section 4.2 OOD data in Section [.3] combined noises in Section 1.
transferability in Section 4.5 and lastly the metric landscape visualisations in Section 4.6
4.1 Experimental Settings
Tasks, Architectures and Datascts: We consider various setups, including computer vision (CV) clas- sification and regression, tabular data classification and regression, and natural language processing (NLP) classification. For CV classification we include datasets such as CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009), SVHN (Netzer et al. 2011), and TinyImageNet (Le & Yang, 2015), along with neural archilectures such as a fully-connected (FC) net and ResNet (He et al.) 2016). For CV regression, we introduce a rotated version of C-FAR-100 to predict the rotation angle, and we also use the WikiFace dataset (Rothe et al. 2015). where the aim is to predict the age based on the image of the face. We use the ResNet model in both cases. We deem the rotation prediction task compelling to evaluate since it is a common task in the literature for self-supervised pre-training (Gidaris et al. 2018). In the realm of tabular data classification and regression, we use an FC network and evaluate noises on diverse datasets, including Wine, Toxicity, Abalone, Students, Adult for classification and Concrete, Energy, Boston, Wine, Yacht for regression (Asuncion & Newman 2007). We explore NLP classification using the NewsGroup and SST-2 datasets (Lang) 1995 Socher et al.
2013) paired with global pooling convolutional NN (Kim) 2014) and a transformer (Vaswani et al. 2017
The Appendix details the datasets and architectures and gives the complete numerical results.
Metrics: To assess the effectiveness of the noise injection methods in classification, we measure their performance using three metrics: Error (J, %), Expected Calibration Error (ECE) (Guo et al. 2017 (4. %) with 10 bins and the categorical Negative Log-Likelhhood (NLL) (L). For regression, we use the Mean Squared Error (MSF) (L) and the Gaussian NLL (4). We test. the generalisation of the models by evaluating their performance on the ID test set. For CV classification and regression, we test the robustness of the models by assessing their performance on an OOD test set by applying corruptions (Hendrycks & Dietterich, (019)) to the ID test set. These corruptions include, for example, adding snow or fog to the image, changing the brightness saturation of the image or blurring the image across 5 intensitics. We created the OOD test set for tabular data by adding multiplying the inputs with Gaussian or Uniform noise or by zeroing some of the input features with Bernoulli noise, similarly across 5 intensities. While vision data has ImageNet- C (Hendrycks & Dietterich, 2019), to the best of our knowledge, there is no similar benchmark for tabular data. Our methodology for introducing perturbations and zeroing out features is designed to simulate a wide range of potential distribution shifts in real-world scenarios, such as instrumentation errors, missing data, and adversarial attacks. We crafted the OOD evaluation be similar to (Hendrycks & Dietterich] 2019). in terms of the magnitude and severity of the noisc, allowing to systematically evaluate the robustness of the models. To summarise the results, we collect the results for each approach for each dataset and metric and rank them relative to the no noise baseline. For example, -1 means that the approach is one rank better than the no noise baseline, and 1 means that the approach is one rank worse than the no noise baseline. We then average the ranks across the datasets for each task and metric.
Hyperparameter Optimisation: We first tune the learning rate and L.2 regularisation of a no noise network, which are reused when tuning the HPs of each noise injection method. By tuning the learning rate and L2 regularisation, we wanted to simulate realistic scenario where the practitioner seeks to add noise to their existing model and does not want to jointly tune the model's hyperparameters and the noise injection method. The tuning was performed with sced, and the winning hyperparameters were retrained 3 times with different seeds. 10% of the training data was used as the validation set to select the best model, with validation NLL used as the selection objective to combine both generalisation and calibration. The tuning is performed using model-based Treo-structured Parzen Estimator method (Bergstra et al., 2011 with successive halving pruning strategy (Jamieson & Talwalkar 2016). We evaluate 50 trials for each setting, which allows us to manage the trade-off between compute costs and reasonable number of trials.
6

---

Published in Transactions on Machine Learning Research (04/2024)
Emer
8.
0.5
02
15
2.0
2.2
15
65
3.0
30
1.2
422
11.
132
23.2
150
1.0
15
0.
32
22
58
2.4
4.0
N5C
1.5
0.5
0.5
3.5
2.5
4.0
9.5
10.0
5.0
3.0
1.0
-55
15
12
0.5
255
2.0
12
42
2.8
-50
02
11. 132 4.5
MLL
2.0
1.6
-3.5
3.8
1.0
-1.0
5.0
2.5
1.0
1.5
1.0
-55
35
1.5
Hoput Wear Gaurtaan
lepat CoS
lopu Auplle
hepph Bhslpp
unaet Setroriing
Actiratinn Clarsiaay
Actosston Degghigg
Gredient Gaorming
Weeger Glsstau
Modil
sengii Geopei
Kan Bireet
Rpo Divect
tap optecieed
Po- oydinited
lopal Aug
toput Cauasian
Inpot Aurltte
rmpe Rerget Chissp
Activaaion Caursiirn
Acdio atter Broghnst
Creditet Cloctiin
thappe Glocetiin
Mide
Weigint Groprrandee
Kigul Direet
app Direct
top Optiviedd
Top oydnoled
(a) CV classification.
(b) CV regression.
500
2.6
2.3
12
3.5
22
033
3.0
200
0.4
ECE
04
2.5
5.2
4.5
0.3
1.5
24
4.2
1.
7.8
10
0.2
1.5
0
MSC
3.4
12
-30
1.0
0.0
22
224
12
12
2.0
0.4
36
4.0
.50
5.2
4.4
25
6.2
02
7.4
44
44
-1.4
-12
-20
-38
-16
-10
5.0
3.
-18
-16
Inpot Gouttaan
Iipud Right Hartp
Labed Smonttinng
Gradeet cearptan
Heiget Gaayslan
Model
Wesgie eeeee
Ryp tiredt
typ Direst
upuz Optontied
Opteniied
Upgel Catostao
Doy Regget Ohisup
Actistton Actoidion Causstan
Droponst
Grasiant Divussann
Midee
Weight Giusstate
Waggle DroiCaeeedd
Rygo Diseet
Repi theet
Kop Detroited
Topo Optnntsed
Actiotton Actrestion Giarr
Tabular classification.
(d) Tabular regression.
Emr
0.2
3.5
02
3.1
2.6
1.5
311
3.6
5.7
455
5.2
CCE
02
5.2
25
0.2
4.
5.0
220
1.0
4.5
4.0
45
3.0
4.4
1.
3.5
5.0
Upputt OEE
Eipul lagett Mteti
Label Smoatinn
Activtton GRoczASo
Anthatun Disgnnal
Grvilesn Geusaaao
Motel
weight DipprrDnn
Fap-2 Dieed
Topul bindd
Top betintrol
Ta Optomaee
Welget
NLP classification.
Figure 1: In-domain evaluation of the differences in rankings compared to not using any noise,
4.2 In-Domain Evaluation
In Figure T we show the in-domain (ID) performance of NNs trained with various noise injection mel hods across CV classification and regression, tabular data classification and regression, and NLP classification. Overall, we observe that the noise injection methods significantly improve the generalisation and calibration in many cases, but different noise types are needed for various tasks. In CV classification, almost all noises improve the error rate, with many simultancously improving calibration. The most beneficial noises are AugMix, weak augmentation and Dropout. MixUp and label smoothing are surprisc to a certain extent they improved generalisation but not calibration. In CV regression, improving generalisation was challenging, with no improvement. However, several noises have improved NLL, with AugMix, weak augmentation, and Dropout achieving the best balance. These results suggest that image augmentation broadly benefits CV, confirming expectations.
Several noises have improved the error rate to a lesser extent or kept it at similar level in tabular data clas- sification. In contrast, almost all noises have improved ECE and NLL. The improvements were particularly impactful in several cases, with model noise, label smoothing, and Dropout being the best. While ODS is designed to improve adversarial robustness, it improved ECF and NLL while slightly improving error rates. All noises improve NLL for tabular regression, and some significantly improve MSE. Gaussian noises applied to the weights, activations, or inputs are the most useful types of noise for improving the two metrics. In NLP classification, about half of the noises improve error, with some also improving calibration simultane- ously. The best noises are Dropout, label smoothing and ODS, which differs from what was the best for CV. These noises significantly lowered error and NLL, while MixUp and model noise were particularly useful for reducing ECE. ODS was beneficial for improving error and calibration via NLL, which can be surprise this technique was not previously considered for improving generalisation or calibration.
In Figure 2 we show detailed results for selecting representative datasets across the 5 lasks. We see the improvements in error can be large for CIFAR-10, for example, halving it in some of the best cases weak augmentation and AugMix, with Dropout also leading a few percentage point improvements. The situation is similar for ECE, where weak augmentation and AugMix make the ECE one-half or one-third. Many errors are slightly better, but certain noises, such as MixUp, label smoothing, or Gaussian noise added
7

---

Published in Transactions on Machine Learning Research (04/2024)
16 10
0.10
0.08
20
ECE (96)
0.06
Error
ONA 8
0.04
0.02
0.00
CIFAR-10 CV classification.
(b) WikiFace CV regression.
14
3.5
0.5
0.25
3.0
50
2.5
0.4
-0.75
2.0
MSE
0.3
2 -1.00
1.5
0.2
-1.50-
1.0
0.5
0.1
-2.00
-1.75
0.0
(c) Adult tabular data classification.
(d) Yacht tabular data regression.
40
25
30
intor (%)
25
15-
20
ECE
10
(e) NewsGroup NLP classification.
Aug.
Imput Goussian
Input Aughix
Input Target
Smoothing
Gaussian
Dropout
Gradient Gaussian
Model
Weight Gaussian
Weight Drepponnect
Top-2 Direet
Top-3 Direct
Top-2 Optinised
Top-3 Optimised
Figure 2: Detailed in-domain performance of NNs trained with various noises across the five tasks.
to the activations, worsen the calibration. For WikiFace, there are more minor improvements in error from weak augmentation and AugMix with overall similar MSE across different noises. Still, the differences calibration as measured using NLL can be considerable, with most noises improving the NLL significantly.
Moving the focus to tabular data, most noises applied to the Adult classification dataset improve the error marginally. In contrast, many improve ECE significantly, with the best ones being Dropout, model noise and DropConnect. Most noises have significantly improved MSE for the Yacht regression dataset, but CMixUp and model noise led to significant increases. The best ones have been gradient Gaussian and Gaussian noise added to the weights. NLL has been improved in several cases, including gradient Gaussian and weight Gaussian, demonstrating solid improvements classification on NewsGroup using the global pooling CNN model. ODS leads to the best improvement, while several noises, specifically Dropout, gradient Gaussian, and model noise, lead worse generalisation. ODS and label smoothing have also noticeably improved ECE.
MSE and NLL. The errors stay similar for NLP
Main Observations: The noises are effective across various tasks and datasets. The shortlist. of the most effective methods is AugMix and weak augmentation in CV, model noise, and Gaussian noise added to weights for tabular data and dropout in NLP. Different task types benefit from different types of noise.
4.3 Out-of- ofDmnn Evaluation
We evaluate the performance the ID test set and augmented OOD set, including average over visual corruptions across 19 categorics and 5 severities (Hendrycks & Dictterich 2019). Likewise, we average the performance across 5 calegories and 5 severities for tabular data. The summary of the results is in Figure g with analysis of correlations between ID and OOD rankings via Kendall Tau score in Table [ For CV classification, we observe that the generalisation improvements also remain for OOD, but improving calibration in terms of ECE turns out to be much more challenging. The overall ranking of the best noises remains similar, with Aux and weak augmentation remaining the best. MixUp prominence thanks
8

---

Published in Transactions on Machine Learning Research (04/2024)
Emer
12
12
20
35
34
15
65
2.5
30
0.6
2.8
12.
11.0
12.2 152
13
23
1.5
1.0
12
02
20
0.
N5C
1.0
13
1.5
2.5
0.3
7.0
15
5.
6.5
5.5
1.0
15
225
85
3.0
02
4.
20
10.8
4.0
53.2-10 122-66
MLL
4.0
1.6
6.0
-3.5
-30
7.0
-..
4.5
4.5
-1.
12.
12.0
15.5
63
Hogut Wear
Cawrtaan
lepa cog
topu Auplle
hepph Bhsllp
unaet Aetrortiig
Actioatin Clarsiiay
Actosston Degghig
Gredient Gaormiig
Modil
Hengii Geopei
Tan Brreet
Rpo Direct
Tap optecieed
Pa- Oydinited
lopal Aug
toput Clussian
Input Augltta
rmpo Rerget Chicspp
Activasioss Cauysiire
Accio attirin Broghnst
Creditet Clock
thappe Glscentin
Midel
Weigint Croperanne
Kiguc Dieeet
app Direet
top Optivied
Top oydnoled
Weiger
(a) CV classification.
(b) CV regression.
3.0
2.6
0.4
4.6
3.5
2.4
1.8
04
0.2
2.6
ECE
0.5
2.2
5.1
4.8
1.2
1.2
14
7.0
20
3.8
0.2
20.4
4.5
1.8
0
MSC
2.4
3A
-1.
-1.
3.0
2.2
-22
-12
1.
18
-1
20
3.4
5.
5.2
5.8
15
5.5
3.0
1.
39
42
18
-3.8
-1.6
3.0
-0
1.8
04
6.2
0.
23
0.4
0.4
Inpat Gouttaan
tapet COO5
Irpud Reget Hartp
Labed Smontting
Actiation trrpeese
Gradend charstan
Hegget Gaurtlan
Model
Wesgie deepe
Rip tieett
top Direct
wau Optondied
topo Opteriieed
logel Catostao
Doy Seggel ChieUp
Actuattnn Actondion Causstan
Graslent Covassane
Droponst
Middo
Beight Glusssan
Weggle DrgiCameed
Rego Disect
Repi theet
kop Detroited
Topo Optionted
Actresiion Glarr
Tabular classification.
(d) Tabular regression.
Figure 3: OOD evaluation of the differences in rankings compared to not using any noise.
Metrie
SWHN
CIFAR-10
CIFAR-100
TinyImageNNet
Average
Metric Rotated CIFAR-100
WikiFace
Average
Esuron
0.912.c..
0.70Gi
0
0.912
0.807
MSE
2.257
2
0.419
ECE
NLL
0.971. ow
1.000.cm
0.510gcom coo
0.618.cm
0.868 sew
0.8822 aeo
0:496
0.809
NLT
-0 -0.238
0.8102 one
0.286
(b) CV regression.
(a) CV classification.
Mutric
Wine
Tosicity
Abalone
Students
Adult
Averrage
EuRoR
0.861
0.529 s
0.532
0.9244 ve
0.718
Metric
Energy
Boston
Wine
Yachi
Concrste
Average
RCB: 0.505 sow
0.906_
0.867 =
0.867 wwn
0.9622
0.901
MSE
0. 0.667,
0.923,
-0 1..56a xen
0.84
0,872 vew
0.508
NLT
0.9261
0.9812co
0.790
0.810
0.9022
0.593
NLT
-0 00.115
0.974,
0.164_
0.590, 0-4301
0.308
(c) Tabular data classification.
(d) Tabular data regression.
Table 1: Kendall Tau correlation between ID and OOD rankings of different noise types for various tasks.
to the bost OOD calibration and improved errors and NLL. Analysis of Kendall Tau correlation in Table la shows that ID and OOD rankings are strongly correlated for error and NLL, while only moderately for ECE, CV regression is similar to classification ranking the best noises, with only AugMix leading to improvements in OOD generalisation. However, calibration is improved by most noises, with AugMix excelling. Only a minor correlation exists between ID and OOD rankings for MSE and NLL metrics. For tabular classification, the noises generally improve ECE and NLL but not the error rale under OOD settings, with model noise, label smoothing, and Dropout being the best. This suggests all of these are among the best noises for both ID and OOD. ID and OOD rankings show strong correlation overall. Several noises improve OOD generalisation and calibration for tabular regression, with DropConnect, Dropout and Gaussian noise added to the inputs, leading to the best overall improvements. The TD) and OOD ranking Kendall Tau correlation is low in this case. MixUp and CMixUp have improved OOD calibration for both tabular classification and regression.
We study selected representative datasets regarding OOD performance in Figure 4 OOD results on CIFAR- 10 show that AugMix significantly improves both error and ECE, making ECE one-third of the no noise equivalent. MixUp leads to similarly considerable improvements in ECE and more minor yet significant improvements in error. Several noises, e.g., Dropout and Gaussian noise, added to activations or weights lead to a few percentages worse ECE. On WikiFace, most OOD MSE values are similar, but OOD calibration in NLL is improved significantly for several noises, including AugMix, weak augneetattin or Dropout. Improvements in generalisation for the Adult tabular classification dataset are minor, but the improvements in calibration can be significant, for example, Dropout and model noise halving the OOD ECE value. For the Yacht tabular regression dataset, the improvements in generalisation have been more critical, with the same being true for calibration measured in terms of OOD NL...
9

---

Published in Transactions on Machine Learning Research (04/2024)
35
20-
30
30
0.08
25-
25
ECE (9)
15
20
Erro
20
MS
0.06
15
15
0.04
10
0.00
(a. CIFAR-10 CV classification.
(b) WikiFace CV regression.
16-
4.0-
3.5
0.5
Error (%)
1.0
3.0
M5E
0.3
0.4-
2.0
1.5
0.2
0.1
0.0
(c) Adult tabular data classification.
(d) Yacht tabular data regression.
No Noiso
Input Weak Aug.
Imput Goussian
Input ODS
Input Muglix
Taput-arget MicUp
Label Smoothing
Activation Gausstan
Activation Dropout
Gradient Claussion
Model
Weight Gaussian
Weight DropConnect
Top-2
Top-3 Direct
Tap-2 Optimised
Top-S Optimised
Figure 4: Detailed OOD performance of NNs trained with various noises across the four tasks.
We include additional OOD invesligation TinyImageNet, where we study the performance on black and white sketches. We use the same classes as in FinyImageNet (Wang et al. 2019) and report the full details and results in the Appendix. The shift from natural images to sketches represents a more significant domain shift than our standard OOD shifts (Hendrycks & Dietterich] 2019). and hence also tests the generalisation of the noises to a larger degree. We do this to disambiguate the performance of AugMix, which uses augmentations that may be similar, but are still distinct (Heudrycks et al. 20020), to the ones used for our OOD evaluation. The results clearly show that AugMix obtains similar rankings for all three metries on both the synthetic OOD and sketch domain evaluations. Overall we see strong Kendall Tau correlation in terms of the error across all noises, but smaller in terms of ECE and NLL.
Main Observations: We see consistent improvements in OOD generalisation and calibration for tabular data. Errors and NLL are improved for CV classification, but calibration is generally not improved when mcasured via ECE. CV regression usually sees improvements in OOD NLL only. The best ID noise types have often remained the best. OOD, but overall, the correlations between ID and OOD rankings were not high in all cases. MixUp, CMixUp for regression, showed surprising behaviour as it was much more helpful for improving OOD calibration than ID calibration.
4.4 Combination of Noises
Next, we evaluate the combination of noises. We construct them from empirical combinations of the Top 2 or 3 noises from the ID evaluation for each task, based average rank across respective datasets and metrics. We consider two cases: 1.) the found hyperparameters of the noises are directly applied, and 2.) the hyperparameters of the noises are jointly tuned. We utilise the same 50-trial budget to tune the selected noises jointly. We restricted our experiments to combinations of two or three noises, as we empirically found that tuning all the noises jointly in our study is not feasible even with a larger computational budget. We consider the individual noise hyperparamcter tuning and then the direct application or joint tuning of their combination to be realistic scenario where practitioner has limited compute budget and wants to improve their model's performance (Godbole et al. 2023) iteratively.
The results are alrcady in Figures I 2 and 4 and denoted as Top-2 Direct, Top-3 Direct for 1.), Top- 2 Optimised and Top-3 Optimised for 2.J. The combinations for Top-2 and Top-3 are in Table 2 To simplify the analysis of how effective the different combinations of noises are, we compute their average rank improvement compared to no noise and report in Table 3 Notice that when we choose a combination of noises involve noises from the same category, for example, ODS and input Gaussian are both input noises, these are applied sequentially.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Task
Top-2
Third Method
CV classification
Input AugMix, Input Weak Augmentation Activation Dropout, Target Label Smoothing Model, Target Label Smoothing Input AugMix, Imput Weak Augmentation
Activation Dropout
NLP classification
Iuput ODS
Tabular classification
Activation Dropout Activation Dropout
CV regression
Tabular regression
Weight Gaussian, Activation Gaussian
Input Gaussian
Table 2:
Top task and noise combinations.
Underlined methods are from the same type.
Scenario
Top-2 Direct
Top-3 Direct
Top-2 Optimised
Top-3 Optimised
ID
-4.75
-3.69
-4.34
-3.30
OOD
-5,24
-4.43
-5.00
-2.59
Table 3: Average rank improvement over no noise for the different combination strategies.
We can draw several observations from Table 3 1.) Directly combining hyperparameters for the top two
three noises is a good strategy when considering the same budget for hyperparameter tuning as for one noise. A significantly larger budget likely needed for jointly optimising hyperparameters of multiple noises. 2.) A combination of two noises performs better than a combination of three noises, suggesting there may be negative interactions when too many noise sources are used without extensive hyperparameter tuning. 3.) The behaviour of different combination strategies consistent across ID and OOD settings.
Commenting on the overall performance of the combinations of noises, the combinations are typically better for classification tasks than the individual noises. Still, the opposite may be truc for regression. As observed in Figures la Ic and Tel the combinations are consistently rauked lowest in comparison to using no noise for classification, showing the effectiveness of the combinations. However, Figures [b and Id show that regression can benefit from only using one noise at time. OOD analysis in Figure confirms the benefits of combinations of noises for classification tasks, and also shows that it can be beneficial for regression, contrary to the TD behaviour. The combinations
generally ranked lower and improve calibration and generalisation, seen in lower MSE, NLL, or error and ECE simultaneously.
The combination of noises can improve both calibration and generalisation simulta- neously. Directly combining two noises is better than three, as too many can lead to conflicts. Combining noises directly with their hyperparameters is generally reasonable and a significantly larger tuning budget for optimising hyperparameters would be needed for optimising their hyperparameters jointly.
Main Observations:
4.5 Transferability of Hyperparameters Across Datasets and Models
Furthermore, we evaluate the transferability of the hyperparameters across datasets and models. We consider two cases: the transfer of hyperparameters to new dataset. and the transfer of hyperparameters to a new architecture. For the dataset transfer, we consider the following combinations: SVHN to CIFAR-10, CIFAR-10 to CIFAR-100, CIFAR-100 to TinylmageNet, and 3 tabular regression datasets combinations, Concrete to Energy, Boston to Wine, Yacht to Concrete. We consider the following combinations for the architecture transfer: FC to ResNet-18 for SVHN and ResNet-18 to ResNet-33 for CIFAR-10, CIFAR-100 and Imay We use NN with an additional layer for tabular data, i.e., five layers instead of four.
4.5.1 Dataset Transfer
Figures and 5c show the dataset transfer results for ID settings, with OOD settings shown in Figures 6a
and Gc We observe generally good transferability of hyperparameters across datasots for CV classification in ID and OOD settings. In particular, weak augmentation, AugMix and Dropout lead to solid improvements in the ID setting. AugMix also excels in OOD scenarios under dataset transfer, but weak augmentation and Dropout are not as strong calibration measured using ECE. Certain noises are less transfcrable, including Gaussian noise added to the input and DropConnect. Hyperparameters for noise in tabular regression are less transferable because of worse generalisation measured using MSF.
11

---

Published in Transactions on Machine Learning Research (04/2024)
10
Error
7.3
1.0
-10
6.3
-13
1.
2.0
4.7
4.3
1.7
43
1.7
10
Errer
-6.
-1.0
6.0
0.7
-.77
1.3
4.00
4.0
-.7
0.7
2.7
ECE
-77
23
37
0.3
0
ECE
-3.0
-2.7
-3.7
-17
00
2.7
0.0
1.7
-30
-3.0
0.0
0
-777 -
0.3
-1.3
7.7
-33
0.3
4.0
-3.0
2.0
3.0
0.7
NLL
6.7
-3.0
-7.0
-2.0
0.0
-1.0
-2.3
2.0
-5.7
-2.0
1.7
-10
Inpatt Weak Aug
Inout Caussian
inputt ODS
Input Aughli
Inguet Tar--rrgett MHUp
Label Smoothhin
Activaten Gauedaa
Activation Dropoodt
Gradent Gatesian
Weight Gaussian
Height DropCanmett
Modell
Irputt Weak Aug.
Input Gaussian
Imput Aughli
Irputt Tarast Hörup
Latel Smoothing
Activation Caussian
Acivation Dropout
Gradent Caussian
Weight Gaussian
Model
Weight DropConnect
(a) ID CV
classification in dataset transfer.
(b) ID CV classification
architecture transfer.
10
-0.7
2.3
13
3.0
2.7
4.7
-.3
2.0
MSE
-1.5
2.2
0.2
0.2
-1.7
1.8
-1.8
-0.8
0.3
2.7
-13
2.2
2.5
3.0
-1.0
2.
2.5
2.8
-10
-10
Inpert Gaussion
Input: Input-Barger Barper CMixUP
Adivation Couniann
Activation Dropout
Gradiant Gaumiin
Model
Weight Gaursian
Weight Drop connec
Input Gauesin
Inpie Parget Catrup
Adivvttio Gaussion
Activation Drapout
Gradient Caussian
Medel
Weight Coumian
Height DropCenned
(c) ID tabular regression in dalaset transfer.
ID labular regression in architecture transfer.
Figure 5: Transfer of hyperparameters on in-domain (ID) data.
10
-.3 4.0
6.
1.3
2.7
-7.7
8.7
-7.7
223
-20
ECE
1.3
-27 -3.0
3.0
-3
4.7
53
2.0
0.0
2.7
0.3
3.0
0
0.7
5.3
-2.7
-.3
4.7
4,0
8.3
-3.7
23
-3.0
-3.7
-13
-10
-10
Ipput Weak Aug.
Input Gaussian
Input ODS
Input AugMli
Input Tout-Tar get MrUp
Label Smoothing
Arthvaton Gauzsian
Activation Dropout
Gradlent Gaussian
Weight Gauasian
Modell
Weight Dropet
Input Weak Aug.
Input Gaussian
Input Augbli
Intuut Input-Target Tarpet Motip
Latel Smoothing
Activation Gaussian
Activation Drapout
Gra Gradtent Gauasian
weight Gautstian
Model
Weight Coopeen
(a) OOD CV classification in dataset transfer.
(b) OOD CV classification in architecture transfer.
10
5.3
-188
0
25
2.0
-10
-10
leputt Caussian
Impul CoRKUP
Activation Coussian
Activatoon Dropout
Gradient Gaustaa
Model
Weight Gaussion
Weight Connect
Input Gaussion
Input-Targe ChixUp
Activation Causiion
Activation Dropoit
Gradient Caussian
Model
Welght Goussian
weight DropConnect
OOD tabular regression in dataset transfer.
(d) OOD tabular regression in architectur transfer.
Figure 6: Transfer of hyperparameters on out-of-domain (OOD) data.
The transfer of hyperparamcters from datasct to datasct generally works well for CV classification. However, caution is advised as it is not the case for all noise types. For labular data regression, tuning of hyperparameters is recommended.
Main Obsorvations:
4.5.2 Architecture Transfer
In Figures 5b and 5d we show the ID results for the archilecture transfer, with Figures 6b and 6d reporting the OOD results. The transferability of noise hyperparameters is lower than across datasets for CV classi- fication, but it still successful, especially for weak augmentation and AugMix for ID settings. Transfer of hyperparameters for tabular data regression works for certain noise types in the ID setting, including adding Gaussian noise to the input the weights, which are among the Top-3 noises for tabular data regression.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Main Observations: Transfer of hyperparameters across architectures appears more challenging than across datasets but can be successful in some instances. Caution is advised, and tuning is recommended.
4.6 Learnt Representation Landscapes
We study the learnt. representation landscapes of NNs trained with various noises through the lenses of ID and OOD performance in terms of error, ECE, NLL or MSE. We consider the noises individually, with the ID-found hyperparamcters starting from the same weight initialisation for fairness. We visualise lincar inter- polation modulated through & parameter between the final, =0 0 and initial model, a = (Goodfellow et al. 2014p. The interpolation empirically investigates the smoothness of the training process. We also visualise the landscape in 2D (Li et al. 2018 [oobrook]] 2020) by saving the network after each epoch and concatenating the weights. Instead of using random coordinates, we use the first two principal components of the weights as the coordinates. We normalise them based on the magnitude of the original weights, and we project all the weights onto these two components in the vicinity of a and 3. The 2D visualisations show us the exploration and exploitation of the training process. In Figures 7 and we compare the metric landscapes of no noise with AugMix and Dropout noises, respectively on CIFAR- 10 and WikiFace datasets. We used 20 points for linear interpolation and 100 points for the 2D plots across five selected OOD augmen- tations and 1000 test data samples for compute efficiency. In red, we show the error or MSE; in green, we offer the NLL or ECE. In the 1D plots, o and stand for ID and OOD error or MSE, and x and stand for ID and OOD ECE or NLL. In the 2D plots, the darker combined contours signify worse performance than the lighter parts, and the * in blue or black denotes the start or end weights, respectively. The Appendix contains the metric landscapes for all other noises, tabular classification Adult, and regression Yacht datasets.
Observing Figures and g we first notice the ID and OOD results are similar, with the OOD results being slightly worse across all metries. This includes both the 1D plots and the 2D plots. Second, as seen in Figures 7a and 7b the curves for error, representing generalisation, and ECE or NLL, representing confidence calibration, do not share the same shapes or curvatures. MSE and NLL curves in Figure Sa are more similar than error and ECE curves. Looking at the 1D plots, for example in Figures 7a 7b
and Sa the error or MSE can be more smoothly interpolated than ECF or NL... Figure [a shows models trained without noise can become overconfident, reflected in large NLL and small MSE. Adding noise such as Dropout
fix this,
leading to low NLL for the final model in Figure [d] Looking at the 2D plots in Figures and [b] the error or MSE valley is wider than the ECE or NLL valley, and they are not aligned. From a detailed comparison between no noise and AugMix or Dropout in Figures 7 and S. we observe that AugMix and Dropout can smoothen the optimisation in the ID plots, but not for ECE, and decrease the gap between ID and OOD performance. The 2D plots show that AugMix and Dropout can explore broader metric landscapes than no noise, shown in ranges of and 3 in the 2D plots, and marginally align the error or MSE with NLL. Seen in the lightness of the 2D contour plots, the noises navigate lower NLL or ECE landscapes than no noise.
Our general observations considering both CV and tabular datasets show that while noises such as Aug- Mix, weak augmentation, MixUp or activation and weight noises based around Dropout can smoothen the optimisation regarding error or MSE, they rarely smoothen the optimisation regarding ECE. The metric landscapes often look similar to no noise, but the optimisation ends in more profound valleys. Across the datasots and tasks, label smoothing, input additive Gaussian and ODS have minimal effect on the 2D land- scapes or 1D interpolation. The model shrink and perturb make the optimisation more "stairs-like", and the metric landscape explored is broader. Together with gradient Gaussian noise, the shrink and perturb noises explore broader metric landscapes than the others. No method drastically changes the metric landscape or the interpolation from the default, but they can make the optimisation smoother or broader.
The metric landscapes for error or MSE and ECE or NLL are different, and the noises can smoothen the optimisation terms of error or MSE but not necessarily terms of ECE or NLL. When a model trained without noise is overconfident, adding noise to the training can resolve it and lead to significantly better-calibrated model at the end of training.
Main Observations:
13

---

Published in Transactions on Machine Learning Research (04/2024)
OolD
Dae F11
Error and ECE.
(b) Error and NLL.
(c) Error and ECE ID.
(d) Error and ECE OOD.
Doe
Error and ECE.
(I) Error and NLL.
(x) Error and ECE on ID.
(h) Error and ECE on OOD.
Figure No noise (top) and Input AugMix (bottom) on CIFAR-10.
make
8.271
E.02 C3L OOW
OOW
(a) MSE and NLL
(b) MSF and NLL on ID.
MSE and NLL on OOD.
0403
04
5006
(d) MSE and NLL.
(e) MSE and NLL ID.
(I) MSE and NLL OOD.
Figure 8: No noise (top) and Activation Dropout (bottom) on WikiFace.
5
Conclusion
Key Takeaways: Noise injection methods can improve NN performance across various tasks and datasets. This is despite the fact L2 regularisation was already tuned to prevent overfitting, indicating noise injection methods can provide additional benefits beyond standard regularisation. The methods were not equally efficient across all tasks and datasets, with significant differences performance between regression and classification. Nevertheless, out of the considered noise types, the proposed methodology identified at least one noise or combination of noises that demonstrated improvements in both calibration and generalisation over the noise baseline for all tasks. The most effective noise for CV AugMix, model shrink and perturb and Gaussian noise added to weights for tabular data classification and regression, respectively. At the same time, Dropout and label smoothing worked the best for NLP. Even though ODS was not designed to improve calibration and generalisation, it has shown promising performance in several cases. Combining noises out performedd individual noises in most classification cases, with regression often benefitting from using only one noise a time. While directly combining hyperparameters of noises is reasonable strategy, tuning them still be valuable if large budget is used. The noises improved ID and OOD performance, but
14

---

Published in Transactions on Machine Learning Research (04/2024)
the ID rankings were sometimes inconsistent with the OOD evaluation. AugMix remained highly ranked for robustness, demonstrating that domain-specifie inductive biases are beneficial when crafting noise injection methods as they can improve generalisation and calibration. Note that our aim was not to add new knowledge about AugMix's performance against image corruptions, given its well-documented performance (Hendrycks) et al. 2020), but to demonstrate that inductive biases should be considered when generating noise for better performance. The visualisation showed noises can smoothen the optimisation in terms of error or MSE but not necessarily in terms of ECE or NLL. It also showed noise can help mitigate overconfidence. Overall, the results indicate practitioners should consider combining noises, e.g. AugMix and Dropout, and tuning hyperparameters for their specific problem.
Limitations: To conduct this study, we had to restrict the experiments" scope. Our scope was limited to experimental datasets, tasks such as classification and regression, and standard NN architectures. Testing on more complex data and downstream tasks such as object detection, segmentation, or reinforcement learning would reveal more profound impacts of noise injection. Furthermore, diving deeper into one particular domain, such NIP,, could provide more insights into the elfectiveness of noise injection methods. Moreover, we also limited the optimisation to SGD with momentum and cosine learning rate schedule, which were tuned beforchand to make the hyperparameter search tractable. To draw practical conclusions, we evaluated the noise performance by minimising the NLL rather than exploring all possible settings. The costs associated with adjusting and evaluating different. noises limited the scope of the experiments. Consequently, certain noises might prove more effective with more thorough tuning and a larger budget. This is especially true for noise combinations, where the number of possible combinations grows exponentially with the number of noises. Nonetheless, the existing findings offer valuable insights for practitioners by giving a preliminary indication of the most promising noise sources. This enables users to concentrate their efforts and compute the budget required for luning, For example, AugMix, incorporating domain-specific inductive biases, was transferable and effective even with a limited budget. Developing methods to choose hyperparameters without the need for extensive tuning would enhance the accessibility of these techniques. Lastly, the out-of- distribution evaluation was focused on synthetic augmentations, which may not fully capture the real-world dist ributiitoon shift, as noticed in (Taori et al. 2020) for computer vision tasks.
The strong performance of AugMix highlights the potential for developing specialised, domain-specific noise techniques such as DeepAngment (Hendrycks et al. 2021). For example, tailored domain-specific noise methods could benefit tabular data-based problems and NLP. Future work should also explore specific data-architeetuure noise interactions, as the transferability of hyperparameters was limited. Inspired by the anncaled gradient noise, anncaling noise levels overtraining may also prove helpful, as carly noise could encourage robustness. In contrast, low late-stage noise could enable convergence on a high- accuracy solution. The potential for combining noises from the same category should also be investigated further. The noises affected the entire architecture, but targeting noise injection methods that only affect specific layers or sections of the network may be possible, requiring more or less regularisation. Specific nosse-based approaches for simultaneously exploring the generalisation and confidence calibration trade-off should be investigated further. As demonstrated across image and tabular domains, the empirical differ- ences between many methods seem minor. This indicates that there are more fundamental determinants of performance. Therefore, theoretical analysis of noise injection methods would be beneficial in understanding the underlying mochanisms and providing guidance for future rescarch. Lastly, testing on wider-scale out- of-distribution datasets and real-world distribution shifts would provide more comprehensive evaluation of the noise injection methods' impact on robustness. We hope our study and framework, embedded in our codebase, will assist further research in this area.
Future Directions:
Acknowledgements
Martin Ferianc was sponsored through scholarship from the Institute of Communccations and Connected Systems al UCL. Ondrej Bohdal was supported by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP EP (P/LO16427/I) and the University of Edinburgh.
15

---

Published in Transactions on Machine Learning Research (04/2024)
References
Jordan Ash and Ryan P Adams. On warm-starting neural network training. In NeurIPS, 2020.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- paper-parameter opti- mization. In NeurIPS, 2011.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1): 108-116, 1995.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In 1CML, 2015.
Alexander Camuto. Understanding Gaussian noise injections in neural networks. PhD thesis, University of Oxford, 2021.
Alexander Camuto, Matthew Willetts, Umut Simsekli, Stephen J Roberts, and Chris C Holmes. Explicit regularisation in gaussian noise injections. In NewrIPS, 2020.
Pratik Chaudhari and Stefano Soatto. On the energy landscape of deep networks. aix 2015.
Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Youngjoon Yoo. An empirical evaluation on robustness and uncertainty of regularization methods. arXiv, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. ICML, 2019.
Terrance DeVries and Graham W Taylor. Dataset augmentation in feature space. arXiv, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In TCML, 2016.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wich- mann. Generalisation in humans and deep neural networks. In NewrIPS, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue, and Zachary Nado. Deep learning tuning playbook, 2023. URL http:::/gittub.coot/
playybook_ Version 1.0.
Tan J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimiza- tion problems. arXiv, 2014.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. TCML, 2017.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classification: An empirical study, arv 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness improve deep neural network robustness against adversarial attack. In CVPR, 2019.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Takshminarayanan. Augmix: simple data processing method to improve robustness and uncertainty, In ICLR, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR, 2021.
Ryan Holbrook, 2020. URL
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimiza- tion. In AISTATS, 2016.
Hojin Jang, Devin McCormack, and Frank Tong. Noise-trained deep neural networks effectively predict human vision and its neural responses to challenging images. PLoS biology, 19(12)::0011418, 2021.
Daniel Kang, Yi Sun, Tom Brown, Dan Hendrycks, and Jacob Steinhardt. Transfer of adversarial robustness between perturbation types. arXiv, 2019.
Yoon Kim. Convolutional neural networks for sentence classification. Xiv, 2014.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In NewrlPS, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In NewrIPS, 1991.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv, 2017.
Ken Lang, Newsweeder: Learning to filter netnews. In JCML, 1995.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018.
Πla Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Rafael Miller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In NewrIPS, 2019.
Arvind Neelakantan, Luke Vilnis, Quoe V. Le, Lukasz Kaiser, Karol Kurach, llya Sutskever, and James Martens. Adding gradient noise improves learning for very deep networks. In OpenReview, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupercised Feature Learning, 2011.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In NewrIPS, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word represen- tation. In EMNLP, 2014.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident. output distributions. In ICLR Workshop, 2017.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Ben Poole, Jascha Sohl-Dickstein, and Surya Ganguli. Analyzing noise in autoencoders and deep networks. arXiv, 2014.
Lutz Prechelt. Early stopping- but when? In Neural Networks: Tricks of the trade. Springer, 2002.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. DEX: Deep EXpectation of apparent age from a single image. In ICCV Workshop, 2015.
Jocelyn Sietsma and Robert JF Dow. Creating artificial neural networks that generalize. Neural networks, 4(1):67-79, 1991.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: survey. Transactions on Neural Networks and Learning Systems, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. In NewrIPS, 2020.
Yusuke Tashiro, Yang Song, and Stefano Ermon. Diversity can be transferred: Output diversification for whitc-and black-box attacks. In NeurIPS, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML, 2013.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global reprosentations by penalizing local predictive power. In NewrIPS, 2019.
Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In ICML, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient. langevin dynamics. In ICML, 2011.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as SGD. In ICML, 2020.
Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving general- ization in regression. In NewrIPS, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Toward understanding the importance of noise in training neural networks. In ICML, 2019.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Appendix
In the Appendix, we first provide the experimental settings and the hyperparameter ranges for all the experiments in Section A. We then provide the full numerical results and visualisations for all the experiments
in Section B
A Settings
A.1 General Settings
We used stochastic gradient descent with momentum of 0.9 to train all the networks. The learning rate and 1.2 regularisation were tuned and reused for each noise injection method. We used a cosine annealing learning rate schedule without restarts (Loshchilov & Hutter 2017) for all experiments. In most cases, we used gradient norm clipping of 20.0 to stabilise the training, with gradient clipping of 10.0 for tabular regression and 5.0 for WikiFace. The batch size was set to 256 for all experiments. The final results are the average of 3 runs with 3 different seeds. We used cross-entropy loss for all classification experiments. For regression, we used the Gaussian negative log-likelihood (NLL) loss, where we modelled the variance as an additional output passed through an exponential function to ensure positivity. We added a small e of 1e to the softmax probabilities to avoid NaNs. We clipped the variance between 1e-4 and le to avoid NaNs. The hyperparameter ranges, and the sampling scale for each datasot-architectur pair are in Table 4 The hyperparamelers and implementations of all the noises and experiments can be found in the code, which will be souree.. We used the default PyTorch weight initialisation for all layers.
For the tabular OOD experiments, we constructed custom augmentations where we applicd Gaussian or Uniform noise scaled by the magnitude of the input features across severities for addition: [0.02, 0.04, 0.06, 0.08, 0.1] multiplication [0.04, 0.08, 0.12, 0.16, 0.2] where the severity scaled the range the standard deviation of the noise applied to the input. Additionally, we zeroed out some input features with probability [0.04, 0.08, 0.12, 0.16, 0.2], denoting 5 severities. In total, there were 5 different input shifts across 5 severities each. To avoid label-flippigg in applying these augmentations, we have introduced an empirically determined scaling factor for the severity of all noises for a particular dataset. They multiply the [0.02, 0.04, 0.06, 0.08, 0.1] or [0.04, 0.08, 0.12, 0.16, 0.2] by a scaling factor to determine the severity of the noise applied to the input based on the dataset. We use a K-nearest neighbour (KNN) classifier, specifically 1-neighbour KNN, trained a dataset's original, unmodified data. This classifier then predicts labels for the augmented data. We adjust the scaling factor for each dataset so that the KNN classifier's accuracy on the augmented data exceeds 99% or the mean squared error is less than 0.01. This approach ensures that the angmentations are subtle enough to maintain the integrity of the data, meaning the nearest neighbour the closest match in the original that the augmentations are not loo severe, and the nearest neighbour's label remains the same. The found scaling factors are shown in Table 5
the same. However, this is not a perfect solution, with an empirical guarantec
Regarding noise implementation details, Dropout, DropConnect, additive weight or activation Gaussian noise, are applied to all linear and convolutional weights throughout the network, excluding the last layer and normalisation layers. Both model and Gaussian gradient noise are implemented on all weights within the network, encompassing affine parameters in normalisation layers. Rotation was omitted from given that one of our tasks involved predicting the rotation angle.
AugMix,
A.2 Vision Experiments
For SVHN we used fully connected network with 4 hidden layers of 150 units followed by ReLU activations.
When we used ResNet-18 we used it with [64, 128, 256, 512] channels in 4 stages with [2, 2, 2, 2] blocks with strides [1, 2, 2, 2). When we used ResNet-34, we used it with [64, 128, 256, 512) channels in 4 stages with [3, 4, 6, 3] blocks with strides [1, 2, 2, 2]. In all cases, we trained the networks for 200 epochs. We only used 0-1 truncation followed by normalisation for each dataset without further data augmentations for training, validation and test sets. For rotation experiments, we enabled uniform rotations between (0, 90%) degrees and rescaled the targets accordingly to [-1, 1]. Gaussian noise, motion blur, snow, elastic
19

---

Published in Transactions on Machine Learning Research (04/2024)
transformation, and JPEG compression were selected as OOD augmentations for visualisation experiments across all 5 severities. For CIFAR-10, CIFAR-100, and SVHN we used the dedicated test sets as the test set, while for TinylmageNee we used the official validation set as the test set. We used 10% of the training data to construct the validation sets. For WikiFace, we used 10% of the data as the test set and 10% of the remainiin data as the validation set and the rest as the training set.
For evaluation on the sketch domain, we utilise ImageNet-Sketch dataset from Wang et al. (2019) and derive our TinylmageVet-Sletchh dataset from it. As the SeageNet-SSketch images are not square, we crop the centre and then resize to 64 64, the same size TinylmageNett. We only keep the images of the same 200 classes as used in TinyImageNet, allowing us to directly evaluate pro-trained models on the new TinylmageNet-Skee dataset.
A.3 NLP Classification Experiments
For our NLP experiments we used NewsGroup and binary SST datasets. In the NewsGroup dataset we aim to classify news texts into one of the 20 available categories based on the topic. The task in the binary SST dataset is to predict the sentiment of a movie review into 2 categories: positive or negative. For SST we only consider the text itself, rather than also considering the available parse trees, making the task more challenging.
Each dataset first pre-processed with respect to glove embeddings (Pennington et al. 2014 into embed- dings of dimension 100 and sequence length 100 and 50 for NewsGroup and SST respectively, In both cases, we trained the networks for 100 epochs. We used the global-pooling convolutional network architecture from Kim (2014) with planes [128, 128, 128] and a transformer decoder (Vaswani et al. (017)) with embed- ding dimensions 100, 6 layers, 8 heads, 1024 hidden dimensions, 64 dimensions per head and no dropout. For the NewsGroup experiments, we used about 5% of the data as the test set, 10% of the remaining data as the validation set and the rest the training set. For the SST experiments, we used the original development set the test set, for validation we took 10% of the original training set and used the remainder as the training set. No OOD Lest was set for the NLP task due to lack of suitable perturbations to construct OOD data.
A.4 Tabular Regression Experiments
For the tabular experiments, used fully connected network with [100, 100, 100, 100) hidden units and ReLU activations. In all cases, we trained the networks for 100 epochs. We normalised the input fcatures and targets to zero mean and unit variance by using the training set statistics and applied the same normalisation to the validation and test sets. For the tabular experiments, we used 20% of the data as the test set and 10% of the remaining data as the validation set and the rest as the training set. The regression targets were normalised zero mean and unit variance.
B Full Results
We provide full results of all experiments in the paper, where the main reported value is the mean across 3 repetitions, followed by the standard deviation. The ranks presented in the main body of the paper can be obtained by ranking the results in each table by the metric of interest. Following the tables, there are the visualisations of metric landscapes for CIFAR-10, Adult, WikiFace and Yacht datasets. We encourage the reader to look at our code for other datasets to regenerate them from there.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Hyperparamcter (6)
Range
Scale
Learning rale (LR)
[0 10-1
Log
L2 weight
[10-7,10
Log
Input Gaussian noise std.
[10-4,10* 10
Log
Inpul. AugMix alpha
[0.1]
Linear
Input AugMix severity
[1,
Linear
Input AugMix width
[1.5]
Linear
Input AugMix chain-depth
[-1,3]
Linear
Input ODS epsilon
[10-1.10-]]
Log
Input ODS temperature
[0.5,5.0]
Log
Input-Target MixUp alpha
[0.1]
Linear
larut-Target CMixUp alpha
[0.1]
Linear
Input-Target CMixUp sigma
1104,109 10
Log
Target Label Smoothing
00.0.25
Linear
Activation Gaussian noise std
[10 10
Log
Activation Dropout rate
0..1
Linear
Gradient Gaussian noise
0.1
Linear
Gradient Gaussian noise y
[0. 1]
Linear
Weight Gaussian noise std
[10 4, 10
Log
Weight DropConnect rate
[0.1]
Lincar
Model noise shrink factor
[000,
Linear
Model noise std
[10-7.10-3]
Log
Model noise frequency
00, 20
Linear
Table 4: Hyperparameters (HPs) optimised for individual noises and their range.
Dataset
Task
Scaling Factor
Adult
Classification
0.1438
Abalone
Classification
0.0886
Conerete
Regression
0.0207
Energy
Regression
0.0546
Wine
Classification
0.0078
Wine
Regression
0.0048
Yacht
Regression
0.0886
Toxicity
Classification
0.3793
Students
Classification
0.2336
Boston
Regression
0.0886
Table 5: Scaling factors for tabular data.
21

---

Published in Transactions on Machine Learning Research (04/2024)
SVHN
CIFAR-10
CIFAR-100
TinyImageNes
Noise Type
ID
OOD
ID
OOD
ID
OOD
ID
OOD
No NOISE
16.33
20.18
12.04
32.32
41.69eas
61.52
54.15
75.65
INPUT WEAK AUG.
+4.67
18.59+314
5.39
27.86+
26.51+au
52.57
39.73 73-
67.73+m.
INPUT GAUSSIAN
20.23+aa
12.02+am
31.97
44.34+-m
61.14+.m
28.28-024
75.12+ta
INPUT ODS
16.35+taai
+0.13
12.0L+a.
30.42
44.28
61.34+am
66.47
82.46+1.1
INPUT AUGMIx
12.28
15.67
7.48
18.75
30.09
46 46.05
42.49 49s
a1.5
INPUT-TAN TRRGET MIXUP
13.95.am
17.71
10.97
28.38 19.70
46. 66.15
63.00aam
54.02_msa
75.33
LABEL SMOOTHING
16.351a12
20.10
11.88
19.6
42.48
58.41
52.47
73.96am.
ACTIVATION GAUSSIAN
16.33
20.14
11.44
30.71 1
14
61.81
53.96
75.45
ACTIVATION DROPOUT
13.83
17.432) 10
8.93gam
29. 19..85
4
58.9221
43.262005
69.322m.
GRADIENT GAUSSIAN
17.59
22.49
16.41
37.57 s2
45.33
62.49aa2
59.99
80.15
MODEL
16.17
20.08
10.65
35.88
56.70 73:
49. 26:666
72.44=to.s
WEIGHT GAUSSIAN
16.60+a.a
20.29+a.
10.53
30.76 76
42.97*
60 28
54.20 64.20mu1
75.71+t0.1
WEIGHT DROPCONNECT
15.82+um
19.53+3
12.20
+ass
42.08+a.
59 16+1....
54.33-am
75.70+z4
Top-2 DIRECT COMBINATION
12.57+tax
16.04+1.9
4.78+aau
16.47 +
24.69
43.04+:a1
37.18 37-18
G8.16+a..
ToP-3 DIRECT COMBINATION
13.49
16.95
5.07
17.72
24.8
42.99
37.43
63.73
Top-2 OPHIMISED COMBINATION
12.14 10.10
15.51a34
5. 5.15gam
15.63
25.28
43.86g1.
36.88
57.90 57.90aza1
ToP-3 OPTIMISED COMBINATION
12.66
16.08130:
6.60
17.91
26.19
43.84
36.59..o
60.36ga47
Table 6: CV classification: Error (4. %) comparison on in-distribution (ID) and out-of-distribution (OOD)
test sets and with tuned hyperparameters.
SVHN
CIFAR-10
CIFAR-100
TinyImageNet
Noise Type
ID
OOD
ID
OOD
ID
OOD
ID
OOD
No NOISE
10
15.5
4.02
16.40gm0
5.76ig
7.432mm
16.43
10.11
INPUT WEAK AUG.
6.35
7.85
1.92
15.91
4.83
13. 13.28
6.49
11.462mm
INPUT GAUSSIAN
13.24
4.22
16. 65.5.
5.42
7.70
15.96amm
10. a
INPUT ODS
t3.03
15.45
13.43zu
4.26 ew:s
15.8 15.38
5.86
7. 78
26.97 =mm
20.20
INPUT AUGMIX
4.81+uu
6.03e
1.18 saice
6.27 ta.n
4.41+a.
11.38
4.87+
15. 78
INPUT-TARGIS MIXUP
2.8l+
49
5.4 +x.c4
7.31+
14.20
5,57++04
16.33+cm
10.26-0.2
LABEL SMOOTHING
8.66
10.61
5. mw12
9. 9.4Ge
21.55
17. 10
29.45
16.02.:mm
ACTIVATION GAUSSIAN
13.09
2.50 gm:3
18 a0
5.12
7.58
14.77
9. 2.
ACTIVATION DROPOUT
5.33+a.9
6,61
4.48 ewns
18 85
5.48
10.31
20. 13:
GRADIENT GAUSSIAN
14.84+um
18.39
6.03 E
18. 79ea
5.54+
9. 9.70
23.56+12
34.33-ro2
MODEL
10.93+a.1
12.82 +0.1s
4.42 #auCa
18. 43
9.06+a.u
11. 11.00+a.
10.80+1ar
9.01+tox
WEIGHT GAUSSIAN
13.38
15.72
6.48 miz4
21.54
5.99
7.79
14.95
9.98
WEIGHT DROPCONNECT
12.51
14.87 a9os
4.94
16, ge.9
5.79
8,21
15.50
10. 10.12
TOP-2 DIRECT COMBINATION
1.79
2.76
1.25g
6.91
6.19
14.21
4.04g3m
15.69
ToP-3 DIRECT COMBINATION
1.31+au
1.86+u
2.08 ew:1
8.57+a.m
6.66
14.67
13. 13.05+am
22. =1.67
Tor-2 OPTIMISED COMBINATION
2.55+um
3. 3.48+am
0.90
5.50+
6.68
14.54+om
3.21+am
15.33-ou
Top-3 OPTIMISED COMBINATION
2.89
3.98
1.15
4.97
5.60
12.56
9.90
20. 10
Table 7: CV classification: ECE (L, %) comparison on in-distribution (ID) and out-of-distribution (OOD)
test sets and with tuned hyperparameters.
22

---

Published in Transactions on Machine Learning Research (04/2024)
SVHN
CIFAR-10
CIFAR- 100
Tiny ImageNet
Noise Type
OOD
ID
OOD
OOD
No NOISE
1.43
1.64gem
m.42_mm
1.21 2m
1.85g
2.69
2.80
3.90aro
INPUT WEAK AUG.
0.622am
0.75gmm
1.10
1.07
2. 2.44gar
1.84 a.
3.462
INPUT GAUSSIAN
1.43
1. 1.65gma
=.4
1.21 201s
1.84
2.67
2. 2.73 25
3.86
INPUT ODS
1.40
1 1.00amz
0.42
1. 1.14
1.84
2.60
4. 4.79
6. 1.
INPUT AUGMIX
0.49+um
0,61+
0.25 2::5u
O, +.m3
1.17 1.17+au
2.0 2.04eum
1.87 28Tuum
3. 3.14 14+4
MIXUP
0.52
0.64
0.40
0.91
2.03
2.77
2.79
3.88
LABEL SMOOTHING
0.75
0.90
0.46
1.09
2.18
2.87
3. 3.252nm
4. 1
ACTIVATION GAUSSIAN
1.41
1. 1.33aee
0.43
1.27
1.84
2.71
3.87
ACTIVATION DROPOUT
0.51
0. 1.63
0.33
1.26
1.71
2.58aumt
1,97
3. 6
GRADIENT GAUSSIAN
1.76
22
0. 0.56
1.39
1.86
2.76um
3. ez
5.36
MODEL
+.02
1.17
0.37
1.27
1.60+avz2
e.w3aa
2.38
3. 6-1++a
WEIGHT GAUSSIAN
1.44+am2
1.65+aaz
1.41+taa
1.79+aus
2.60+ara
2.76
3. 3.89+caaz
WRIGHT DROPCONNECT
1.32
+.52
1.21 taau
e.73
2.58
2.78
3.90
ToP-2 DIRECT COMBINATION
0.44
2.54
0.16 20.0
0.57 19.0
0.97
1.9G
1.G0
2.98
Tor-3 DIRECT COMBINATION
0.45
0.55
0.17 =1
0.97
1.97 sum
3.25
Tor-2 OPTIMISED COMDINATION
0.43+u
0 0.54+tae
0.53
0.98
2.00
1858mm
2. 2.96+coz
Top-3 OPTIMISED COMBINATION
0.44+am
0 e.55e
-0.0
0.57 +aum
0.99+sa
1.93 eus
1.63
3.25+tm:
Table 8: CV classification: NLL (L) comparison on in-distribution (ID) and out-of-distribution (OOD) test
sets and with tuned hyperparameters.
Scores
Ranks
Noise Type
ID
OOD
Sketch
ID
OOD
Sketch
No NOISE
54. 20
75.65
89.61 to
13.0
13.0
16.0
INPUT WEAK AUG.
39.73
67.7320.
85.90
5.0
6.0
6.0
INPUT GAUSSIAN
133.28-a2
75.12
a0.2
10.0
10.0
13.0
INPUT ODS
66.47+14
82. 82.46+8.72
89. $9.54+aa
17.0
17.0
15.0
INPUT AUGMIx
42.49 1974
61 01.581030
83. 12
6.0
5.0
4.0
INPUT-TARGET MIXUP
54.02_0.0
75.33
89.271990
12.0
11.0
12.0
LABEL SMOOTHING
52.47 -01
73.961015
$8.1520u
9.0
9.0
9.0
ACTIVATION GAUSSIAN
53.96=o.m
75.45
89.01+0ac
11.0
12.0
10.0
ACTIVATION DROPOUT
43 43.26=0aa
09.3224..
87.60 o0.
7.0
7.0
8.0
GRADIENT GAUSSIAN
59.99 =a3
80.15 1001
92.16:
16.0
16.0
17.0
MODEL
49.66
72.44
86.98
8.0
8.0
7.0
WEIGHT GAUSSIAN
54.20=o.n
75.71 10.33
89.4629az
14.0
15.0
14.0
WEIGHT DROPCONNECT
54.33_00
75.7010.1
89. 99.241
15.0
14.0
11.0
ToP-2 DIRECT COMBINATION
37.18-oos
58, 16+4.22
82.52+a.a
3.0
2.0
2.0
Tor-3 DIRECT COMBINATION
37. 37.43
58.75 43.73
83,46++.17
4.0
3.0
5.0
TOP-2 OPTIMISED COMBINATION
36.88 =
57.90+024
82.30+o.15
2.0
1.0
1.0
Tor-3 OPTIMISED COMBINATION
36.59
60.36
83.021a.0
1.0
4.0
3.0
Table TinylmageNeet classification: Error (L, %) comparison on in-distribution (ID) and distiiiiitttiitution (OOD) and Sketch test sets and with tuned hyperparameters.
23

---

Published in Transactions on Machine Learning Research (04/2024)
Scores
Ranks
Noise Type
ID
OOD
Sketch
ID
OOD
Sketch
No NOISE
16.43
10.11
8.67 20.45
14.0
5.0
INPUT WEAK AUG.
20.3
11.46
18.10
4.0
8.0
10.0
INPUT GAUSSIAN
15.96
10. 15
7.53
12.0
6.0
3.0
INPUT ODS
26.97
28. 20
11.3
16.0
16.0
9.0
INPUT AUGMIX
4.87 +0.31
15.78+tu +8+
20.08
3.0
11.0
11.0
INPUT-TARGET MIXUP
16. 16.33++ 14.33++um
10.20 +0.52
7,28+1o8
13.0
7.0
2.0
LABEL SMOOTHING
29. 20001
16.622n.c
5.28
17.0
12.0
1.0
ACTIVATION GAUSSIAN
14.77
9.83
9.041am9
9.0
2.0
7.0
ACTIVATION DROPOUT
10.3 =0,00
34.71
6.0
14.0
15.0
GRADIENT GAUSSIAN
23.56
34.33
47 47.82
15.0
17.0
17.0
MODEL
20.80
9.01
9.90
7.0
1.0
8.0
WEIGHT GAUSSIAN
14.95 1.00
9.98
8.99
10.0
6.0
WEIGHT DROPCONNECT
15.50 +o.sa
10.12++..s
7.76+a.72
11.0
5.0
4.0
ToP-2 DIRECT COMBINATION
2.04
15.69
25.121a.9
2.0
10.0
13.0
TOP-3 DIRECT COMBINATION
13.05
22.67
39.5410.43
8.0
15.0
16.0
ToP-2 OPTIMISED COMBINATION
3.21 -0.00
15.33+40.5
24.93+1.2
1.0
9.0
12.0
ToP-3 OPTIMISED COMBINATION
=.90
20.11+0.00
34.6440.0
5.0
13.0
14.0
Table 10: TinyImageNet classification: ECE (J, %) comparison on in-distribution (ID) and out-of-
distribution (OOD) and Sketch test sets and with tuned hyperparameters.
Scores
Ranks
Noise Type
ID
OOD
Sketch
ID
OOD
Sketch
No NOISE
2.80
3.90
4. 4.68aam
14.0
14.0
8.0
INPUT WEAK AUG.
1.84
3.4610.0
2.69
5.0
6.0
9.0
INPUT GAUSSIAN
2.75-ux
3.86+
4002
10.0
5.0
INPUT ODS
4.79-2.0
6.23
4. 4.72+aun
17.0
17.0
12.0
INPUT AUGMIX
1.87 =001
3. 4.000
4.52
6.0
3.0
1.0
INPUT-TARGET MIxUP
2.79
3. 3.881
4. 160
13.0
11.0
6.0
LABEL SMOOTHING
3.25=
4. 4.10
4. poc0
16.0
15.0
10.0
ACTIVATION GAUSSIAN
2.74=0om
3..7+
4. Doca
9.0
10.0
3.0
ACTIVATION DROPOUT
1.97 =002
3.69 t0.0s
5.48 2.19
7.0
8.0
15.0
GRADIENT GAUSSIAN
3.20
5.36
7.53 7
15.0
16.0
17.0
MODEL
2.38 =00a
3. 3.64++0.0.
4. 4.54 4
8.0
7.0
2.0
WEIGHT GAUSSIAN
2.76
3.89 10.07
2005
11.0
12.0
7.0
WEIGHT DROPCONNECT
2.78
3.901a.01
4.6 2
12.0
13.0
4.0
TOP-2 DIRECT COMBINATION
1.60=a.0
2.98 +0.00
4. 4.72+000
2.0
13.0
TOP-3 DIRECT COMBINATION
1.73=002
3. 3.25++.10
5.75
4.0
5.0
16.0
Tor-2 OPTIMISED COMBINATION
1.58_aa
2.96
4.71 2005
1.0
11.0
Tor-3 OPTIMISED COMBINATION
1.63
3.25
5.22=007
3.0
4.0
14.0
Table 11: TinylmageNet classification: NLL (4) comparison on in-distribution (ID), out-of-distribution (OOD) and Sketch test sets and with tuned hyperparamcters.
Metric ID vs OOD
ID vs SKETCH OOD vs SKETCH
ERROR
0.912
0. 79.944
0.824
ECE
0. 1e.18 tose
-0.397 +0.07
0.397 -0.027
NLL
0.878
0.029 -oa0s
0.081 +0.660
Table 12: TinylmageNet classification: Kendall Tau correlation between ID, OOD and Sketch rankings of different noise types.
24

---

Published in Transactions on Machine Learning Research (04/2024)
CIFAR-10
CIFAR-100
TinyImageNet
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
16.09ae1s
20
40.97
61.56aam1
54.70
75.96
INPUT WEAK AUG.
7.99+a.m
27.60=o.z
24.03
52.82+a.a
39.60+a.s
67.41+aa
INPUT GAUSSIAN
16.72
30.28
41.07
61.04+a.4
54.01+a.
75.31+aa8
INPUT ODS
15.79
29.54 20:71
11.13gto.
60.43
52.77 10.54
74.61 2011
INPUT AUGMIX
10.2616.0
20.95
30.18 2:0
45.9110.
10.041...
60. 10:6
INPUT-TARGES MIXUP
16.90
32.62 =0.06
39.04 00.004
48.48
51.49 40.97
LABEL SMOOTHING
17.18
31.10 =0:0000
42.00 20.000.:.co
61.4940.2
52.33
23.95=0008
ACTIVATION GAUSSIAN
16.34
30.52 20220mmm
38. .8.9 20.30
59.70
52 12,49 2.033
ACTIVATION DROPOUT
12.45+t.12
27.90=o.u
31.58 t0.54
56.38
51.85 t0.10
74.08 #0o1
GRADIENT GAUSSIAN
18. 18.70+0.15
34.04-o.a
47.64+tm +0.31
67.79
55.91 +0.25 +0.25
76.96+a00
MODEL
13.1 10.35
32.40
79.31
87.90
48.8610.1
72.07
WEIGHT GAUSSIAN
16.5410m
30.86
37.14 42..0
58.3210.1
52.88 12.888.0.
74.82
WEIGHT DROPCONNECT
16. +0.225
28. 28.80=0a0
41.18
61.39+a..2
53.67 +0.11
75.28=014
Table 13: CV classification: Error (L, %) comparison in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across datasets.
CIFAR-10
CIFAR-100
TinyImageNet
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
9.762007
20.73
12.77
10.73
20.11
11 11.6610m
INPUT WEAK AUG.
5.40=0.0
20.98 20.02
3.65=oas
11.12+a.0
5.97=a40
11.75+a.4
INPUT GAUSSIAN
10.39
20. +0.78
12.81=026
10.66
16.74
10.72+0.16
INPUT ODS
9.89
20. 2o.rs
11.18 1211
10 10.28 Lm
15.98
10.52
INPUT AUGMIX
5.53=000
11.65 4o.0z
8.94-007
8.86
3.97
15.43
INPUT-TARGET MIXUP
6.27 20:11
8. 2.122o.00
14.76
10.99
19.33
10.08
LABEL SMOOTHING
1.65 70
6.05
21.53_o.0
15.64
27.1 77.18am07
15.49
ACTIVATION GAUSSIAN
10.03 00.00
20.36 10.00
5.07
9. +0.55
14.04+a72
9.88+
ACTIVATION DROPOUT
9.41 +0.17
21.79+0.11
7.94-0.0
19.81 +0.47
16.67=#00
11.00++.12
GRADIENT GAUSSIAN
12.77
24.59 +0.50
8.38=
18,02
13.41+a7a
10.07
MODEL
2.53
21.33
0.80=1.0
4.07 25.10
14.45
10.68
WEIGHT GAUSSIAN
10.05
20. 70.70
11.79
22 2
15.55
10. 10.38
WEIGHT DROPCONNECT
10.93
19.75
11.24=070
9.93
18.10
11.11
Table 14: CV classification: ECE (J, %) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across datasets.
25

---

Published in Transactions on Machine Learning Research (04/2024)
CIFAR-10
CIFAR-100
TinyImageNeet
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
0.71 2001
1.55gam
1.88_a02
2.87
2.962a12
3.99
INPUT WEAK AUG.
0.402z00
1.04
1.02=am
2.45
1.80
3.4
INPUT GAUSSIAN
0.72
1.50
1.89=001
2.84
2.81
3.91
INPUT ODS
0.71
1. 1.55
1.87=007
2. 2.80
2.72
3. 2..00
INPUT AUGMIX
0.39=0.0
1.30-001
2.02+a..
1.73+a01
3.10+
INPUT-TARGET MIXUP
0.56+0.0
+.04++0.2
1.71-001
2.59 +0.00
2.62+xo2
3. 3.65
LABEL. SMOOTHING
0.56
1.01
2.16-001
3.04
3.162a00
4.1010.
ACTIVATION GAUSSIAN
2.70
1.51 10.05
1.66_001
2.73
2.64
3.81 10.m
ACTIVATION DROPOUT
0.000
1.96+0.12
1.24=002
2.69
2.70=
3. 3.83++..0
GRADIENT GAUSSIAN
0.93 =0
1.99
1.93=001
3. 3.20
2.83=
3.9 +.98
MODEL
0.52
1.43
3.59=144
4.07 20.75
2.46#a01
3.69
WEIGHT GAUSSIAN
0.72
1.54
1.54-ams
2..l
to.m
2.72=ac4
3.86+a..2
WEIGHT DROPCONNECT
0.79=0o1
1.48
1.86-aat
2.85
2.83=ao1
3.92+t..
Table 15: CV classification: NLL ()) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across datascts.
SVHN
CIFAR-100
EinyemageNet
Noise Type
OOD
ID
OOD
ID
OOD
No NOISE
5.12 =0.88
9.20+
43.87 -0.4
61.25+to.
53.63+aur
75.20+a.2
INPUT WEAK AUG.
4.13
8.51 1033
27.33_o.t
51.09
38.21 gn0e
64.7540m
INPUT GAUSSIAN
5.01 1008
9.09
43.05
58.71 10:37
54.15
74.6111..
INPUT AUGMIX
3.51 =0.06
3,27 +0,04
30.23
45.51+0.r +0.17
42.05+022
59.93++.10
INPUT-TARGES MIXUP
5.58
12. +0.111
43. 43.93=066
59. 199.98.a +0.21
52.95=1am
73.82+0.44
LABEL SMOOTHING
5.04
8, 8..8
42.99
57.27 +0.43
53.77 nua2
74.25+
ACTIVATION GAUSSIAN
5.142000
9.1 20.06
13.74=aam
59. 29.299c. 20.0
56.32
76.49
ACTIVATION DROPOUT
4.37 20002
16.
42.89 =0.0
58.26 t0.51
42.47 20as
67. 07.788
GRADIENT GAUSSIAN
6.25
11.43
44.39
59.5610.97
57.92
77.81 Lo.at
MODEL
3.98
37. 37.14m0.1
56.97 10.10
47.112053
69. 76.m
WEIGHT GAUSSIAN
5.04+004
9.08
44.28
59. 19.884+a.0
54.11+241
74.52
WEIGHT DROPCONNECT
5.11+0.1
9,03++.11
45.75
59.7 199.1++0 +.1
56.44+s4
76.12+2.5
Table 16: CV classification: Error (L, () comparison in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across architectures.
26

---

Published in Transactions on Machine Learning Research (04/2024)
SVHN
CIFAR-100
TinyImageNet
Noise Type
ID
OOD
ID
0OD
ID
OOD
No NOISE
2.73+0.0
+..3
5.76
+.50+a.1
14. 14.68+a4
9.93++ae
INPUT WEAK AUG.
2.66=0.6
5.75+
6.33
15.26+0.s:
8.20=128
9..8
INPUT GAUSSIAN
2.68
5.09
5.07
13.79 2m
8.75
INPUT AUGMIX
1.462000
3.04
6.56=071
14.17
6.23
16.60
INPUT-TARGET MIXUP
5..1
5.2 5.27 10,00
2.68=0226
6.77
12.24
8, s..12
LABEL SMOOTHING
0.
1.1620003
10.18=284
1.54
17.47
9.76.1..1
ACTIVATION GAUSSIAN
2.75=000
5. 1.11 +0.07
5.45-1128
10,46+
10.66=3e5
8.42+ta.m
ACTIVATION DROPOUT
3.03
5.72+o.a
5.49-o.22
10.00+a.41
14.68+air
25.16+0.15
GRADIENT GAUSSIAN
3.61+0.11
7.07 70.7+
5.59-120
11.16+178
21.10+a45
31.33++a. +a.35
MODEL
2.38
4.781001
7.59
13.21 10.00
5.632ma1
10.34
WEIGHT GAUSSIAN
2.67 20
5.062o.m
5.30=ma0
9.87
12.422aa7
1.24
WEIGHT DROPCONNECT
2.91 =0:08
5.38
6.98-1.6
11.51+1.08
11.42=214
7.99
Table 17: CV classification: RCE (J, %) comparison on in-distributtin (TD) and out-of-distribution (OOD) test sets and with hyperparameters transferred across architectures.
SVHN
CIFAR-100
EinyImageNet
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
0.23
0.41 to.00
1.83-as
2.69
2.72=am
3.86+am
INPUT WEAK AUG.
0.23
0.48
1.09-aa1
2.36++am
1.84+am
3.28+
INPUT GAUSSIAN
0.23
0.41 10:00
1.75=0m
2.562o.m
2.67
3. 3.76.m.
INPUT AUGMIX
0.16g000
0.31 10:00
1.19
2.05
1.90
3.051
INPUT-TARGET MIXUP
0.24goo1
0.462gom
1.76_003
2.58
2.562003
3. 3.68 18.0
LABEL SMOOTHING
0.19=0.00
0.31 +0,00
1.99=000
2. 2.65+
2.82=
+0.00
ACTIVATION GAUSSIAN
0.23=0000
0, 41 +0,00
1.78=002
2.59
2.72=a19
3. 8.84
ACTIVATION DROPOUT
0.26
0.48
1.76-ao4
2.55
2.03=a00
3. 3.700..
GRADIENT GAUSSIAN
0.30
0.57 20.02
1.79-ams
2.60
2.94=am
4.79
MODEL
0.18 00
0.35 no
1.59_a01
2.67
2.10
3.461am
WEIGHT GAUSSIAN
0.
0. 0.4010.00
1.81 g00e
2.61 10.0
2.63
3. 7..400m
WEIGHT DROPCONNECT
0.24
0.43
1.87
2.61 10.00
2.73
3.8220.15
Table 18: CV classification: NLL (L) comparison on in-distribution (ID) and out-of-distributiin (OOD) test set and with hyperparameters transferred across architectures.
Wine
Toxicity
Abalone
Students
Adult
Noise Type
OOD
ID
OOD
OOD
OOD
OOD
No NOISE
35.73.ss
36.37
47.06
11.98
13.30
13.31
65.10
65.89...
15.5
73
INPUT GAUSSIAN
35
36. 28
47.06
44.78473
43.3 13.30_a
43.53
63.4024-
60.81
15.58144
15. 74
INPUT ODS
35.83.m
36.17
18.01
45.65
13.38
13.61
64.56ian
65.74_1m
15.47
15.07
INPUT-TARGET TARGET MIXUP
ISPUT-
38.02
38. 12:us
50.96
46.47
43.46-ae
43.55
65.40
68.13-
15.28
15.44
LABEL SMOOTHING
35
36.05gra
50.00a
46.58
43.22
64.98 43.41
65.69
15.55
15.74gia
ACTIVATION GAUSSIAN
96. 122
47.062xa
44.82a74
43 1
43.41
64. 56
66 46
16.77amm
15.91
ACTIVATION DROPOUT
56
56. 27
13.54..1.
71.73 1346
72. :2.4
14.82
14.92
GRADIENT GAUSSIAN
32.60
33,97 Lis
40.00
49.10
43.74
43.86
68.78 sse
68. 1d.5d
15.38 teas
15.52
MODEL
37.92
38. 15gmm
45. 10
43.49h
43.42_aa
43.60
65.82gaa
65. 86
14.72
14.81
WEIGHT GAUSSIAN
35.32g
36, 162
50.00accs
49.0622a4
43. 43.10
43.30
64.982
66 80
15.13g
15. 23
WEIGHT DROPCONNECTT
38.85
38.99
47. 47.06
44.90
43.42_am
43.36
64.56
66. 28.58
14.93
15.04g
ToP-2 DIRECT COMMINATION
37.50g
37.8821
46.082201
44.002
43.18
43.41
64.56gaa
82.82
14.61
14.77gmm
TOP-3 DIRECT COMBINATION
38.23
38. 4Gz0
31.37amm
31.37417
43. 13.00
13.39
69, 20
69 10.35-11
14.831om
14.91mam
ToP-2 OPTIMIEED COMBINATION
37.60:s.:
37.79.2x
50.00
47.02
43.22
43.40
05.82
66.
14.61iess
14.74
ToP-3 OPTIMISE COMPENATION
39.90ssae
29.96
49.02...a
44.47
43.26
43.584sm
82,70
82.75
14.64
14.74gua
Table 19: Tabular data classification: error (L, %) comparison on in-distribution (ID) and distrr-iist
(OOD) test sets and with tuned hyperparameters.
27

---

Published in Transactions on Machine Learning Research (04/2024)
Wine
Toxieity
Abalone
Students
Adullt
Noise Type
ID
OOD
ID
OOD
ID
OOD
ID
0OD
ID
OOD
No NOISE
8.75
9.07
45.78
44.18
3.
4.19
12.46
13
3.47
3,85
INPUT GAUSSIAN
8.56
8.52.tm
45.84
43.
3.60
4.05 te40
13.36
14.37-L-
3.50
3,84
INPUT ODS
5.75
6.02
3.04
3.55
14.04
14 =4
3.64
3.89
INPUT-TARGIT MIXUP
4.43
4.76gmm
43. 18.18
41 2223
3.41g
3.80
11.07a
11.74gr
2.79
3.00g
LABEL SMOOTHING
1.82
5.47a
02.08
38, 38.75
2.77
3.42
12. 22.
13.81
3.32
3.62
ACTIVATION GAUSSIAN
9.27 a.93
1.5
46.46
41.11
3.17
3.79
11.9
13.62
3.74
t.05
ACTIVATION DROPOUT
7.26
21. 26
23. 21.81=3mm Slmm
3.48
3.75
9.75
9.25
0.95
1.10gu
GRADENT GAUSSIAN
16.06
18.15
48.00 1
5.30 v
5.87
17.58a
18.03
2.81
3.13
MODEL
4 4.73
3.30
39 G6aas
37 27
2.80 2.80
3.40a
12. 35mm5
18.65
1.49zor
1.62
WEIGIT GAUSSIAN
9.56
9.362
48.77am
46.122.
3.48
3.90
12.27
13.49
2.36ggm
2.6
WEIGHT DROPCONNECT
2.66
3.26224
46.47a7o
44.012002
3. 3.
3. 3.96
12. 73
18.75
1.70amm
1.
1.92
TOP-2 DIRECT COMBINATION
1.58
4.19
32.77
32.
2.80.am
3.07
11.78
11.99.
1.39
1.50
TOP-3 DIRECT COMBINATION
3.77
3.95
15.99
16.13sse
2.65-1.
2.88
12.44
12.01
0.95
1.10
Tor-2
OPTIMISEn COMMINATION
5.01gmm
5.51gm
45.30ama
43.41
2.98=
3.69
10.68
11.72
1.255
1.43g
TOP-3 OPTIMISED COMBINATION
7.39210
7.20gt
38.88a00
36.78am1
3.08
3.62a
2.56g
2.70gt
1.172om
1.38
Table 20: Tabular data classification: ECE %) comparison on in-distribution (TD) and out-of-distribution (OOD) test sets and with tuned hyperparameters.
Wine
Toxicity
A Abalone
Students
Adult
Noise Type
OOD
OOD
OOD
ID
OOD
OOD
No NOIS
0,94
1.95
1655
1.87
0,8-iaw
1.86
180
1.98
0.35
0.36
INPUT GAUSSIAN
0.94
0,95
4.85
4.87
0,84
0.36
1.87
1.97
0.34
g.36
INPUT ODS
0.91
0.92g
2.20
2.35
0.94
0.85
1.87
1.98
0.35g
0.36gmm
INPUT- TAROST Mp
0.91
0.97 zom
2.3092e7
2.442x
0. 8.84
0. 0.86
1888
1 1.
0. 0.34
0.36
SMOOTHING
0.93..x
0,91
1.761es2
0.8
0.85
1.87
1.97.
0.34a
0.35
ACTIVATION GAUSSIAN
0,94-ax
0.96
4.78
4.82-1.3
0,84-ax
0,56
1.90
2 2.00-
0.35
0.36
ACTIVATION DRoPour
0.92
0.93
1.03gaa4
1.03gxx
0.84
0.85gmm
2.23
2:25
0.32gkm
0.32
GRADENT GAUSSIAN
1
1.30ggme
5.412e
5.65
0.86
0.88
1.96aers
2.05g
0. 0.34gm
0.30ggm
MODEL
0.92.ax
1,93
2.061ea7
2.14:ro
0,84
0. 0.85
1.87
1.98
0.31aam
0.32
WEIGHT GAUSSIAN
0,95
1.96
3.02
3.15-a1:
0.84-ax
0,56
1.91
2.01
0.33-am
i.34
WEIGHT CROFCON
0.94gau
0.94
4.82
4.80
0.84
0.85gmme
1.91 aasn
2.01
0.32gam
0.33
TOP-2 DIRECT COMBINATION
0.94
1,95
1.57
0.81
0.85
1.87
1.97
0.31
0.32 cou
ToP-3 DIRECT COMBINATION
0.94
0.95
0.93aes2
0.93
0.84
0.84
2.20
2.20
0.32
0.32
Top-2 OPTIMISRD COMMINATION
0.
0.93
2.47
2.50
0.84gmm
0.85gmm
1.862a
1.962
0.31gm
0.32gmm
TOP-3 OPTIMISED COMBINATION
0.96
0.96
2.05ae57
2.25
0,84g00
0.80azoe
2.54am
0.31gmm
0.322am
Table 21: Tabular data classification: NLL (4) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with tuned hyperparameters.
NewsGroup
SST
Noise Type
GP-CNN
Transformer
GP-CNN
Transformer
No NOISE
35.6 31.67+1.1
36.56+aas
19.15
21.60+0.24
INPUT GAUSSIAN
35. 40.57
36.59
24.
21.52+0021
INPUT ODS
23.56
34.44
18.462114
21.52
INPUT-TARGET MIXUP
35. go.
36. 36.70
28.11
21.33
LABEL SMOOTHING
35.59 20.m 20.m
36.56-11.
21.75
21,29
ACTIVATION GAUSSIAN
35.562116
36. 27
19.61 11:00
21.64
ACTIVATION DROPOUT
39.19 29
36.48.9m
19.61 2::35
21.021o.1
GRADIENT GAUSSIAN
40.56+0.r
36.52 +
21.9 21.90+a.a2
21.41+a.0
MODEL
40.22+0.a +0,04
36, 36752+077
19.57+0.2
21.67+0.2 +0.25
WEIGHT GAUSSIAN
35.48 20.53
36.89
20.1 8 10.19
21.561005
WEIGHT DROPCONNECT
35.19 15.11920.0 20.00
37 220ass
20. 15.5
21. 21.18
TOP-2 DIRECT COMBINATION
39.52+0.o
36, 36.41+a.. +
18. #1.00
20. 20.54
TOP-3 DIRECT COMBINATION
36.9620.7
34.67
17.5820.44
20.22 20.0
ToP-2 OPTIMISED COMBINATION
38.0441.22
36.52+1a0
20.41 10.50
21.0644..1 20.33
Tor-3 OPTIMISED COMBINATION
36.00+o.4s
35. 19.+a
18.85+t0.a
19.07 +0.01
Table 22: NewsGroup NLP classification: Error (J, %) comparison on in-distribution (ID) test set and with tuned hyperparameters.
28

---

Published in Transactions on Machine Learning Research (04/2024)
NewsGroup
SST
Noise Type
GP-CNN
Transformer
GP-CNN
Transformer
No NOISE
5.12
3.47
13.99 20.7
11.61
INPUT GAUSSIAN
4..8
3.59
15. 14
11.32
INPUT ODS
2.57 +0.81
+.99+am0
11.80+aaa
14.70+201
INPUT-TARGTT MIXUP
5.07+0.50
2.54+127
7.76+0.u
11.75+aa6
LABEL SMOOTHING
3.78 20:50
1.132am
9.10gest
10.261330
ACTIVATION GAUSSIAN
5.75
3.42
13.99
11.39
ACTIVATION DROPOUT
a.19
2.26++000
11.50
7.42
GRADIENT GAUSSIAN
24.26
3. 8.24
17.40
12.20
MODEL
2.91
3.55
24.00
11.161a.2
WEIGHT GAUSSIAN
4.57 +0.48
4.01+aae
+0.7
12.09 +3.05
WEIGHT DROPCONNECT
5.41+0.78
4.43+
16. +6.08+102
11.27
TOP-2 DIRECT COMPINATION
8.25
2.91
4.55gmm
6.4022.0
ToP-3 DIRECT COMRINATION
13.7610.7
11.191005
2.1220.7
8.701288
TOP-2 OPTIMISED COMBINATION
3.44
+.07+
8.866
8, 8.02+2.2 +2.12
TOP-3 OPTIMISED COMBINATION
2.75
2.89+
9.33
9.18
Table 23: NewsGroup NLP classífication: ECE (4, %) comparison on in-distribution (TD) test and with tuned hyperparameters.
NewsGroup
SST
Noise Type
GP-CNN
Transformer
GP-CNN
Transformer
No NOISE
1.14
1.132a0
0.81 1001
0.61 10:00
INPUT GAUSSIAN
1.12
1. 1.132
0.85
0.59
INPUT ODS
1.03
1. 10
0.62+
0.73+t012
SPPUT-TARGE MIXUP
1.13
1. 13
0.48
0.59
LABEL SMOOTHING
1.12
1. 1302
0.52
0.54
ACTIVATION GAUSSIAN
1.14
1.13
0.83
0.59
ACTIVATION DROPOUT
1.18
1. 1 Lov0
0.55
0.48
GRADIENT GAUSSIAN
1.87 20.07
1.13 1.13
1.11 10.00
0.60.m.e
MODEL
1.21+
1.1400
0,85+0022
0.59 +o.0s
WEIGHT GAUSSIAN
1.14+0.0
1,13+am2
0.84+0.02
0.62++o.1
WEIGHT DROPCONNECT
1.13
13
1.05
0,58 +0.07
Tor-2 DIRECT COMBINATION
1.19
1. 12mm
0.43
0.46.2o.0
Tor-3 DIRECT COMBINATION
1.18
1. 15250
0.402not
0.5 1 20.0
TOP-2 OPTIMISED COMBINATION
1.14
1. 1.12
2oot
0.49 20.02
ToP-3 OPTIMISED COMBINATION
1.09
1.10
0.48
0.50
Table 24: NewsGroup NLP classification: NLL (4) comparison on in-distribution (ID) test set and with tuned hyperparameters.
29

---

Published in Transactions on Machine Learning Research (04/2024)
Rotated CIFAR-100
FaceFace
Noise Type
OOD
OOD
No NOISE
0.03
0.13
0.03
0.04
INPUT WEAK AUG.
0.03
0.13
0.03
0.04
INPUT GAUSSIAN
0.03-ac0
0.14
0.04++0.0
0.05+omm
INPUT AUGMIX
0.03-ax0
0.07+
0.03
0.04+am0
INPUT-TARGET CMIXUP
2.03
0.09 10:00
0.04.1.0
0.04gmm
ACTIVATION GAUSSIAN
0.03
0.1 1:00t
0.04
0.04gam
ACTIVATION DROPOUT
0.03=n.co
0.14+00.0
0.04
0.05=0.00
GRADIENT GAUSSIAN
0.04=0.c
0.1 +0.00
0.04
0.00
MODEL
0.04=n.o
0.1622000
0.04
0. 0.04
WEIGHT GAUSSIAN
0.03=a.0
0. 5 t0.00
0.04-to.00
WEIGHT DROPCONNECT
0.03=acn
0.12 +0.01
0.10 +0.04
0, +0.04
TOP-2 DIRECT COMBINATION
0.03
0.08 10:00
0.04
0.04 0.04
ToP-3 DIRECT COMRINATION
0.03
0.08
0.04 1044...0
0.04gom0
TOP-2 OPTIMISED COMBINATION
0.03=aco
0. +6.0
0,03 0.03+
0. 04.00000
TOP-3 OPTIMISED COMBINATION
0.24
0.29
0.04
0.04
Table 25: Rotated CV regression: MSE (L) comparison on in-distribution (TD) and out-of-distributtion (OOD) test sets and with tuned hyperparameters.
Rotated CIFAR-100
WikiFace
Noise Type
OOD
OOD
No NOISE
-4.81
6.90
27.03
31.432
INPUT WEAK AUG.
-4
3.41
-0.8210a1
0.312om
INPUT GAUSSIAN
-4.67
7.57+214
27.96+2m
31.50=215
INPUT AUGMIX
-4
-1.70
0.78
-0. 02
INPUT-TARGET CMIXUP
-4.62-aos
1.86
21.83
25.622.05
ACTIVATION GAUSSIAN
-4.27
2.94
14.93
17.61
ACTIVATION DROPOUT
-3.81 2
1.04
-1.3510.0
-0.3310m
GRADIENT GAUSSIAN
-3.70
-0.44
25.87
29.69
MODEL
-4.66
3.44+n.0
-1.08
-1.08+cns
WEIGHT' GAUSSIAN
-4.23
2.53++.30
4.42++.10
5.65+0.0
WEIGHT DROPCONNECT
-2.28
39.88+2478
4.83+a.7
6. 6.00+418
Tor-2 DIRECT COMBINATION
-4.14=03
-1.93
-1.3-
1.152oo2
Tor-3 DIRECT COMBINATION
-4.21
-1.762no1
-1.32
-1. -1.162ooz
ToP-2 OPTIMISED COMBINATION
-4.27
-1.71 incs
-1.394mar
-1.124o02
ToP-3 OPTIMISED COMBINATION
1.84
0.57
-1.34
-1.07
Table 26: Rotated CV regression: NLL (4) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with tuned hyperparameters.
30

---

Published in Transactions on Machine Learning Research (04/2024)
Energy
Boston
Wine
Yacht
Coucrete
Noise Type
OOD
OOD
No NOISE
0.04
0.05
29. 42
a.07
0.16
0.10
0.10gmm
INPUT GAUSSIAN
1.04
0.05 Loom
0.11
0.13
0.25
29. 39
0.02 100z
1.09
0.10
0.10:an
TARCIEE Cp
0.04
0.00
0.13
0. Danan
0. 2.
28. 26
0.15 0.152a17
0.19
0.10
0.11amm
ACTIVATION GAUSSIAN
0.04
0.05
29. aour
g.03
0.
0.10
0.10gmm
ACTIVATION DROPOUT
0.04gmm
0.05amm
0.1 0.16
0.
0.36
17. 17.02
2.03
0.09
0.10
0.11gmm
GRADIENT GAUSSIAN
0.04
a.04
0. g.
0. aa
0.25
28. 34 gacin
0.01 zum
0.09
0.10
0.10g
MODEL
0.05
0.15 Loaz
0,25 1ass
29. 11 asve
0.5-1
0.10
0.11
WEIGHT GAUSSIAN
0.04gmm
0.05
0.12am
0.26
29.
0.02
0.06anm
0.10
0.10gam
WEIGHT DROPCONNECT
0.04g
p.05
0.13
0.1 Baiaa
20.53ga..
g.05
0.07
0.10gmm
0.10gmm
Top-2 DIRECT
0.05 cooe
+.13
0.18 1oa
0.25
29. -asa
0.03
0.10
0.10cam
0.10:an
TOP-3
0.13
g.25
29. saaee
0.04
0.09
0.10pmm
Top-2 OPTIMIED
1.0
0.05 1000
1.13
0.16
0.25 24
29. 10.35.ss
0.01
0.07
0.10
0.10:an
ToP-3 OPTIMISD
0.04gmm
0.00gmm
0.144gm
0.1 72.7
0. 1.25
29.31 am
0.04gna:
0.10
0. 0.10gmm
0.11amm
Table 27: Tabular regression: MSE (4) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with tuned hyperparameters.
Energy
Wine
Concrete
Noise Type
OOD
OOD
OOD
No
6.64
-..9
0.04-a12
-0.22
0,03-a12
-1.18-1m
10.45-ma
-0.56
-0.01
INPUT GAUSSIAN
-1.552em
6.55
-0.54
-0.40amm
-0.22
0.03
-1.66am
-1.04amm
-0.53
0.21
INPUT- TARGRT CMixUP
-1.53
6.03
-0.422
-0.20
19
81.98
-1.05gm
-0.87
-0.63
0.26
ACTIVATION GAUSSIAN
-1.56
6.64
-0.20eir
0.04-mm
-0.22
0.04-aa
-1.81emm
-1.21
-0.56
0.08
ACTIVATION DROPOUT
-1.53
4.44
-0.62em1
-0.59
0.01
0.03
-1.18211
-1.07
-0.58
0.07
GRADIETT GAUSSIAN
-1,56
8.61 ram
0.55
0.80
-0,22 iua
0.01:am
-2.01-1m
-1.49
-0.55
0.13:a0
MODEL
-1.532om
7.20
-0.16am
0.06
-0.22ama
0.03
-0.77a:m
0.152mm
-0.592wm
WEIGHT GAUSSIAN
-1.56gmm
6.
-0.36
-0.1 10em
-0.22amm
0.04gm
-2.08
-1.71
-0.54gmm
0.14gm
WEIGHT DROPCONNECT
-1.552em
m.
-0.022rm
-0.49gmm
-0.22
0.02gam
-1.3621n
-1.252a
-0.57
0.07
TOP-2 DIRECT
-1.55
6.09
0.32g
0.63
-0.22
0.02
-1.37
0.35
-0.57
0.11
DIRECT
-1.33
1,312a
-0.18
-0.22
0.03
-1.82
-1.39tam
-0.59
Tor-2 OPTIMIEDD
-1.562em
8. 8.60ass:
0.16
0.46
-0.22
0,03
-2.03
-1.56aam
-0.57
0. 0.22
Top-3 OPTIMIEED
-1.55
6.68
-0.27
-0.041eas
-0.22
0.03
-1.77
-0.101112
-0.60
0.17aam
Table 28: Tabular regression: NLL (1) comparison on in-distribution (ID) and dut-of-iistributio (OOD) test sets and with tuned hyperparameters.
Energy
Wine
Concrete
Noise Type
ID
OOD
ID
OOD
ID
OOD
NO NOISE
0.03=
0.03+a.0
0.21+
37.20+81.2
0.09
0.10+a.0
INPUT GAUSSIAN
0.03=
0.03+amm
0.21 2002
45.36=4145
0.08=aa
0.09 aa.0:
INPUT-TARGET CMiXUP
0.03=2000
0.04 0.01
0.18
19.23
0.10=aa1
0. 1.11iam.
ACTIVATION GAUSSIAN
0.03
0.03 0.03 zo.0
0.22
43.40=
0.09
0.10
ACTIVATION DROPOUT
2.03
0. 0.03 20.00
0.24
42. 6
0.09
O. 0.10.a.0
GRADIENT GAUSSIAN
0.03
0. 2.04
311
1858.
44
0.09
O. 0.10...02
MODEL
0.03+aa0
0. +o.o
0.21 +a00
37.65 s2.88
0. 0.11 1
0. +a.0.
WEIGHT GAUSSIAN
0.03+nx0
0.03+aa
0.20+ax0
41.60+wam
0.09
0.10+
WEIGHT DROPCONNECT
0.03+na0
0.03+amm
0.20+
46.44+
0.09
0.10+a.0
Table 29: Tabular regression: MSE (4) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across datasets.
31

---

Published in Transactions on Machine Learning Research (04/2024)
Energy
Wine
Concrete
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
1020
1.24
4.78
6.6621.2
-0.54
0.32
INPUT GAUSSIAN
-1.74
1.06
5.24
19.
-0.40
0.23
INPUT-TARGES CMiXUP
-1.6620t0
0.42
10.06
95014.61
-0.63
0.53
ACTIVATION GAUSSIAN
-1.71+ao0
0,51+
1.84+122
177076.
4976..994 12
-0.50+m.14
0.46++.74
ACTIVATION DROPOUT
11.66+a12
0.02 +.55
-0.27+a.05
-0. 19.0
-0.56+0.11
1,19+2.0
GRADIENT GAUSSIAN
-1.70go11
5.71 10.20
1.97
2. 2.06 23
-0.11
0.63
MODEL
- -1.642002
-0.1620m1
4.68
6.22
-0.64
-0.492017
WEIGHT GAUSSIAN
1..71
-0.26+n.3
4.51=1.08
206785.
200088
-0.55
-0. 0..03
WEIGHT DROPCONNECT
1. 1073
0.00
0.37=044
777100.06+aaa0
-0.67
-0.56...1
Table 30: Tabular regression: NLL (4) comparison in-distribution (TD) and out-of-distributioo (OOD)
test sets and with hyperparameters transferred across datasets.
Boston
Yacht
Concrete
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
0.13
0.16+a04
0.56+a.4
0.59
0.1 0.11+oma
0.11=000
INPUT GAUSSIAN
+a.0s
0.15+tna
0.55+c77
0.59+a.2
+.10
0.11-oa1
INPUT-TARGER CMixUP
0.501a.47
0.52
0.05
0.1 10 06
0.1 10:00
0.12_me1
ACTIVATION GAUSSIAN
0.1 1 1000
0. 3a0e
0.5616.7
0.
0. 11 10:0
0.1 =001
ACTIVATION DROPOUT
0.17+0.07
0.18
0. +.05
0.08=
0. +0.0
0.11=901
GRADIENT GAUSSIAN
0.16+0.04
0. 18
0.02+c.m
0.06
0. 1 +0.02
0.11=0.00
MODEL
0.15+0.05
10.000
+0.00
0.13=2000
0. 12 +0.00
0. 02
WEIGIT GAUSSIAN
t.12
0. 1 20o0
t.03
0.07 20022
0. 1 toons
0.1 10c0
WEIGHT DROPCONNECT
0.13
0.04
0.10 20o0
0. to.m
0.11=aa1
Table 31: Tabular regression: MSE (4) comparison on in-distribution (ID) and out-of-distribution (OOD)
test sets and with hyperparameters transferred across architectures.
Boston
Yacht
Concrete
Noise Type
ID
OOD
ID
OOD
ID
OOD
No NOISE
-0.251om
-0.01
-0.422oms
-0.0220.0
-0.57 10.0s
-0.02gaat
INPUT GAUSSIAN -0.35++.1
-0.17=0.0
-0.94
-0.72+100
-0.58 +0.0s
0.06+048
INPUT- INPUT-TARGET CMIXUP
-0.09
-0.02=0.z
11200
-1.06=a11
-0.60 +0.60
-0.25+a.a2
ACTIVATION GAUSSIAN
1820ms
0.02
-0.44
-0.09
-0.56.2o.7
-0.0920.0
ACTIVATION DROPOUT
0.562om
-0.54
-1. -1.11 11mm
-0.81 2nn
-0.58 10.07
0.08
GRADIENT GAUSSIAN
0.97
1.22-1..
-1.52+0..5
-1.25
-0.57 to.0s
-0.02
MODEL
-0.1
-0.1020.1
-0.37
-0.562om
-0.58
-0.33
WEIGHT GAUSSIAN
0..42a115
-0.28201
-1.622oar
-0.77
-0.564..7
0. 12
WEIGHT DROFCONNECT
0.466+mm
-0.34+a.0
-1.72+o.1
-1.20+020
-0.55
0.07
Table 32: Tabular regression: NLL (4) comparison on in-distribution (ID) and out-of-distribution (OOD) test sets and with hyperparameters transferred across architectures.
32

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on (d) Error and ECE on OOD,
Figure 9: Input Random Crop, Horizontal Flip on CIFAR-10. Observations: Did not change the smoothness of the 1D curves the 2D metric landscape trajectory compared no noise.
cen
(a) Error and ECE.
(b) Error NLL.
(c) Error and ECE on (d) Error and ECE on OOD.
Figure 10: Input Additive Gaussian on CIFAR-10. Observations: Changed the smoothness the 1D curves where NLL became less smooth and removed the bumps ECE for a approaching the initial model. The 2D metric landscape trajectory did not change in comparison to no noise.
call
CUM
Gun wncte
Senw
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 11: Input ODS on CIFAR-10. Obsercalions: Marginally changed the smoothness of the 1D curves. The 2D metric landscape trajectory did not change in comparison to noise.
Call
CUU
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 12: Targe-Target MixUp on CIFAR-10. Observalions: Both the NLL and ECE 1D curves changed in comparison to noise, and the 2D plots to explore wider valleys compared to noise.
33

---

Published in Transactions on Machine Learning Research (04/2024)
000
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 13: Target Smoothing on CIFAR-10. Observations: The NLL became more aligned with the error, not the ECE. The 2D plots show slightly more variation in the trajectory than no noise.
Dnw 36
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE OOD.
Figure 14: Activation Additive Gaussian on CIFAR-10. Observations: Did not change the smoothness of the 1D curves or the 2D metric landscape trajectory compared to no noise.
D00
Sonw P1
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and BCE OOD.
Figure 15: Activation Dropout on CIFAR-10. Observations: Dropout narrowed the gap between ID and OOD results; nevertheless, the shape of the 1D curves is similar to no noise. The trajectories in 2D plots did not seem to converge into a narrow local minimum.
OVED
30
DTO
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 16: Gradient Gaussian on CIFAR-10. Observations: The 1D and 2D figures changed curvature and
shape drastically, and NLL and ECE follow a non-linear pattern. The 2D plots show a circular curvature, perhaps suggesting difficulty in convergence.
34

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on (d) Error and ECE on OOD.
Figure 17: Model Shrink and Perturb on CIFAR-10. Observations: The 1D and 2D figures changed curvature and shape drastically, and all metrics show non-linear optimisation path hypothesised. The point cluster around centres created by shrinking and perturbing the weights.
Soo
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE ID. (d) Error and ECE OOD.
Figure 18: Weight Additive Gaussian CIFAR-10. Observations: The 1D curves marginally changed their shape. However, the difference between ID and OOD metrics became more profound. The 2D plots suggest that the optimisation was not able to converge.
000
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 19: Weight DropConnect on CIFAR-10. Observations: Did not change the smoothness of the 1D
curves or the 2D metric landscape trajectory compared to no noise.
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE OOD.
Figure 20: No noise Adult.
35

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and BCE OOD.
Figure 21: Input Additive Gaussian Adult. Obsercations: Did not change the smoothness of the ID curves or the 2D metric landscape trajectory compared to noise.
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE ID. (d) Error and ECE OOD.
Figure 22: Input ODS on Adult. Observations: Did not change the smoothness of the ID curves or the 2D metric landscape trajectory compared to no noise
(a) Erro and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and BCE OOD.
Figure 23: Input-Targel MixUp Adult. Observations: Did not change the smoothness of the 1D curves or the 2D metric landscape trajectory compared to no noise.
10
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and BCE OOD.
Figure 24: Target Smoothing on Adult. Obsercations: Did not change the smoothness of the 1D curves the 2D metric landscape trajectory compared to no noise.
36

---

Published in Transactions on Machine Learning Research (04/2024)
- 040
W
COD
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 25: Activation Additive Gaussian on Adult. Observations: Did not change the smoothness of the ID curves or the 2D metric landscape trajectory compared to no noise.
+d- 000
C0D
000
CC
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD,
Figure 26: Activation Dropout on Adult. Observations: Changed the ECE curvature and made the NLL plots smoother in the 1D case. In the 2D plots, the ECE and error appear aligued during optimisation. The curvature of the 2D plots has changed and there is a higher alignment between the ECE and error.
COI
1850
X
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 27: Gradient Gaussian on Adult. Observations: Did not change the smoothness of the 1D curves,
but the 2D trajectory appears more exploratory compared no noise.
000
GOD
000
000
1
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 28: Model Shrink and Perturb on Adult. Observations: Did not change the smoothness of the 1D
curves, but the 2D trajectory appears more exploratory compared no noise.
37

---

Published in Transactions on Machine Learning Research (04/2024)
(00
CO0
(0 180 000
000
13
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD.
Figure 29: Weight Additive Gaussian on Adult. Observations: Did not change the smoothness of the ID
curves or the 2D metric landscape trajectory compared to no noise.
1975 <e +
(a) Error and ECE.
(b) Error and NLL.
(c) Error and ECE on ID. (d) Error and ECE on OOD,
Figure 30: Weight DropConnect on Adult. Observations: Changed the ECE curvature and made the NLL and ECE plots smoother in the 1D case. In the 2D plots, the ECE and error appear aligned during optimisation.
-26 ID
Stact willo
Ead vilgo
Stxct vicati
Eal vilate
00032
0007
mrDAL
DANSS
0003
0007
545
00064
Shaet
485
mrea
orea
0re37
crear
10044
00084
18.0
112
012
000
010
E.003
Ex000
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 31: Input Additive Gaussian on WikiFace. Observations: Did not change the smoothness of the ID curves, or the 2D trajectory.
38

---

Published in Transactions on Machine Learning Research (04/2024)
Stan redi
Red vilgoe
0100
010
0.00
0.7
0.00
0/01
D000
00112
0110
60020
E001a
010
(a) MSE and NLT.
(b) MSE and NLL ID.
(c) MSE and NL. OOD.
Figure 32: Input Random Crop, Horizontal Flip on WikiFace. Observations: Surprisingly, the NLL starts decreasing compared to MSE as the model is interpolated between the final and the initial model in the ID plots. The 2D plots demonstrate that the model was able to explore a deeper optimal from the start where NLL was slower to converge than MSE.
-de- OOD
No. TD
CO0
Ziam wish
Kod wets
Zhet wigh
End velgre
0.30
83030
0.033
0003:
0002
0.88
DOOX
0.87
O
0.110
0.00
0.82
00000
-0.6
-000S
(a) MSE and NLL.
(b) MSE and NLL ID.
(c) MSE and NLL OOD.
Figure 33: Input AugMix WikiFace. Obsermations: Surprisingly, the NLL starts decreasing compared to MSE as the model is interpolated between the final and the initial model in the ID plots. The 2D plots demonstrate that the model was able to explore a deeper optimal from the start where NLL was slower to converge than MSE, and it did not converge in the optima from the perspective of NLL.
101
DO0
-36 ID
OGD
Siee widde
Ral wripte
2.0
8:00
0310
6386
tuse
2.00
0.00
-2/02: 1000
-DUKSS naaic 0.822 CACOD
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 34: Imput-Targe CMixUp on WikiFace. Observations: Did not change the smoothness of the 1D curves, or the 2D trajectory appears more exploratory compared to no noise.
39

---

Published in Transactions on Machine Learning Research (04/2024)
Start midgle
Red miglt
midde
110
21.5
CABRO
0.16
203
2.0
DU0525
126
600G
000770
Ti.
0.000
CABR25
0.800
4.
015
022
000
110
tuee 3.02 D004 0.000 DUCS 0.810
MUKO 3.000 OUK0 EIMM 0.003 0.000
CUUTB
(a) MSE NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 35: Activation Additive Gaussian on WikiFace. Observations: Did change the smoothness of the 1D curvos, but the 2D trajectory appears exploratory compared to noisc
Start wedde
Eed wadex
Star widee
Dad mnidee
110
0.82
0.030
0.30
3030
210
0.32
2.7
0.
0.32
325
230
0.0
0.00
3.000
0.2
012
06
-5068 -0000
-5464 0A0C
0.30
-5.00 -0000
336 -1462 C,00 0.30
(a) MSE NLT.
(b) MSE and NLL on
(c) MSE and NL. on OOD.
Figure 36: Gradient Gaussian on WikiFace. Observations: Did not change the smoothness of the 1D curves, but the 2D trajectory appears to align MSE and NLL. However, it seems that the optimisation missed local minimum during training.
Surt wights
Fril wigles
UP
Scar wieks
Fai wigles
DP
0.3
OGD
000
24
24
0.34
200
220 207TR
280
210
CID
Mon.
80
3.00
100
03
6.3
0.00
(a) MSE and NLL.
(b) MSE and NLL ID.
(c) MSE and NLLL OOD.
Figure 37: Model Shrink and Perturb WikiFace. Observations: Due to shrinking and perturbation, the experiment appears to converge in narrow basin and as seed in the 1D plots, the optimisation completely non-linear and unrecoverable.
40

---

Published in Transactions on Machine Learning Research (04/2024)
113
000
-20
000
medio
Rat wingle
viiilee
1.0
S1ON)
0108
D0099
1.6
0012
00090
00037
-000
3.4
-005
00051
BA072
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 38: Weight Additive Gaussian on WikiFace. Observations: The 1D curves look similar to no noise, although with respoct to a different scale for NLL. The 2D plots explore a similar trajectory to no noise: however, the 2D landscape appears more distorted.
1.10
1
000
-35
ID
OOD
0008
madia
Rad viggee
1542
madii
Eat volple
BIGAL
0011
DOVT2
BAVIO
DUNTO
EORR
086-69
1.0466
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 39: Weight DropConnect on WikiFace. Observations: The 1D curves look similar to no noise, although with respect to a different scale for NLL. The 2D plots explore similar trajectory to no noise.
veght
Invivggo
06
283
0.5
80
17-1
61
-01
0.
R12
1.3
(a) MSF NL..
(b) MSE and NLL on
(c) MSF and NL. on OOD.
Figure 40: No noise on Yacht.
41

---

Published in Transactions on Machine Learning Research (04/2024)
-de- (00)
010
06
0.5
0.3
08
(a) MSE and NLL.
(b) MSE and NLL ID.
(c) MSE and NLL OOD.
Figure 41: Input Additive Gaussian Yacht. Obsercations: While the shape of the 1D curves looks similar to no noise, the MSE and NLL magnitudes are different. The OOD NLL is substantially higher than the OOD NLL for no noise. The 2D plots demonstrate a wider landscape of feasible solutions than no noise.
-30-
DOD
230
1.4
3.2
1:15
00
01
0.2
085
3.5
10
0100
1.30
0100
6.500
(a) MSE NLL.
(b) MSE and NLL on
(c) MSE and NLL on OOD.
Figure 42: Input- Target CMixUp on Yacht. Observations: While the shape of the 1D curves looks similar The 2D plots demonstrate wider landscape of
to noise, the MSE and NLL. magnitudes different. feasible solutions than no noise.
Eallo
Vegio
0.3
02
(a) MSE and NLL.
(b) MSE and NLL ID.
(c) MSE and NLL OOD.
Figure 43: Activation Additive Gaussian Yacht. Observations: While the shape of the 1D curves looks similar to noise, the MSE and NLL magnitudes are different. The 2D plots close to the no-noise ones, showing marginal differences.
42

---

Published in Transactions on Machine Learning Research (04/2024)
604
Squand Erer
06-
80
(a) MSE and NLL.
(b) MSE and NLL on
(c) MSE and NLL on OOD.
Figure 44: Activation Dropout on Yacht. Observations: The 1D curves look similar to no noise but Dropout converged in narrow valley, demonstrated in the 2D plots.
(00
000
724
0.5
0.4
0.4
NLLL
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 45: Gradient Gaussian on Yacht. Observalions: The 1D curves remained unchanged except for the magnitude of NLL or MSE. Nevertheless, the 2D plots show us that the optimisation trajectory significantly differed from no noise where the landscape of potential optimal solutions was wider.
150000
50000
1.0
5
00
66
86
300000
200000
2100
33000
28000
115
(a) MSE NLL.
(b) MSE and NLL on
(c) MSE and NLL on OOD.
Figure 46: Model Shrink and Perturb on Yacht. Observations: The model jumped between narrow valleys seed in the 2D plots and the 1D plots show smoother behaviour from the OOD perspective for MSE but not NLL.
43

---

Published in Transactions on Machine Learning Research (04/2024)
-A- 000
04
Hoa Spatod fine
144
Noa Eie
X
61
3
022
160
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 47: Weight Additive Gaussian on Yacht. Observations: Did not change the smoothness of the 1D curves or the 2D trajectory.
0GD
5 X
0.4
04
More Spaind Banee
04
3.
NOE
T
01
12
00
0.0
(a) MSE and NLL.
(b) MSE and NLL on ID.
(c) MSE and NLL on OOD.
Figure 48: Weight DropConnect on Yacht. Observations: While the shape the 1D curves looks similar no noise, the MSE and NLL. magnitudes are different.. The 2D plots demonstrate wider landscape of feasible solutions than no noise.
44


Reviews:
[{"review_id": "iCh8lEcU90", "reviewer": "Reviewer_bpxd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The study has an interesting research aim, which aims to systematically investigate the effects of diverse noise injections on NN generalization and calibration across multiple datasets, tasks, and architectures. This problem is largely understudied. \n+ The study draws specific conclusion on the practical usage of the noises, giving suggestions on how to use the model in practice. \n\n\nWeakness\n\n- While the ideal scope of this study is admirable, as the authors admit themselves, a practical study need to limit the study scope due to many practical limitations. For example, a limited opitimization scope that is tuned before hand, while different techniques are will likely benefit from different parameters, tuning strategies etc. Thus, the conclusions are less likely to be meaningful as in practice, because in practice, people who choose to use this method for a specific application is likely to tune heavily. \n\n- The conclusion that AugMix is the most promising is no surprising and likely misleading. The OOD datasets the authors choose to evaluate is where AugMix is defined for. In fact, there are many other datasets that AugMix is no longer the best performing one. \n\n- There are also more powerful augmentations, such as DeepAug, that are not compared by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the impact of various noise injection techniques on the generalization and calibration of neural networks (NNs) across different tasks, datasets, and architectures. The study reveals that methods like AugMix, weak augmentation, and Gaussian noise enhance NN performance diversely, highlighting the importance of task-specific noise application. The research shows that combining different noises can be more effective than using them individually, particularly in classification tasks, while regression might benefit from a singular noise approach. Despite variations in effectiveness across tasks, the study confirms that noise injections contribute to improved performance both in-distribution (ID) and out-of-distribution (OOD), with notable differences in their efficiency across different tasks and datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Recent studies have shown the limitation of evaluating a model with fixed benchmark [1], and as an example, AugMix, which is primarily designed for the noises it's introduced for, shows weaker performances when the evaluation is more dynamic. I will recommend the authors to consider such evaluation protocol for a more dynamic evaluation process. \n\n2. more powerful methods should probably be in the discussion, e.g., DeepAug [2], there are many other follow-ups also, but probably less prominent. \n\n\n[1] Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models. \n[2] The many faces of robustness: A critical analysis of out-of-distribution generalization", "broader_impact_concerns": "none noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fauZgJ55LB", "reviewer": "Reviewer_EoNq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper's experiments cover quite a range of \"noise\" methods spanning different types (input, labels, weights, gradients), and are evaluated over several domains and datasets from each domain. Hyper-parameter tuning is also done. Overall it is quite an effort to perform all the experiments.\n- Investigation of transferability of hyperparameters is something quite useful for practitioners, though it would be more interesting to see whether there is a default set of hyperparameters that generally performs well, which could then serve as a starting point for further tuning.\n- The submission includes code to reproduce the results, which is to be commended.\n\nWeaknesses:\n- The OOD evaluation for tabular data does not seem well-justified. Zeroing out features and adding noise could change the labels, unlike the common corruptions used for image data. More broadly, it is advised to use real-world distribution shifted data for a more realistic evaluation (e.g. WILDS - https://wilds.stanford.edu/datasets/). There is also no OOD analysis provided for the text-domain.\n- Some of the datasets/tasks used in the experiments seem rather \"toy\", such that the conclusions obtained from these may not generalize in practice. In particular, the image regression task of rotation prediction and the NewsGroup text classification task; in addition for the text-domain only a single dataset is used.\n- When combining different noises, using the top-k noise combinations requires knowing which of them is the top (i.e., tuning on all methods chosen) - how does this compare to doing a search over all combinations of methods but with the same tuning budget (# methods * budget per method)?\n- It is unclear why there is an emphasis on simultaneously optimizing for generalization and calibration as calibration can be done post-hoc using various methods (e.g. temperature scaling).\n\nOther comments:\n- It is arguable whether data augmentation methods should be termed as \"noise\" or perturbations.\n- Would be useful to explain how the methods used in the evaluation were selected\n- Clarify the testing protocol - what train/validation/test splits were used? \n- Some of the stated conclusions are not very surprising (e.g. \"Noise injection methods can improve NN performance across various tasks and datasets\" - this would be shown when the methods were published in the first place; \"AugMix remained highly ranked for robustness\" - AugMix was specifically designed for robustness)\n\nOverall, the paper presents results from many experiments, but is largely descriptive and would likely benefit from being more focused on particular domains/tasks to provide more generalizable conclusions and to provide a more complete picture (e.g. lack of OOD analysis and only a single dataset for text data); more analysis/insight into the differences between methods would also enrich the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs an extensive evaluation of an array of data augmentation and regularization techniques (termed as \"noise\" in the paper) on multiple datasets in the image, text and tabular data domains, on classification and regression tasks, in both in-domain and out-of-distribution settings. Both prediction performance and calibration are evaluated. Based on these results, the paper makes several general observations, for instance, that adding \"noise\" helps for the most part except on the tested image regression tasks and that combining multiple \"noises\" further improves performance. The relative ranking of the methods could be of interest to a practitioner.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the testing protocol - what data splits were used \n- Perform more realistic OOD evaluation at least for tabular data, if not for all domains\n- Clarify the motivation for simultaneously optimizing generalization and calibration\n- Include additional datasets/tasks for text (e.g. sentiment analysis) \n- Include more realistic regression tasks on image data (e.g. facial keypoint detection) (optional)\n- Explain how the noise methods included were selected (optional)\n- Provide further insight into why the different methods perform differently (optional)", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cEscj4PWIV", "reviewer": "Reviewer_QK5J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The motivation of this paper is strong. \n- The writing and organization of this paper are overall good. \n- Experimental results are overall comprehensive. \n\n**Weaknesses**\n- The technical details of the proposed method are not clear. \n- There are some unclear descriptions, explanations, and justifications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Enhancing the generalization capabilities of neural networks (NNs) by incorporating noise, such as MixUp or Dropout, during training has become a potent and adaptable technique. Despite the established effectiveness of noise in NN training, a consensus is lacking regarding the optimal sources, types, and placements of noise that yield maximum benefits in generalization and confidence calibration. This study extensively explores diverse noise modalities to assess their impacts on NN generalization and calibration in both in-distribution and out-of-distribution settings. Additionally, experiments delve into the metric landscapes of the learned representations across various NN architectures, tasks, and datasets.\n\nThe research findings indicate that AugMix and weak augmentation demonstrate effectiveness across different tasks in computer vision, underscoring the importance of tailoring noise to specific domains. While it highlights the success of combining different noise sources and achieving hyperparameter transfer within a single domain, challenges arise in extending these benefits to other domains. Furthermore, the study emphasizes the intricacy of concurrently optimizing for both generalization and calibration, stressing the importance for practitioners to thoughtfully consider noise combinations and hyperparameter tuning to achieve optimal performance in specific tasks and datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In the “Introduction”, this paper should give more details about the definition of noise. \n- It is somewhat confusing about the target noise such as label smoothing, as it seems that this way does not introduce noise. Also, it would be better if the paper could add some discussions about label noise [R1-R2] as target noise.\n- It seems that there is an overlap between different types of noise. For example, input-target noise includes target noise. A more accurate classification would be better. \n- Could the different noise produce conflicts that finally hurt the model generalization? \n- The green color is not clear in the algorithm flow. \n- It seems that the proposed method of this paper sometimes does not work well. However, there is no clear description of this phenomenon. More details about this are expected to be provided. \n\n----\n[R1] Are Anchor Points Really Indispensable in Label-noise Learning? NeurIPS 2019.      \n[R2] Robust Early-learning: Hindering the Memorization of Noisy Labels. ICLR 2021.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has mixed opinions. Most negative opinions are from the fundamental limitations of the study. In my opinion, two claims can be critical "AugMix noises are exactly defined on the target OOD dataset (which is hard to say "OOD")" and "the evaluation is conducted on the fixed benchmark". One possible direction to resolve this concern is an additional OOD experiment on completely unknown distribution shifts. For example, one can use adversarial OOD samples [1] or the NINCO (No ImageNet Class Objects) dataset [2] for a fairer evaluation. I suggest to examine these datasets (especially [2]) for enhancing the results.

- [1] NeurIPS 2020. Certifiably Adversarially Robust Detection of Out-of-Distribution Data https://github.com/j-cb/GOOD
- [2] ICML 2023. In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation https://github.com/j-cb/NINCO

Note that NINCO [2] is only for ImageNet-1k classifiers. Hence, it would be better to resize NINCO images and evaluate them using TinyImageNet classifiers. I also clarify that it is still okay if the NINCO results (or GOOD results) are not perfectly aligned with the original results, because there are other experiments (e.g., Tabular, NLP classification and Tabular regression) that show consistent results. Plus, I think it is an optional experiment to resolve the concerns raised by the reviewers.

Another concern from me (not raised by the reviewers) is that similar studies with similar settings already exist. In my opinion, it is better to cite the previous studies and add a discussion of the difference between the previous studies and this study. In my point of view, using various datasets is a valuable contribution compared to the previous studies.

- [3] NeurIPS 2018. Generalisation in humans and deep neural networks
- [4] ICML workshop 2019. Transfer of Adversarial Robustness Between Perturbation Types
- [5] ICML workshop 2019. An Empirical Evaluation on Robustness and Uncertainty of Regularization methods
- [6] NeurIPS 2020. Measuring robustness to natural distribution shifts in image classification

In summary, I suggest to the authors revise the paper by (1) adding a discussion of the difference between the previous studies and this study and (2) considering to add more OOD experiments to resolve the concerns regarding to AugMix. I think these changes will be minor modifications that can be sufficiently handled by a minor revision.
Decision: Accept (poster)