Predicting Out-of-Domain Generalization with Neighborhood Invariance
Authors: Nathan Hoyen Ng
Neha Hulkund
Kyunghyun Cho
Marzyeh Ghassemi
Venue: Accepted by TMLR ()

Abstract: Developing and deploying machine learning models safely depends on the ability to char- acterize and compare their abilities to generalize to new environments. Although recent work has proposed a variety of methods that can directly predict or theoretically bound the generalization capacity of a model, they rely on strong assumptions such as matching train/test distributions and access to model gradients. In order to characterize generalization when these assumptions are not satisfied, we propose neighborhood invariance, a measure of a classifier’s output invariance in a local transformation neighborhood. Specifically, we sample a set of transformations and given an input test point, calculate the invariance as the largest fraction of transformed points classified into the same class. Crucially, our measure is simple to calculate, does not depend on the test point’s true label, makes no assumptions about the data distribution or model, and can be applied even in out-of-domain (OOD) settings where existing methods cannot, requiring only selecting a set of appropriate data transformations. In experiments on robustness benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our neighborhood invariance measure and actual OOD generalization on over 4,600 models evaluated on over 100 train/test domain pairs.


Published in Transactions on Machine Learning Research (06/2023)
Predicting Out-of-Domain Generalization with Neighborhood Invariance
Nathan Ng University of Toronto
nothanng@mit..cm edu
Vector Institute
Massachusetts Institute of Technology
Neha Hulkund
ahalkaunddddd edu
Massachusetts Institute of Technology
Kyunghyun Cho
kyunghyun.
cho cho Dryu edu
New York University
Prescient Design, Genentech
CIFAR Fellow
Marzyeh Ghassemi
mghassem@mitt @mit edu
Massachusells Institute of Tecknology
CIFAR AI Chair
Vector Institute
Reviewed on OpenReview: https: // openreview. net/ forum? id= VokkdJZTmm
Abstract
Developing and deploying machine learning models safely depends on the ability to char- acterize and compare their abilitics to generalize to new environments. Although recent work has proposed variely of methods that can directly predict or theoretically bound the generalization capacity of model, they rely on strong assumptions such as matching train/ted /est distributions and access model gradients. In order characterize generalization when these assumptions are not satisfied, propose neighborhood invariance, measure of a classifier's output invariance in a local transformation neighborhood. Specifically, we sample set of transformations and given input test point, calculate the invariance the largest fraction of transformed points classified into the same class. Crucially, our measure is simple to calculate, does not depend on the test point's true label, makes no assumptions about the data distribution or model, and can be applied even out-of-domain (OOD) settings where existing methods cannot, requiring only selecting set of appropriate data transformations. In experiments robustness benchmarks in image classification, sentiment analysis, and natural language inference, demonstrate strong and robust correlation between our neighborhood invariance measure and actual OOD generalization on over 4,600 models evaluated on over 100 unique train/ test domain pairs.
1 Introduction
As deep neural networks find increasing use in safety-critical domains such as autonomous driving (Gupta et al. 2021) and healthcare (Wiens el al. 2019). it is important to develop methods to undersland and compare how these models generalize to new environments. Although empirically these models generalize in many sottings [Hendryeks et al. 2020al Allen-Zhi
al.) 2018 Neyshabur et al., [2017a] they also exhibit
numerous failure cases. For example, models have been shown overfit to dataset's meta characteristies
Recht al.] 2019) arbitrarily corrupted labels (Zhang et al. 2016); learn spurious correlations [Liang

---

Published in Transactions on Machine Learning Research (06/2023)
3
3
3
Gylar)
(,(x))
Less Invariant Classifier
More Invariant Classifier
Figure 1: The transformation neighborhood (-(x) around x contains the set of points reachable from a transformation in 9, (pictured here as a rotation of r degrees). As increases, the transformation neighborhood is partitioned into a distinet set of decision regions. More invariant classifiers (right) classify most points in the neighborhood into the same class and should generalize better compared to less invariant classifiers (left). Our measure of invariance is independent of what specific region a lies in.
& Zou 2022 and change their predictions even with small adversarial perturbations (Goodfellow al.) 2014 Papernol et al.] 2017). Many methods have been proposed to miligate these issues, but precisely characterizing the generalization properties of model in diverse settings remains open problem.
One line of work aims to theoretically bound generalization capacity (Vapnik & Chervonenkis, 1971: Bartlett & Mendelson 2003 McAllester 1999 Neyshabur et al.]
Dziugaite & Roy] 2017: Neysha bur el al. 2015b) or directly predict generalization (Keskar et al. 2016 Liang et al. 2019 Neyshabur et al.) 201 5a Schiff et al.j 2021 Jiang et al., 2019),, and are useful in reasoning about a model beyond its performance on a specific known test set. However, these methods work only when train and test distributions are the same, and often rely on strong set of assumptions such as access to labelled test. data. (Schiff et al.) 2021 model al. 2015b; Bartlett et al.) 2017 Neyshabur et al., 2017b), model gradicnts (Jiang el al. 2019), and training data (Keskar et al. 2016). More recent work aims to estimate the generalization of trained model on unlabelled test data directly (Deng & Zheng 2021 Jiang et al., 2021 Deng et al. 2021 Garg et al. 2022). However, these metrics are typically calculated based on the output logits of a model on individual examples, which can become poorly calibrated 2022). Tn real world settings, we require a robust measure of generalization that can be applied across a wide range of test distributions and where we are often given access only a black box model.
2017a
weights (Ncyshabur
out-of-domain (OOD) settings (Morteza & Li
In this paper propose neighborhood invariance, a complexity measure that correlates well with generalization and that only assumes access to set of suitable data transformations. Given test data point, we define the transformation neighborhood as the set of points that can be generated from a set of transformations with a given maximum magnitude. classifier's neighborhood invariance then the proportion of points that are classified into the most commonly predicted class in this neighborhood. Intuitively, a classifier that is more invariant this neighborhood should have be able represent examples with lower dimensionality and thus lower complexity, leading to stronger generalization. Different from other similar methods (Aithal K al., 2021), define invariance with respect to the neighborhood itself rather than relative to the prediction at the test point and do not require manually tuning weights, mcaning our measure can be applied even when test distributions vary. In addition, since our measure makes so few assumptions it is applicable in wide range of experimental settings and can be used to compare the generalization properties of multiple models even when labeled data is unavailable.
We investigate the correlation of a model's neighborhood invariance with its capacity generalize, focusing experimental settings with OOD dataset. shifts ( Taori et al.] 2020) where test data is sampled from
2

---

Published in Transactions on Machine Learning Research (06/2023)
distribution different from the training distribution. We select common OOD benchmark datasets in image
Lu et al. 2020; Recht et al. 2018 Deng 2012 Darlow et al. 2018 Netzer Taori et al. 2020), sentiment analysis (Ni et al. 2019), and natural language inference (Williams et al. 2018 which totals over 100 pairs of training/ test domains. We consider a large
classification (Krizhessky
2009;
et al. 2011 Arjovsky et al.
2019
pool of over 4,600 models trained on these datasets with varying architectures and generalization properties, and sample sets of transformations commonly used for data augmentation (Ng et al., 2020) [ubuk et al., 2020; Wei & Zou 2019 Xie al., 2019). Across a wide set of correlation metries, we find that neighborhood invariance measures outperform or match baselines in almost all experimental settings.
2 Related Work
Characterizing Model Invariance Ensuring various kinds of model invariance is well studied aspect of
learning generalizable models and has been analyzed extensively from a causality perspective (Bühhmann
2018 Peters et al. 2015 Haavelmo 1943). At the largest scale, models trained on a wide support of training
data and domains have demonstrated robust zero-shot and few-shot abilities (Radford et al. 2021 Brown
ct al.] 2020; Wortsman et al. 2022). At a smaller scale, models that are invariant across data domains
or interventions (Arjovsky et al. 2019 Gulrajani & Lopez-Paz 2020 Bühhmannn 2018) are able to learn
representations that do not depend on spurious correlations. Finally, at the smallest scale, local invariance to
data angmentations (Cubuk et al. 2020). local changes (Rifai et al. 2011), augmentation graphs (HaoChen
ct al. et al. invariance at this local scale.
2021). similar neighbors (Luo et al. 2018), or interpolation between points (Verma et al. 2019 Zhang
2018) have demonstrated improvements in model generalization. In our paper, we consider model
Recent work has shown that models that are invariant to local transformations factorize the input space into base space and the set of transformations (Sokolić et. al. 2017 Sannai el. al. (021), effectively reducing the
2015). Measuring this input dimensionality and thus model complexity (Anselmi et al. 2016 Anselmi et al.
decrease in complexity can be performed by analyzing the sample cover ZZhu et al.
2021 A similar line of
work derives estimation error bounds based on the intrinsic dimensionality of deep ReLU networks in Hölder
(Schmidt-Hieber 2019; Nakada & Imaizumi] 2020 Chen et al. 2019), Besov, mixed smooth Besov (Suzuki
2018), and anisotropic Besov (Suzuki & Nitanda 2021 function spaces.
Most similar to our work. Aithal K et al. (2021) measures a model's robustness to perturbations as a proxy for generalization. Our method generalizes theirs and differs in a few key ways. We calculate our measure on the test set relative to a transformation neighborhood and can thus adapt to any specific domain for which we measure complexity and predict generalization. In cont Aithal K et al. (2021) calculate their measure on the training set, use the models' prediction as ground truth, and require manually tuning the weights of augmentations, mcaning it is relatively brittle and can only be applied to in-domain data. In addition we analyze the correlation of our neighborhood invariance measure on a wide range of OOD benchmarks on image classification, sentiment analysis, and natural language inference, while Aithal et al. (2021) consider only image classification tasks with matching train/test distributions.
Measures of Complexity and Predicting Generalization Traditional methods of analyzing the generalization bounds of neural networks use theoretical measures of complexity. VC dimension (Vapnik
& Chervonenkis 1971] | and Rademacher complexity (Bartlett & Mendelson 2003) can be used to bound the generalization of particular function classes, although they are often vacuous at the scale of deep neural
networks (Dziugaite Roy| (0177). The PAC-Bayes framework (McAllester 1999: Neyshabur et al. 2017a Dziugaite & Roy] 2017 Garg et al., 2021) can be used to build tighter gencralization bounds by considering the "sharpness" of the local minima. Norm-based measures (Neyshabur et al. 2015b Bartlett et al. 2017 Neyshabur et al. 2017b) bound generalization by considering different norms of the weights of learned networks More recent analyses have focused on empirically motivated measures that do not provide theoretical bounds.
These include the sharpness of minima in parameter space Keskar et al. (0016), Fisher-Rao norm Liang et al.
(2019), distance from initialization (Nagarajan & Kolter 2019), path norm (Neyshabur et al. 2015a). layer
margin distributions (Jiang et al. 2019), and perturbation response curves Schiff et al. 20021.
However, these measures are only applicable when train and test distributions match Although some generalization bounds have been derived for these OOD settings (Garg et al. 2021 Ben-David et al. 2007
3

---

Published in Transactions on Machine Learning Research (06/2023)
Neighborhood Examples {g,()) yy
More Inveriant Classifier
Less Invariant Classifier
OOD Test Example
Sample from Neighborhood
f2
Figure 2: To estimate neighborhood invariance sample sel of transformations {g:) Gr and generate set of nearby examples {9x(z));= for every test example This set of examples is then evaluated using each classifier. We expect classifiers with more points classified in the most common class to generalize better to the given test set.
Zhang al. 2019; they rely on access to the test data distribution. In addition, many testbeds examine only synthetic shifts, whereas natural shifts such as WILDS (Koh et al., 2021] are much more difficult. In real world scttings where test distributions
often unknown, a separate line of work aims to directly predict generalization from unlabelled test data. These methods either predict the correctness individual examples
[Deng & Zheng, 2021 Jiang et al. 2021 Deng et al. 2021). directly estimate the total error (Garg et al., 2022
Guillory et al. 2021 Chen * et al. 2021 Chuang et al. 2020 Vedantam et al. 2021 or learn linear
models relating ID and OOD accuracy Miller et al. 2021 h or agreement (Back et al. 2022
3 Neighborhood Invariance Measure
In this section we introduce our neighborhood invariance measure. We start by defining the transformation neighborhood of a point, then motivate our formulation of invariance in this neighborhood, and finally show how to estimate in practice.
3.1 Motivation
Consider a classification task from an input space X' to an output space y with & classes. We are given a model f X - y trained on an in-domain training dataset D; = {(r), v.)., (x. x; sampled from distribution P)), and an out-of-domain test dataset D. = (x ve),..., (IT 25 "²0) sampled from a distribution Po(X V). We assume further that domains are covariate shifted such that P()))))) does not change between domains
We consider set of data transformations G, transformation with an associated measure of the magnitude of the transformation m(g). For example, a set of image rotation transformations with a maximum angle of degrees might be denoted G30 where m(g:) = a is the angle of rotation for specific gi- For a given test point x E Do. we define the transformation neighborhood (x) (9(x) X Gr) as the set of outputs after applying all transformations in G,- Defining the neighborhood this way allows us to consider wide range of nearby points in controllable way without needing access to the underlying data distribution. As shown in Figure | for given r, we can define a neighborhood decision distribution as
m(g) < r) where is a particular data
(.) [(((x') =j = (x()))
(1)
(,,(
We then define our neighborhood invariance measure
p(Jxx) = max (x))
(2)
JEY
We assume that data transformations are selected such that the label for the transformed point g(x) is still well defined. For example, flipping MNIST digits horizontally would cause most examples have an undefined label. If the label is undefined, then (g(()) should produce close to random outputs and thus
4

---

Published in Transactions on Machine Learning Research (06/2023)
constant invariance values of 1/4 regardless of the generalization properties of causing measure to fail. We empirically measure this phenomenon across wide range of transformations in Section 5.4
Intuitively, a classifier that is more invariant in the neighborhood of x should be able to represent it with a lower input dimensionality and thus be less complex, leading to stronger generalization capabilities. contrast, less invariant classifier will need higher input dimensionality to represent the neighborhood of æ and thus be more complex, leading weaker generalization. Crucially, since our invariance is measured with respect to the neighborhood around the test point rather than the test point itself, it does not rely the ground truth label. In addition, it makes assumptions about the model or the distribution from which test data was sampled, making it applicable in many scttings where existing complexity measures cannot be calculated, including common OOD robustness settings.
3.2 Estimating Neighborhood Invariance
Calculating neighborhood invariance exactly is typically intractable since evaluating all possible transforma- tions is impossible. Instead, we perform Monte Carlo estimation by sampling a set of N transformations {ge /N from Gr (including the identity transformation 1(x) = x) and calculating
N Σ
x(f
(f(a(())) = o(f.z)).
(3)
N
where
0(f.z) arg max JEV
Σ
(f(g(())) =)
(4)
is the most commonly output label. The average neighborhood invariance across the entire dataset D. is then E
((f,
4 Experimental Setup
Empirically evaluating the quality of complexity measure is difficult and requires careful experimental design. Typically, evaluation is done by generating a large pool of models with sufficiently varied generalization propertics, but we generate these models by varying only a few hyperparamcters, our observed correlation may be an artifact of these factors alfecting both generalization and our measure. To this end, we follow a similar experimental setup Jiang et al. (2019).
4.1 Data
For our experiments we focus on three tasks: large and small scale image classification, sentiment analysis single sentences and natural language inference sentence pairs. For cach task we construct set of datasets sampled from different data domains.
Image Classification For image classification we begin by considering datasets domain shifted from ImageNet (Deng et al.) |2009] Russakovsky et al. 2015) These include ImageNet (Recht et al., 2019) and
Saagenet-Sketdl (Wang et al.
2019) with the same output classes, as well as ObjectNet (Barbu et al. 2019
ImageNetVid, YTBB anchors (Gu et al. 2019 Recht et al. 2019), ImageNet-A (Hendryeks et al. 2021b
and ImageNet-R (Hendrycks al.] 2021a with smaller subset of output classes.
In addition to ImageNet datasets construct two sets of smaller scale datasets. The first we call
CI10
and consists of CIFAR10 (Krizhevsky 2009), CINIC10 (Darlow et al.) 2018 CIFAR10.1 (Recht et al.
201
and CIFAR10.2 et al. 2020). The second we call Numbers and consists of SVHN (Netzer et al.
2011
MNIST Deng 2012), and Colored MNIST (Arjovsky el al. 2019 Domains in each set. share the same set of output classes.
Sentiment Analysis (SA) We use the datasets subsampled from the Amazon reviews dataset NNi al. 2019) which contains product reviews from Amazon. Following Hendrycks al. (2020b) and Ng et al. 2020
5

---

Published in Transactions on Machine Learning Research (06/2023)
Training Domain
Batch Size
Weight Decay
Learning Rate
Model
Dataset
Depth
Width
Dropout
Noisc
Norm
Seed
Aug
# Converged
Evaluations
CNN
MNLI
Amazon
10
3
3
3
3
3
2
3
2,418
796
39,800
24,150
RoBERTS
MNLI
5
3
2
3
213
10,630
Various
em
481
2,406
SVHN
CIFATLIO
2
2
2
32
128
VGG
CIFATLIO
3
3
2
51
2:6
ResNet
CIFARIO
3
864
CNN
CINICTO
512
4,644
112,318
Table 1: Number possible hyperparameter values for each architecture and task. Fields denoted with a indicate that this hyperparameter fixed or not applicable. We also list the total number of models converged and evaluations run in each model pool. In total we consider 4,644 models and 112,118 evaluations.
we split the dataset into 10 different domains based on review category. For all domains and datasets, models are trained to predict a review's star rating from to 5.
Natural Language Inference (NLI) We use the MNLI (Williams et al. 2018) dataset, a corpus of NLI data from 10 distinct genres of written and spoken English. We train on the 5 genres with training data and evaluate on all 10 genres. Models are given two sentences, a premise and hypothosis, and predict whether the hypothesis is entailed by, is neutral to, or contradicts the premise.
4.2
Model and Hyperparameter Space
For large scale image classification ImageNet,
pretrained models from the ImageNet Testbed (Tori
ct al.
2020) which covers a wide range of architectures including ResNext (Xie et al. 2016), EfficientNet
(Tan & Le 2019 BiT (Beyer et al.) 2021,, Vision Transformers @Dsovitskiy et al. 2020). CLIP (Radford
el al. 2021) and many more models. We provide a full list of models evaluated in Appendix A.2. For smaller
scale image classification tasks, we use models trained for the tasks 1, 2, 4, 5, and from the Predicting
Generalization in Doep Learning competition (PGDL) (Jiang et al. 2020 as well as models from Jiang ct al. (2019), which covers Network in Network (NiN) (Lin et al., 2013), VGG (Simonyan & Zissermann 2015). ), ResNet (He et al.) 2015), and CNN models trained on CIFAR10, CINIC10, and SVHN. On natural language tasks we consider CNN Kim 2014 Mou et al. 2016) and RoBERTa (Liu et al. 2019) based models. On natural language models we apply label noise by randomly replacing fraction of training labels with uniform
samples from the label space. We argue that label noise is not an artificial training setting as stated in Jiang
et al. 2019) but rather a method of entropy regularization (Pereyra et al. 2017 Xie et al. 2016 which prevents models from becoming overconfident.
In order to control for the varying convergence rates and learning capacities of our different models, follow Jiang al. (2019) and carly stop the training of models when they reach given training cross entropy loss (usually around 99% training accuracy), or if they reach the max number of training epochs. We discard all models which do not converge within this time. The total number of models trained and converged in each pool as well as details on hyperparamoter variations for cach task and model provided in Table [ We include further details on model training, the hyperparameter space, and specific choices hyperparameters in A Appendix A.4 A.2 and A.3
4.3 Evaluation Metrics
Given a set of domains defined by distributions {P1, P2....PP, Pn) and a set of datascts (D; ~ P. sampled from these domains, we train a set of models F. = models J E F; on all OOD test. datasets D, o i, generating a set of invariance and generalization values (utt st.). We define generalization as the top-1 accuracy of fM on D..
on each dataset D,. We evaluate all
We evaluate our measure first by predicting the generalization of given model to an OOD test set. Specifically, select 0OD test set D, and in-domain training set D; o and predict the OOD generalization gi.
6

---

Published in Transactions on Machine Learning Research (06/2023)
of a model f trained on Dj and evaluated on Do from its invaraince value pho To generate these predictions we use linear model 9 == ap b with parameters o, b E R. To estimate our parameters and b, we select a pool of models {. E F3 j # i, o, that are trained on all remaóning datasets. Each model fy in this pool is evaluated on the OOD dataset D. to give us set of pairs {( gfa)]. We then find a, by minimizing the mean squared crror (a", b*) arg min Ej (apt) + b 9 on all models the pool.
We use the learned parameters to make generalization predictions g = apt. + for every model f E F; on Do and measure the coefficient of determination R² (Glantz et al. 1990). We also measure the residuals of our linear model by calculating the mean absolute error (MAE) between our predictions and the actual generalization. For every pair of training domaín i and OOD test domain o, we evaluate R² and MAE then average each metric across all pairs. We report MAE values as percentage points.
We also consider the rank correlation between neighborhood invariance and actual generalization. Specifically, for a pair of models fi with mcasure and generalization pairs (p4. 9s) and (fj,9j), we want 9i > 9j if P4 > Mg. We use Kendall's rank 7 coefficient (Kendall 1938) to measure how consistent these sets of rankings are. We measure four different r values:
ID This metric evaluates the corrclation of our mcasure with in-domain generalization. We select a training dataset D, and consider pairs f(u oht)) generated from the set of models F; trained on D.. T values are averaged across all training domains.
Macro This metric evaluates the correlation of our measure individually on each training /OOd test domain pair. We select a training dataset D; and a OOD test dataset D, and consider pairs { g generated from the set of models F; trained on D; T values are averaged across all pairs of training and OOD test domains.
Micro This metric evaluates the correlation of our measure on a given OOD test domain across models trained on all other domains. We select single OOD test domain D. and consider pairs {(r oha)) generated from the set of models {f,h F; # of trained on all other datasets {D, i * o}. 7 values are averaged across all test domains. We use this metric only when different models are trained on different training sets.
Arch This metric evaluates the correlation of our measure on models trained with different architectures. Arch r is calculated similar to Micro t, except F, now includes models from all architectures. T values are averaged across all Lest domains.
4.4 Data Transformations
Defining the transformation neighborhood requires defining a set of data transformations with associated magnitudes. For image classification, we consider four transformations: RandAugment (Cubuk et al., 2020 which randomly combines various transformations, random translation in the X- and Y-axes, random patch erasing (Zhong et al., 2020) which removes randomly sized patches from the image, and horizontal flips and crops. We call neighborhood invariance measures based on these transformations Aug, NI-Translate, NI-Erase, and NI-FC respectively. For natural language tasks, we also consider four transformations: SSMBA (Ng et al.) 2020) which generates examples in manifold neighborhood using denoising autocncoder, EDA (Wei & Zou 2019) which applics random word level operations, backtranslation (BT) (Rioo Sennrich 2016 Xie et al. 2019) which translates back and forth from a pivot language, and a transformation that randomly replaces a percentage of tokens. We call neighborhood invariance measures based on these transformations NI-SSMBA, NI-EDA, NI-BT, and NI-RandRep respectively. For all experiments we sample n = 10 transformations in addition to the identity tranfsormation, although ablations in section 5.4 show that our method is relatively robust to the specific number of transformations sampled. We provide further details on specific transformation magnitude values and implementations for all methods in Appendix A.5
4.5 Baselines
Since our experimental setting makes so few assumptions, there are very few complexity measures that we can compare against. This includes Aithal K et al. (2021). which requires matching train/test distributions.

---

Published in Transactions on Machine Learning Research (06/2023)
We thus consider complexity measures that require only model weights, specifically the Spectral (oshiida & Miyato 2017 Neyshabur et al. 2017b) and Frobenius (Neyshabur et al. 2015b) norms. However, in our experiments we find close to 0 or negative correlation for these measures, so we do not report their performance. We also compare our method against output based methods that directly predict OOD generalization. We use ATC-MC and ATC-NE Garg et al. (2022) as our two baselines, which calculate a threshold on in-domain validation data based on max confidence and negative entropy scores respectivcly. To calculate metrics on these methods we treat the generated accuracy predictions as a score. To calculate ID values we select threshold value based on validation data then calculate predicted accuracy values on test data from the same domain. For ImageNet domain shift datasets where the output classes are a subset of the original 1,000 ImageNet classes, we do not recompute a subclassed ATC threshold as we do not assume prior knowledge of the OOD output classes.
5 Results
We now present the results of our experiments evaluating the quality of our neighborhood invariance measure. We begin by analyzing the effect of dataset distance on the correlation of neighborhood invariance with generalization in a toy setting. Our main set of results evaluate neighborhood invariance on OOD benchmarks in image classification, sentiment analysis, and natural language inference. Finally, we examine the correlation of our measure in extreme OOD settings and analyze the factors that affect the quality of our neighborhood invariance estimates.
5.1 Dataset Distance: Toy Analysis
In general, as with any complexity measure or generalization predictor, we expect neighborhood invariance to perform more poorly as we move farther from the training domain. In the worst case, if classifier becomes degenerate constant classifier in far enough OOD domain, then neighborhood invariance reaches constant maximum value of 1 while generalization becomes random. In order for our measure to work well, we assume that test, domains are sufficiently close to training domains so that model predictions are non-constant. Tn this section we present an analysis of the effect of dataset distance on the quality of our measure in a toy setting.
We consider binary classification task of points inside and outside unit hypersphere in X' R", as shown in Figure We define different data domains univariate gaussian distributions P(X), centered at point p on the hypersphere. The distance between two datasets D1 (X') and D2 P2(X") can then be measured as the distance along the hypersphere between μ1 and μ2 in radians. We generate a training dataset Do by sampling points around the north polc of the hypersphere, then generate out-of-domain datasets D, at varying distances from Do- Given a model trained on Do, we can calculate its generalization to Dj, as well as its neighborhood invariance.
In our experiments, we consider 16 dimensional hypersphere and sample 1000 points per data distribution for each dataset. The univariate gaussian distributions that sample data points from have fixed variance 0.005, ensuring models cannot generalize fully across the entire hypersphere. We train 200 single hidden-layer MLPs with hidden dimension of 16 on the training dataset Do. each with a random level of label noise between 0-30% to ensure a wide range of generalization properties. We consider 40 different dataset distances, equally spaced along the hypersphere between opposite poles. For given dataset distance, we select 5 points at random from the corresponding circumference and generate 5 datascts from univariate gaussians centered at these points. To measure neighborhood invariance for a given x, we sample 10 transformations from the set of transformations defined as perturbation along the hypersphere of radius ||||| with maximum distance m(g) = [=(x)) - xx|1 & 0.01. For each datasct we measure the neighborhood invariance and generalization for each of the 200 trained models and calculate the Kendall 7 correlation between them. For a given dataset. distance, the 7 values are then averaged across all datasets. Results are presented in Figure 3b
For datasets closest to the training datasot, the correlation between generalization and neighborhood invariance is high. However, as dataset distance increases, correlation decreases. At a distance of around x correlation becomes nearly 0, and iontinues decreasing until the two values are negatively correlated on data sampled
8

---

Published in Transactions on Machine Learning Research (06/2023)
Do
NI
D
ATC
0.5
D2
Kendall 7
D3
-0.5
D4
D5
5/4
7/2 3r/a
Dataset Distance
(a) We generate data from univariate gaussians whose means lie hypersphere. We train models on a dataset Du to classify points as inside or outside the hypersphere then test them out of domain datasets D, that lie at various distances from Do as measured by radiance distance along the hypersphere.
(b) The correlation between neighborhood invariance and OOD generalization remains high near the train- ing domain but quickly decreases as dataset distance incerases, becoming negatively correlated for datasets further than "/4 away on the hypersphere. We observe almost identical results for baseline ATC methods.
Figure 3: Toy analysis of the effect of dataset distance the correlation between neighborhood invariance and OOD generalization.
from the opposite side of the hypersphere from the training domain. This behavior almost identical for both neighborhood invariance and ATC methods (ATC-MC and ATC-NE perform the same so we report only one). These results demonstrate that the correlation of neighborhood invariance with generalization should decrease as dataset distance increases. To investigate the degree to which this happens in practice, we consider a set of extreme OOD experiments (Section 5.3) where we observe a surprisingly small decrease in correlation, indicating closer dataset distance than might initially be assumed.
5.2
Correlation with OOD Generalization
We first present results in Table analyzing the correlation of proposed neighborhood invariance mcasure with OOD generalization. We report 123, MAE, Macro т, Micro т, ID т, and Arch as detailed in Section We omit results on Spectral and Frobenius norm measures as they are close to 0 or negative for all metrics. We do not report Micro t values for image classification models since each model type is trained on only one domain. Additional experiments and results are presented in Appendix B
4.3
ImageNett-Salle Image Classification Results on TmageNet-scale image classification datasets are presented in Table 28a]and and are averaged across all architectures. On standard domain shifts, NI-RandAug performs slightly better than ATC methods on R and Macro т with similar MAE. On the adversarial ImageNet-A dataset, ATC methods fail completely whereas NI methods maintain strong performance and still correlate well with accuracy. However, both methods exhibit large MAE and fail to accurately predict actual OOD accuracy. On ID NI methods perform slightly worse than ATC methods although they still show very strong correlations.
CIFARIO-Scale Image Classification Results on smaller scale image classification datasets are presentod in 28b and are averaged across all architectures. On CI10 datasets, NI-RandAug significantly outperforms ATC baselines and all other measures all metrics. NI-RandAug also exhibits only small decrease in correlation when moving from in-domain (ID r) to OOD datasets (Macro r), compared to ATC methods which suffer much larger drop. On Numbers datasets, NI-RandAug outperforms all other methods on I² and MAE, although it performs slightly worse on Macro т compared to NI-Translate and on ID compared to ATC baselines.
For both CI10 and Numbers, using patch crasing and flip and crop transformations cause our method perform worse or fail entirely, in contrast to ImageNet where they perform similarly or better. For these
9

---

Published in Transactions on Machine Learning Research (06/2023)
Domain Shifts
ImageNet-A
Measure
R²
MAE
Macro
R²
MAE
Macro
ID
NI-RandAug
0.709
11.87
0.724
0.577
31.17
0.586
0.845
NI-Translate
0.587
13.58
0.604
0.468
29.54
0.439
0.834
NI-Erase
0.492
0.555
0.446
33.06
0.517
NI-FC
0.603
12.28
0.691
0.679
31.84
0.589
0.867
ATC-NE
0.607
11.40
0.703
0.209
32.00
0.248
0.931
ATC-MC
0.622
11.97
0.691
0.159
32.85
0.190
0.942
(a) Results on ImageNet scale models and datasets. On standard domain shift datasets NI-RandAug performs slightly
better than ATC methods, and maintains strong performance on adversarial data where ATC methods fail completely.
CT10
Numbers
Measure
R²a
MAE
Macro
ID
Arch
R²a
MAE
Macro
ID
NI-RandAug
0.899
3.11
0.768
0.793
0.837
0.764
5.33
0.642
0.733
NI-Translate
0.732
3.56
0.607
0.661
0.786
0.685
6.12
0.667
0.881
NI-Erase
0.518
4.41
0.411
0.406
0.299
0.153
10.17
-0.135
0.324
NI-FC
0.417
4.47
0.371
0.344
0.683
0.208
10.42
-0.316
-0.033
ATC-NE
0.655
3.61
0.548
0.693
0.689
0.616
0.637 0.682
0.859
ATC-MC
0.640
3.65
0.544
0.682
0.685
0.692
6.19
0.844
(b) Results on small scale image classification, averaged across model architectures. No Micro T is reported since models are trained single domain and Arch is reported for the Numbers dataset since only consider single architecture. NI-RandAug beats all other methods on almost all metrics.
CNN
RoBERTa
Measure
n²
MAE
Macro
Micro
ID
n²
MAE
Macro
Micro
ID
Arch
NI-SSMBA
0.662
1.93
0.677
0.689
0.629
0.972
1.29
0.832
0.829
0.838
0.588
NI-EDA
0.641
2.04
0.664
0.649
0.611
0.968
1.45
0.830
0.810
0.830
0.512
NI-BT
0.550
2.99
0.592
0.501
0.538
0.961
1.47
0.813
0.801
0.801
0.523
NI-RandRep
0.409
2.64
0.544
0.554
0.439
0.967
1.27
0.821
0.816
0.822
0.537
ATC-NE
0.760
2.47
0.514
0.633
0.467
0.852
2.38
0.707
0.691
0.749
0.660
ATC-MC
0.761
2.46
0.517
0.634
0.467
0.869
2.26
0.722
0.705
0.749
0.663
(c) Results on sentiment analysis (SA) datasets. NI-SSMBA beats all other methods almost all metrics.
CNN
RoBERTa
Measure
R²
MAE
Macro T
Micro т
ID т
R²
MAE
Macro T
Micro T
ID T
Arch
NI-SSMBA
0.575
2.09
0.570
0.584
0.704
0.933
1.19
0.750
0.730
0.771
0.301
NI-EDA
0.577
2.04
0.581
0.511
0.709
0.941
1.26
0.789
0.757
0.799
0.572
NI-BT
0.509
2.11
0.470
0.449
0.584
0.944
1.07
0.759
0.740
0.778
0.563
Ne-RandRep
0.451
2.20
0.452
0.428
0.570
0,890
1.70
0.688
0.647
0.710
0.401
ATC-NE
0.576
2.52
0.568
0.446
0.705
0.737
2.22
0.557
0.541
0.739
0.635
ATC-MC
0.576
2.52
0.568
0.446
0.706
0.769
2.10
0.581
0.567
0.748
0.636
(d) Results on natural language inference (NLI) datasets. NI measures beat baselines on all metrics except Arch т.
Table Evaluation metrics measuring the correlation of our neighborhood invariance measure with ID/OOD generalization. The best performing measures for each metric are bolded. On all tasks, neighborhood invariance achieves strong generalization and beats baseline methods almost all metrics. Full tables for R², Macro T, Micro r, and ID individual train/test domains are in Appendix C
10

---

Published in Transactions on Machine Learning Research (06/2023)
CNN
RoBERTa
CNN
RoBERTa
Measure
Ra
Micro
R
Mícro
Measure
R
Micro
R
Micro
NI-SSMBA
0.584
0.566
0.941
0.816
NI-SSMBA
0.083
0.080
0.691
0.463
NI-EDA
0.575
0.567
0.884
0.715
NI-EDA
0.202
0.110
0.739
0.540
NI-BT
0.538
0,470
0.906
0.766
NI-BT
0.213
0.247
0.730
0.527
Ne-RandRep
0.277
0.373
0.918
0.776
NI-RandRep
0.096
0.102
0.030
0.012
ATC-NE 0.271
0.495
0.329
0.356
ATC-NE
0.077
-0.107
0.719
0.345
ATC-MC
0.295
0.506
0.437
0.436
ATC-MC
0.076
-0. 101006
0.734
0.354
(a) Results on the Drugs. com dataset.
(b) Results the MedNLI dataset.
Table 3: Evaluation metrics measuring the correlation of our neighborhood invariance measure with gen- eralization extreme OOD datasets. N1-* methods beat baselines both tasks, with RoBERTa models exhibiting only a slight degradation in correlation compared to more typical OOD scttings.
smaller datasets, since these transformations are more likely to generate images that cannot be classified (e.g. images without an object in frame), they produce more similar invariance values between classifiers that cannot be used to rank them properly. This effect is more pronounced for Numbers datasets because flipping the image or removing even small portions of the number to be classified can render the task impossible. In contrast, random image translation which almost always preserves label information performs similarly well across both datasets and almost matches NI-RandAug, We provide larger set of ablations on the Numbers dataset exploring this phenomenon in Section 5.4 The strong performance of NI-RandAug indicates that combining multiple transformations is helpful for mitigating datasee--pecific transformation sensitivities, as in the case of Numbers.
Sentiment Analysis (SA) Results Sentiment Analysis datascts are presented in Table 28c In
experiments both architectures, neighborhood invariance measures achieves strong correlation with OOD generalization and beats all basclines on almost all metrics. Of the transformations considered, NI- SSMBA performs the best across both architectures. NI-EDA, N1-BT, and even NI-RandRep achieve strong results as well, often beating ATC baselines. We observe particularly strong correlation on RoBERTa models, with a nearly perfectly linear R² value of 0.972 and large Micro т of 0.829. We hypothesize that this is due to the pretrained initialization of RoBERTa models, which gives strong inductive bias towards learning a space invariant to transformations that preserve meaning. In contrast, CNN models are trained from random initializations and may not learn closely aligned space. On cross architecture analysis, observe strong Arch for our ncighborhood measures, although they are outperformed by both ATC methods. Compared to image classification results, our results on sentiment analysis tasks are overall less sensitive to the data transformations selected because they are less likely to destroy information necessary for classification.
Natural Language Inferenee (NLI) Results on Natural Language Inference tasks are presented in Table 28d Similar to our sentiment analysis results, our neighborhood invariance measures achieve strong correlation with OOD generalization
both architectures and beat all baselines. Correlations in general on NLI are lower than those of sentiment analysis because it is more difficult to maintain the complex relationship between the two sentenees during a transformation. For example, changing a single word can easily change a sentence pair from entailment to contradiction, whereas many words must be changed to modify a 5 star review to a 1 star review. We observe exceptionally high correlation on RoBERTa models, for which we offer a similar hypothesis as in our sentiment analysis experiments. On cross architecture analysis, we observe strong correlation for our NI-EDA and NI-BT although they are outperformed by both ATC methods.
5.3 Extreme OOD Generalization
We now consider more extreme generalization to data domains with specialized and knowledge intensive data. We consider only natural language tasks as it difficult to find sufficiently specialized image classification dataset that maintains the same output classes. For sentiment analysis we use the Drugs com review dataset Gräßer et al. 2018), and for natural language inference
MedNLI (Romanov & Shivade 2018), an
11

---

Published in Transactions on Machine Learning Research (06/2023)
0.0
Micoo
0.4
Mien
0.4
NISSMBA NI DT
0.2
NLSSMBA
0.2
N1 DT
0
10
0
100
01
20
60
80
Test Datcset Exaaples
of Neighbothodd Samples
Corrapiion %6
(a) Correlation improves as the size of the test dataset that we evaluate increases.
(b) Correlation is constant the mumber transformations de- creases, even when we sample only single transformation in addition to the identity transformation.
(c) Correlation increases until optimal corruption percentage is reached, then decreases as the cor- ruption continues to increase.
Figure 4: Micro for our neighborhood invariance measure calculated with varying ablations on CNN models evaluated on Amazon toy reviews.
NLI dataset generated from clinical notes and patient history. Both datasets contain highly specific medical language not seen in any of our training domains. All models from all original training domains are evaluated on each of these extreme OOD domains, and we report R² and Micro т. Results are shown in Table 3
On the Drugs. com dataset, we observe a small decrease in correlation for neighborhood invariance methods compared to results on AWS datasets. However, ATC methods begin to fail, with Micro 7 on RoBERTa models dropping significantly from 0.706 0.356. This suggests that models become poorly calibrated in extreme OOD settings, making ATC methods fragile. On MedNLI we observe much larger disparity in performance. For CNN models, most of our measures fail to correlate at all, and ATC methods degrade so much they became anti correlated with generalization. For RoBERTa models we observe only minor drops in correlation for all measures. For both lasks Ni-RandRep exhibits almost no correlation with OOD generalization. This suggests that the choice of transformation becomes much more important as we move farther from our training domain.
5.4 Ablations
In this section we examine factors that may affect the quality of neighborhood invariance estimation and its correlation with actual generalization. Since rerunning of our experiments is too costly, we evaluate on toys Amazon reviews using pool of CNN models trained on all other domains for the first three ablations, and on the Numbers datasets using NiN models trained on SVHN for the final two.
Test Dataset Size: Does neighborhood invariance measure still correlate well when the test dataset is small? We randomly and iteratively subsample our test dataset of 2000 examples to reduce our dataset. size down to 10 examples. We then measure our models' neighborhood invariance on each subsampled dataset and calculate the Micro 5 on all models. Results are shown Figure 4a We find that for all neighborhoods, smaller datasets lead to noisier invariance estimates and lower correlation. As dataset size increascs, corrclation incrcases as well.
Number of Transformations:
How many transformations do need to sample in order to generate reliable neighborhood invariance estimate? We sample varying number of transformations for each test example, from minimum of two transformations to maximum of 100 transformations. then estimate our neighborhood invariance measure with each set of transformations on the entire test datasel and calculate the micro т. By default we always include the identity transformation. Results are shown in Figure [b] We find that our mcasure surprisingly robust the number of samples, with only a small difference between 100 and transformations sampled. For all measures, correlation slightly increases as the number of samples increases and achieve better estimation of the true invariance value.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Training Regime
R²
MAE
Macro t
ID
Standard Training
0.684
12.37
0.763
0.852
Augmentation/ Robustness
0.707
12.23
0.650
0.855
Pretraining/ Extra Data
0.799
11.73
0.707
0.836
All Models
0.709
11.87
0.724
0.845
Table 4: NI-RandAug results on different subsets of the ImageNet Testbed models trained with different. training regimes. Training with augmentation and robustness interventions decreases macro T compared to standard training but increases all other metrics. R and MAE are higher for models ceinod/pretraincd with large amounts of data.
Transformation Magnitude: How sensitive our measure to the maximum magnitude of transformation considered? We use the SSMBA transformation for which the magnitude of transformation is defined by the percentage of tokens corrupted, which vary from minimum of 5% to maximum of 85%. After sampling set of transformations from cach corruption level, we estimate neighborhood invariance on the test dataset using each set and calculate the micro T on all models. Results are shown in Figure 4c We find that as we begin to increase the corruption percentage, correlation begins to increase as well. Correlation reaches a maximum, then decreases as we continue to increase our corruption percentage. However, even 85% corruption, our method is quite robust and achieves a micro of 0.416.
Selecting Transformations: How do we ensure that transformations do not destroy label information? We consider the set of NiN models trained on SVHN and set of image transformations including RandAngment (Cubuk et al. 2020), rotations, translations, shears, brightness jittering, contrast jittering, color jittering, patch erasing, and flips and crops. For each transformation and both OOD datasets ColoredMNIST and MNIST we calculate the Macro correlation between accuracy and neighborhood invariance, as well as the average entropy difference between a model's output on transformed image and on the original image. Results are shown in Figure
suitable for given dataset and
Neighborhood invariance is relatively insensitive to the transformation selected, with most transformations performing similarly up to a certain entropy difference threshold around 0.1, after which it fails. The transformations that fail, erase and flip crop, both tend to destroy label information and lead to much higher entropy outputs. We propose this method of examining entropy differences as a simple way to diagnose whether a given transform is appropriate for a specific dataset.
Pretraining and Training with Augmentation: Does self-supervised pretraining or training with data augmentations, which should make models more invariant to certain transformations, make our neighborhood invariance measure ineffective? We begin by examining our metrics subsets of models from the ImageNet Testbed TTaoi et al., 2020) split. by models trained on standard ImageNet (81 models), models trained on ImageNet with augmentations and robustness interventions (74 models), and finally models trained with extra data or pretrained with self-supervised objectives (41 models). Results are shown in Table 4 We find that compared to evaluating only on standard training models, the addition of augmentations or pretraining slightly degrades Macro т, but improves R² and MAE. Compared to the overall results on all models, we do not observe any large decreases in performance.
Since the augmentations considered in the set of ImageNet Testbed models are not the same across models, also consider models from PGDL ęJiang et al.) 2020) trained with and without single type of augmentation: flip crop. Calculating our evaluation metrics on each set of models allows us to isolate the effect of data augmentation. Results are shown Table 5 We find that measuring neighborhood invariance using the same transformation (NI-FC) that models are trained with causes only a slight degradation compared models trained without. When measuring invariance using other transformations (Ni-RandAug, NI-Erase), evaluation metrics actually improve slightly.
13

---

Published in Transactions on Machine Learning Research (06/2023)
RundAngmnn Afline
Measure
Flip Crop
R²
MAB
Macro T
TD>
Roteto 20
ExpCrop
Eraze
NI-RandAug
/
x
0.844
0.833
3.16
3.32
0.732
0.719
0.753
0.744
Maaco
Trazaliee
NI- TT-arnsstte
x 0,874
2.61
0.768
0,831
Translate 50
v
0.877
2.52
0.794
0.845
Shrar 15
Cikee
x
0.790
3.28
0.716
0.702
Contzter
NI-Erase
0.5
Beightuss
V
0.812
3.17
0.722
0.719
Asseago Eatropey
NI-FC
x 0.681
4.34
0.620
0.587
0.663
4.23
0.608
0.554
Figure 5: Transformations whose neighborhood invariance measures cor- relate well with OOD generalization exhibit smaller differences in oulpul entropy.
Table 5: Training with and without flip crop augmentation has a minimal effect on the effectiveness of our method, even when the transformation neighborhood aligns with those used to train the model (NI-FC).
6 Discussion
In this paper, motivated by the limited settings in which existing complexity measures be applicd, propose simple to calculate neighborhood invariance measure that can be applied even when test distributions are unknown and model training data, weights, and gradients are unavailable. We evaluate our method on image classification, sentiment analysis, and natural language inference datasets, calculating a variety correlation metrics with both in-domain and out-of-domain (OOD) generalization. Across almost all tasks and experimental settings, we find that our neighborhood invariance measure consistently outperforms baseline methods and correlates strongly with actual generalization. However, our method has several limitations. Data transformations must be selected such that labels for transformed points still well defined, although examining entropy differences can diagnose poor transformation choices In settings where such transformations are difficult to define, our method may not be applicable or provide inappropriately high estimates, so practitioners must be careful to verify their estimates with a labelled test set. In addition, our neighborhood invariance measure may fail sufficiently OOD settings where a model may become poorly calibrated or degenerate, although we find in practice on our tasks that even extreme OOD scttings are similar enough for our measure to perform well. In future work we plan to explore using similar measures caleulated over transformation neighborhoods method for OOD detection.
References
Sumukh Aithal K, Dhruva Kashyap, and Natarajan Subramanyam. Robustness to Augmentations Generalization metric. arXiv e-prints, art. arXiv:2101.06459. Jamuary 2021.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers. arXiv e-prints, art arXiv:1811.04918, November 2018.
Fabio Anselmi, Lorenzo Rosasco, and Tomaso Poggio. On Invariance and Selectivity in Representation Learning. arXiv e-prints, art. arXiv:1503.05983 059338, March 2015.
Fabio Anselmi, Joel Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio. Unsuper- vised learning of invariant representations. Theoretical Computer Science, 633:112 121, 2016. ISSN 0304- 3975. doi: httpss//daooorr//...000//// 2ss.2015.06.043 URL https: wwwp//wwwww../ www.://ceeneeerret.ccom wwwpeettt..oocceccncceeeeeeeeeee article/11/8000445755555657 Biologically Inspired Processes in Neural Computation.
Martin Arjovsky, Léon Bottou, Tshaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXin, 2019.
Christina Back, Yiding Jiang, Aditi Raghunathan, and Zico Kolter, Agreement-on-the-linne Predicting the performance of neural networks under distribution shift, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d' ''Alcé--Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associales, Inc., 2019. URL httpp::/proceeddng.. wewpeeeddnnnn.uuiip... nwwrips. wwtppppr..ills//ppeeeeeeeeeeeeee ftt.ee//aaaee/21/////222 pdf
Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3(mull):463 482, mar 2003. ISSN 1532-4435.
Peter L Bartlelt, Dylan J Foster, and Matus J Telgarsky. Spectrally-normallzed margin bounds for neural net- works. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), ), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https
pdf
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Pross, 2007. URL lutops://proceedings. neeraiddddinnnnnnns...... www..ps..c.pappee////ee pappre/20/// #ile//1104422eeafb07774444466643 14852a17 pdf
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. CoRR, abs/2106.05237, 2021. URL
https://arxia http:///axxxvv.oraa/a/////sss (arrgabbs/11e 2106. 052337
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Newral Information Processing Systems, volume 33, pp. 1877-1901 Curran Associates, Inc., 2020. URL Paper
Peter Bühlmann. Invariance, causality and robustness, 2018.
Mayee Chen*, Karan Goel*, Nimit Sohoni*, Fait Poms, Kayvon Fatahalian, and Christopher Re. Mandoline: Model evaluation under distribution shift. International Conference of Machine Learning (ICML), 2021.
Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Nonparametric Regression Low-Dimensional Manifolds using Deep ReLU Networks: Function Approximation and Statistical Recovery. arXiv e-prints, art. arXiv:1908.01842 August 2019.
Ching-Yas Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. International conference on machine learning, 2020.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Lc. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18613- 18624. Curran Associates, Inc., 2020. URL httpe://prooceedingg. nwwrips. cc/ /2/papee/2220/fille 00bb6eef00c014444300//7/00000000 PaP-
Luke N. Darlow, Elliot. J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not ImageNet or CIFAR-10. arXiv e-prints, art. arXiv:1810.03500 October 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.109/CVPP..200.52266
15

---

Published in Transactions on Machine Learning Research (06/2023)
Li Deng. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proc. CVPR, 2021.
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? In ICML, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference Uncertainty in Artificial Intelligence (UAI), 2017.
Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton, Behnam Neyshabur, and Hanic Sedghi. Leveraging Unlabeled Data to Predict Out-of-Distribution Performance. arXiv e-prints, art. arXiv:2201. 0022444, January 2022.
Vikas Garg, Adam Tauman Kalai, Katrina Ligett, and Steven Wu. Learn to expect the unexpected: Probably approximately correct domain generalization. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 PMLR, 13-15 Apr 2021. URL https:
of Proceedings of Machine Learning Reseavch, pp. //proceedings.o mlr. prress/130//gar2220
Stanton A Glantz, Bryan K Slinker, and Torsten B Neilands. Primer of Applied Regression and Analysis of Variance. Health Professions Division, McGraw-Hill, New York, 1990.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572, 12 2014.
Felix Gräßer, Surya Kallumadi, Hagen Malberg, and Sebastian Zaunseder. Aspect-based sentiment analysis drug reviews applying dross-domain and cross-data learning. In Proceedings of the 2018 International Conference on Digital Health, DH '18, pp. 121-125, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450364935. doi: 10. 10.1165/314468.1146677. URL httpp:://doi.org///
01994658.314661
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoe Le, and Jonathon Shlens. Using videos to evaluate image model robustness. In SafeML Workshop at ICLR, 2019.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with Confidence on Unseen Distributions. In International Conference on Computer Vision, 2021.
Ishaan Gulrajani and David Lopez-Paz. In Scarch of Lost Domain Generalization. arXiv e-prints, art. arXiv:2007.01434, July 2020.
Abhishck Gupta, Alagan Anpalagan, Ling Guan, and Ahmed Shaharyar Khwaja. Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array, 10: 100057, 2021. ISSN 2590-0056. doi: http:://bo.orrr/0..00/////aaa aaieaaccareettcco///ceeee///eeee
URL https:
Trygve Haavelmo. The statistical implications of a system of simultancons equations. Econometrica, 11(1): 1-12, 1943. ISSN 00129682, 14680262. URL http::/www.jstot/ /tt.://ww...ooo//tta////////////
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable Guarantees for Sulf-vupee Deep Learning with Spectral Contrastive Loss. Xiv e-prints, art. arXiv:2106.04156, June 2021.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arxiv e-prints, art XXiv:1512.033883 December 2015.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp. 2744-2751, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/ 10.18653/w1/2020.cem 1112000.acllmmain 244. URL httpp://aedaaattooooooooo/////// aclp:22220cccc11.0. mol-mminn.. 244
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Association for Compulational Linguisties, 2020b.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Yiding Jiang, Dilip Krishman, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. Tn International Conference on Learwing Representations, 2019. URL
https://openreview.nn http://wwww....ttoooorrrtttatooo
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. arXiv e-prints, art. arXiv:1912.02178, December 2019.
Yiding Jiang, Pierre Foret, Scott Yak, Daniel M. Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Sany Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur. NeurIPS 2020 Competition: Predicting Generalization in Deep Learning. arXiv e-prinds, art ar arXiv:2012.07976, December 2020.
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and J. Zico Kolter. Assessing Generalization of SGD via Disagroement. Xixi e-prints, art arXiv:2106.1377 June 2021.
M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938. ISSN 00063444. URL http://www.jattt org/staalee/332226
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang, On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprinl arXiv: 04836, 2016.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL http: //trxxv.orrg/////e.e..e..oooollt (988 cite 1ww::44112.980 Published as conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML,, 2021.
Alex Krichevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 888-896. PMLR, 16-18 Apr 2019. URL attps://proceeddngs.t Thass/v89/1iang199 html
17

---

Published in Transactions on Machine Learning Research (06/2023)
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, 2022. URL
Min Lin, Qiang Chen, and Shuichong Yan. Network In Network. arXi e-prints, art arXiv:1312.4400, December 2013.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar Xiv: 1692, 2019.
Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? closer look at distribution shift in datasct reproduction. In ICML Workshop Uncertainly and Robustness in Deep Learning, 2020.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8896-8905, 2018. doi: 10.1109/CVV CCCCCC20....00.00000
David A. McAllester. Pac-bayesian model averaging. In Proceedings of the Twelfth Annual Conference Computational Learning Theory, COLT '99, pp. 164-170, New York, NY, USA, 1999. Association for Computing Machinery. ISBN 1581131674. doi: 10.1145/304400.307444 URL https://doi.orr org/10...111.... 1145/
307400..07435
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization, 2021.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding oft-of- detection. Proceedings of the AAAI Conference on Artificial Intelligence, (6(7)):7811-844)) Jun. 2022, doi: 10.16 nnai.v361.22752.. URL https://ojs aaaii org/ index. hhtp/AAA//aattcll///e////22
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language inference by tree- based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 130-136, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: (411663/vv1, 2022262202.. URL
org/P16-200 2022
Vaishnavh Nagarajan and J Zico Kolter. Generalization in deep networks: The role of distance from initialization. In NeurIPS Workshop on Deep Learning: Bridging Theory and Praclice, 2019.
Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1-33 38, 2020. URL http:
7/jmr. cemlr.orgge otgppapprrs/v21/200022 html
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng, Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Behnam Neyshabur, Russ R Salakhutcinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015a. URL https: pdf
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm based capacity control in neural networks. In Peter Grünwald, Elad Hazan, and Satyen Kale (cds.), Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 1376-1401, Paris, France, 03 06 Jul 2015b. PMLR. URL wwwp:linnn..euriis...m.m... http://wwwppp...........o/aaaaaa
mlr http:///g.......pae/////////aa/a
18

---

Published in Transactions on Machine Learning Research (06/2023)
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. In Proceeding of NewrIPS, 2017a.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. In International Conference on Learning Representations, 2017b.
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. Samba: Self-supervised manifold based data augmen- tation for improving out-of-domain robustness. In Proc. of EMNLP, 2020.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and find-grained aspects. In Proceedings of EMNLP, 2019.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS '17, pp. 506 519, New York, NY, USA, 2017. URL
Association for Computing Machinery. ISBN 9781450349444. doi: 10.1145/30529733303383 http:://ddi.org/110 /ttp///....////793335555553
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR, 2017.
Jonas Peters, Peter Büihlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020,, 2021. URL https:
//axxiv. http:/.orrg/abb22200000022
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? 2018. https://arxiv.. http://aaxivv.gggbb////// wttp:///wrr//aaa//.....oo
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet. classifiers gen- eralize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389 5400. PMLR, 09-15 Jun 2019. URL
mlr ppesss/v97/rectt11/00 html
Alexandra Birch Rico Sennrich, Barry Haddow. Improving neural machine translation models with monolingual data. In Proc. of ACL, 2016.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In J. Shawe- R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https:
pdf
Alexey Romanov and Chaitanya Shivade. Lessons from natural language inference in the clinical domain. 2018. URL http::/aaxiv.oort http://.rgggab////b/eeaa wttpaaab./111./////
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)) Dec 2015. ISSN 1573-1405. doi: 10.10077/111223-01--000
1007 URL http:://ab...rr//0..0//////aa
19

---

Published in Transactions on Machine Learning Research (06/2023)
Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant equivariant deep networks via quotient feature spaces. In Conference on Uncerlainly in Arlificial Intelligence, 2021.
Yair Schiff, Brian Quanz, Payel Das, and Pin-Yu Chen. Predicting deep neural network generalization with perturbation response curves. In Newral Information Processing Systems, 2021.
Johannes Schmidt-Hieber. Deep ReLU network approximation of functions on a manifold. arXiv e-prints, art. 1rXiv:1908.00695, August 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations, 2015.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Generalization error of deep neural networks: Role of classification margin and data structure. In 2017 International Conference on Sampling Theory and Applications (Samp TA): pp. 147-151, 2017. doi: 10.1
Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2018.
Taiji Suzuki and Atsushi Nitanda. Deep learning is adaptive to intrinsie dimensionality of model smoothness in anisotropic besov space. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 6609-3621. Curran Associates, Inc., 2021. URL https://prooeeedingg wwwripe..rccoooo ccpp///////p///////e///e//// (tle:100063337777778377788888883
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105 16144 PMLR, 09-15 Jun 2019. URL htttps:/ wtapiiiiinnmmmmm.ppmmmeemm prtsserr9//t//ttntttte html
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 18583- 18599. Curran Associates, Inc., 2020. URL htttps://proceedinggs http:/eewccnnnnneoeerr..s.oooooo couripps.c/ppaer/2000/ pdf
file #111/688004551775555211111166660
V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probubility & Tts Applications, 16(2):264-2)) 280, 1971. doi: 10.1137/1116025. URL
https://aoi.org// /0t.:/0001.0////11111///6611515 1137/1116025
Ramakrishna Vedantam, David Lopez-Paz, and David J. Schwab. An empirical investigation of domain generalization with empirical risk minimizers. In Neural Information Processing Systems, 2021.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6438 6447. PMI.R, 09-15 Jun 2019. URL https://proceedingg: mlr. wwwpp//ee////// wwtpv////vve....../ html
Haohan Wang, Songwei Gc, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL
pdf
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natural Language Processing (EMNIP-IJCNLPP)
20

---

Published in Transactions on Machine Learning Research (06/2023)
pp. 6383-6389, Hong Kong, China, November 2019. Association for Computational Linguistics. URL https://www.aclweet www.wawlww...... wwtp:acllee..or//aattoooooo org./anthollggy/199-1555
J. Wiens, S. Saria, M. Sendak, M. Ghassemi, V. Liu, F. Doshi- Velez, K. Jung, K. Heller, D. Kale, M. Saeed, Ossorio, S. Thadaney-Israni, and A. Goldenberg. Do no harm: A roadmap for responsible machine learning for healthcare. Nature Medicine, 25(10):1337 1340, 2019.
Adina Williams, Nikila Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112 1122. Association for Computational Linguistics, 2018. URL http://aclweb. owglwweeb.rro.ee.......o anthology/ N18-1101
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidtt Robust fine-tuning of zero-shot models. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971, Junc 2022.
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. DisturbTabel: Regularizing CNN the Loss Layer. arXiv e-prints, art arXiv:1605.00055, April 2016.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe V Le. Unsupervised data augmentation for consistency training. arXiv preprint 1XX,:19004128400 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, arXiv preprint arXi::1611.05533 2016.
Yuichi Yoshida and Takeru Miyato. Spectral Norm Regularization for Improving the Generalizability of Deep Learning, arXiv e-prints, art. arXiv:1705.10944 May 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv e-prints, art. arXiv:1611.03530 November 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https::/openreviee:
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adapiation. In Kamalika. Chaudhuri and Ruslan Salakhutdinov (eds.), Proccedlings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7404 7413. PMLR, 09 15 Jun 2019. URL https::/proceeddng htaproceedinn..m..rreeeee///////
mlr wpwppes/97//haagiiiii
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. Proceedings of the AAAI Conference Artificial Intelligence, 341(07):13001-1300 Apr. 2020. doi: 10.1609 va4.7.70000 URL https://ojs. aaai. aeeii.orr/indee wwt.ndx...ppp/eee hhtppp///////rrrraal////////////
Sicheng Zhu, Bang An, and Furong Huang. Understanding the generalization benefit model invariance from a data perspective. In NewrIPS, 2021.
A Experimental Setup
In this section we present full details of our experimental setup, including data preprocessing and specifics on model architecture and hyperparameter space. All models are trained single RTX6000 GPU.
21

---

Published in Transactions on Machine Learning Research (06/2023)
A.1 Data Preprocessing
Large ImageNet scale datasets are preprocessed using the pipeline provided by Taori et al. (2020). Small scale image classification datasets are preprocessed by normalizing pixel values and resizing to 32 x 32 if
necessary. We use the same preprocessing steps for sentiment analysis and NI. experiments. All data is first. tokenized using GPT-2 style tokenizer and BPE vocabulary provided by fairseq (Ott et al. 2019 This
BPE vocabulary consists of 50263 types. Corresponding labels are encoded using a label dictionary consisting of as many types as there are classes. Input text and labels are then binarized for model training.
A.2 Model Architecture
The full list of the 196 models we evaluate from the Imagenet-Testbed (Taori et al. 2020 is provided below:
alexaat 1pf2, vit, large_ http_aar__.paaaaa_____ _334,, 384, vide reamet101_2 2, resmat50, aes basaline, resmat50, Saanatt0ffeatrre effricientteet---aarooong vegii_bo, resnet50_ reenet500_itth_jpegg withh_jpgg.omppeesiioo awa, BiT-W-R101x3-nonfinee resnet18, 1pt5, mobilanet_ v2, reanstão saal, denseneti21 B1T-4-81D1x1-nnffttttun reanext101 3228d_aaaal aval, reanet50_ _itts___tt____eeeeee___e fog_aw, reanet50, teroo_trreeedddddeeeee _0__81 IN, resmexti01_ reanet50_ _itth_-__it__----
densenat:69,
efficientuet--6- -a-utoou,,
contrast_ave, FizfeeelesSSCCMMii, resuet50, smagenet_eubeeett subsample___of_... $2__atcche__rrgghe siiiies,,,nee,,,,, respext101_32ke -_xx_dd_______---- exsl, squeszeneti_1, comet101_cctmiiin efficientnet-b3, resnst50, with_notion blubbblurrssssseer lat_larre__atchh pato_pptcch___e efficiantnat-b4, resnat50_-
Sannaattt_innngeess wwwsssaaabameeeeee_e__ 16, hatche4_originall images, rematti--roaatioon-ncrpppp
1pf3, dpo107, reamext101 _3xxdd__d__ deaaa_ddppaggeent. augnix, vgg19, resnst18_ sal, vagl3_bn, vgg13, reanet50 siaersss_iitt_pppee wttpisiiiittttpiiiitteeeeeee _aaa, semet154, resaet:18_lpf2, @buffleett_v_x1__ reanet50 augnix, resnet50 seamet5__ttmiann, aimaian, (rfecitttett----avroo-a-aaa
se se_resnet101, slexmet_ppfS, densemet121,
resnat50, inagenet_ www.snuuaaaeeeeee.e 5000 wlwpselaaaaaaeeeeee _laasses_batchee_ Sigginallinagee, vgg16_1pf2, umaemeti_0, 0, reemet34_ dpa68b, vobilemet__v__pppe efficientmet-bo, inceptionv3, reeneti8-rotattioo-orrstt
reemet101_1p33, slexmet, vgg16_ bu,
resmet152_8; sxs_siinnrr22_eeeeeuueee 100pet______ppp_______r_ resmet50_ mmagenet_subsapple 1f batch6c prigimal images, vida_resmet50_2 2, polynet, Sfficiinttett---raddee
dpn131, vggi6_-
bn, Jpt2, instagran-roanssiir.. 32x16d, "ggi6_nn_lpt resnat50 linf_ sint_apsB__nneeeeee lit__aaB_robbusse robust, inceptionv4, 6_32_clip___r_000000_ inages, reemet50_ withss__tttf inages, afficismttaet----attouuu reanet50_aal, vgg16_Ipf5, vit. base__aas___aeeeeeee _atae__tcchi____ 224, reamet34_ Iprs, resnet152, resnext50_-
resnet50, inagenstt 100peeeeepeeeeeebeeee batch644 originai--
wesnet50_inageet___ee
sfficienteet-b--attouu resnet50_inagenn
hacpppeee_______eeee 1et_eclaaeee_baaee_eaaaae aasch66_original_
32x4d, FixPMASKet, sithettt_with_eaaturat aatcchet_rriginn__. aarpttttbb--arpro--aattuuu resnet50, satees00_xit____ppeee vith_ _ass, remmet50, trained_an. on an_SIN, remnet50 strcllre_ffnntuuedd
FiafeeNee500uMMxx. v2, densenet122__pp5, reemet50, ALT---HB003-noofieetuee
remert8-rotatiigrraddd. 40, resnet50_ady- traiiiii--- rosnet18_1pf3,
finatuned, ft0p tf
port, pnasnetSlarge, B1T-N-95013-115WR222
reanet50, iaaattto__nnnnee_____ hatpameetubbanppe___. 250, classes batch64 priginal_inages. efficientnet-Ds,
resnet50 reenot50_ deepaugnent, efficientaet- -55 resmet50_1pf2,
wer.eeeeet0__2xee....ee reemet50_ aelpzeerohhot, images, vit_- exsl, nonfinttee,, NIT---R10113--LSSA22020 resnet50
1eexxt5__3xxd__-
large_patch16_ 224, baaiiiientutt---aattauu
sranet5o_vith.o goon_- blur_avs, resmext50_32xta s2xid_ssl, Fixiesliet50_12 v2, resmet50_1pfs resnet101, resmet18_svell officientaet- squeezeneti_0,
afficientaet-DS-uuttn
cemeet62-inngemetlll _uneet5__iincrr2___ liw.arrr_iieeerprbolo pronetr_rooe_tt_po poprr, alexmet_lpf3, bninception, afficciettatt----arrpp-----t _pp___ops__rrttt,, resnet50, 1inf epa4 robust, FixRanKet50, masnato_5, 5, resnet50, nixup, densanet121 1pt2, rematt 40, se_- reanext101 32x4d, saanett@-ottatiooraaddo...
officciettett-O--ateee mentrast__noton_bae btt_r_ippe_coppessiiooo.
efficiientee-
vggll, veanext101_32x66
151---B50x-ILLLWR0000 resuet50, reszet50_
sbufflenet_ 22__2_neee___________1_
dpa92, xception, resmet162_ 1ariit__xx__nnnere sx__iiclrr2, 1ineellnnrr____e
Linearrproboo__ probo_roobbb....eo popor, 4pn96, masncogrioa--magmmee21... sfficiinttett----drrro----
(es0ext101_32x166__ee0 32x66_eee, vit_base_patccee jase___ttch3__800 384, demsenet201, inceptionresseet2,, caffereenet101, rettattrr
resmet34, (ifteellet50_oo_ddpttttee
resmet101_1pf5, mobilenet 12__pp, mustagran-reseext100..
nasnetanobile, mobilanet, v2_ipf2, resmst101_ 1pp2, re_resnt550, dpn68, resnet50, waarrt00_ttt__triieeee wataetto0ettbb.rrgtteeeees
resmext101 _4x11110__6,4,,,_-111011111__,11
reanext101 321, vgg19 bn, fbreanat162, reanet50,
remetto0_dappuguae despaugnent, augnix, se resnati52, resnet50 cutout, coanat50_cutmii
reenet50 12_1____ppp trained_on, on, sxav, watpatt00xxtttgrrrrseeo
efficientnet-bl, reamet50 cittsss__tt__ddoooeeeee blur_ave, 111---100111-1155222222,, "gg16_ba_lpf3 resuet50_- Siteede_oo___i_e__eeeeeeeeeeee Sid__Iiiiddii_ii ai___nnd_______ theen_fnetumed_.cm rgglé, resmet34_lp$3,
naenetalargo, resnet50_ aaath__ausrinn_ane waausiin__oiie__eee moise_ave, vit_baree_atcchi__ee resuet50_
vggi6_1pf3, resneti8, densenet161
Our small image classification models are Network in Network (NiN) [Lin et al. 2013). VGG (Simonyan & Zisserman 2015), and CNN models. Training and hyperparemeter details for these models are provided in
Jiang et al. 22020).
For natural language tasks, our CNN models are based on the architecture Kim (2014). Our input embeddings are 512 dimensional, which we trcat as our channel dimension. Our base model applics set of three one dimensional convolutions of kernel size 3, 4, and with 256 output channels. We modulate the number of stacked convolutions (depth) as well as the channel size (width). Each convolution generates a separate representation that is max pooled across the sequence and concatenated together. We feed this representation into a MLP classifier with a single hidden layer of 512 dimensions. We apply dropout of 0.2 to our inputs and MLP classifier.
Our RoBERTa models use a pre-trained RoBERTaBASE model provided by fairseq. Classification token embeddings are fed into an MLP classifier with a single hidden layer of 512 dimensions. All models are written within the fairseq framework (Ott et al. 2019) and trained on a single RTX6000 or T4 GPU.
22

---

Published in Transactions on Machine Learning Research (06/2023)
CNN
RoBERTa
Hyperparameter
SA
NLI
SA
NLI
Batch Size
{32, 64, 128)
{32, 64, 128 } {1, 2, {128, 256, 512 {0.0, 0.25} 0.0
{8, 16, 32} 1
{8, 16, 32}
Depth
{1, 2,
Width
{ [228, 256, 512) {0.0, 0.25, 0.5)
768
768
Dropout
{0.0, 0.1) {0.0, 0.0001, 0.0005)
(0.0, 0.13 {0.0, 0.0001, 0.0005) (0.0, 0.2, 0.4)
Weight Decay
{0.0, 0.0001, 0.0005)
Label Noise
0.0
(0.0, 0.2, 0.4)
{0.0, 0.2, 0.4)
Table 6: Possible hyperparameter values for each architecture and task.
A.3 Model Hyperparameters
Hyperparameter values for image classification models are provided in Jiang et al. (2020) For natural language models we vary the following hyperparameters: training domain, batch size, depth, width, dropout, weight decay, and label noise. For training domains, on sentiment analysis we choose between books, clothing, home, kindle, movies, pets, sports, tech, tools, toys. For training domains on NLI, we choose between slate, government, fiction, telephone, travel. NLI datasets include additional test sets oup, nineeleven, facetoface, verbatim, letters. Possible values for all other hyperparameters are provided in Table
A.4 Model Training
All models are trained with the Adam optimizer (Kingma & Ba 2014 with 3 = (0.9, 0.98) and 1 x 10-6 CNN models trained with learning rate x 10-3 and RoBERTa models are trained with learning rate 1 x 10-5, 5, We use inverse square root learning rate scheduler to anneal learning rate over training, We early stop CNN models on sentiment analysis at 0.04 cross entropy and on NLI at 0.03 cross entropy. We carly stop RoBERTa models on sentiment analysis at 0.05 cross entropy and on NLI at 0.03 cross entropy. Training details for image classification models are provided in Jiang et al. (2020).
A.5 Transformation Magnitudes
We define how to determine the magnitude of each data transformation below, and use a maximum value based on best practices provided in their respective papers.
The magnitude of a transformation is determined by the magnitude parameter in the RandAugment algorithm as well the number of augmentations applied. In experiments we consider transformations with maximum magnitude of 15, with 3 augmentations for larger ImageNet models and 1 augmentation for smaller models.
Augment:
The magnitude of transformation is determined by the maximum percentage of the image the image will be translated in both the X- and Y-axes. In our experiments we consider translations of up to 10% of the size of the image in both axes.
Translate:
Erase:
The magnitude of the erase transformation is determined by the percentage of the image erased. In our experiments we consider transformations that remove a maximum size of 33% of the total image area and an aspect ratio between 1/3 and 10/3.
Flip and Crop: The magnitude of the flip and crop transformation is determined by the flip probability and the crop size. our experiments we consider transformations that flip the image 50% of the time and crop the image with lower bound of 8% of total image area and an aspect ratio between 3/4 and 4/3. Images are resized to 32 x 32 after cropping.
SSMBA: The magnitude of a SSMBA transformation is determined by the percentage of tokens corrupted, where of the tokens selected, 10% are unmasked, 10% are randomly replaced, and the
23

---

Published in Transactions on Machine Learning Research (06/2023)
remaining 80% are masked. In our experiments consider SSMBA transformations with maximum of 15% of tokens corrupted.
EDA: The magnitude of an EDA transformation is determined by the percentage of tokens noised. In experiments we consider transformations with maximum of 10% of the tokens.
The magnitude of a backtranslation operation is determined by the temperature of the softmax-ed distribution from which we sample tokens. In our experiments we consider transformations with maximum temperature of 0.7.
Backtranslation:
The magnitude of a random replacement operation is determined by the percentage of lokens replaced. In our experiments we consider transformations with maximum of 15% of tokens replaced.
Random Replacement:
Additional Experiments
B.1
Complexity Measures
Following (Jiang et al. 2019), we calculate our spectral norm measure as II w and Frobenius norm measure II W. We do not list results on these measures as the correlations are often negative or 0.
B.2 Cross-Domain Correlation
In this set of experiments we measure the correlation between neighborhood invariance and generalization values of a single model trained on single training domain evaluated across different OOD test domains. For natural language experiments we average correlations across all CNN and RoBERTa models and training domains and call these the CNN and Roberta т. Since these results rank correlations over only 9 values, they are quite noisy. Results are presented in Table 7
Neighborhood invariance performs quite poorly on both models, although they still outperforms ATC basclines. We hypothesize different regions of the input space may have different optimal levels of smoothness that achieve the lowest generalization error. Our value of interest is then not the absolute smoothness, but the relative smoothness compared to this optimal value. These values are the same when comparing different. models evaluated on the same domain, but not the same for the same model evaluated on different. domains, making correlating across domains difficult. On the natural image manifold where domains more well behaved and uniform compared to the natural language manifold, the relative smoothness may not differ much between domains allowing us to correlate our measure across domains.
B.3 Negative Entropy Results
As an alternative to defining the invariance the maximum value of the neighborhood decision distribution in Eq. we also consider defining it using the negative entropy of the same distribution:
v(fxx) = (p)())log))
jEY
A full table of results including metrics calculated on neighborhood invariance measured with negative entropy is provided Table 8 We refer to measures calculated with entropy as NE-SSMBA, NE-EDA, NE-BT, and NE-Random. For most metrics, NE** methods perform similarly or slightly worse.
C Full Results
We provide a full breakdown of results on the correlation metrics R² (Tables 10 (Tables
12
13
14
15
macro 7
17
18
19
20
21
22 miero т (Tables 23
24 and ID T (Tables 25
each set of datasets
and models. We also provide additional standard deviations for all main results Table
24

---

Published in Transactions on Machine Learning Research (06/2023)
Task
Measure
CNN
RoBERIa. 7
NI-SSMBA
0.360
0.010
NI-EDA
0.431
0.266
NI-BT
0.505
0.245
SA
NI-RandRep
0.570
0.150
ATC-NE
0.543
0.228
ATC-MC
0.539
0.224
NI-SSMBA
0.022
0.260
NI-EDA
0.102
0.335
NI-BT
0.089
0.333
NLA
NI-RandRep
0.226
0.440
ATC-NE 0.219
0,231
ATC-MC 0.223
0.239
Table 7: Correlation metrics evaluating the ability of our smoothness measure to predict OOD generalization across test datasets. Our smoothness measures achieves strong correlation in image classification tasks but fails in natural language tasks.
CNN
RoBERTa
Task Measure
R²
MAE
Macro т
Micro т
R2
MAE
Macro
Micro
NI-SSMBA
0.662
1.93
0.677
0.689
0.972
1.29
0.832
0.829
NI-EDA
0.641
2.04
0.664
0.649
0.968
1.45
0.830
0.810
NI-BT
0.550
2.99
0.592
0.501
0.961
1.47
0.813
0.801
Re-RandRep
0.409
2.64
0.544
0.554
0.967
1.27
0.821
0.816
NE-SSMBA
0.595
2.53
0.708
0.713
0.971
1.32
0.830
0.824
NE-EDA
0.534
2.71
0.698
0.674
0.965
1.55
0.825
0.801
SA
NE-BT
0.471
3.59
0.618
0.541
0.961
1.46
0.813
0.799
NE-RandRep
0.283
3.37
0.570
0.552
0.964
1.34
0.818
0.809
ATC-NE
0.530
3.80
0.506
0.642
0.8/1
3.59
0.681
0.706
ATC-MC
0.528
3.76
0.507
0.642
0.863
3.54
0.698
0.716
NI-SSMBA
0.575
2.09
0.570
0.534
0.933
1.19
0.750
0.730
NI-DDA
0.577
2.04
0.581
0.511
0.941
1.26
0.789
0.757
NI-BT
0.509
2.11
0.470
0.449
0.944
1.07
0.759
0.740
N--RandRep
0.451
2.20
0.452
0.428
0.890
1.70
0.688
0.647
NE-SSMBA
0.588
2.20
0.579
0.520
0.941
1.39
0.738
0.711
NE-EDA
0.606
2.11
0.597
0.512
0.937
1.48
0.767
0.732
NLI
NE-BT
0.536
2.27
0.480
0.422
0.954
1.12
0.761
0.750
NE-RandRey
0.457
2.36
0.451
0.397
0.904
1.79
0.665
0.591
ATC-NE 0.378
3.57
0.430
0.294
0.673
2.35
0.536
0.52
ATC-MC
0.382
3.57
0.433
0.297
0.718
2.21
0.570
0.556
Table 8: Correlation metrics evaluating the quality of our neighborhood invariance measure on two tasks, sentiment analysis and natural language inference, and architectures, CNN and RoBERTa. Details on metric calculations and baselines are provided in sections 4.3 and 4.5 This full table of results includes metrics calculated with neighborhood negative entropy measure as well.
25

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Measure
I2
Imagen
ObjectNet
ImageNet- Vid
YTBB
ImageNet-IR
ImageNet-A
Aug
0.810
0.641
0.613
0.767
0.570
0.763
0.377
NI-Translate
0.794
0.436
0.461
0.642
0.395
0.560
0.468
NI-Erase
0.715
0.222
0.381
0.635
0.399
0.398
0.146
NI-FC
0.706
0.679
ATC-MC ATC-NE 0.991
0.980
0.710
0.151
0.552
0.200
0,463
0.159
0.709
0.392
0.484
0.229
0.441
0,209
Table 9: Full R² metrics for all ImageNet test domains.
We average values across models.
Test Domain
Model
Train Domain
Measure
SVHN
Colored MNIST
MN18T
NI- NI-RandAug
0.785
0.744
NI- -Ta'saasltte
0.728
0.642
NiN
SVHN
NI-Erase
01088
0.218
NI-FC
0.134
0.283
ATC-NE
0.506
0.725
ATC-MC
0.615
0.769
Table 10: Full R² metrics for all test domains for image classification models on SVHN, Colored MNIST, and MNIST.
Test Domain
Model
Train Domain
Measure
CIFARIO
CINICI0
CIFAR10.1
CIFARI0.2
Al-HandAug
0.898
0.927
NI-Translate
0.688
0.864
0.730
NIN
CIFARIO
NI-Brase
0.404
0.326
0.547
NL-FC
0.109
0.028
0.000
ATC-NE ATC-MC
0.237
0.575
0.435
0.252
0.538
0.390
Ml-RandAug
0.8
0.844
0.8553
NI-Translste
0.889
0.899
0.836
NosNot
CIFARIO
NI-Erase
0.807
0.812
0.783
NI-FC
0.726
0.628
0.0G0
ATC-NE ATC-MC
U.736
0.782
0.693
NI-RandAug
0.969
0.950
0.929
NI-Translste
0.8:9
0.844
0.809
VGG
CIFAR10
NI-Erase
0.096
0.100
0.104
NL-FC
0.637
0.524
ATC-NE
0.557
0.774
ATC-MC
0.669
0.764
0.709
NI-RandAug
0.922
0.876
0.865
NI-Translate
0.516
0.449
0.407
CNN
CINICIO
EraEra
NI-FC
0.416
0.397
0.395
ATC-NE
0.869
0.750
0.724
ATC-MC
0.568
0.741
0.716
Table 11: Full RD metrics for all test domains for image classification models on CIFAR10, CINIC10, CIFAR1O.1, and CIFAR10.2.
26

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trii Domain
Measure
books
clothing
kindle
novies
pets
sports
tech
toys
NI-SSMBAA
0.688
0.771
0,752
0.685
0.823
0.762
0.697
0.711
0.729
NI-EDA
0.720
0.725
0.699
0.689
0.810
0. 0.693
0.663
0.727
0.806
NI-BT
0.560
0.663
0.639
0.583
0.675
0.602
0.591
0.592
0.658
booka
NI- Nl-RandHep
0.413
0.418
0.331
0.343
0.479
0.373
0.504
0.401
0.496
ATC-NE
0.785
0.859
0.828
0.777
0.834
0.834
0.791
0.814
0.872
ATC-MC
0.759
0.854
0.821
0.771
0.828
0.830
0.790
0.808
0.871
NI-SSMBA
0.517
0.001
0.531
0.525
0.535
0.506
0.484
0.549
0.631
NL-DA
0.409
0.419
0.452
0.408
0.395
0.457
0.388
0.486
0.534
0.234
0.164
0.267
0.273
0.121
0.191
0.164
0.193
0.355
clothing
NI- Nl-
0.236
0.237
0,223
0.250
0.212
0.213
0.156
0.246
0.335
ATC-NE
0.480
0.673
0.478
01.667
0.466
0.739
0.299
0.772
018447
ATC-MC
0:477
0.675
0.177
0.106
0:174
0.742
0.300
0.772
0.848
NI-SSMBA
0.533
0.545
0.524
0.511
0.665
0.648
0.576
0.604
0.592
NI-EDA
0.412
0.554
0.451
0.433
0.586
0.496
0.417
0.525
0.657
NI-BT
0.313
0,260
0.527
0.316
0.346
0.366
0.304
013385
0.408
home
Ni-RandRep
0.273
0,260
0,236
0.325
0.286
0.299
0,270
0.282
0.370
ATC-NE
0.608
0,861
0.675
0.618
0,838
0.838
0.720
0.863
0.804
ATC-MC
0.612
0.861
0.679
0.626
0.838
0.839
0.719
0.863
0.895
NL-SMBA
0.645
0.680
0,650
0.626
0.765
0.659
0.551
0.634
NI-DA
0.722
0.688
0.005
0.602
0.785
0.600
0.658
0.623
0.783
NI-BT
0.655
0.674
0.684
0.625
0.745
0.704
0.695
0.649
0.735
kindle
Ni-RandRep
0.325
0.361
0,253
0.269
0.412
0.292
0.241
0.274
0.145
ATC-NE
0.717
0.659
0.701
0.690
0.792
0.717
0.505
0.610
0.776
ATC-MC
0.759
0.642
0.600
0.687
0.784
0.708
0.507
0.504
0.765
NL-SMMAA
0.541
0.640
0.658
0.615
0.633
0:656
0.583
0.653
0.741
NI-EDA
0.542
0.662
0.571
0.678
0.629
0.594
0.574
0.663
0.789
NI-BT
0.602
0.679
0.677
0.653
0.723
0.739
0.700
0.709
0.751
novies
N-Randdep
0.194
0.276
0,281
0.228
00.166
0.236
0.316
0.369
ATC-NE
0.698
0.719
0.708
0
0.718
0.707
0.611
0.752
0.818
ATC-MC
0.707
0.725
0.713
0.664
0.732
0.711
0.624
0.755
0.820
NLSSMMBA
0.458
0.543
0.578
0.529
0.548
0:606
0.528
0.590
0.721
NI-EDA
0.426
0.352
0.554
0.467
0.513
0.546
0.463
0.549
0.677
NI-BT
0.185
0.523
0.177
0.5:9
0.563
0.581
0.550
0.580
0.587
Ne-RandRep
0:200
0.333
0.284
0.302
0.320
0.304
0:271
0.345
0.432
ATC-NE
0.641
0.851
0.849
0.651
0.680
0.825
0.591
0.812
0.883
ATC-MC
0.6-44
0.846
0.8-18
0.648
0.683
0.824
0.589
0.810
0.883
NI-SMMBA
0.499
0.330
0.569
0.524
0.463
0.573
0.517
(1500)
0.721
NLEDA
0.464
0.514
0.480
0.489
0.381
0.508
0.448
0.541
0.594
NI-BT'
0.406
0.295
0.331
0.383
0.381
0.325
0.331
01,993
0.411
sporta
Ni-RandRep
0.253
0.173
0.212
0.256
0.219
0.239
0.204
0.231
0.331
ATC-NE
0.712
0.900
0.926
0.744
0.656
0.891
0.828
0.931
0.953
ATC-MC
0.723
0.897
0.926
0.754
0.665
0.896
0.833
0.933
0.952
NI-SMMA
0.610
0.308
0.635
0.620
0.506
0.604
0.636
0.545
0.848
NI-EDA
0.550
0.475
0.608
0.581
0.587
0.532
0.619
NI-BT
0.588
0.447
0.572
0.632
0.597
0.558
0.603
0.523
0.648
Lech
Nl-RandRep
0.310
0.241
0.262
0.307
0.337
0.285
0.291
0.200
0.132
ATC-NE
0.706
0.820
0.904
0.791
0.817
0.866
0.874
0.882
0.893
ATC-MC
0.771
0.823
0.904
0.794
0.817
0.864
0.875
0.882
0.893
NI-SMMAA
0.554
0.505
0.548
0.556
0.605
0.651
0.806
0.577
0.680
NI-DDA
0.466
0.413
0.589
0.587
0.512
0.465
0.593
NI-BT
0.451
0.385
0.447
0.476
0.482
0.501
0.483
0.48-
0.462
toole
Ne-RandRep
0.323
0.241
0.204
0.306
0.390
0.316
0.253
0.227
0.297
ATC-NE
0.661
0.875
0.901
0.679
0.719
0.867
0.908
0.795
0.916
ATC-MC
0.670
0.874
0.903
0.684
0.729
0.866
0.909
0.801
0.916
NI-SSMBA
0.625
0.693
0.656
0.620
0.613
0.661
0.708
0.618
0.650
NI-EDA
0.473
0.582
0.487
0.481
0.498
0.552
0.535
0.429
0.528
NI-BT
0.311
0.408
0.331
0.351
0.321
0.355
0.420
0.331
0.357
toys
NI- Re-RandRep
0.272
0.307
0.234
0.254
0.286
0.257
0.301
0.215
0.248
ATC-NE
0.656
0
0.855
0.742
0.712
0.838
0.985
0.561
0.866
MTC-MC
0.691
0.933
0.858
0.742
0.718
0.843
0.886
0.574
0.869
Table 12: Full R² metrics for all pairs of training and test domains for CNN models trained on AWS.
27

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trii Domain
Measure
books
clothing
kindle
novies
pets
sports
tech
toys
NI-SSMBAA
0.973
0.983
0.987
0.963
0.973
0.985
0.985
0.977
0.968
NI-EDA
0.961
0.970
0.982
0.966
0.966
0.970
0.957
0.978
0.981
NI-BT
0.946
0.979
0.985
0.961
0:972
0.972
0.975
0.970
0.955
books
NI- Ni-RandHep
0.956
0.970
0.981
0.958
0.967
0.970
0.973
0.952
0.961
ATC-NE
0.845
0
0.957
0.927
0.910
0.852
0.925
0.853
0.876
ATC-MC
0.838
0.8-1
0.947
0.921
0.929
0.840
0.927
0.845
0.876
NI-SSMBA
0.944
0.974
0.942
0.980
0.981
0.984
0:973
0.969
0.978
NL-DA
0.974
0.970
0.922
0.965
0.978
0.961
0.971
0.953
0.972
0.944
0.977
0.921
0.971
0.983
0.982
0.980
0.970
0.993
clothing
NI Nl-onnRep
0.950
0.965
0.933
0.982
0.977
0.985
0.981
0.974
0.973
ATC-NE
0.792
0.934
0,764
0.913
0.933
0.963
0.846
0.914
0.973
ATC-MC
0.801
0.9-1
0.789
0.917
0.918
0.933
0.887
0.926
0.971
NI-SSMBA
0.973
0.972
0.976
0.972
0.979
0.976
0.980
0.972
0.984
NI-EDA
0.961
0.975
0.967
0.950
0.960
0.949
0.917
0.963
0.979
NI-BT
0.947
0.958
0.944
0.947
0.948
0.977
0.972
0.971
0.974
home
Ni-RandRep
0.978
0.976
0,970
0.967
0.975
0.973
0.961
0.975
0.983
ATC-NE
0.878
0.959
0.958
0.908
0.930
0.962
0.860
0.928
0.951
ATC-MC
0.885
0,960
0.963
0.913
0.915
0.961
0.872
0.938
0.952
NL-SMBA
0.992
0.963
0.961
0.957
0.963
0.971
0.984
NI-DA
0.963
0.968
0.951
0.966
0.962
0.963
0.974
0.963
0.960
NI-BT
0.973
0.947
0.975
0.954
0.949
0.954
0.973
0.923
0.973
kindle
NI-RandRep
0.979
0.916
0.956
0.967
0.961
0.916
0.970
0.934
0.975
ATC-NE
0.931
0.350
0,510
0.861
0.162
0.200
0.121
0.179
0. 0.608
ATC-MC
0.930
0.534
0.623
0.871
0.294
0.302
0.551
0.316
0.672
NL-SSMBA
0.984
0.976
0.969
0.981
0.962
0.983
0.958
0.961
0.971
NI-EDA
0.974
0.975
0.947
0.985
0.928
0.977
0.950
0.972
0.969
NI-BT
0.971
0.952
0.965
0.965
0.951
0.970
0.957
0.950
0.977
novies
N--anddep
0.981
0.969
0.971
0.958
0.969
0.957
0.958
0.962
ATC-NE
0.918
0.861
0,902
0.919
0.746
0.908
0.813
0.817
0.917
ATC-MC
0.949
0.893
0.928
0.948
0.802
0.929
0.861
0.859
0.926
NLSSMMBA
0.943
0.974
0.981
0.945
0.942
0.979
0.982
0.978
0.969
NI-EDA
0.965
0,980
0.985
0.958
0.952
0.983
0.975
0.976
0.983
NI-BT
0.986
0.971
0.971
0.931
0.926
0.980
0.963
0.979
0.977
Ne-RandRep
0.941
0.976
0.971
0.926
0.931
0.976
0.908
0.976
0.979
ATC-NE
0.594
0.967
0.900
0.577
0.862
0.947
0.959
0.929
0.882
ATC-MC
0.611
0.968
0.909
0.589
0.861
0.965
0.959
0.939
0.894
NI-SMMBA
0.979
0.952
0.994
0.980
0.979
0.970
0.982
0.984
0.978
NL-DA
0.979
0.987
0.986
0.980
0.984
0.985
0.985
0.986
0.987
NI-BT'
0.966
0.977
0,989
0.919
0.969
0.979
0.986
(..984
0.988
sporta
Ni-RandRep
0.981
0.979
0.994
0.978
0.980
0.978
0.985
0.985
0.984
ATC-NE
0.655
0.945
0.963
0.688
0.850
0.890
0.938
0.960
0.957
ATC-MC
0.705
0.951
0.965
0.722
0.882
0.905
0.944
0.972
0.953
NI-SMMA
0.944
0.979
0.977
0.947
0.967
0.958
0.975
0.990
0.981
NI-DDA
0.917
0.969
0.953
0.937
0.967
0.963
0.972
NI-BT
0.804
0.982
0.987
0.924
0.987
0.933
0.978
0.987
0.972
Lech
Nl-RandRep
0.929
0.959
0.977
0.918
0.952
0.914
0.963
0.984
0.971
ATC-NE
0.937
0.939
0.983
0.933
0.946
0.961
0.961
0.974
0.981
ATC-MC
0.941
0.941
0.980
0.928
0.955
0.958
0.963
0.971
0.982
NI-SMMAA
0.977
0.988
0.975
0.987
0.971
0.986
0.985
0.978
0.983
NI-DDA
0.975
0.985
0.978
0.969
0.980
0.986
0.985
0.975
0.982
NI-BT
0.940
0.976
0.970
0.941
0.944
0.969
0.978
0.948
0.976
toole
Ne-RandRep
0.963
0.985
0.978
0.954
0.963
0.988
0.984
0.977
0.987
ATC-NE
0.857
0.945
0.964
0.883
0.865
0.925
0.964
0.963
0.922
ATC-MC
0.568
0.941
0.965
0.887
0.876
0.936
0.967
0.971
0.919
NI-SSMBA
0.955
0.971
0.985
0.980
0.961
0.966
0.988
0.978
0.971
NI-EDA
0.965
0.968
0.984
0.970
0.985
0.973
0.981
0.975
0.983
NI-BT
0.890
0.916
0,959
0.911
0.925
0.935
0.965
0.943
0.934
toys
NI- NI-RandRep
0.950
0.970
0.979
0.960
0.965
0.967
0.977
0.978
0.974
ATC-NE
0.883
0.878
0.885
0.763
0.906
0.798
0.938
0.815
0.890
MTC-MC
0.889
0.897
0.895
0.754
0.920
0.818
0.947
0.833
0.914
Table 13: Full R² metrics for all pairs of training and test domains for BERT models trained on AWS.
28

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Train
Measure
verbatin
oup
nineslaven
talephona
latters
government
NLSSMBA
0.544
0.642
0.706
0.727
0.640
0.714
NI-EDA
0.574
0.723
0.565
0.620
0.708
0.731
0.5419
0.595
0.706
NL-BT
0.862
0.920
0.942
0.930
0.913
0.878
0.936
0.949
0.940
slate
NI-RandRep
0.242
0.336
0.501
0.415
0.121
0.170
0.301
0.652
0.170
ATC-NE
0.681
0.677
0.594
0.599
0.741
0.609
0.664
0.661
0.773
ATC-MC
0.682
0.675
0.590
0.600
0. 737
0.670
0.685
0.661
0.771
NI-SMMBA
0.581
0.492
0.765
0.451
0.530
0.683
0.499
0.502
0.614
NI-EDA
0.637
0.601
0.728
0.620
0,530
0.676
0.527
0.508
0.612
NI-BT
0.889
0.887
0.917
0,950
0.921
0.863
0.928
0.935
0.941
fiction
NI-RandRep
0,360
0.114
0.459
0.666
0.531
0.188
0.461
0.680
0.631
ATC-NE
0,530
0.425
0.718
0.323
0.536
0.556
0.425
0.597
0.522
ATC-MC
0.525
0.118
0.719
0.326
0.531
0.553
0.122
0.592
0.521
NI-SSMBA
0.559
0.439
NI-EDA
0.677
0.636
0.801
0.192
0.639
0.881
0.557
0.653
0.568
NI-BT
0.882
0,901
0.928
0,936
0.927
0.912
0.937
0.916
0.919
telaphone
NI-RandRep
0.332
0.198
0.488
0.584
0.538
0.310
0.586
0.677
0.576
ATC-NE
0.557
0.554
0,468
ATC-MC
0.561
0.518
0,762
0.521
0.467
0.698
0.372
0.560
0.170
NI-SSMBA
0.503
0.506
0.588
0.419
0.501
0.543
0.597
0.670
0.507
NLEDA
0.444
0.429
0.502
0.388
0.342
0.532
0.478
0.438
0.377
NI-BT
0.802
0.923
travel
NI-RandRep
0.315
0.333
0.385
0.643
0.516
0.182
0.221
0.690
0.026
ATC-NE
0.561
0.516
0.498
0.686
0.546
0.581
0.606
0.585
0.598
ATC-MC
0.563
0.517
0. 0.503
0.690
0.514
0. 0.588
0.611
0.582
0.007
NI-SSMBA
0.592
0.497
0.618
0.596
0.648
0.593
0.539
0.676
NLEDA
0.577
0.574
0.550
0.476
0.618
0.600
0.491
0.518
0.526
NI-BT
0.818
0,812
0.859
0.909
0.893
0.813
0.809
0.909
0.917
government
NI- Re-andRe
0.010
0.088
0.207
0.306
0.279
0.011
0.027
0.123
0.360
ATC-NE
0.683
0.337
0.630
0.653
ATC-MC
0.666
0.405
0.574
0.721
0.341
0.633
0.636
0.509
0.654
Table 14: Full R² metrics for all pairs of training and test domains for CNN models trained MNLI.
29

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trai Domain
Measure
slate
verbatin
facetoface
oup
nineeleven
fiction
telephone
travel
letters
goverament
NL-SSMBA
0.931
NLEDA
0.701
0.754
0.816
0.784
0.790
0.708
0.787
0.820
0.837
NL-BT
0.862
0.920
0.942
0.930
0.913
0.878
0.936
0.949
0.940
slate
NI-RandRep
0.212
0.336
0.501
0.415
0.121
0.170
0.301
0.652
0,170
ATC-NE
0.890
ATC-MC
0.871
0.884
0.953
0.908
0.932
0.914
0.911
0.587
0.949
NI-SSMBA
0.954
0.928
0.895
0.952
0.946
0.952
0.975
0.946
0.954
NLEDA
0.661
0.687
0.600
0.840
0.686
0.683
0.764
0.799
0.797
NL-TT
0,889
0.863
riction
NI-RandRep
0.360
0.414
0.459
0.666
0.531
0.188
0.464
0.680
0.631
ATC-NE
0,808
0.585
0.881
0.726
0.939
0.836
0.826
0.850
0,890
ATC-MC
0.817
0.652
0.889
0.732
0.936
0.841
0.830
0.861
0.892
NI-SSMBA
0.943
0.962
0.948
0.953
0.917
0.954
0.921
0.934
0.956
NLEDA
0.776
0.787
0.790
0.852
0.823
0.847
0.853
0.877
0.852
NI-BT
0.882
0.901
0.928
0.936
0.927
0.912
0.987
0.916
0.919
telephone
NI Re-RanRRep
0.332
0.498
0.488
0.584
0.538
0.310
0.586
0.677
0.576
ATC-NE
0.062
0.576
0.884
0.904
0,762
0.605
0.771
0.856
0.889
ATC-MC
0.714
0.868
0.902
0.924
0.807
0.732
0.795
0.857
0.899
NI-SSMBA
0.938
0.935
0.949
0.924
0.948
0.946
0.958
0.962
0.953
NI-EDA
0.538
0.553
0.171
0.711
0.602
0.638
0.533
0.735
0.699
NLBT
0.806
0.799
0.860
0.828
0.802
0.923
0.007
travel
NL-RandRep
0.315
0.333
0.385
0.643
0.516
0.182
0.224
0.690
0.626
ATC-NE
0.319
0.417
0.500
0.889
0.768
0.396
0.494
0.928
0.911
ATC-MC
0.412
0.506
0.591
0.906
0.789
0.525
0.572
0.921
0.917
NI-SSMBA
0.916
0.960
0.919
0.816
0.882
0.914
0.950
0.920
NI-EDA
0.181
0.572
0.514
0.728
0.715
0.655
0.506
0.611
0.681
NL-BT
0.818
0.812
0.859
0.909
0.893
0.813
0.800
0.909
0.917
government
NI-RandRep
0.010
0.083
0.207
0.306
0.279
0.011
0.027
0.123
0.360
ATC-NE
0.544
0.385
0.439
0.731
0.449
0.538
0.842
ATC-MC
0.624
0.181
0.478
0.919
0.797
0.243
0.505
0.576
0.856
Table 15: Full n2 metrics for all pairs of training and test domains for BERT models trained MNLI.
Test Domain
Measure
ImageNetV2
ImagenNettSSettch
ObjectNet
ImageNet-Vid
YTBB
ImageNet-R
ImageNet-A
RandAug
0.816
0.677
0.646
0.790
0.692
0.724
0.586
NI-Translat
0.785
0.515
0.516
0.683
0.511
0.581
0.419
NI-Erase
0.779
0.322
0.52
0.706
0.560
0.444
0.517
NL-FC
0.842
0.552
0.645
0.780
0.649
0.679
0.589
ATC-MC ATC-NE
0.708
0.756
0.761
0.521
0.926
0.717
0.671
0.651
0.491
0.569
0.248
Table 16: Full macro 7 metrics for all ImageNet. Lest domains. We average values across models.
Test Domain
Model
Train Domain
Measure
SVHN
Colored MNIST
MNIST
NI-RandAug
0.068
0.616
NI- Tr-arasseeee
0.699
0.635
NiN
SVHN
NI-Erase
-0.102
-0.168
NI-FC
-0.233
-0.398
ATC-NE
0.577
0.695
ATC-MC
0.646
0.716
Table 17: Full macro metrics for all test domains for image classification models on SVHN, Colored MNIST, and MNIST.
30

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Model
Train Domain
Messure
CIFAR10
CINIC10
CIFAR10.1
CIFAR10.2
Aug
0.785
0.830
0.783
NI-Translate
0.621
0.739
0.644
NiN
CTFARIO
NI-Frase
0.385
0.485
0.499
NI-FC
0.194
0.077
-0.037
ATC-NE ATC-MC
0.250
0.543
0.484
NI-RandAug
0.706
0.722
0.742
NI-Translate
0.800
0.796
0.741
ResNet
CIFAR10
NI-Erase
0.723
0.727
0.699
NI-FC
0.585
ATC-NE
0.687
ATC-MC
0.631
0.672
0.624
NI-RandAug
0.868
0.831
0.772
NI-Translate
U.700
0.679
0.632
VGG
CIFARIO
NI-Brase
NI-FC
0.597
0.569
0.512
ATC-NE
0.531
0.656
0.599
ATC-MC
0.533
0.648
0.586
NI- Aug
0.756
0.698
0.695
NI-Translate
0.318
CNN
CINICI0
NI-Erase
0.619
0.539
0.577
NI-FC
0.249
0.286
0.252
ATC-NE
0.607
0.510
0.447
ATC-MC
0.611
0.500
Table 18: Full macro т metrics for test domains for image classification models on CIFARIO, CINIC1O, CIFAR10.1, and CIFAR10.2.
31

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trii Domain
Measure
books
clothing
kindle
novies
pets
sports
tech
toys
NI-SSMBAA
0.708
0.741
0.687
0.678
0.754
0.738
0.738
0.721
0.689
NI-EDA
0.722
0.738
0.640
0.685
0.756
0. 0.688
0.709
0.719
0.725
NI-BT
0.615
0.631
0.549
0.596
0:656
0.602
0.644
0.584
0.621
booka
NI- Nl-RandHep
0.391
0.586
0.499
0.587
0.567
0.530
0.530
0.548
0.589
ATC-NE
0.605
0.612
0.534
0. 096
0.629
0.605
0.538
0.618
0.673
ATC-MC
0.004
0.639
0.534
0.489
0.630
0.606
0.544
0.616
0.670
NI-SSMBA
0.769
0.774
0.773
0.793
0.726
0.603
0.716
0.692
0.709
NL-DA
0.681
0.670
0.787
0.721
0.696
0.675
0.694
0.690
0.784
0.574
0.542
0.654
0.505
0.481
0.506
0.501
0.503
0.679
clothing
NI- Nl-
0.623
0.591
0.603
0.612
0.557
0.496
0.507
0.549
0.640
ATC-NE
0.420
0.366
0.408
0.450
0.308
0.501
0.242
0.560
0.566
ATC-MC
0.421
0.363
0,109
0.151
0.315
0.198
0.239
0.558
0.563
NI-SSMBA
0.786
0.352
0.731
0.731
0.769
0.790
0.789
0.775
0.614
NI-EDA
0.734
0.659
0.767
0.649
0.757
0.707
0.723
0.696
0.748
NI-BT
0.670
0.498
0.641
0.639
0.634
0.706
0.688
0.662
0.630
home
Ni-RandRep
0.700
0.519
0.685
0.705
0.686
0.603
0.661
0.632
0.651
ATC-NE
0.450
0.450
0.459
0..47
0.455
0.559
0.497
0.588
01.45
ATC-MC
0:159
0.480
0.160
0.152
0:458
0.557
0.195
0.557
0.418
NL-SMBAA
0.546
0.679
0.557
0.609
0.656
0.608
0.630
NI-DA
0.663
0.644
0.717
0.637
0.743
0.699
0.676
0.643
0.696
NI-BT
0.556
0.648
0.652
0.562
0.686
0.683
0.668
0.643
0.674
kindle
Ni-RandRep
0.454
0.519
0.428
0.355
0.431
0.115
0.415
0.535
ATC-NE
0.370
0.517
0,168
0.338
0.583
0.540
0.383
0.172
0.545
ATC-MC
0.358
0.512
0.469
0.342
0.578
0.538
0.384
0.467
0.541
NL-SMMA
0.572
0.689
0.717
0.660
0:732
0.741
0.675
0.714
0.749
NI-EDA
0.500
0.717
0.711
0.629
0.733
0.725
0.660
0.719
0.748
NI-BT
0.619
0.681
0.687
0.595
0.738
0. 0.729
0.717
0.717
0.717
movies
N--Rnddeep
0.435
0.491
004660
0.5228
0.436
0.511
0.370
ATC-NE
0.136
0.605
0.623
0,398
0.571
0.622
0.502
0.669
0.701
ATC-MC
0.449
0.612
0.631
0.403
0.586
0.628
0.520
0.661
0.704
NLSSMMBA
0.747
0.500
0.653
0.766
0.700
0.719
0.696
0.697
0.731
NI-EDA
0.762
0.684
0.682
0.794
0.773
0.709
0.668
0.689
0.745
NI-BT
0.711
0.350
0.625
0.751
0.776
0.701
0.676
0.702
0.651
Ne-RandRep
0.619
0.375
0.572
0.696
0.602
0.580
0.532
0.500
0.710
ATC-NE
0.570
0.545
0.462
0.533
0.543
0.511
0.333
0.503
0.634
ATC-MC
0.575
0.541
0.461
0.535
0.546
0.510
0.332
0.502
0.632
NI-SMMBA
0.774
0.361
0.657
0.812
0.764
0.683
0.689
0.746
0.651
NLEDA
0.770
0.633
0.659
0.791
0.710
0.712
0.701
0.732
0.692
NI-BT'
0.670
0.484
0.598
0.689
0.672
0.545
0.534
01.704
0.538
sporta
NI-RandRep
0.684
0.472
0.562
0.704
0.638
0.589
0.614
0.622
0.693
ATC-NE
0.502
0.456
0.454
0.506
0.443
0.374
0.397
0.565
0.577
ATC-MC
0.513
0.459
0.459
0.521
0.453
0.393
0.410
0.574
0.574
NI-SMMA
0.784
0.674
0.768
0.793
0.779
0.755
0.785
0.690
0.746
NI-DDA
0.718
0.698
0.750
0.752
0.817
0.772
0.726
0.781
NI-BT
0.715
0.576
0.719
0.732
0.746
0.716
0.726
0.687
0.683
Lech
Nl-RandRep
0.680
0.647
0.632
0.686
0.608
0.677
0.601
0.553
0.742
ATC-NE
0.547
0.688
0.681
0.560
0.640
0.653
0.606
0.637
0.723
ATC-MC
0.553
0.690
0.678
0.567
0.630
0.650
0.664
0.633
0.721
NI-SMMAA
0.783
0.541
0.604
0.777
0.782
0.757
0.758
0.730
0.682
NI-DDA
0.716
0.341
0.735
0.764
0.722
0.696
0.745
NI-BT
0:691
0.342
0.454
0.681
0.677
0.567
0.575
0.636
0.513
toole
Ne-RandRep
0.661
0.550
0.581
0.675
0.685
0.668
0.623
0.584
0.687
ATC-NE
0.621
0.472
0.480
0.647
0.648
0.536
0.508
0.448
0.619
ATC-MC
0.630
0.465
0.491
0.649
0.655
0.538
0.507
0.456
0.622
NI-SSMBA
0.760
0.702
0.758
0.763
0.731
0.775
0.782
0.750
0.753
NI-EDA
0.680
0.678
0.061
0.757
0.646
0.736
0.71
0.661
0.700
NI-BT
0.562
0.601
0.683
0.626
0.610
0.695
0.723
0.600
0.658
toys
NI- NI-RandRep
0.629
0.680
0.575
0.661
0.600
0.605
0.500
0.543
0.582
ATC-NE
0.401
0.617
0.309
0.479
0.370
0.323
0.485
0.143
0.526
MTC-MC
0.408
0.614
0.315
0.176
0.368
0.329
01.888
0.146
0.533
Table 19: Full macro metrics for all pairs of training and test domains for CNN models trained on AWS.
32

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trii Domain
Measure
books
clothing
kindle
novies
pats
sports
tech
toys
NI-SSMBAA
0.886
0.906
0,895
0.901
0.900
0.925
0.918
0.931
0.857
NI-EDA
0.883
0.851
0.890
0.862
0.863
0.854
0.855
0.895
0.811
NI-BT
0.830
0.887
0.803
0.883
0.884
0.871
0.878
0.960
0.824
booka
NI- Nl-RandHep
0.882
0.871
0.908
0.906
0.897
0.894
0.900
0.913
0.846
ATC-NE
0.803
0.814
0,851
0.821
0.885
0.756
0.843
0.774
0,832
ATC-MC
0.789
0.806
0.850
0.794
0.874
0.700
0.845
0.768
0.824
NI-SSMBA
0.797
0.882
0.714
0.773
0.896
0.881
0.826
0.883
0.824
NL-DA
0.828
0.848
0.813
0.787
0.862
0.741
0.839
0.848
0.777
clothing
0.817
0.903
0.696
0.837
0.832
0.849
0.635
0.823
0.778
NI- Nl-onnRep
0.762
0.858
0.691
0.826
0.832
0.872
0.822
0.868
0.786
ATC-NE
0.729
0,805
0.672
0.681
0.817
0.728
0.698
0.792
0.791
ATC-MC
0.735
0.791
0.683
0.701
0.858
0.711
0.736
0,808
0.791
NI-SSMBA
0.736
0.800
0.786
0.780
0.868
0.896
0.876
0,868
0.885
NL-DD
0.680
0.885
0.768
0.830
0.880
0.879
0.843
0.877
0.930
NI-BT
0.749
0.758
0.756
0.778
0.819
0.571
0.835
0.906
0.866
home
Ni-RandRep
0.729
0.811
0.701
0.759
0.853
0.981
0.831
0.876
0.871
ATC-NE
0.691
0.788
0,750
0.802
0.791
0.797
0.739
0.824
0.824
ATC-MC
0.681
0.782
0.738
0.793
0.800
0.772
0.751
0.826
0.819
NL-SMBA
0.735
0.850
0,885
0.701
0.913
0.875
0.569
0.871
NI-DA
0.840
0.830
0.824
0.700
0.879
0.856
0.665
0.864
0.790
NI-BT
0.727
0.848
0.892
0.763
0.856
0.926
0.871
0.879
0.861
kindle
Ni-RandRep
0.773
0.818
0.735
0.875
0.857
0.843
0.817
ATC-NE
0.695
0.335
0,186
0.614
0:211
0.274
0.126
0.238
0.589
ATC-MC
0.729
0.455
0.558
0.671
0.337
0.366
0.509
0.380
0.652
NL-SMMAA
0.845
0.897
0.888
0,792
0.850
0.931
0.877
0.881
0.896
NI-EDA
0.764
0.881
0,852
0.863
0.859
0.901
0.863
0.882
0.938
NI-BT
0.810
0.891
0.882
0,719
0.837
0. 1.891
0.864
0.872
0.870
movies
N-Randdee
0.851
0.875
0.742
0.850
0.905
0.864
0.861
0.874
ATC-NE
0:692
0.745
0,807
0.712
0.561
0.768
0.580
0.620
0.763
ATC-MC
0.698
0.770
0.847
0.722
0.622
0.813
0.637
0.678
0.784
NLSSSMBA
0.728
0.737
0.806
0.737
0.820
0.883
0.667
0,850
0.773
NI-EDA
0.732
0.831
0.880
0.601
0.788
0.853
0.634
0.903
0.841
NI-BT
0.697
0.685
0,816
0.758
0.816
0.867
0.814
0.866
0.810
Ne-RandRep
0.753
0.731
0.813
0.735
0.841
0.528
0.871
0.860
0.774
ATC-NE
0.623
0.740
0.719
0.553
0.737
0.815
0.788
0.748
0.781
ATC-MC
0.662
0.753
0,723
0.566
0.711
0.831
0.781
0.752
0.750
NI-SMMBA
0.715
0.743
0.879
0.764
0.837
0.863
0.879
0.860
0.870
NLEDA
0.757
0.731
0.807
0.673
0.798
0.786
0.815
0.827
0.833
NI-BT'
0.728
0.753
0,766
0.664
0.826
0.832
0.783
0.884
0.865
sporta
NI-RandRep
0.735
0.702
0.816
0.728
0.808
0. 0.802
0.830
0.849
0.836
ATC-NE
0.460
0.734
0.738
0.567
0.720
0.726
0.726
0.725
0.862
ATC-MC
0.484
0.728
0.766
0.582
0.754
0.727
0.725
0.741
0.860
NI-SMMA
0.784
0.763
0.845
0.710
0.769
0.853
0.755
0.886
0.877
NI-DDA
0.771
0.867
0,881
0.801
0.855
0.886
0.869
NI-BT
0.763
0.750
0.867
0.719
0.805
0.860
0.787
0.842
0.835
Lech
Nl-RondRep
0.819
0.721
0.843
0.711
0.854
0.889
0.787
0.854
0.871
ATC-NE
0.737
0.744
0.787
0.751
0.732
0.795
0.727
0.830
0.863
ATC-MC
0.771
0.730
0.764
0.749
0.704
0.809
0.771
0.780
0.843
NI-SMMAA
0.792
0.854
0.786
0.707
0.746
0.875
0.888
0.792
0.866
NI-DDA
0.859
0.846
0.790
0.837
0.886
0.819
0.894
NI-BT
0.709
0.802
0.763
0,709
0.750
0.832
0.824
0.74-1
0.856
toole
Ne-RandRep
0.792
0.825
0.816
0.717
0.724
0.885
0.877
0.801
0.845
ATC-NE
0.745
0.789
0.775
0.744
0.632
0.698
0.856
0.737
0.799
ATC-MC
0.749
0.798
0.772
0.745
0.634
0.737
0.876
0.777
0.793
NI-SSMBA
0.779
0.830
0.813
0.766
0.695
0.808
0.872
0.872
0.891
NI-EDA
0:606
0.790
0.805
0.821
0.790
0.837
0.534
0.600
0.840
NI-BT
0.748
0.762
0.746
0.755
0.707
0.8-10
0.809
0.773
0.811
toys
NI- NI-RandRep
0.813
0.788
0.783
0.770
0.660
0.788
0.938
0.873
0.891
ATC-NE
0.856
0.489
0.743
0.550
0.628
0.515
0.792
0.703
0.771
MTC-MC
0.694
0.482
0.726
0.539
0.651
0.577
0.805
0.730
0.777
Table 20: Full macro 7 metrics for all pairs of training and test domains for BERT models trained on AWS.
33

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Train
Measure
facetoface
oup
nineslaven
talephona
latters
government
NLSSMBA
0.675
0.608
0.650
0.661
0.622
0.641
NI-EDA
0.561
0.675
0.563
0.599
0.658
0.672
0.537
0.599
0.619
NL-BT
0.669
0.742
0.736
0.756
0.727
0.754
0.751
0.738
0.791
slate
NI-RandRep
0.393
0.526
0.529
0,480
0.398
0.438
0.455
0.640
0.531
ATC-NE
0.650
0.632
0.577
0.594
0.683
0.625
0.602
0.621
0.099
ATC-MC
0.652
0,632
0.578
0.591
0.681
0.625
0.603
0.622
0.697
NI-SMMBA
0.577
0.537
0.692
0.482
0.539
0.629
0.520
0.546
0.598
NI-EDA
0.632
0.581
0.691
0.608
0.557
0.631
0.529
0.540
0.608
NI-BT
0.783
0.704
0.771
0.693
0.671
0.776
0.680
0.707
0.693
fiction
NI-RandRep
0.195
0.116
0.522
0.531
0.437
0.431
0.461
0.531
0.193
ATC-NE
0,538
0.461
0.630
0.428
0.513
0.542
0.469
0.581
0.536
ATC-MC
0.530
0.150
0.631
0.127
0.513
0.513
0.163
0.576
0.531
NI-SSMBA
NI-EDA
0.637
0.633
0.726
0.537
0.615
0,735
0.579
0.611
0,580
NI-BT
0.657
0.668
0.722
0.637
0.663
0.639
0.621
016602
0.614
telaphone
NI-RandRep
0.504
0.170
0.573
0.170
0.510
0,480
0,482
0.576
0.410
ATC-NE
0.533
0.564
ATC-MC
0.531
0.507
0.681
0.522
0.501
0.637
0.435
0.561
0.191
NI-SSMBA
0.524
0.528
0.585
0.462
0.517
0.533
0.569
0.617
0,521
NLEDA
0.492
0.484
0.523
0.461
0.418
0.550
0.524
0.507
0.458
NI-BT
0.645
0.713
travel
NI-RandRep
0.165
0.476
0.467
0.514
0.492
0.472
0.526
0.569
0.497
ATC-NE
0.579
0.528
0.533
0.643
0.541
0.571
0.591
0.572
0.597
ATC-MC
0.576
0.528
0.531
0.641
0.536
0.573
0.591
0.570
0.002
NI-SSMBA
0.578
0.531
0.600
0.568
0.565
0.617
0.580
0.563
0.616
NLEDA
0.581
0.582
0.565
0.535
0.594
0.624
0.519
0.552
0.566
NI-BT
0.719
0.693
0.686
0.687
0.755
0.701
0.651
0.712
0.719
government
NI- R-nddRRp
0.274
0.268
0.357
0.444
0.464
0.301
0.247
0.395
0.452
ATC-NE
0.624
0.440
0.616
0.641
ATC-MC
0.621
0.465
0.583
0.667
0.446
0.602
0.616
0.522
0.642
Table 21: Full macro metrics for all pairs of training and test domains for CNN models trained MNLI.
34

---

Published in Transactions on Machine Learning Research (06/2023)
Test Domain
Trii Domain
Mensure
facetoface
oup
nineeleven
fiction
telephone
goverament
NI-SSMBA
0.743
0.746
0.709
0.767
0.742
0.727
0.771
0.782
0,819
NI-EDA
0.580
0,693
0.674
0.61%i
0.703
0.655
0.651
0.671
0.737
NL-TT
0.669
0.742
0.736
0.756
0.727
0.754
0.751
0.738
0.791
slate
NI-RandRep
0.526
0,480
0.398
0.438
0.640
0.531
ATC-NE
0.689
0.768
0.786
0.728
0.752
0.761
0.744
0.743
0.766
ATC-MC
0.684
0.761
0.794
0.730
0.761
0.789
0.765
0.741
0.785
NI-SSMBA
0.755
0.673
0.690
0.726
0.729
0.769
0.806
0.770
0.727
NI-EDA
0.609
0,532
0.608
0.554
0.624
0,629
NL-BT
0.783
0,704
0.771
0.693
0.671
0.776
0.680
0.707
0. 0.633
Ni-RandRep
0.195
0.146
0,522
0.531
0.437
0.431
0.461
0.531
0.193
ATC-NE
0.618
0.377
0.707
0.478
0.711
0.506
0.558
0.658
0.711
ATC-MC
0,631
0.712
0.707
0.603
0.670
0.706
NL-SSMBA
0.720
0,768
0.793
0.792
0.668
0.720
0.813
0.758
0. 0.788
NLEDA
0.670
067
0.706
0.617
0.718
0.795
0.642
0.732
0.661
NL-BT
0.637
0.668
0.722
0.637
0.663
0.639
0.621
0.602
0.614
telephone
Ni-RandRep
0.501
0.573
0.510
0,480
0,482
0.576
0,1410
ATC-NE
0.462
0.486
0.682
0.575
0.641
0.542
0.644
0.697
0.702
ATC-MC
0.519
0.551
0.717
0.641
0.637
0.597
0.685
0.684
0.708
NI-SSMBA
0.700
0.806
0.737
0.676
0.744
0.707
0.848
0.789
0.783
NLEDA
0.483
0.486
0.493
0.480
0.515
0.483
0.522
0.495
NL-BT
0.619
0.634
0.630
0.660
0.681
0.655
0.645
0.713
0.699
travel
NI-RandRep
D.465
0.476
0.467
0.514
0.492
0.472
0.526
0.569
0.497
ATC-NE
0.155
0.320
0.234
0.539
0.533
0.146
0.360
0.633
0.630
ATC-MC
0.175
0.320
0.576
0.219
0.443
0.648
0.646
NI-SSMBA
0.769
0.777
0.727
0.652
0.640
0.783
0.784
0.805
0.754
NLEDA
0.521
0.513
0.480
0.578
0.661
0.657
0.494
0.565
0.598
NLBT
0.719
0.693
0.686
0.687
0.755
0.701
0.651
0.712
0.719
govermment
RandRRep
0.274
0.357
0.464
0.247
0.452
ATC-NE
0.377
0.176
0.472
0.660
0.558
0.239
0.497
0.302
0.638
ATC-MC
0.427
0.231
0.487
0.653
0.600
0.275
0.496
0.323
0.629
Table 22: Full macro metrics for all pairs of training and test domains for BERT models trained on MNLI.
Test Domain
Model
Measure
books
clothing
home
kindle
movies
pets
sports
tech
tools
toys
NI-SSMBA
0.677
0.688
0.758
0.641
0.691
0.786
0.784
0.758
0.722
0.683
NI-EDA
0.637
0.672
0.704
0.612
0.658
0.746
0.739
0.680
0.703
0.711
NI-BT
0.601
0.532
0.572
0.509
0.521
0.509
0.587
0.536
0.571
0.529
CNN
NI- RenddRR
0.485
0.542
0.501
0.522
0.440
0.491
0.650
0.477
0.507
0.550
ATC-NE
0.406
0.727
0.728
0.362
0.499
0. 72222
0.726
0.679
0.730
0.752
ATC-MC
0.410
0.726
0.729
0.359
0.500
0.724
0.727
0.681
0.730
0.752
NI-SSMBA
0.790
0.850
0.868
0.661
0.899
0.864
0.855
0.874
0.880
0.815
NI-EDA
0.739
0.848
0.825
0.731
0.793
0.836
0.825
0.828
0.837
0.841
NI-BT
0.702
0.845
0.830
0.668
0.792
0.839
0.843
0.813
0.819
0.858
BERT
NI-RendRep
0.839
0.841
0.891
0.694
0.864
0.834
0.747
0.825
0.850
0.831
ATC-NE 0.613
0.721
0.712
0.618
0.695
0.666
0.733
0.693
0.708
0.755
ATC-MC
0.621
0.735
0.730
0.616
0.711
0.691
0.749
0.710
0.726
0.764
Table 23: Full miero I metrics for all test domains for CNN and BERT models trained on AWS.
35

---

Published in Transactions on Machine Learning Research (06/2023)
Domain
Mrasure
facetoface
oup
nineeleven
telephone
travel
letters
government
NI-SSMBA
0.540
0.493
0.562
0.193
0.431
0.607
0.541
0.579
0.563
NI-EDA
0.560
0.355
0.127
0.520
0.488
0.435
0.499
0.552
0.532
0.542
NI-BT
0.698
0.684
0.611
0.713
0.713
0.595
0.676
0.714
0.697
0.722
CNN
NI RandRep
0.448
0.417
0.427
0.503
0.472
0.324
0.395
0.450
0.550
0.509
ATC-NE
0.461
0.413
0.549
0.427
0.350
0.517
0.531
0.307
0.197
0,401
ATC-MC
0.460
0.411
0.550
0.429
0.351
0.517
0.532
0.307
0.496
0.404
NI-SSMBA
0.782
0.710
0.742
0.632
0.724
0.756
0.779
0.691
0.768
0.700
NI-EDA
0,563
0,356
0.481
0.622
0.586
0.545
0.529
0.G07
0.607
0.630
NI-BT
0.698
0.684
0.611
0.713
0.713
0.595
0.676
0.714
0.697
0.722
BERT
NI-Ra RendReR
0.448
0.417
0.427
0.503
0.172
0.321
0.395
0,450
0.550
0.509
ATC-NE
0.375
0.389
0.600
0.606
0.593
0.414
0.523
0.511
0.655
0.687
ATC-MC
0.676
0.486
0.554
0.580
0.663
Table 24: Full micro t metrics for all test domains for CNN and BERT models trained on MNLI.
Domain (Model)
Measure
SVHN (NIN)
CIPARIO (NiN)
CIFARIO (ResNet)
CIFARIO (VGG)
CINIC10 (CNN)
NI- Al-HandAug
0.733
0.797
0.746
0.869
0.759
Ni-asaasltee
0.881
NI-Erase
0.324
0.409
0.708
-0.183
0.606
NI-FC
-0.033
-0.015
0.568
0.628
0.289
ATC-NE 0.859
0.648
0.757
0.773
0.548
ATC-MC
0.843
0.767
Table 25: Full in-domain т metrics for all test domains for image classification models and datasets.
Train Domain
Model
Measure
books
clothing
home
kindle
movies
peta
sports
tech
toola
toys
NI-SSMBA
0.646
0.613
0.643
0.577
0.597
0.591
0.703
0.648
0.643
0.620
NI-EDA
0.653
0.558
0.615
0.660
0.447
0.666
0.708
0.646
0.663
0.565
NI-BT
0.569
0.449
0.582
0.537
0.626
0.605
0.568
0.653
0.640
CNN
NI-RandRep
0.560
0.537
0.539
0.559
0.593
0.596
0.572
0.545
0.566
0.615
ATC-NE
0.575
0.413
0.561
0.478
0.294
0.464
0.539
0.561
0.446
0.341
ATC-MC
0.566
0.405
0.558
0.478
0.300
0.461
0.540
0.569
0.451
0.338
NI-SSMBA
0.735
0.874
0.868
0.693
0.780
0.871
0.884
0.851
0.851
0.884
NI-EDA
0.875
0.788
0.814
0.700
0.870
0.894
0.818
0.858
0.882
0.810
NI-BT
0.801
0.811
0.857
0.638
0.835
0.813
0.860
0.846
0.803
0.750
BERT
NI-RandRep
0.839
0.841
0.891
0.694
0.864
0.834
0.747
0.825
0.850
0.831
ATC-NE 0.733
0.724
0.822
0.640
0.721
0.711
0.778
0.783
0.799
0.783
ATC-MC
0.742
0.746
0.813
0.630
0.730
0.725
0.814
0.780
0.774
0.739
Table 26: Full in-domain 7 metrics for all test domains for CNN and BERT models trained on AWS.
36

---

Published in Transactions on Machine Learning Research (06/2023)
Train Domain
Model
telephone
government
NI-SMMBA
0.661
0.635
0.753
0.692
0.758
NI-EDA
0.629
0.723
0.786
0.652
0.755
CNN
Ne-RandRep
NL-BT
0.751
0.437
0.765
0.471
0.546
0.637
0.786
0.570
0.826
0.491
ATC-NE
0.722
ATC-MC
0.703
0.660
0.722
0.716
0.728
NI-SSMBA
0.713
0.754
0.766
0.785
0.839
NI-EDA
0.608
0.664
0.781
0.575
0.726
NI-BT
0.751
0.765
0.637
0.786
0.826
BERT
NI- Re-
0.437
0.471
0.546
0.570
0.491
ATC-NE
0.722
0.796
0.740
0.737 0.719
0.701
ATC-MC
0.745
0.791
0.791
0.696
Table 27: Full in-domain metrics for all train domains for CNN and BERT models trained on MNLI.
37

---

Published in Transactions on Machine Learning Research (06/2023)
Domain Shifts
ImageNet-A
Measure
N²2
Macro
R² Macro
I1)
NI-RandAug
0.091
0.061
NI-Translate
0.159
0.099
NI-Erase
0.174
0.153
NI-FC
0.139
0.095
ATC.NE
0.272
0.136
ATC-MC
0.279
0.135
(a) Standard deviations for ImageNet scale results. No standard deviations are reported for ImageNet-A or ID 7 since report only single value.
CTI0
Numbers
Measure
R
Macro
ID>
R²
Macro
ID)
NI-RandAug
0.044
0.054
0.048
0.021
0,026
NI-Translate
0.170
0.181
0.197
0.043
0.032
NI-Erase
0.254
0.348
0.345
0.065
0.033
NI-FC
0.239
0.225
0.255
0.074
0.083
ATC-NE
0.169
0.115
0.091
0.109
0.059
ATC-MC
0.168
0.101
0.093
0.077
0.035
(b) Standard deviations for small scale image results. No standard deviation is reported for Numbers TD) since report only a single value.
CNN
RoBERTa
Measure
R²
Macro T
Micro
ID т
R²
Macro 7
Micro т
ID т
NI-SSMBA
0.079
0.067
0.046
0.035
0.012
0.063
0.064
0.065
NI-EDA
0.108
0.051
0.041
0.072
0.016
0.058
0.040
0.055
NI-BT
0.161
0.078
0.032
0.060
0.021
0.060
0.061
0,063
NI-RandRep
0.069
0.080
0.054
0.021
0.015
0.060
0.055
0.055
ATC-NE
0.126
0.110
0.143
0.092
0.168
0.135
0.044
0.031
ATC-MC
0.125
0.109
0.143
0.091
0.139
0.112
0.048
0.050
(c) Standard deviations sentiment analysis results.
CNN
RoBERTa
Measure
R²
Macro т
Micro 7
ID r
R²
Macro 7
Micro T
ID
NI-SSMBA
0.098
0.060
0.048
0.043
0.035
0.046
0.040
0.041
NI-EDA
0.107
0.068
0.046
0.060
0.108
0.087
0.044
0.075
NI-BT
0.045
0.049
0.042
0.063
0.045
0.049
0.042
0.063
NI-RandRep
0.196
0.079
0.061
0.049
0.196
0.079
0.061
0.049
ATC--E
0.108
0.068
0.076
0.022
0.206
0.180
0.111
0.032
ATC-MC
0.108
0.068
0.076
0.024
0.174
0.164
0.100
0.038
(d) Standard deviations on NLI results.
Table 28: Standard deviations for reported values in Table
38


Reviews:
[{"review_id": "tTq0HXlabW", "reviewer": "Reviewer_8wmA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ **Versatility**: The proposed neighborhood invariance measure is versatile, as it can be applied to various tasks (image classification, sentiment analysis, and natural language inference) and experimental settings. Unlike existing methods, the neighborhood invariance measure does not depend on the test point's true label and makes no assumptions about the data distribution or model.\n+ **Strong correlation with generalization**: The measure consistently outperforms baselines and correlates strongly with actual generalization across tasks and experimental settings. The measure is particularly useful in OOD settings, where existing methods often struggle.\n\n## Weaknesses\n\n- **Reliance on appropriate data transformations:** The main limitation of this work is the reliance on selecting suitable data transformations. In certain tasks or domains, this might be challenging, and poor transformation choices can lead to inappropriately high estimates. A key challenge of real-world OOD shifts is their difficulty to characterize, as they often cannot be expressed through common data augmentations.\n\n- **Limited Model & Dataset Scale:** The proposed evaluation metric is not surprising to researchers in invariant representation learning. Thus, the main contribution of this paper is its empirical evaluation. However, the models and datasets used in the study fall behind modern practice, making the paper less interesting or useful to the community. For image classification, the authors use CIFAR-10 variants and pre-trained networks up to ResNet, while current OOD robustness studies focus on larger pretrained networks and higher-resolution datasets. For text evaluations, the sentiment analysis (SA) and natural language inference (NLI) datasets also fall behind modern practices, as do the BERT family models. The study and evaluation results on small models and datasets may not be applicable to modern practices.\n\n- **Insufficient discussion of related works:** The authors have missed distribution shift works in the related works discussion. It is recommended to add a paragraph discussing these works in the revision. See the section of requested changes below for details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce \"neighborhood invariance,\" a simple and versatile measure to evaluate the generalization capacity of machine learning models, particularly in out-of-domain (OOD) settings. This measure calculates the proportion of points classified into the most common predicted class within a transformation neighborhood. Unlike existing methods, it does not rely on strong assumptions or test point labels. The paper evaluates the measure on various tasks, using over 4,000 models and 100 train/test domain pairs. The neighborhood invariance measure consistently outperforms baselines and correlates strongly with actual generalization. Despite limitations related to data transformations and extreme OOD settings, the measure provides an alternative approach to characterizing and comparing model generalization across different experimental settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "## Scaling up models and datasets\n- For image classification, it is suggested that the authors follow prior works [1,2,3,5,6] and evaluate modern pre-trained networks (e.g., DenseNet, ViT, CLIP) on modern distribution shifts datasets (ImageNetV2/ObjectNet/ImageNet-A/ImageNet-R/ImageNet-C/WILDS-IWildCam/WILDS-FMoW, etc.).\n\n- For text evaluation, the authors should consider following BIG-Bench [4] to evaluate modern models (e.g., GPT-2, GPT-Neo, T5, FlanT5, LLaMA) on up-to-date text OOD datasets (such as those in BIG-Bench).\n\n## Discussion of Related Works\n- It is recommended to add a discussion of previous distribution shift works on benchmark curation or large-scale empirical studies, such as [1,2,3,4,5,6].\n\n[1] Miller et al. Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. ICML 2021.\n\n[2] Wortsman et al. Robust fine-tuning of zero-shot models. CVPR 2022.\n\n[3] Baek et al. Agreement-on-the-line: Predicting the Performance of Neural Networks under Distribution Shift. NeurIPS 2022.\n\n[4] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.\n\n[5] Koh et al. WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021.\n\n[6] Gulrajani et al. In Search of Lost Domain Generalization. ICLR 2021.\n", "broader_impact_concerns": "I don't think there are any ethical concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BqRufhd78g", "reviewer": "Reviewer_yuPF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n+ The idea is simple but intuitively reasonable. \n+ The implementation is straightforward.\n+ The paper is generally well written and easy to follow.\n\n**Weaknesses:**\n\n - My main concern is how to select a set of **appropriate** data transformations. In my opinion, it is the key to the proposed measure. The appropriateness entirely determines the validity of the measure. Unfortunately, it seems that the authors skip this significant point in the paper. At least, the authors should provide some general rules to guide how to select them in different tasks. \n- How to guarantee the selected data transformations are still well defined? That is, applying the transformation will not lead to an undefined label. This is another key technical point, but not well discussed in the paper. \n- In light of the two issues above, it is hard to justify the extent to which the proposed measure works in practice. \n- In the section of related work, the authors missed a large body of work on invariance in the causality community. \n- In the image classification task, the types of data transformations considered in the experiments are too simple to cover many real-world scenarios. I suggest the authors conduct experiments on more real-world datasets, e.g., DomainBed (https://github.com/facebookresearch/DomainBed).  \n- The description of the experimental setup, eps the sections of Data, and Model and Hyperparameter Space, are too vague, which should be more detailedly explained. \n- The baseline methods are too simple to demonstrate the proposed measure. The authors should compare with those in the community of adversarial robustness. \n- Some typos: 1) pp2: \"our measure can applied\" -> \"our measure can be applied\"? 2) pp3:\"then motivate our formulation\"?\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new measure, namely *neighborhood invariance*, to characterize out-of-domain generalization. It is implemented by first selecting a set of data transformations and then calculating the percentage of points classified into the most commonly predicted class in the neighborhood of the input test point.  Extensive experimental results show the proposed measure is consistent with the out-of-domain generalization. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See **Weaknesses** above.", "broader_impact_concerns": "The authors did not discuss the broad impact of their method in the paper. I also do not see any obvious ethical implications of this work in the near future. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qn0t5i8dkK", "reviewer": "Reviewer_qQw6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is easy, compelling, and seems effective without making extra assumptions as opposed to the methods in the literature.\n- The paper is well-written and easy to follow.\n- The paper addresses an important issue in machine learning and provides an extensive set of experiments.\n\nWeaknesses:\nI have a few concerns in the experimental section:\n\n1. The vision datasets used in the experiments are extremely small in pixels (32x32). Even CINIC that includes some images from ImageNet downsize them to 32x32. Since the main contribution of the paper comes from data transformations it would be more informative and conclusive to see results on more realistic datasets such as ImageNet and then generalizing to ImageNet variants which are standard DG testbeds. \n2. Another note on my previous point, although results presented in Figure 6 (by the way, it should be titled as Table 6?) are shown to support the claim that \"training models with data augmentations doesn't interfere with the outcomes of the proposed method and the method is still effective\", I am curious to see results shown in Table 6 for a dataset with better resolution (eg. ImageNet but I am open to other datasets too)\n3. I am curious to know why authors use CNN models for NLP tasks given that CNNs in NLP are neither the common practice nor the state of the art. Also why not having a transformer based models for vision experiments?\n4. Another missing point I think is to see the effect of pretraining on the effectiveness of the proposed method. Pretraining is believed to increase generalizability so is the proposed method still effective under circumstances where DG is stronger? Especially contrastive pertaining methods in vision which leverage lots of augmentation techniques. \n5. Last but not least, it would be very helpful to see the standard deviations of the numbers in your paper to get a better sense of how spread out they are. Sometimes they are too close which makes it hard to assess.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the correlation between neighborhood invariance and generalization capacity. The proposed neighborhood invariance measure is a simple and effective way to calculate a classifier's output invariance in a local transformation neighborhood, making it applicable even in out-of-domain settings, where other methods cannot be used.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please read the above weaknesses. They are critical to secure my recommendation for acceptance. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission introduces neighborhood invariance, a measure of classifier output invariance to "local" transformations which is designed to characterize model generalization without making strong assumptions. Given a test example and some transformation neighborhood, the approach computes the proportion of transformations of the example classified as the most commonly predicted class within the neighborhood. The metric is then averaged over all examples of the test dataset.

Neighborhood invariance is evaluated on image classification, sentiment analysis, and natural language inference tasks, where it is shown to correlate well with OOD generalization.

Following the author-reviewer discussion, and after changes to the related work discussion and the incorporation of new experimental results, the reviewer consensus is that the submission is interesting to a wide audience and makes a valuable and well-supported empirical contribution.
Decision: Accept (poster)