The Frechet Distance of training and test distribution predicts the generalization gap
Authors: Julian Zilly
Hannes Zilly
Oliver Richter
Roger Wattenhofer
Andrea Censi
Emilio Frazzoli
Venue:  (2020)
Keywords: Generalization
Transfer learning
Frechet distance
Optimal transport
Domain adaptation
Distribution shift
Invariance

Abstract: Learning theory tells us that more data is better when minimizing the generalization error of identically distributed training and test sets. However, when training and test distribution differ, this distribution shift can have a significant effect. With a novel perspective on function transfer learning, we are able to lower bound the change of performance when transferring from training to test set with the Wasserstein distance between the embedded training and test set distribution. We find that there is a trade-off affecting performance between how invariant a function is to changes in training and test distribution and how large this shift in distribution is. Empirically across several data domains, we substantiate this viewpoint by showing that test performance correlates strongly with the distance in data distributions between training and test set. Complementary to the popular belief that more data is always better, our results highlight the utility of also choosing a training data distribution that is close to the test data distribution when the learned function is not invariant to such changes.

Under review as a conference paper at ICLR 2020
THE FRÉCHET DISTANCE OF TRAINING AND TEST DISTRIBUTION PREDICTS THE GENERALIZATION GAP
Anonymous authors Paper under double-blind review
ABSTRACT
Learning theory tells us that more data is better when minimizing the generaliza- tion error of identically distributed training and test sets. However, when training and test distribution differ, this distribution shift can have a significant effect. With novel perspective function transfer learning,
are able to lower bound the change of performance when transferring from training to test set with the Wasser- stein distance between the embedded training and test distribution. We find that there is trade-off affecting performance between how invariant a function is to changes training and test distribution and how large this shift in distribution is. Empirically across several data domains, we substantiate this viewpoint by show- ing that test performance correlates strongly with the distance in data distributions between training and test set. Complementary to the popular belief that more data is always better, our results highlight the utility of also choosing a training data distribution that is close to the test data distribution when the learned function is not invariant to such changes.
INTRODUCTION
Imagine there are two students who are studying for an exam. Student A studies by diligently learning the class material by heart. Student B studies by learning the underlying reasons for why things are the way they are. Come test day, student A is only able to answer test questions that are very similar to the class material while student B has no trouble answering different looking questions that follow the same reasoning. Distilled from this example, we note there is trade-off between how "well" a student studied, i.c., how indifferent the student is to receiving exereise or test questions, and how close the test questions are to the exercise questions.
While most machine learning work studies the generalization error, i.e., the error when testing different samples from the same distribution, we do not take the match of train and test distribution given. In fact, it appears that the distance between train and test distribution may be critical for successful "generalization". Following a similar line of thought, lggrrolu & Carbonell (201T) devised distribution measurement select only features that do not vary from domain to another. In contrast, we are interested in linking performance directly to the distance between train and test distribution.
Invariance to distribution shifts: We say that function is invariant to a given input perturbation when the corresponding output does not change with the perturbation. This is desirable when try- ing to achieve robustness to irrelevant data variations which are called nuisances (Achille & Soatto, 2018). As outlined by Achille & Soatto (2018; Censi & Murray (20IT), the "optimal" learned fune- tion from input to output is maximally invariant to all data variations that do not contain information about the output. To the extent to which learner reacts to such nuisance variations, which carry no information about the output, it will incur performance change in expectation. The difficulty lies in knowing what can be ignored and what cannot.
Similarity between training and test distribution: Another strategy would be to ensure that the train- ing and test distribution match which has been investigated a number of diverse settings (Tzeng ett al. 2017: Arjovsky et al.) 2017). Variations of this theme encountered by Zhang et al. (2016), where they show that networks are able to fit random labels perfectly, yet understandably fail to generalize to the test set of the correct label distribution.

---

Under review as a conference paper at ICLR 2020
Contribution: We frame the learning problem of training a training set then shifting to test test set as a transfer learning problem that goes beyond the usual generalization which assumes the same distribution for training and testing. Based on analysis, we aim to explicitly express the trade-off of the distance between both data set distributions given by the Wasserstein distance, which measures how close training and testing distribution are, and how invariant the learned function is to the training to test distribution shift. Joined together this is expressed via the Wasserstein distance between training and test samples embedded in the feature space of the learned function. Our experiments show a strong negative linear correlation between the distribution distances and network performance. This corroborates the notion that as long as the true function class is not found, it is best to ensure that training and test distribution are close. While this conclusion seems intuitive, may often be overlooked in the hunt for more data.
2 RELATED WORK
The trade-off and effect on performance between invariance distribution shifts and the magnitude of such distribution shifts has been an active field of research, albeit mostly implicitly. Given the correct invariance, changes in distribution do not have strong effect and learning "generalizes". What has not been investigated so far, to the best of our knowledge, is a more systematic treatment of measuring distribution distances and invariances of learned functions. These assumptions currently remain mainly implicit in the literature.
Invariance: Invariance is key component of Deep Learning (Goodfellow et al. 2009; Achille & Soatto 2018) with many works focusing on increasing invariance by incorporating structure into their models (LeCun et al. 1998 Bruna & Mallat, 2012 Cohen et al.) 2018 Kumar et al. 2017 van der Wilk et al.
2018). Other works apply GANs (Goodfellow al. 2014), e.g., to domain Hoffman et al. 2017). to map source and target distributions to the same space while being invariant to occurring distribution shifts. As stand in for many similar works, Tzeng et al. (2015) employ invariance to improve visual adaptation to domain changes. In this work, rather than trying to beat benchmark, we aim to establish the quantitative and qualitative trade-offs between invariance and distribution shift. In contrast to adapting the structure of a model, many works achieve invariance through data augmentation (Krizhevsky et al. 2017b] Shorten & Khoshgoftaar 2019 Herrández-Gariaa et al. 2019). While most augmentations are handpicked based on domain knowledge, automated methods based on GANs have been successfully applied to data augmentation (Antoniou et al. 2017] Kim et al. 2017). Any of these transformations can be considered a canonization (Censi & Murray 2011). l.e., generalization of data normalization to group actions, by removing unwanted nuisances to increase robustness. Many other techniques such as regularization Leen (1995), and dimension reduction can also be considered part of the umbrella of invariance learning (Jolliffe & Springer- Verlag 2002) Hadsell et al. 2006 Saul et al., 2006). In this work, we simply take the inherent invariance of learned embeddings to approximate the invariance of our function approximator and focus on the distributional shift from train test distribution.
adaptation problems (Tzeng et al. (
Distribution shifts: Shifts in data set distribution (Storkey 2009
Citooner-CCandel et al. 2009
can take a number of forms where either the input distribution p(x)) the target distribution p(y) or combination thereof changes. Likewise imbalanced datasets (Japkowiez & Stephen 2002), methods of over and undersampling (Bowyer et al. 2011), and domain shifts (Zhang et al. 2013) can be described in this framework. By knowing how the distribution changes from one setting to another, Zanardi et al. (2019) were able to define an operating domain for different vision sensors and employ this for semi supeisupeeeieed learning setup. Part of evaluating distribution shifts is measuring these probability distribution distances. A popular use for this is the Maximum Mean Discrepancy (MMD) distance (Borgwardt et al. (2006) that has found application in anomaly detection (Zou et al.) 2017) and two-sample testing (Gretton et al., 2012). Other distances such as the Wasserstein distance have made an impact in computational graphics modeling (Solomon et al. 2015) and in generating mealisttc-looking synthetic images using GANs (Arjovsky al.) (017). In this work, we apply the Wasserstein distance as a powerful tool to investigate the performance drop when moving from train to test samples.
2

---

Under review as a conference paper at ICLR 2020
Training Data
Ground Truth
Prediction
Testing Domain
Training Domatn
110
-20
10
20
-20
-10
10
20
-20
-10
10
20
Figure 1: Function approximations of 2 layer neural network with changing training and testing domains. (First row) Testing domain input shifted by the scalar a = [0, -05,, 0.5) from left to right respectively. (Second row) Similar configuration as the first row, but with an additional zero mean normalization on the input data. (Third row) Function approximations of a nonlinear, piecewise linear and linear function (left to right). (Fourth row) The training set domain is either smaller, equal, or larger than the test domain. Best viewed in color.
INSIGHTS FROM SMALL-SCALE EXPERIMENTS
Before jumping ahead, we gain intuitive understanding of the behavior of neural network in the case of mismatched train and test distributions. In particular we are interested in the effect of distribution shift from training test distribution and the effect of invariance of the learned function to such changes. Looking at this through a different lens, we also investigate what impact the type of function are approximating. and the type of function
approximating with can have.
First, we run experiments detailing changes in the input distribution as illustrated in the top row of Fig. []] From left to right, we increase input values by adding scalar a. As can be readily observed, the quality of the function approximation deteriorates under such a distribution shift. If on the other hand, we normalize the input data by subtracting the mean of the distribution, the function approximation remains unchanged as seen in the second row of Fig. This added invariance through normalization made the function approximation robust to the given distribution shift.
Secondly, we run function approximation experiments in which we have a training set in one interval and test set in another. We focus on what happens when the true function class the same as the one used for approximation and what happens when this is not the case. We observe in the third row of Fig. [ that when the correct function is found, as in the second and third image, the approximated function works even in domain in which it was not trained. On the other hand, the first plot of the third row and the plots in the fourth row show that if the true function is not found exactly, then outside of the training domain errors accumulate. Further, Fig. [ that a oversized training domain (bottom right plot) can hurt performance on the test domain, when compared to the middle plot with overlapping train and test domains.
conclude from the bottom row in
3

---

the probability space Under review as a conference paper at ICLR 2020
Our takeaway is that invariance indeed influences how distribution shifts affect changes in training and testing performance. We however also note that this topic is more nuanced. The difficulty lies in the fact that finding the "correct" invariances also means finding the true function visualized in the third row of Fig. | These smaller experiments make us expect that it may be unlikely to find the true invariances of underlying functional relationship. In such case, ensuring that train and test distribution match thus becomes viable option to enforce a small change from train to test performance.
4 THE TRADE-OFF BETWEEN INVARIANCE AND DISTRIBUTION SHIFT
P1
Op
f(1))
af
((f(p)))
P2
[p2)
((((p2)
Wasserstein metric
Wasserstein metric
Le metric
Figure 2: Illustration of the relationship between dataset distributions p; to learned induced distribu- tion fp;) to associated expected loss (pf(p,)). Noted at the bottom of the spaces are the assumed underlying metrics of the space.
We begin our investigation by stating that learning is a process that maps training dataset Duain with distribution Prrain1;; (2, y) to learned function approximation f (Prrain: X) Rs V(x, y) E Derain in- ducing target distribution (Prain minimizing a loss function on the training set using stochastic gradient descent (Robbins & Monro 1951). Of interest however is the performance of the learned function on testing dataset Diest with possibly different distribution Pres (2. v). By modeling the learned relationship / (Prrann x. E Y as a function on the data distribution Puzin (Ir y), we are then able to obtain relationship between changes in the data distribution and changes in the learned function and its performance.
depicted in Fig. 2 In deep learning this usually achieved by
Theorem 1. We are given two dataset distributions pı (x. y) and p2 (z. y) of input data x E X and oulput E Y situated in the probability space with Wasserstein metric Wa(-,") and assume that [p, x + Y is the function of a function family F continuous P that minimizes the norm loss [[f(p1,) z), [[f(p1, yll2 when the data is distributed according to Pi(z, (z,y). Then the expected absolute difference in loss for distribution p2 has the following lower bound.
1|
[
Epp ((I((((((2))))))))) ((((), Ry
W2(p1,P2)
(1)
Op
where 0f(p.z) denotes the average value of @f(p..) bp
along the integral as part of the intermediate
Op value theorem.
Proof. First we note that we can write the learned function for distribution pa as the function learned for distribution yı plus the line integral of the change of the function along the Wasserstein geodesic between p1 and p2.
f(p2, x.) =f(p1 + f p2 0f(p,)) x) Op, 0p: VpW2(p.p)
(i)
0p
1Th P22 Op = n Vy(Ws(p.P))ddp = Wo(p2, We(pi, p1) = W2(p2.P1)
(ii)
4

---

Under review as a conference paper at ICLR 2020
Then the change in loss is lower bounded as follows:
Ep, AC = Ep, ((f(2 x., v) [(f(p). z).
Epp
[p x) ull - (1)(p1, - wll
2 Epp (P2,
- f(11,)))
①
pa f x) Op
Intermodiate value
@f(p.a)
f Op(x))
ps
Eyp
3p
Op
@ Epg
0f(p, x) W2(P1. p2) = = Öp
[0flr.>1 2)
(P1.P.P2))
Op
From the above lower bound in Ineq. I we can deduce two ways of achieving minimal change in loss when moving from training distribution P1 to testing distribution P2.
(1) Either the difference in distribution from training set to test set is small (2 P2) Rs In this case, by assuming the learned function fits the training distribution perfectly and is not ill-conditioned, we can deduce that also the test distribution fit perfectly, or
(2) The function fp) does not change when the dataset distribution changes from pr to P2 by 0.
being invariant to such changes s.t. Ep. 0p
af(p.))
Clearly, if the learned function cannot distinguish between, i.e. is invariant to, changes in training and test set, then even shift of distribution the test set cannot alter the quality of the performance. If on the other hand, as may be more common, the function class is not invariant, then the difference in distribution from training to test set becomes decisive. Note that since we regard changes in per- formance, it is vital to already have a good performance on the training distribution. From the above analysis, we believe that it may be essential distribution to the test set. If this is the case one does not need to worry about finding the right func- tion class with the right invariances to approximate the true function. In the case where the learned function has "too much invariance", the function is not noticing some of the relevant variations in the data, e.g. in the extreme a constant would not change from training to testing distribution. While its performance does not change from training to test distribution, its training performance already suffers from unnecessary invariance. on the other hand, the function is not invariant enough might fit the training set well yet lose performance when switching to the test distribution.
employ training sets that are as close as possible in
Instead of either calculating the distribution distance on the raw data (W2(Pi,P2) we compute the distance an embedding of the raw data that aims to mirror the distances the function f(-) would perceive. This has the advantage of avoiding to compute demanding high dimensional distribution distances from raw data on the one hand, and having to train networks for both training and test distribution on the other hand. Additionally, it may be hard to properly compare the distribution of a function trained on the training distribution vs. one trained on the test distribution since training and test set oftentimes are of very different size.
To measure distribution distances, we employ the Fréchet distance (FD) (Fréchet 1957). also called the 2- 2-Wasserstein distance (Vaserstein, 1969), by assuming all probabilities to be normally dis- tributed. The FD a distance measure between two normal distributions, also adopted in the Fréchet Inception Distance (FID) (Heusel et al., 2017) to evaluate the quality difference of real and synthetic images. Instead of comparing raw data, we use the metric by exploiting domain- using an embedding network such as the inception network (Szegedy al. 2015). The FD of two normal distributions. N (11,, Σn) and N(u2, Σz) is given by:
features
FD d ((1,, Ex), (12, Σx)) = | 1el13 Tr(E, + Σg - -0EEE))))))
Heusel et al. (2017) show that the distance correlates well with human judgement in the vision domain, it can also be applied to other domains if domain relevant model replaces the inception network. We apply the FD to measure the mismatch of two sets in the following sections.
5

---

Under review as a conference paper at ICLR 2020
sets
Voice
GloVe3300
GloVe100
VoeCaleb2
Libei
85
11
800
SDRU (dB)
10
65
0.000
0.005
0,010
0.015
0.1
0.2
0.3
0.4
0.5
0.6
0.7
FD
(a)
(b)
Figure 3: (a) Sentiment classification accuracy versus FD the Amazon review dataset. Each point corresponds to a train and test set pair. Two independent runs are displayed with different GloVe embeddings used to compute FD scores. The Pearson correlation is -0.78. (b) Scatter plot of the relation between SDRi and FD scores on the speech separation task. The Pearson correlation between all SDRi and FD points is -0.61, while the average correlation given a fixed test set is -0.82.
5 EXPERIMENTS AND RESULTS
Next, aim show that the distribution distance between training and test distributions matters in many practical problems. We underline this perspective by running experiments in text clas- sification, vision data augmentation, and speech separation. Crucial for these experiments is that we ensure that the output distribution stays the same while only the input distribution changes. In the following we offer the general experimental setup for each dataset. For more comprehensive implementation details please refer to appendices B Bnd C.
5.1 TEXT CLASSIFICATION
We begin our investigation in text classification. To this end we employ different subsets of large text classification dataset which classifies product reviews into binary good or bad product ratings. To measure the effect of distribution shift, we train our rating classifier on texts of one product category and test it on another. This yields many pairwise train/test distances and performances.
Specifically, we use the Amazon Review dataset (He & McAuley 2016 McAuley et al., 2015) which contains text reviews and ratings divided into various product categories. In our experiments, we select categories of the 5-core least reviews per item) subsets. As input we use review texts and as output create binary rating categories of good bad by splitting the rating categories (1-5) into negative (1-2) and positive (4-5) labels. Reviews with rating of are removed. pretrained GloVe (Pennington et al. 2014)
To infer the FD score between datasets categories, embeddings of size 100 and 300. The word embeddings of each sentence are averaged to obtain a single embedding for each review.
we can see that the classification accuracy the binarized ratings decreases the further the training set is away from the test set measured by the FD score. The relationship correlates strongly (Pearson correlation -0.78), and notably there is no high accuracy for large FD scores.
From Fig. 3a
5.2 VISION CLASSIFICATION
Next examine the effect of distribution shift in vision classification tasks. Special about this problem is that the task is rarely the same across many datasets. Instead we study distribution shifts that are created when applying data augmentation techniques (Leen) 1995).
We study the problem on the popular CIFARIO (Krizhevsky et al. 2009), Street View House Num-
bers (SVHN) (Netzer et al. 2011), and Fashion MNIST dataset Xiao al. 2017). As classifica-

---

Under review as a conference paper at ICLR 2020
Table Classification accuracy and FD scores on the CIFARIO (CIO), SVHN, and Fashion MNIST (FM) data sets. Compared are various data augmentation techniques and their effect on classification accuracy and distribution distances.
Training Set
C10
C10 FD
SVHN
SVHN FD
FM
FM FD
No Augmentations
81.71%
3.34
93.89%
13,96
93.00%
1.50
+ Flipping
83.12%
3.31
93.13%
13.93
93.35%
1.87
+ Contrast
81.36%
3.92
94.21% %
14.27
93.22%
1.50
Contrast Flipping
84.15%
3.88
93.09%
14.01
93.40%
1.85
+ Salt and Pepper Noise
77.19%
131.82
94.20%
217.96
92.73%
226.51
Crop
84.24%
3.34
95.69%
11.85
93.15%
3.16
+ Flipping
85.77%
3.35
95.18%
11.23
92.88% %
3.44
+ Contrast
85.00%
4.32
95.90%
11.59
92.74%
3.19
+ Contrast Flipping
86.36%
4.31
95.25%
11.69
93.13%
3.47
+ Salt and Pepper Noise
81.55%
125.62
95.46%
194.43
91.53%
216.93
Crop Resize
79.77%
20.99
90.08%
22.89
89.83%
17.77
+ Flipping
79.74%
20.91
88.94%
22.63
89.72%
18.19
+ Contrast
79.07%
22.78
90.23%
23.77
89.09%
17.74
+ Contrast + Flipping
80.34%
22.68
89.32%
23.59
89.70%
18.29
+ Salt and Pepper Noise
78.08%
224.61
90.35%
266.05
88.79%
255.05
12
train sets
15
All triin
Common Voice
8.15
9.27
8.78
7.61
6.97
Comman Voice
1.21
20
35
Training set
SDRI (8
Training sat
Libri
9.01
7.48
12.49
8.67
7.41
Libri
3.35
5.88
0.85
6. 6.26
10.14
FDX
Tedtium
9.32
9.93
10.62
6.37
Tediuu
1.37
17
0.67
2.74
VoxCeteba
10.21
9.14
VoxCeleb2
test sets
Common Voice
Libri
Tediuu
VoxCeleb2
test seto
Common Vokce
Libri
Tedkuu
VoxCeleb2
Test set
Test set
(a)
(b)
Figure 4: (a) SDR improvement scores for multiple training and test set configurations. (b) Pairwise FD scores between training and test sets. The matching scatter plot and correlations are reported in Fig. 3b
tion network, we apply simple convolutional neural network (LeCun et al. 1998) (see appendix B for more details) and use the Inception-v3 (Szegedy et al.) 2015) network compute the FD score.
In order to create distribution shifts, we apply data augmentation techniques such as flipping, chang- ing contrast, random cropping, and adding salt and pepper noise. We report the resulting classifica- tion accuracies and FD scores in Tab. [
Intriguing about the results displayed Tab. | is that yet again we see that larger FD scores lead to worse performance. The numbers however are more nuanced than in the previous text experiments.
The example of salt and pepper noise shows how nonlinear the relationship between distribution distance and performance can be. While the performance does decrease with increasing distance, it does so at a different scale than observed for the other data augmentation techniques. Nevertheless, these results suggest direct application of our insights: Selecting the augmentation methods applied based on the FD score between the augmented training set and a validation set. We leave an empirical validation of this use future work.

---

Under review as a conference paper at ICLR 2020
5.3 SPEECH SEPARATION
As final experiment, we venture into speech separation which deals with separating the speech of 2 or more speakers from audio data. This task is unusual because, through the combinatorial way the dataset is generated, it allows for very large datasets. This lets us gauge the influence of distribution shift especially when more than sufficient data is available. Supervised speech separation requires mixed speaker samples of at least two speakers and the corresponding single speaker streams as labels. In the monaural (single-channel) case, one can simply add two single speaker streams to create overlap. The resulting combinatorial possibilities create large datasets.
We collect data from the Libri Speech Corpus (Panayotov et al. 2013), the Voxceleb2 data set (Chung et al._ 2018), Common Voice data set (Firefox, 2019) and the TED-LIUM speech corpus (Hernandez et 2018)) We balance the number of speakers and time per speaker for the first four data sets, keeping 707 speakers for each data set and 7.5 minutes per speaker leading to roughly 88 hours in the training set. For the test set, we sample 10 hours from each data set, from a held back set of data with new speakers. We create mixes from equal datasets since they have more consistent recording settings, which resemble real world situations more closely. We adopt the strongest performing configuration of the Conv-TasNet model (Luo & Mesgarani] 2018) with SDRi score of 15.6 on the 2-mix WSJ (Hershey et al. (015) dataset. Identically to Luo & Mesgarani (2018), we use 4 second long samples with a sampling rate of 8k Hz. We report scores improvement score (SDRi), obtained through the difference between the SDR (Vincent et al.) 2006) of the processed and unprocessed sample. In place of the inception network, we we leverage the encoded features of speaker embedding network to compute FD) scores. Detailed hyperparameters and layer configurations are reported in the appendix in Tab. 2
SDR
Again, as depicted in Fig. 4 we find a strong correlation (-0.61) between loss in performance and distribution distance between different training and testing distribution pairs. appears that larger data sets resulting in better performance is not the whole story. Fig. 4a shows that a large combined dataset performs worse on the individual test sets than a model trained only on the corresponding training set. The presented experiments substantiate theorem by showing strong correlation between network performance and data distribution distances, especially visible in Fig. 3
6 DISCUSSION AND CONCLUSION
Following popular wisdom, one would be led to believe that more data all you need. The presented theory and experiments however clearly detail that, while the amount of data is important, ensuring that train and test distribution are close may be similarly significant to perform well on the test set.
From the scall-scale and real-world experiments we are left with the startling observation that, frequently, neural networks do not find the "true" functional relationship between input and output. If this were the case, distribution shifts between training and testing should have a smaller impact. Whether this problem can be remedied by finding richer function classes or whether it may be inherently unsolvable will have to be investigated.
An important aspect of this work is how we measure distribution distances. By using a represen- tation network, we obtain low dimensional embeddings and reduce the effect of noisy data. This embedding is however in itself limited by its features, training data, and objective. To apply the in- sights of this work, will therefore be paramount to carefully choose an embedding for each dataset and task, whose features are able to meaningfully model the various data shifts a desired learning algorithm would react to. As an example, a word embedding trained only on English texts will not provide meaningful results on other languages and hence is useless for modeling a distribution shift.
Through this work, we emphasize the consequences of using models for predictions, which do not share the invariances of the true functional relationship. In this case, data distribution shifts lead to deterioration of performance. As a remedy to this issue, we propose applying the Fréchet distance to measure the distance of the dataset distributions to infer the degree of mismatch. With this measure, we can deduce important criteria to choose training sets, select data augmentation techniques, and help optimize networks and their invariances. We believe that making the problem explicit and having a way to measure progress through the FD score may allow for a new wave of innovative ideas on how address generalization under data shifts.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research, 19(1):1947-199 1990, 2018.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXi::1711.04344 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv e-prints, art. arXiv: 101.0777 Jan 2017.
Karsten M. Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schlkopf, and
Alex J. Smola. Integrating structured biological data by Kernel Maximum Mean Discrepancy. Bioinformatics, 22144) 07 2006. ISSN 1367-4803. doi: 10. 10.105/biionffomattsshbbb...
Kevin W. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE:
synthetic minority over-sampling technique. CoRR, abs/ 1111106..1111 1813, 2011.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. CoRR, 10s/1203.1513,, 2012.
A. Censi and R. M. Murray. Uncertain semantics. representation nuisances, and necessary invariance properties of bootstrapping agents. In 20/ / IEEE International Conference on Development and Learning (ICDL), volume 2, pp. 1-8, Aug 2011. doi: 10. 10.1109/DEVLRN.201 21..20011.6331.
Joon Son Chung. Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. CoRR, Abs//800.05622, 2018.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns.
CoRR,
2018.
J. Deng. W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR09, 2009.
Mozilla Firefox. Common Voice. 2019-08-14.
2019. Accessed:
Maurice Fréchet. Sur la distance de deux lois de probabilité. COMPTES RENDUS HEBDO- MADAIRES DES SEANCES DE L ACADEMIE DES SCIENCES, 244(6):689-697 1957.
Ian Goodfellow, Honglak Lee, Quoe Le, Andrew Saxe, and Andrew Y. Ng. Measuring invariances in deep networks. In Bengio. D. Schuurmans. J. D. Lafferty, C. K. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems 22, pp. 646-654. Curran Associates, Inc., 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680. 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13(1):723-773, March 2012. ISSN 1532-4435.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR' 066), volume 2, pp. 1735-1742. IEEE, 2006.
Ruining He and Julian J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends 2016.
with one-class collaborative filtering. CoRR,

---

Under review as a conference paper at ICLR 2020
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia A. Tomashenko, and Yannick Estève. TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation. CoRR, abs/1805500l 00499, 2018.
Alex Hernández-Garria. Peter König, and Tim C. Kietzmann. Learning robust visual representations using data augmentation invariance. CoRR,
2019.
John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrimi- native embeddings for segmentation and separation. CoRR, abs/1508.04306, 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a nash equilibrium. CoRR, Abs/1706.08500 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Judy Hoffmman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexci A Efros, and Trevor Darrell Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03213. 2017.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1 (08.00083.... 2016.
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelli- gent Data Analysis, pp. 429-449, 2002.
I.T. Jolliffe and Springer-Verlag. Principal Component Analysis. Springer Series in Statistics. Springer, 2002. ISBN 9780387954424.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. CoRR, abs/1703.05119) 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. cite 1xxiv1442.69880Commes Published as conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep volutional neural networks. Commun. ACM, 60(6):84-90, May 2017a. ISSN 0001-0782. doi: 10.1145/3065386.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun. ACM, 60(6):4499, May 2017b. ISSN 0001-0782. doi:
10.1145/3065386.
Abhishek Kumar, Prasanna Sattigeri, and Thomas Fletcher. Improved semi-supervised learning with gans using manifold invariances. CoRR, abs/1 088800, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
Todd K Leen. From data distributions to regularization in invariant learning. In Advances Neural information processing systems, pp. 223-230 1995.
Yi Luo and Nima Mesgarani. Tasnet: Surpassing ideal time-frequency masking for speech separa- tion. CoRR, abs/ 2018
Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations styles and substitutes. CoRR, abs/1.50600457, 2015.
10

---

Under review as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th interaational conference on machine learning (ICML-10), pp. 807-814, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Read- ing digits in natural images with unsupervised feature learning. In In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011, 2011.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-52188 5210, April 2015, doi: 10.1 10.1109/ICASPP.015... 1779644
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
Joaquin quionero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
Herbert Robbins and Sutton Monro. stochastic approximation method. Ann. Math. Statist., 22(3): 400-407, 09 1951. doi: 10. 10.1214//a /
Lawrence K Saul, Kilian Q Weinberger, Jihun H Ham, Fei Sha, and Daniel D Lee. Spectral methods for dimensionality reduction. Semisupervised learning, pp. 293-308, 2006.
Connor Shorten and Taghi M. Khoshgoftaar. A survey image data augmentation for deep learn- ing. Journal of Big Data, 6(1):60, Jul 2019. ISSN 2196-1115. doi: 10. 00.118664003377019901977
David Snyder, Daniel Garcia-Romenn Daniel Povey, and Sanjeev Khudanpur. Deep neural network embeddings for text-independent speaker verification. Interspeech, pp. 1003, 2017.
Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. Convolutional wasserstein distances: Efficient optimal transporta- tion on geometric domains. ACM Trans. Graph., (6((4))66:1166111 1,, 2015. ISSN 0730-0301. doi: 10. 00.1145/2766963
Amos Storkey. When training and test sets are different: characterizing learning transfer. Dataset shift in machine learning, pp. 3-28, 2009.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shiens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4068 00884076,370 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. CoRR, abs/1702.05464, 2017.
Selen Uguroglu and Jaime Carbonell. Feature selection for transfer learning. In Dimitrios Gunop- ulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 430-442, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-2380866
Mark van der Wilk, Matthias Bauer, S. T. John, and James Hensman. Learning invariances using the marginal likelihood. CoRR, abs/1 808.05563, 2018.
Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatii, 5(3):64-72, 1969.
E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source sepa- ration. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462-1 146-1169, July 2006. ISSN 1558-7916. doi: 10.110//AA.2000.55555

---

Under review as a conference paper at ICLR 2020
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized End-to-End Loss for Speaker Verification. arXiv e-prints, art, arXiv:1710.10467, Oct 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.007747 2017.
A. Zanardi, J. Zilly, A. Aumiller, A. Censi, and E. Frazzoli. Wormhole learning. In 20/9 termational Conference on Robotics and Automation (ICRA), pp. 7899-7905, May 2019. doi: 201.://CCAA.211.8993333363
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 16 1.03500, 2016.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827, 2013.
Shaofeng Zou, Yingbin Liang, H Vincent Poor, and Xinghua Shi. Nonparametric detection of anomalous data streams. IEEE Transactions on Signal Processing, 65(21):5785-5797, 2017.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
The appendix comprises extended details of the work that may add to the understanding without being critical follow the main text.
A
SMALL-SCALE EXPERIMENTS
The following specifies the experimental details of the small-scale insight experiments from sec-
tion 3 Throughout these experiments, a model needs to approximate a function f(() given a point R, which sampled from the uniform distribution. We limit sampling of æ during training specific interval [a, bj and add gaussian noise from (0, 0.4) to the training labels. We further elaborate the experiments based on the rows of Fig. !
In rows one and two of Fig. [ model is trained to approximate the function f(z) = (2Txx)
on the training interval [-2: 2, 2]. During testing, we shift the test interval by the scalar [0, -0.5, 0.5] (images from left to right respectively). Additionally, we apply a zero mean nor- malization to the input data in the experiments of the second row. Note that the ground truth is invariant to change in the input distribution and thus stays consistent.
In the third row of Fig. 1 x is drawn from Uniform [-6 6) during training, while the test set samples lie outside of the training domain in [-10, -6]
6] and [6, 10). The functions from left to right are given
as follows:
f(x)
0.006. 0.027 0.018 0.015
- -
x x
<2<r<0
f(x) =
-2x
0<
1.5x
10.5
2
f(x) = -0.2x +1
representing nonlinear, piecewise linear, and linear function.
The last experiments approximate f(x) = e cos(2mx) with changing training domains while keeping the test domain consistent. The training intervals from left right are [0. 1]. [-1, 1], and [-2,2] with the test interval set to 1-1,
For all task we the same neural network model. We train two-layer feedforward neural network with 8 hidden units in its first layer followed by ReLU (Nair & Hinton| 2010) activation function and linear layer with output unit in its second layer. In total the network consists of 25 parameters including biases, which we optimize using the mean squared error and Adam optimizer (Kingma & Ba 2014).
B TEXT CLASSIFICATION
To complete the description of the experiment in section 5.1 of the preprocessing and network details. Before a model can use the text data from the amazon review dataset, several processing steps are necessary convert text to numeric representations. Each review contains several information fields. However, we only utilize the overall score and the review text, from which we remove capitalization, punctuation and multiple digit numbers. The overall score (1-5) is split into positive (4-5) and negative (1-2) reviews, while ratings of 3 are removed. The vocabulary size is limited to 10,000 words resulting in less frequent words being replaced by the UNK > symbol. To achieve consistent sentence lengths, to fixed length of 256 with padded words set to the symbol < PAD >. As a last processing step, tokenize the training data and apply the mapping to the paired test set. We report the complete classification accuracies and FD scores in Tab. between each category used as a training and test set.
we offer more detailed description
pad reviews
The classification network consists of an embedding, a global average pooling, and three dense layers of size 32, 16 and respectively. ReL.Us (Nair & Hinton 2010) applied after the first two
13

---

Under review as a conference paper at ICLR 2020
dense layers and Sigmoid activation function is applied to the last layer. We use Adam (Kingma & Ba 2014) and the binary cross-entropy loss for optimization with learning rate of 0.01.
C
VISION CLASSIFICATION
Again we offer complementary details to the vision classification experiments with data augmenta- tion. Especially, we provide more details the examined datasets, augmentations, and classification network. The CIFAR 10 dataset (Krizhevsky et al.] 2009) consists of 32 x 32 colored natural images divided into 10 classes. The training and test consists of 50,000 images and 10,000 images, respectively. The Street View House Numbers (SVHN) (Netzer et al., 2011) dataset, also contains 10 categories, one for each digit between I and 10. For SVHN, training and test set consist of 73,257 images and 26,032 images. respectively. The Fashion MNIST dataset (Xiao et al.) (017)) consists of 60,000 train and 10,000 test images. The dataset contains grayscale images of clothes again classified into 10 classes.
For data augmentation we offer complementary details. Random cropping reduces the image size to 24 x 24 and is only applied to the middle of images in the test set. We differentiate between only cropping and cropping and resize, which the images are rescaled back to 32 x 32. However, the test set remains unchanged when rescaling. For salt and pepper noise, we set distorted pixels equiprobably to black or white with a probability of 33%. For random contrast adjustment, we set the upper and lower limit to 1.8 and 0.2, respectively.
The classification network is based on a CNN (LeCun et al. 1998) structure. Convolutional layers of size 64 use kernel size of 5 followed by ReLu (Nair & Hinton 2010) activation function, pooling laver with a kernel size of 3 x 3 and stride 2, and a local response normalization (Krizhevsky al. 2017a) with depth radius of 4, bias of 1, a equal to 0.001 and / set 0.75. After two blocks three fully-connected layers of size 384, 192 and 10 are used, with ReLu activations (Nair & Hinton 2010) after the first and a softmax layer following the last layer. For optimization, we use the cross-entropy loss and gradient descent (Robbins & Monro 195T) with a initial learning rate of 0.1 and an exponential decay rate of O. Additionally, we use exponential moving average on the weights with decay factor of 0.9999.
9
In order to compute FD scores, we use the Inception-v3 (Szegedy et al. 2015) network, which is available pretrained in the Tensorflow library (Abadi et al. (2066). Images are resized to 299 x 299 fit the input dimension the network, which was pretrained on Imagenet (Deng et al. 2009). The resulting feature representations are retrieved from the pool-3 layer with 2048 output dimensions.
D SPEECH SEPARATION
Making the description of the speech separation experiments more comprehensive, we offer the following additional implementation details for the speech embedding network for which we adopt the generalized end-to-end loss (GE2E) proposed by Wan al. (2017). We train network build on convolutional and LSTM (Hochreiter & Schmidhuber 1997) layers. The inputs are based on second long samples transformed into the Short- Time Fourier Transform (STFT) spectrogram and the Mel-frequency cepstral coefficients (MFCC). For the spectrogram creation, we use a hop size of 6 ms and a window length of 23 ms. We apply simple 1-dimensional convolutional layer with kernel size of to the MFCC to match the number of frequency bins of the STFT, such that we can use the concatenation of both an input to network. Tab. summarizes the network following the input, predicting a 100 dimensional speaker embedding.
14

---

Under review as a conference paper at ICLR 2020
Table 2: Detailed specifications of the speaker embedding network trained with the GE2E Loss (Wan
al. 2017) applied to calculate FD scores in section 5.3 The network is based on the densely
nected network (Huang et al. 2016), fully-connected layers (FC), LSTMs (Hochreiter & Schmid-
huber 1997). and Statistical Pooling (Snyder al. 2017). As optimizer, use Adam (Kingma
& Ba 2014 with learning rate of 0.0001.
Layer
Filter Sizes
Filters
Additional Specifications
1 x
64
Dense Block
2
densely connected
3 x 3
16
CNN
3 x 3
32
stride of 2
1
64
Dense Block
3 x
16
CNN
3 x
64
stride of 2
1 x
64
Dense Block
x 12
3 x
16
CNN
3 x
128
stride of
1x1
64
Dense Block
12
3 x 3
16
CNN
3 x
256
stride of 2
1 1 1
64
Dense Block
3 x
16
x
12
CNN
3 x
512
stride of 2
1
64
Dense Block
12
3 x 3
16
CNN
3 x
1024
stride of 2
CNN
1
512
5
LSTM
512
Stats Pooling
with max, mean, variance features
CNN
3
200
FC
100
with flattened time dimension applied on the last strided CNN output
Stats Pooling
CNN
3
200
FC
100
flattening the input beforehand on both stats pooling results after the FC layer
Addition
L2 normalization
15

---

Under review as a conference paper at ICLR 2020
Table 3: Detailed accuracy and FD the Amazon review categorics. Models are trained by using different categorics train and test pairs. The FD is obtained with the GloVe 100 100 embedding.
Supplies
82.04%
31.78%
66. 66.49%
83. 83.4%
80.68%
77.01 %
82. 62.47%
78.35%
0.2014
0.0836
0.4481
0.0862
0.1029
0.353
0.1526
0.1656
Per
Office Products
84.54% 44.54%
84.84%
66.61% 66
84.05%
82.2%
78.06%
83.21%
81.11%
0.0888
0.1113
0.4596
0.0954
0.1616
0.3888
0.2128
0.1656
Clothing Shoes Jewelry
88. 12.
85.83%
67.56%
87.56%
83.87%
82.01%
81.43%
85.83%
0.2415
0.174
0.5008
0.1
0.1695
0.4082
0.2128
0.1526
Movies and TV
19.73%
78.69%
79.2%
79.09%
78.46%
77.69%
74. 74.94%
76.05%
0.3539
0.4044
0.1932
0.3598
0.2511
0.4278
0.3888
0,3676
Toys and Games
Accuracy
87.0%
86 66.88%
73.79%
86.6%
85 22%
84.65%
81 1..6%%
86.36%
0.1739
0.1455
0.3624
0.1275
0.2366
0.1695
0.1616
0.1029
FD
Sports and Outdoors
86.71%
85 54.49%
71.02%
79.75%
78.58%
84.58%
81 11.24%
83.96%
0.099
0.0635
0.4355
0.1275
0.3598
0.1132
0.0954
0.0862
Digital Music
76.34%
75.62%
75.37%
73.93%
87.24%
74.48%
67. 67.59%
69.21%
0.4055
0.4678
0.4355
0.3624
0.1932
0.5008
0.4596
0.448
Home and Kitchen
87.92%
67.78%
86.54%
81.91% 9%
78.97%
85.18%
83.00%%
84.89%
0.1602
0.4678
0.0635
0.1455
0.4044
0.174
0.1113
0.0836
Electronics
44.83%
68. 12%
82.7%
66.08%
77.11%
83.166 16%
79. 47%
81 11.06%
0.1602
0.4055
0.099
0.1739
0.3411
0.2415
0.0888
0,2014
Clothing Shoes Jewelry
Training Set Test Set
Sports and Outdoors
Sports and Outdoors
Home and Kitchen
Home and Kitchen
Toys and Games
Movics and TV
Office Products
Toys Games
Movies and TV
Office Products
Digital Music
Digital Music
Training Set
Pet Supplies
Training Set
Pet Supplics
Electronics
Electronics
Clothing
16


Reviews:
[{"review_id": "ryx1DuaTKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to relate the performance of a classifier under distribution shift using a quantity called Frechet distance. It is common belief that the further apart the training and test distributions are, the more difficult it is to transfer a learned classifier. They give simple bounds via gradient norm/Lipschitz constants and distribution distance in Theorem 1. The authors try to capture it with Frechet distance, but I struggle to understand what is new in this work. \n\nFirst, there are a lot of assumptions in the computation of the Frechet distance: \n  1. The authors use the embeddings given by the neural networks instead of the raw data since density estimation is hard. This makes the distance model-dependent \n  2. The authors assume the embeddings are normally distributed in their computation, which have not been justified. \n\nMost importantly, they do not relate the Frechet distance to the lower bound in Theorem 1. There is no estimation on how the learned changes across distributions in the gradient norm term. This makes the evaluation nothing more than a confirmation of the general idea that the closer the distribution, the better the transfer. The lower bound is not used in any quantitative manner. \n\nThe authors should make the connection of the bound and its computation clear, with proper connections to the experiments. The current paper looks like separate theoretical and experimental results that do not tie together. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1euOzraKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the relation between Frechet distance of training and test distribution and the generalization gap. The authors derive the lower bound for the difference of loss function w.r.t. training and test set by the Wasserstein distance between embedding training and test set distribution. Empirically, the authors illustrate a strong correlation between test performance and the distance in distributions between training and test set.\n\nThe motivation to find the relation between generalization gap and the Frechet distance of training and test distribution is sound. However, I am not sure that the lower bound as in Equation (1) is enough. I am curious that one can derive the upper bound for the relation or not. The finding about choosing a training data distribution should be close to the test data distribution seems quite trivial in some sense. I am not clear about its important since it is quite popular that the distribution shift affects the performance and many learning approach assumes same distribution for training and test data. Overall I feel that the contribution may be quite weak, and I lean on the negative side.\n\nBelow are some of my concerns:\n\n1) About the lower-bound in Equation (1), it seems unclear to me that when the W_2(p1, p2) = 0, we can inference any information about the test performance (It seems quite trivial for this case, the left hand side time is greater than or equal 0?) In my opinion, the upper-bound is more important which one can inference much information about the difference of generalization gap.\n\n2) In the proof of Theorem 1, it is quite hard to follow with the current notation, for the integral in (i), (ii) as well as in the proof using the intermediate value theorem, which variables are used? I am confused which one is variable, which one is constants in those integrals.\n\n3) In page 5, at the interpretation (1), for W2(p1, p2) = 0, the learned function fits training distribution perfectly and is not ill-conditioned ==> why one can deduce that the test distribution is fit perfectly? What we have in Theorem 1 is the lower-bound only?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1goCQ9J9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of how the mismatch between distributions of training data and test data would affect the generalization gap in machine learning tasks. This phenomenon has been observed many times in previous literature and has gathered significant attention in the machine learning community.\n\nThe paper took a step in relating the change in the performance of the learned function to the Frechet distance (FD), also known as 2-Wasserstein distance, between the input and output distributions and proved that the former is lower bounded by the latter multiplied by a term related to the sensitivity of learning algorithm to distribution shift. The paper also provides empirical evidence that the testing error is correlated with the FD between input and output distributions based on tasks including text classification, image classification, and speech separation.\n\nI find the idea of the paper interesting but the content not convincing enough. The theory proved in the paper does not provide additional quantitive insight beyond intuition. Specifically, the term about the sensitivity of the algorithm is not justified enough in the paper. The experiments provide some evidence but not convincing, especially for the part about image classification.\n\nI also find the statement about the generalization gap a bit misleading. Generally, the generalization gap refers to the gap between the expected error and the empirical error.  But the experiments are mostly presenting the performance on the test data. \n\nOverall, I don't think the paper meets the standard for publication at ICLR.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors discuss how to predict generalization gaps. Reviews are mixed, putting the submission in the lower half of this year's submissions. I also would have liked to see a comparison with other divergence metrics, for example, L1, MMD, H-distance, discrepancy distance, and learned representations (e.g., BERT, Laser, etc., for language). Without this, the empirical evaluation of FD is a bit weak. Also, the obvious next step would be trying to minimize FD in the context of domain adaptation, and the question is if this shouldn't already be part of your paper? Suggestions: The Amazon reviews are time-stamped, enabling you to run experiments with drift over time. See [0] for an example. 

[0] https://www.aclweb.org/anthology/W18-6210/
Decision: Reject