Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data
Authors: Yaoqing Yang
Ryan Theisen
Liam Hodgkinson
Joseph E. Gonzalez
Kannan Ramchandran
charles h martin
Michael W. Mahoney
Venue: Submitted to ICLR 2023 (2023)

Abstract: The search for effective and robust metrics has been the focus of recent theoretical and empirical work on generalization of deep neural networks (NNs). In this paper, we discuss the performance of natural language processing (NLP) models, and we evaluate various existing and novel generalization metrics. Compared to prior studies, we (i) focus on NLP instead of computer vision (CV), (ii) focus on generalization metrics that predict test error instead of the generalization gap, (iii) focus on generalization metrics that do not need the access to data, and (iv) focus on the heavy-tail (HT) phenomenon that has received comparatively less attention in the study of deep neural networks. We extend recent HT-based work which focuses on power law (PL) distributions, and we study exponential (EXP) and exponentially truncated power law (E-TPL) fitting to the empirical spectral densities (ESDs) of weight matrices. Our empirical studies are carried on (i) hundreds of Transformers trained in different settings, in which we systematically vary the amount of data, the model size and the optimization hyperparameters, (ii) a total of 51 pretrained Transformers from eight families of Huggingface NLP models, including BERT, GPT2, ALBERT, etc., and (iii) a total of 28 existing and novel generalization metrics. From our detailed empirical analyses, we show that shape metrics, or the metrics obtained from fitting the shape of the ESDs, perform uniformly better at predicting generalization performance than scale metrics commonly studied in the literature, as measured by the average rank correlations with the generalization performance for all of our experiments. We also show that among the three HT distributions considered in our paper, the E-TPL fitting of ESDs performs the most robustly when the models are trained in experimental settings, while the PL fitting achieves the best performance on well-trained Huggingface models, and that both E-TPL and PL metrics (which are both shape metrics) outperform scale metrics.

Under review as a conference paper at ICLR 2023
EVALUATING NATURAL LANGUAGE PROCESSING MOD- ELS WITH GENERALIZATION METRICS THAT DO NOT NEED ACCESS TO ANY TRAINING OR TESTING DATA
Anonymous authors Paper under double-blind review
ABSTRACT
The search for effective and robust metrics has been the focus of recent theoretical and empirical work on generalization of deep neural networks (NNs). this paper, we discuss the performance of natural language processing (NLP) models, and we evaluate various existing and novel generalization metrics. Compared to prior studies, we (i) focus NLP instead of computer vision (CV), (ii) focus on generalization metrics that predict test error instead of the generalization gap, (ii) focus on generalization metrics that do not need the access to data, and (iv) focus the heavy-tail (HT) phenomenon that has received comparatively less attention the study of deep neural networks. We extend recent HT-based work which focuses on power law (PL) distributions, and we study exponential (EXP) and exponentially truncated power law (E-TPL) fitting to the empirical spectral densities (ESDs) of weight matrices. Our empirical studies are carried on (i) hundreds of Transformers trained in different settings, in which we systematically vary the amount of data, the model size and the optimization hyperparameters, (ii) a total of 51 pretrained Transformers from eight families Huggingface NL.P models, including BERT, GPT2, ALBERT, etc., and (iii) total of 28 existing and novel generalization metrics. From our detailed empirical analyses, we show that shape metrics, or the metries obtained from fitting the shape of the ESDs, perform uniformly better predicting generalization performance than scale metrics commonly studied in the literature, as measured by the average rank correlations with the generalization performance for all of our experiments. We also show that among the three HT distributions considered in paper, the E-TPL fitting of ESDs performs the most robustly when the models are trained in experimental settings, while the PL fitting achieves the best performance on well-trained Huggingface models, and that both E-TPL. and PL. metrics (which are both shape metrics) outperform scale metrics.
INTRODUCTION
Recent years have wide array of large-scale empirical studies the various metrics used
quantify generalization (Dziugaite et al., 2020), Diang et al. 2019) Martin & Mahoney 2021a Martin
ct al. 2021). On the one hand, eicory-driven metrics have the potential reveal more information
than test error, bringing us one step closer to unpacking the black box of deep NNs (Frankle & Carbin
2018 Nakkiran al. 2019 Zhang al. 202T). On the other hand, wide variety of generalization
metrics have been applied to predict the quality of pretrained models (Martin & Mahoney 2019
Martin al.) 202T), , design effective training procedures (Foret et al., 2020; Izmailov al. 2018).
improve network efficiency (Chen et al., 2020 Dong et al. 2019), quantify network robustness TTnaa
Griftin 2016 Yang al. 2020), improve ensemble learning techniques (Fort al. 2019 Garipov
al.) 2018), analyze and improve large-scale machine learning contests (Martin & Mahoney 202
and so on.
Despite advances in the study of generalization, however, several recent papers point out the defi-
ciencies of many of these "fantastic" generalization metrics. These include a lack of "robustness" to the changes of environmental hyperparameters (Dziugaite et al.) 2020 Diang et al. 2019) (such as data, network architecture and training schemes), or the Simpson's 's paradox that generalization metrics perform differently (i.e., predict opposite trends) when applied to each sub-part of collection

---

Under review as a conference paper at ICLR 2023
of learning models to the holistic study (Martin & Mahoney 2021 aa. Another drawback is the
over-reliance on experiments with CV models, which are relatively eepl-ex,, and which are not representative of many other application areas. Despite a few counterexamples (Martin et al. 202T Nakkiran et al. 2019 Yang et al. (02T), systematic studies of generalization in other fields, such as NLP, are largely missing.
Generalization metrics for NLP. The objective of this paper is to provide a systematic study of generalization metries in NLP, addressing several deficiencies in prior studies (Dziugatte et al. 2020 Jiang et al. 2019; Martin et al. (02T). Compared to CV, predicting generalization in NLP has several important differences that require careful consideration. The training data from standard CV benchmarks can often be easily obtained, while NLP pretraining datasets are typically web-scale and are challenging to access. Therefore, generalization metries that can measure the quality of learning models without access to data are ideal for NLP. Indeed, recent work has demonstrated that access to training or testing data is not necessary for assessing the model quality of learning models (Martin et al. 202I. though these have yet to be evaluated at scale in the NLP domain. Furthermore, it is typically infeasible to train NLP models to interpolate the (frequently large) training set. This becomes issue when applying most existing generalization metries they often estimate the generalization gap (i.e., the difference between training and test performance) rather than the test error itself. Metries that focus predicting the generalization gap include most of the well-known metrics in CV, such as those based on the PAC-Bayesian framework (McAllester
1999 Neyshabur
al. 2018 and margins (Bartlett et al. 2017 Jiang al. 2018 Pitas al. 2017).
To illustrate the issue, consider the problem of model selection between two models CJang et al. 2020: Martin & Mahoney (02la).Suppon we are given two classification models. Then even if we have i) access to both models' training errors, and ii) a metric which is guaranteed to perfectly rank correlate with the generalization gap, then we still cannot determine which model as smaller test error. This means that, if our objective is to construct metric that correctly predicts which model has lower test error, rank correlation with the generalization gap is not sufficient. In this paper, we aim to study how generalization metrics correlate with model quality, for which we use test error close approximation. As we will demonstrate (in Figure 4. rank correlation with the generalization gap indeed does not imply rank correlation with model quality in practice, and in fact often orders models in the opposite order of their test errors. From a practical point of view, for NLP tasks, we prefer generalization metrics that can directly predict trends in test error (or similar evaluation metrics in NLP, such as the test BLEU score (Papineni et al. 2002)) rather than trends in the generalization gap.
Naturally, cannot expect metric be universally correlated with test error if evaluating the metric does not need data. However, within certain classes of models (e.g.. stages of training in one model or across pre-trained models), they may be effective diagnosing model quality. With these objectives in mind, among the generalization metrics in the literature, we take particular interest in those derived from the heavy-tail self regularization (HT-SR) theory (Martin & Mahoney 2019 2021b), which (i) predicts test error directly instead of the generalization gap and (ii) does not require access to training (or testing) data.
HT-SR theory. The core principle of HT-SR theory is that HT structures arise naturally in the ESDs of the weight matrices [1] as the result of extracting various correlations in data during optimization (Martin & Mahoney 2019 2021 la D. Martin et al. 202T). Its primary practical consequence is that by estimating the PL coefficient from the ESDs (requiring only weights), one can predict model quality, as smaller coefficients are reported to correspond to higher test accuracy. However, these estimators can be unstable, and so one must be careful not to rely on them alone. The quality of the PL fit itself should also point similar conclusions (Martin & Mahoney 2021b which can be sanity check.
The principles of HT-SR theory extend beyond fitting the PL coefficient, however, as ESDs can take many forms. To this end, we study three different types of distributions to fit to the ESDs of weight matrices, including power laws (PL.) in Eqn. ①.. exponentially truncated power laws (E-TPL.) in Eqn. ( and exponential laws (EXP) in Eqn. ( These all commonly considered families of distributions in classical studies of PL (Clauset et al., 2009), and it is often hard in practice to predict which family fits data the best (as we show in this paper, this is true for deep NNs especially).
'The ESD of weight matrix W refers to the empirical density of the eigenvalues of the squared weight matrix W W. See "Preliminary of ESDs of weight matrices" the end of the Introduction.
2

---

Under review as a conference paper at ICLR 2023
Figure| shows examples of comparing different HT fittings the same ESD. Following Martin & Mahoney 2021a), we refer to the various metrics derived from HT-SR as shape metrics.
pha-1.76,
distance-0.09
alpha- 3.46, ks, distance- 0.07
alpha-4.88 4.84, ks distance-0.11
10
10"
10"
10-1-
10 3-
10
30"2
10-3
10-3
10
10
10
10P
10-2
30
10'
30
10
10-2
10:1
10"
10'
10
Eigenvalues of correlation matrix
Eigenvalues of correlation matrix
Eigenvalues correlation matrix
(a) Small
(b) Mediocre ks ks_distance.
(c) Large ks_distance.
E-TPL betam1.50, ks distance=0.02 E-TPL lambda=0.02
E-TPL beta=1.11, ks distance=0.02 E-TPL lambda-
E-TPL betam1.00 000 ks wld.sstnnc=0.cccm E-TPL lambda=0.00
10
10
16
10P
109
16f
B5D
1071
10a
20"3
10-1
10
10"
10:
10a
10:
10a
10-1 Eigenvalues of correlation matrix
10-1
30
Eigenvalues of correlation matrix
Eigenvalues of correlation matrix
(d) E-TPL fitting of the ESD.
(c) E-TPL fitting of the ESD.
(f) E-TPL fitting of the ESD.
Figure 1: Comparing PL and E-TPL fitting, (First row). Good, mediocre, and bad PL fittings measured by the us.distance (Second row). E-TPL fitting of the ESD on the same column. Blue histograms represent the ESDs. Solid vertical lines represent the lower threshold Emin of the PL distribution found by the fitting procedure. Solid curves represent ESDs truncated using Tiin and dashed curves represent the fitted HT distributions.
Contributions. The following summarizes our main contributions.
Deviating from prior work examining generalization metrics in CV DZiugaate et al. 2020 Jiang jet al. 2019), we provide the first systematic empirical study on various generalization metrics in NLP. Our detailed studies include the following:
We consider 360 transformers trained with varying hyperparameters. and eight families of
pretrained SOTA transformers downloaded from Huggingface (Wolf et al. 2020). including
BERT (Kenton & Toutanova 2019), GPT2 (Radford et al. 2019). ALBERT (both v1 and v2)
Lan et al. 2019). etc.
measuring the correlation between 28 generalization metrics and the model quality (measured
by test-time performance) over three different model classes: (i) models trained with the optimal hyperparameters: (ii) a single model at different stages of training; and (iii) model
trained with different typerparameters (similar to Diang et al. (2019).
We extend prior studies on HT-SR theory and investigate alternative models to fit heavy-tail/light tail distributions. Our results show that R-TPL. fits comparatively robust alternatives to PI. fits for predicting trends in test error on suboptimally-traine. models.
We find that, applied appropriately, HT-based shape metrics consistently perform better than scale metrics (or norm-based metrics) for predicting model quality.
We provide results for data- data-dependent metries motivated by margins and PAC-Bayesian bounds (Dziugaite et al.) 2020; Jiang et al. 2019). While these metrics perform well in predicting the generalization gap, we show that none of them satisfactorily predicts test error directly.
Preliminary of ESDs of weight matrices. Consider NN with d layers and corresponding weight matrices Wi, W.... Wa. For each weight matrix W; with shape N x M, assume without loss of generality that N > M (otherwise, consider W.). We define the correlation matrix as X, W. W, ,. and denote the cigenvalues of X; as (Ay [ so that Xy of, where {ay),4, the singular values of W, Furthermore, we use Ai,max to denote the maximum eigenvalue of the correlation matrix X, The ESD of the weight matrix W, refers to the empirical density of the
1.M
3

---

Under review as a conference paper at ICLR 2023
eigenvalues of X,, typically represented through a histogram. We let p(x) denote the density function to fit the ESD taking values in the interval (Xmin Tmax). For a power law, p satisfies
p o x x.
Tmin < x < mmax.
From Martin & Mahoney a Umax is chosen to be the maximum eigenvalue of the empirical correlation matrix. However, Zmin is variable to be optimized to improve the quality PL fitting, and it is not equal to the minimum eigenvalue in general.
2 HEAVY-TAIL SELF-REGULARIZATION THEORY
Here, we provide brief overview of the HT-SR theory, and discuss several metrics that can be derived from it. According to HT-SR theory, the ESDs the weight matrices become more heavy- tailed during training as they become increasingly correlated. One can quantify the extent of these correlations by fitting a PL to the ESD of a weight matrix, for example, by using the open-source WeightWa ther tool (Martin et al. 202T After computing the ESD of weight matrix, the maximum likelihood estimate from Alstott et al. (014)) to fit the PL distribution, the specific form of which has been defined in 1 Let PL_a pha denote the PL coefficient averaged over layers; effectively the slope of the tail of the ESD of the pooled weights,
log-log scale.
Correctly identifying and fitting PL distributions well-known to be a challenge in practice. For example, a density that appears as straight line on log-log scale plot need not follow power law, as there are many other distributions that could show similar behavior, including lognormal and exponential-tye combine the pure PL and other distributional assumptions, can often improve the quality of fitting (Alstott et al., 2014 Clauset et al.) 2009). Therefore, in addition to the PL distribution defined in we consider several other distribution classes from the literature.
distributions (Clauset et al. 2009). Nested distributions such as E-TPL, which
TE.TPL_lambl and E. T.L. beta) The ESDs assumed take "nested" form.
p(x) x - exp(- (xr),, Zmin < < Tmaxx-
After fitting the E-TPL, we call the exponential truncation coefficient the E.TPL. ambda metric, and we call the PL coefficient the E. TPL beta metric.
(Emb_lambda). The ESDs are assumed to take the following form.
p(x) xx exp(-Aan Tiin < < Tmmx-
(3)
After fitting the EXP, we call the exponential coefficient À the EXP -lambd metric.
For more details of the various metrics considered in this paper, see Table All of the metrics derived from HT-SR do not require access to data, and they are relatively cheap to compute. Our primary comparisons are between shape metries (derived from HT-SR), and scale metries (mostly norm-based). For the precise definitions of these metrics, see Appendix A
Issues of PL. fitting. It is well-known that subtle issues arise when fitting the ESDs Alstott je al. 2014 Clauset et al. 2009 Martin & Mahoney 2017 2021ap. To best mitigate these issues in PL fits, we adopt the fitting strategies used in WeightWatcher (arrtin & Mahoney 2017). For example, as in Clauset et al. (2009), it is common to choose the lower threshold Imin which coincides with the best quality fit under the Kolmogorov-Smiroof statistic (referred as PL-ks.d. wiisssiisstti ance for PL and E.TPL.ks www.stanccee. for E-TPL in the sequel; see Eqn. 12)). However, this method cone-consu,ing, especially for E-TPL as there are two parameters to fit. Instead, we adopt the fix-finger method (see We W ghttatcher) which selects Tiin as the peak of the ESD when fitting E-TPLs. More than a simple speed improvement, we find this method also yields more stable results.
Comparing PI and E-TPL fitting. Referring to Figure 1 we now discuss how E-TPL could partially address these fitting issues. On the first row of Figure we show three typical cases of PL fitting. In Figure Ia the log-log scale reveals a "linear region" of the histogram, which the PL fitting correctly locates. The quality of fit, measured by the ks.distann reported in Table 5 of Martin & Mahoney 222 In Figure Ib and Figure Ic the ESDs do not exhibit a clear linear region on the log-log scale. Following Martin & Mahoney (20216). is ill-advised consider metrics derived from a Pl. fit in these scenarios. In practice, this typically
is within a typical range,
4

---

Under review as a conference paper at ICLR 2023
Need initial
Scale
Need
Need
Prediching model quality or generalization gap?
Name
Eqn
Ref
weights?
shape
data?
gpu?
paran.rocm
2019
No
Scale
Generalization gap
fro_dist
Jaang et
2019
Yos
Scale
No
No
Generalization gap
Legunorm
Generalization gap
ww.ppactrrll.nnn morm
Martin & Mahoneyy
64021
No
Scale
No
No
Generalization
dist.apec_int
Piir 2019
Scale
Generalization gap
Neyshahur el A015
No
Scale
No
Generalization gap
mp. rank
10
Martin & Mahoney
412 b
No
Scale/Shape
No
No
Model quality
Mahoney
Scale/Shape
Model quality
PL.alpha
Mariin & Mahiony
22211b
No
Shape
No
No
Model quality
paper
E.IPL bets
WeightWatcher paper
No
Shape
No
No
Model quality
Weigh/Watcher
Shape
quality
This paper
EXP -ap_iambda PL.ksdisteene
WeighrWatcher
No
Shape
No
No
Model quality quality
Mahoney
2021b
Shape
This paper
E. E.TPL ks distance
Martin & Mahoney
2021b
Shape
Model quality
sipha.wwighte
Martin & Mahonev
2021b
No
Hybrid
No
Model quality
log.alpäe_torm
LG
Martin & Mallony
2221a
No
Hybrid
No
No
Model quality
inverse_margin
2019
Maybe
Generalization gap
harnieit di +0:00
loggreeldf..pee..vveerarggg
Pilas et al
201
No
Scale
Yes
Maybe
Generalization gap
Barlieit ot El
201
Pilas ct ml
11180
No
Scale
Yes
Maybe
Generlizatton gap
Bartlett ct
bis.pro__af_free htt.:ffooooovoorrrgg..o
21
Prias et ml
201
No
Scale
Yes
Maybe
Generalization
Log.sum.of. fro.fooveerr margin
Pilas al
2117)
No
Scale
Yes
Maybe
Generaization gap Generalization gap
22
Neyshabur
2015
No
Scale
Yes
Maybe
pecbayes
Scale
Yes
Generalization gap
pachayes orig
2G
Nevslaauu el
2017
No
Scale
Yes
Yes
Generalization
pachayas,flatmass
Neyshahur 2l
Generalization gap
pacbayes.mag.init
2019
Yes
Scale
Yes
Yes
Generalization gap
macbayes.mag.ore
Jaang ct 2019
No
Scale
Yes
Yes
Generalization gap
pachayes flatneas
Jiang
No
Scale
Yes
Yes
Generalization gap
Table Overview the generalization metrics considered. We focus on the shape metrics derived from the ESDs of weight matrices. See Appendix A for the details of these metrics.
occurs when PL_appha > 4 (e.g., see Figure Tcl. On the other hand, in these two cases, the corresponding E-TPL fits (shown on the second row in Figure D still closely match the empirical density function (see Figure Ic and Figure MT]. and the ks.distan ance on the second row using E-TPL fit is smaller than that for the PL fit on the first row, even when the fit on the second row clearly covers a larger part of the ESD. In these two cases, the E.TPL.lambdk plays a similar role as the PI pha in PL. fitting, and provides an effective alternative when the ESD does not exhibit a proper PI..
Between these three PL and E-TPL fittings, we would like to point out that the important thing in HT-SR is not the PL fitting per but that the spectral distributions exhibit HT or other non- standard shapes. The particular forms of the distributions fit here simply constitute different ways to quantify this property in practice. These details, such as selecting the most appropriate distributional assumptions, clearly matter we would like to engineer the tools HT analysis to effectively measure the ground truth. However, the primary concern in predicting generalization is measure the shape information, and the shape information independent of the fitting procedure, although better fitting procedures may capture the shape information better.
EMPIRICAL RESULTS
3.1 EXPERIMENTAL SETUP
Dataset. We consider the WMT14 German English (DE-EN) dataset (Bojar et al. 2014). commonly used us a benchmark for neural machine translation (Edunov et al. 2018; Ott et al.) 2018 Shen et al. 2020 Vaswani al. 2017). WMT14 consists of 4.5 million sentence pairs for training.
5

---

Under review as a conference paper at ICLR 2023
model in group
Best model group
model group
Num comples
Num samplecs
Num samples
160000
160000
ELEE score
140000
320009
522030
525020
640020
540000
B40200
1150000
B171)
1280000
L280000
30
10
0.2
0.4
6.5
0.04
0.c0
0.08 oin
0.32
E_TPL_ IPP__aabda
PL_ -_aappha
E_ __TPL_ks_ee did.ttatane
ks_
model in group
Best model group
Best model group
score
25
Num comples
160000
Num samreics
160000
BLEU score
Num samples
160000
320009
532030
320020
20
646026
540000
540200
1150000
1280000
L200000
30
10
0.0
0:00
0.09
0.10
0.4 0.6
PL kss distance
stable baabbbaankkk rank
mp solttaatkkk
Figure 2: BLEU-score vs. six shape metrics for 200 Transformers trained on WMT14 with varying hyperparameters. HT-SR theory applies for optimally-ture models (black stars), that is, models that have better BLEU scores exhibit heavier-tailed ESDs. For suboptimal models, the HT-SR metrics can be anti-correlated with model quality, see e.g. the grey dotted line in the first subfigure.
Hyperparameters. To capture the relationship between the generalization metries and model quality in number of different settings, we vary several hyperparameters: the number of samples (either 160K, 320K, 640K, 1 1,28M, 2.56M samples), the initial learning rate during training (across eight different rates), the model width (embedding dimension either 256, 384, 512, 768, or 1024), and the model depth (f4, 5, 6, 7, 8J-layer transformers). We also construct a nigh-ddmmensioal grid of different hyperparameters @ = {(O1..... 0k): 01 E ① I..... 0x E O J. so that we can compare models when one of the hyperparameters varied. Two separate high- high-dimensional grids with dimension K are considered: (1) xamplexxearniinn rate x width; (2) sample Iearning rate x depth. Each grid contains 5x8x5=200 of these training settings. In total, there are 360 trained models because the high-dimensional grids overlap each other, and 40 models belong both grids. We will consider three subtasks to evaluate the considered generalization metrics.
Task one, correlation evaluated on optimally trained models. In the first task (Section 3.2.1 we study the relationship between model quality and generalization metries on models trained with the optimal choice of hyperparameters. This task mimics the grid-search method often employed in large-scale (pre)training tasks.
Task two, correlation in time. In the second task (Section 3.2.2p. we track BLEU score and generalization metrics during training, assessing time-wise correlation to model quality. This task has been considered the literature (Bartlett al., 2017), and from practical point of view, capturing the time-wise dependence during training could potentially lead to better ways of early stopping or regularizing the model.
Task three, correlation when hyperparameters are varied. In the third task, we study the rela- tionship between the model quality and the generalization metrics when a single hyperparameter varied. Metrics that achieve high (rank) correlation for all the hyperparameters are good candidates for model selection. Constrained by space, we discuss this result in details in Appendix
Training and model setup. For the details of the training settings, see Appendix
3.2 EVALUATING THE METRICS ON TRANSFORMERS TRAINED DIFFERENT SETTINGS
In this subsection, we study 28 generalization metrics (with details provided in Table and examine their correlations with BLEU score (Papineni et al. 2002), the most commonly used metric to evaluate

---

Under review as a conference paper at ICLR 2023
Num samples
160000
Num samples 320000
Num samples 640000
samples 1280000
0.30
0:30
0.30
ETPL lambad
BLEU score
E-TPL lambaa
BLEU score
lambda
BLEU score
TPP lambda
0.00
0.0G
0.06
0.06
0.02
0.02
0.02
20
Epoch number
number
Epoch number
Epoch number
Num samples 160000
samples 320000
Num samples 640000
samples 1280000
ETPL lombdo
0.08
BLEU score
E-TPL lambdd
BLEU score lambda
score
T- lambda
0.00
0.00
0.06
0.02
0.02
0.02
0.02
20
10 number
38 Epech number
Epoch number
Epoch number
Figure 3: E.IPL_labdda closely tracks the BLEU score, i.e., BLEU score increases when the E.TPL.lambde drops. Results are shown for Transformers trained on WMT14 with different number of samples. (First row). Training with dropout 0.1. (Second row). Training without dropout.
machine translation We also consider correlation between these metrics and the generalization gap, defined as the BLEU score for training data subtracted by the BLEU score for test data.
3.2.1 TASK ONE: EVALUATING CORRELATIONS ON OPTIMALLY TRAINED MODELS ONLY
Here, we group models using the number of training samples, and select the best model from each group when the model depth and the learning rate are varied. In Figure 2 each curve represents a group of models trained with certain number of training samples. The black star each curve represents the model trained with the optimal choice of hyperparameters (learning rate and depth in our setting), obtained by searching for the optimum on third-order polynomial fit of each curve. From Figure 2 we see that the shape metrics correctly predict the model quality for models trained with the optimal hyperparameters, i.c., the BLEU scores should be higher when the metric values are smaller.
3.2.2 TASK TWO: TIME-WISE CORRELATIONS AND RANK CORRELATION RESULTS
In this subsection, we study time-wise correlation between our chosen metrics and the BLEU scores.
TPL lambda tracks the BLEU score. As warm-up, consider how well the E TPL. ammoda metric defined in 2 tracks the BLEU score (recalling that E. E.TPL.1 ambda assumes the ESDs follow E-TPLs). We training with and without dropout to study the effect of training schemes, and we consider different quantities of data to test robustness in the dataset. In Figure 3 the first row considers models trained with dropout, while the second row considers models trained without dropout. The multiple columns track A.TPL.llabb different amounts of data. We can see that E E_TPL ambda not only successfully tracks BLEU scores but also differentiates underfitting (first row, with dropout) from overfitting (second row, without dropout) this experiment.
and the BLEU score throughout training for
Shape metrics predict model quality, while scale metrics predict the generalization gap. Now we consider the rank correlations between our chosen metries and the test BLEU score. The rank correlations are evaluated across training, i.c., for each of the 360 settings of the hyperparameters, calculate the Spearman's 's rank correlation between BLEU scores and the values of each generalization metric over all epochs. The summarized results are presented in Figure 4a A positive Spearman's rank correlation (with BLEU) suggests that the generalization metric is useful in tracking BLEU during training. A negative Spearman's 's rank correlation, on the other hand, implies that the metric
Several empirical metries have been designed measure the quality text generation, such as BERTScore Zhang et al. 2019) and BARTScore (Yuan et all 2021). Our work is different because we do not need any data, and we do model selection using the weight matrices only. BERTScore and BARTScore evaluate the text quality. and thus they need source or reference texts generated by humans. These metrics can serve as alternatives to BL BLEU,, which is viewed as ground truth in our work.
7

---

strong Under review as a conference paper at ICLR 2023
Correlations with model quality
Correlations with generalization gap
esp_lambde
leg_prod_ _rod_of_spee log, sum spec margin leg_spectral, norm
ofsspe, sppe__vee__eteeeeeeee over_mergin
Shape Hybrid
Shape
Ettp ks_ distance
Hlyrid
Inverse, margin tpl beta
Scale
log_norm norm log, prod, of fro, oveeveer _aargin log_sum, of fro, maagrnn pacbayes, init
Scale
mp. softrank
stable rank
plaipha
fro, dist
margin
pacbayes pacbayes, mag, magalaaaaa pacbayes, flatness param norm dist spec. Init pacbayes, mag. mag_init init path norm alpho weighted pachayes_ Baccaays_mag.coe mag_s_maarri
alpha weighted
log alpha norm
pacbayes, Beabayes_mag_ooe mag_artg path norm dist spec Init pacbayes mag Init pacbayes flatness
pacbayes_on
pl margin
path, norm, pl plaippa
pacbayes_
pacbayes, mag flatness
stable rank
log, sum, of fro, over margin
mp softrank
leg poo_pod_ot. of or_.fr_over__ne ooooovermmma margin log, sum of spec over marginn log, regp of www.spec_oee_eeee margin
betpl_beta
inverse, margin
t.tpl ..tpllambda
ks distance
leg_spectral
-15 -10 -05 00 a5 Spearman's rank correlation
-1. -0.5 00 05 La 1.5
Spoarman's rank correlation
(a) Correlations with model quality. Spearman's 's rank correlation between various generalization met- rics and BLEU.
(b) Correlations with generalization gap. Spear- man's 's rank correlation between various generalization metries and the generalization gap.
Figure 4: Comparing multiple generalization metrics for predicting BLEU score (on the left) or the generalization gap (on the right). Lines each box delineate the 25/50/75 percentiles of the rank correlations in 360 different settings (including different amount of data, different network depths, different network widths, and different initial learning rates).
often gives the incorrect prediction. Figure 4a we use the average rank correlations for all settings to study the effectiveness of each metric, and present 25% quantile rank correlations to indicate robustness across runs.
In Figure 4a E.TPL.ks.distance and E.IPL.beta, exhibit some of the highest rank correlations with BLEU score. The EXP lambda metrie, which assumes EXP distribution the ESDs, achieves the highest median rank correlation, while the E TPL. ambda metric, which assumes a E-TPL. distribution the ESDs, achieves the second highest. We discuss the inverse.margin metric in Appendix D
find shape metrics, such as EXP @xP_lambda, E_TPL_lamdda,
In Figure we plot the rank correlations the generalization gap across our chosen metrics. While is encouraging that most existing generalization metrics yield correct predictions, as previously discussed, correct predictions of the generalization gap do not imply accurate predictions the best- pest-performing models here.
Details of the rank correlation calculations. When calculating the rank correlation with the test accuracy, we associate a negative sign to all the generalization metrics, i.c., a positive rank correlation in Figure 4a means that generalization metric negatively correlated with the BLEU score. We use this procedure to follow the conventional wisdom that smaller value of the complexity metric leads to better generalization. On the other hand, for Figure 4b a positive rank correlation means that metric is positively correlated with the generalization gap. Thus, for both Figure 4a and 4 positive correlation corresponds to the expected trend.
Rank-correlation results when varying hy perparameeter. We also assess whether the generaliza- tion metrics can predict trends in BLEU score when a single hyperparameter is changed (hang et al., 2019). Our findings shown in Appendix B Again, shape metrics have better rank correlations with model quality, while scale metrics are better correlated with the generalization gap.
Corroborating results. We extend our empirical evaluations to other datasets and evaluation methods. In Section E. we consider three other language processing tasks trained with different Transformers. In Section evaluate correlations using Kendall's instead of Spearman's 's rank correlation.
8

---

Under review as a conference paper at ICLR 2023
distance
Hidden dimmennsion size
tpl lambba
e. in disance
128
pl Lapph
Hidden dimmenssio 522
5.03
710 522 236
D207
eap lnrelds
3.07
alphe, secigbbed
log. altt nom
8011
BOTT
2001
00
Suece
2.06
nom
Eem
Bxm
Spcermas
cordotiinn Propertinn medecb concelly Soleched
Number of layers
12
Number of layers
12
(a) Model selection Huggingface Transform- ens. Metrics on the lefl and right are aligned.
(b) B_TFL,kG. di stance evaluated on (c) E.TPI_lambda evaluated on BERT
BERT models different size.
models different size.
Figure 5: Generalization metrics evaluated on pretrained Transformers. (a) Model selection results on eight Huggingface Transformer model series: BERT, GPT2, ALBERTv1, ALBERTv2, T5, DialoGPT, FlauBERT, Funnel Transformer. Left shows the rank correlation averaged over different Transformers. Right shows the proportion of the best Transformers correctly selected using different metrics. Shape metrics outperform scale metric only except stable_rand which is strongly influenced by the matrix size. (b and c) Evaluating two metrics on the "Smaller BERT" series. While E_TPL_ks _di st stance predicts the correct trends, E_TPL_ lambaa shows the reversed trends with depth.
3.3 EVALUATING THE GENERALIZATION METRICS USING HUGGINGFACE TRANSFORMERS
Finally, we evaluate the data-free generalization metrics on pretrained Transformers. Eight series of models downloaded from Huggingface (Wol' et al., 2020) are considered see Table for details. We also include 24 BERT models from the "Smaller BERT" series @Ture et al. 2019) produced from a "pretrained distillation" " pipeline that combines masked language modeling pretraining (Kenton & Toutanova 2019) and knowledge distillation from a single BERT teacher model. In total, there are 51 pretrained Transformers.
Model series
Models
(Kenoon Toutanoval
2019
Tiny,
Small, Medium, BERT LaRg-
Smaller BERT (ure of 2019
24 smaller BERT models (English only, uncased, trained with WordPiece masking)
GPT2 Radford et (019)
GPT2, GPT2-medium. GPT2-large, GPT2-xl
ALBERIVI
2019
ALBERT-base-i. ALBERT- large-y.. ALBERT-xtageer ALdggT-x
ALBERTv2 Lah ct al 2019
ALBERT-basc-te ALBERT-large--2 ALBERT-xlargo-2. ALBERT-xxlarg--2
(Ratte)
T5-small, 15-base. T'--large
DialoGiPT (Zlang et mi 4020
DinlaGPT-small,, DialoGPT m-mdiumm DialoGPT-lange
FlauBERT [Le
AUA)
FlauBERT small cased, FlauBERT cased, FlauBERT large cased
Fannel Transformer (Dar et 2020
FunnelModel-amall, Funnel ModellModel-mediig
FumelModel-large FunnelMedelxxtarg
Table 2: Pretrained Transformers considered in this paper.
We report rank cank-correlations averaged over these 8 model series in Figure Sa (left subplot), i.e., larger/deeper models should have smaller generalization metric values. Again, find that the shape metrics outperform scale metrics (except for stable. bank,, which is strongly influenced by the size of the weight matrix). The hybrid models achieve performance in-between the shape and scale metrics. In Figure 5a (right subplot), we compare different metrics in their ability to select the best model. That is, we report for each metric the proportion that the best model is selected from one model series when this metric is used as the model selection criterion. Note that the rankings of metries on the two subplots in Figure 5a are the same.
From Figure 5a we can see that, while the shape metrics perform better than scale metrics, none show a particularly strong rank correlation. To understand this, we examine the "Smaller BERT" series (Ture et al. 2019), which contains a more fine-grained structure of different model sizes. Specifically, these models are arranged in 4-by-6 grid, where represents (2,4,6,8,10, 12) trans- former layers and 4 means different hidden embedding sizes { [128,256.51] 12,6651..768 From Figure 5b see that the E_TPL_ks_de sic.ssiiissss ance correctly predicts the trend that wider and deeper models perform better. On the other hand, from Figure 5c lambdaa correctly predicts that wider models are better, but incorrectly predicts that shallower models are better (yet another form of Simpson's 's paradox in data set of neural network model quality; see also Martin & Mahoney (2021a)).
Another curious observation from Figure 5a is that, for the pretrained transformers, PL metrics, such PT. a 1 pha and PI, ka di stance, outperform E-TPI. metrics, such as TPL lambda,

---

Under review as a conference paper at ICLR 2023
E.TPL.beta and www......isstacee.. This phenomenon may seem surprising expect E-TPL fits to be more flexible than PL fits. These pretrained models are likely trained with much larger datasets and over many more epochs than the models we have otherwise considered. Here, PI.s appear to provide a more natural fit. This is further evidence that HT-SR theory particularly well-suited for evaluating the quality of relatively high-quality models.
may
4 CONCLUSION
After conducting large-scale empirical studies on a variety of metrics, we find that shape metrics derived from HT-SR theory are more effective for model selection and model evaluation, particular for evaluating models even without access to training or testing data. Poor correlations between existing generalization metrics and test-time performance have been reported in prior work (Dziugaite al., 2020 Jiang et al. 2019 Nagarajan & Kolter 2019). Rather than providing a "lump sum" to rank existing and novel generalization metrics (Figure 4. we evaluated these metries in several ways: quantifying correlations only on optimally-traines models (Figure 2 examining the time-wist correlation during training (Figure 5) differentiating between the correlation with test accuracy versus generalization gap (Figure B. evaluating these metries pretrained Transformer models where do not have any control over the training process (Figure 5): and thoroughly investigating the rich correlational structures when different hyperparameters
to M in the appendix). By thorough empirical investigations, we show that shape metrics perform consistently better than scale metrics in model selection-they correlate primarily with test accuracy instead of generalization gap, and they display better correlations with models' test performance. These metrics from HT-SR theory provide value to practitioners, allowing one to assess pretrained NLP models without training or testing data, even when their corresponding training loss is not small. Also, there are many large linear layers for Transformer models typically used in modern NI.P tasks, which allows for greater accuracy in the PI. estimators. That being said, further exploration of the Transformer architecture can lead to improved metrics on NLP models, complementing existing metrics designed explicitly for convolutional layers (Long & Sedghi 2019: Sedghi et al. 2018). We expect our current and future studies to be relevant and useful for improving existing generalization metrics in NLP moving forward.
varied (Figures 6
REFERENCES
Jeff Alstott, Ed Bullmore, and Dietmar Plenz. Powerlaw: python package for analysis of heavy- tailed distributions. PloS one, 9()):85777 2014.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948, 2020.
Peter Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normmalzzeed margin bounds for neural networks. Advances in Neural Information Processing Systems, 30:6241-6250 2017.
Jason Baumgartner. Reddit comments dumps. https://filess pushshif.... .tciffftticie. itw.abshhitt.iirddddi....... reddit/
comments/ Accessed: 2022-11-9.
Paul T Boggs and Janet E Rogers. Orthogonal distance regression. Contemporary Mathematics, 1 112:
183-194, 1990.
Ondrej Bojar, Christian Buck. Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop statistical machine translation. In Proceedings of the ninth workshop statistical machine translation, pp. 12-58, 2014.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations, 2020.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661-703 2009.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel- transforrass.ommer: Filtering sequential redundancy for efficient language processing. Advances in neural information processing systems, 33:4271-4282. 2020.
10

---

Under review as a conference paper at ICLR 2023
Zhen Dong. Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. HAWQ: Hes- sian aware quantization of neural networks with mixed -recissne In IEEE/CVF International Conference Computer Vision, pp. 293-302, 2019.
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, loannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. Advances in Neural Information Processing Systems, 33, 2020.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500, 2018.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in Neural Information Processing Systems, 31:842-852, 2018.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminnraaanann Deep ensembles: A loss landscape perspec- tive. Technical Report Preprint: arXiv:1912.02757 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wil- son. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Conference on Neural Information Processing Systems, pp. 8803-881 8812, 2018.
P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, pp. 876-885, 2018.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2018
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2019.
Yiding Jiang. Pierre Foret, Scott Yak, Daniel M Roy, Hossein Mobahi, Gintare Karolina Dziu- gaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur. Neurips 2020 competition: Predicting generalization in deep learning. arXiv preprint arXiv:2012.07976 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4188 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2019.
Hang Le, Loic Vial, Jibril Frej. Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. FlauBERT: Unsupervised language model pre-tnai for French. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 2479-2499 2020.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004. 08249, 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. Technical Report Preprint: arXiv:1907.1 1166 2019.
Philip M Long and Hanie Sedghi. Size-free generalization bounds for convolutional neural networks. Technical report, 2019.
11

---

Under review as a conference paper at ICLR 2023
Charles H. Martin and Michael W Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.09553, 2017.
Charles H Martin and Michael W Mahoney. Traditional and heavy tailed self regularization in neural network models. International Conference on Machine Learning, Pp. 4284 4293, 2019.
Charles H Martin and Michael W Mahoney. Post-mortem on deep learning contest: Simpson's ' paradox and the complementary roles of scale metrics versus shape metrics. Technical Report Preprint: net.::1000007774 202
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73,2D0 2021b.
Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):1-13, 2021.
David A McAllester. PAC-Bayesian model averaging. In Anmual Conference on Computational Learning Theory, pp. 164-170, 1999.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. Technical Report Preprint: 1EXiv:16090078. 2016.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. International Conference on Learning Representations, 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401. PMLR, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring general-
ization in deep learning. Advances in Neural Information Processing Systems, 30:5947-5956,
2017.
Behnam Neyshabur, Srinadh Bhojanapalli,, and Nathan Srebro. A PAC-Bayesian approach to spectrally-normalzzed margin bounds for neural networks. In International Conference Learning Representations, 2018
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 3-1--18, 2002.
Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst. Pac-bayesian margin bounds for convolutional neural networks. Technical Report Preprint: 1.Xi::1001.00111,, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, (1(14)):1 2020.
Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. In International Conference on Learning Representations, 2018
Dinghnn Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. simple but tough-to-
beat data augmentation approach for natural language understanding and generation. Technical
Report Preprint: arXiv:2009. 13818, 2020.
Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. Technical Report Preprint: 1xXiv11688 668.006600 2016.
Iulia Ture, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance pre-training compact models. Technical Report Preprint: arXiv: 1008.1190..... 08962, 2019.
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Colin Wei and Tengyu Ma. Improved sample complexities for deep neural networks and robust classification via an all-layer margin. International Conference on Leurning Representations, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, Volume (Long Papers), pp. 1112-1122, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing: System Demonstrations, pp. 38-45, Association for Computational Linguistics, October 2020.
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang. Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020.
Yaoqing Yang, Rajiv Khanna, Yaodong Yu. Amir Gholami, Kurt Keutzer, Joseph Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Boundary thickness and robustness in learning models. Advances in Neural Information Processing Systems, 33, 2020.
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 270-278, 2020.
13


Reviews:
[{"review_id": "m0OYStysLPD", "reviewer": "Reviewer_j1Vx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provide the first systematic empirical study on various generalization metrics in NLP, including 400 transformers trained with varying hyperparameters. The authors measure the correlation between 28 generalization metrics and the model quality. The authors find that shape metrics consistently perform better than scale metrics for predicting model quality.\n\n", "summary_of_the_review": "Overall, I think the authors over-claimed too much on the contribution. I would suggest the authors either add more NLP tasks in the experiments or change the title. I appreciate the large-scale experiments on machine translation tasks, while the metric mainly comes from CV and the novelty is little bit limited.", "strength_and_weaknesses": "Strength:\n1. The authors have done widely exploration of generalization metrics on machine translation tasks.\n2. The metric can be used for machine translation model evaluation. \n\nWeakness:\n1. The paper over-claimed the contribution. The authors only work on machine translation task but claim \"Evaluating natural language processing models \" in the title. I would suggest authors work on GLUE/SuperGLUE/AdvGLUE dataset to further explore the robustness.\n2. Not sure why the authors want to emphasize \"do not need access to any training or testing data\". It is ambiguous. Some scores like BERTScore (Evaluating Text Generation with BERT), BARTScore (BARTScore: Evaluating Generated Text as Text Generation) also don't need. Can the metric evaluate model only based on the model initialization?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XlwByJ0C6ff", "reviewer": "Reviewer_oVEt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a metric to measure generalization ability for transformer models that is not data-dependent. Their proposed metric is based entirely on a spectral analysis of the weight matrices, and the authors seek to show that this metric correlates directly with test accuracy (as a proxy for model generalization) rather than the 'generalization gap' considered by other works. They demonstrate the effectiveness of their generalization metric on transformers trained with a wide variety of hyperparameter settings on WMT '14, as well as analyzing its predictions on a number of pretrained models from the literature such as BERT, GPT-2 and T5.", "summary_of_the_review": "Overall I think the paper makes a valuable contribution in thoroughly cataloguing generalization metrics for NLP models and demonstrating the effectiveness of HT-SR methods at this task in a data-agnostic way. If these results had been demonstrated on a broader array of natural language tasks and datasets I would have no qualms about recommending it be accepted. As is, I believe this is a significant enough limitation for a work whose primary novelty is in the scope of its empirical studies that I view it as more of a borderline paper.", "strength_and_weaknesses": "**Strengths**\n\nThe idea of producing a comprehensive study of generalization metrics for NLP models is an interesting and valuable contribution, and the proposed metric does indeed perform well in the thorough empirical studies they performed. The authors tested a wide variety of hyperparameter settings and convincingly showed strong performance on predicting generalization ability. The analysis of existing pretrained models also led to some interesting results, and considered a wide variety of models from the literature with different variants.\n\t\n**Weaknesses**\n\nMy primary criticism of this paper is that almost all this analysis was performed only on a single dataset:  the WMT '14 english to german translation dataset. As such, it is difficult to be entirely confident that the results shown in the paper are truly general, and not specific to either this dataset or machine translation models on the whole. An analysis of multiple datasets with diverse task settings and dataset sizes that could show consistent results would be significantly more convincing. I certainly understand that this must already have been an enormous computational undertaking, but surely some of the resources used in the broad hyperparameter search might be better spent on replicating these results with other datasets. This is particularly critical given that this paper's primary contribution seems to be the broad scope of empirical analysis, rather than technical novelty.\n\nMy one other criticism is that I believe it is extremely important that papers relying on large-scale experiments accurately report the computational cost of their experiments and particularly the approximate greenhouse gas emissions of said experiments (there are a number of calculators online for estimating this). It is important for us to be cognizant of the environmental implications of our research and I believe transparency about the environmental cost of our experiments is ethically significant.\n\nQuestions/Minor errors:\n1) I don’t believe ESD is defined anywhere in the introduction of the paper outside of the abstract\n2) Figure 2 is a little bit unclear. You state that \"the shape metrics always correctly predict the model quality, i.e., the BLEU scores should be higher when the metric values are smaller\". Is this true only for the black starred models (the optimal configurations)? The wording makes it seem like this holds more generally, but this is not visually obvious and is contradicted by the caption.\n3) In Figure 4 the 'scale' metrics seem near-perfectly anticorrelated with model quality. Shouldn't that mean these metrics are actually *more* predictive of model quality than the shape metrics, as a perfect negative indicator should be just as valuable as a perfect positive one?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2dq0Y8Bssf", "reviewer": "Reviewer_rJjc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically measures correlations between various generalization metrics and model quality (BLEU score) on a machine translation dataset across a range of hyperparameters. Compare to prior work, the main differences are:\n- Focusing on NLP, rather than computer vision\n- Focusing on metrics that predict test error rather than generalization error\n- Focusing on a class of metrics based on whether the empirical spectral density (ESD) of the weight matrix follows a power law.", "summary_of_the_review": "My recommendation is to reject this paper. The paper addresses an interesting problem, evaluating generalization metrics for NLP, but it is difficult to interpret the results due to the secondary focus of the paper on advocating the use of a certain class of metrics, and I find that some of the analysis does not seem to support the main claims of the paper. I think the paper would be stronger if it placed less emphasis on shape metrics and focused more on evaluating generalization metrics for NLP, using a similar evaluation methodology to prior work.", "strength_and_weaknesses": "**Strengths**\n- This paper extends the study of generalization metrics to NLP. This is a valuable contribution, because unlike in computer vision, NLP models are not typically trained to zero training error. As the authors note, even if a generalization metric is rank-correlated with the generalization gap, it may not be correlated with the test error due to differences in training error. There are also differences in neural network architecture (transformers vs. CNNs).\n- The paper considers a fairly wide range of metrics and hyperparameters.\n\n**Weaknesses**\n- There are two main components in the paper (1) Evaluating natural language processing models with generalization metrics, and (2) advocating the use of a certain class of generalization metrics based on heavy-tail self-regularization theory (HT-SR), including several new metrics based on fitting exponentially truncated power laws to the ESD of the weight matrix. In my opinion, the second point is largely orthogonal to the first, and it's difficult to assess whether these HT-SR metrics are preferable to existing metrics due to methodological differences with prior work [1, 2]. I think the paper would be stronger if it placed less emphasis on shape metrics and focused more on evaluating generalization metrics for NLP, using a similar evaluation methodology to prior work.\n- Interpretation of Figure 4 (comparing generalization metrics): Figure 4 indicates that scale-based metrics are positively correlated with generalization gap but have a strong negative correlation with model quality, meaning a lower value of the generalization metric is associated with higher value of model quality. Isn’t this to be expected? We would expect that a low generalization gap is correlated with a high quality score. From this point of view, the metrics at the bottom of Figure 4a seem better. The other evaluations in the main paper only compare shape metrics, or compare correlations with hyperparameters rather than quality (see next point), so overall I am not convinced that shape metrics are preferable to other metrics.\n- The evaluation setting in 3.3 is puzzling to me. From what I can understand, this section does not measure the correlation between generalization metrics and model quality, but instead measures correlation with architecture parameters like depth and hidden dimension. It's not clear how this section supports the argument of the paper. This section might be more convincing if it compared models that were trained on the same data and then evaluated on a consistent quality metric.\n- Justification for HT-SR metrics: In particular, is there any theoretical justification for these metrics? The authors propose several different shape metrics (based on PLs and E-TPLs, different parameters, and goodness-of-fit metrics). Which of these should be preferred?\n- Methodological differences from prior work: Prior work [1, 2] has explored methodological questions about comparing generalization measures and proposed metrics and methodologies aimed at controlling for noise and unwanted correlations---in particular using Kendall's tau correlation and controlling for Monte Carlo noise across environments. This paper reports Spearman's rank correlation and does not explicitly address the questions raised by [1, 2] about how to robustly evaluate generalization metrics.\n- \"Generalization metrics that do not need access to any training or testing data\": The authors argue that shape metrics are preferable \"without access to training or testing data\". It would be valuable to be more clear about how existing metrics depend on the training data: to my understanding, metrics based on generalization bounds depend only the number of samples in the training data, which is a relatively mild requirement (you need to know the size of the data, but don't need access to the data itself). Also, none of the other generalization metrics considered use the testing data, so that should not be considered a special property of shape metrics.\n- The authors note that \"NLP pretraining datasets are typically web-scale and are challenging to access\", but their evaluation is focused on a machine translation dataset (where the data is available), so the argument about NLP pretraining is not entirely relevant.\n- Correlation evaluated on optimally trained models: It's not clear to me that this is a reasonable evaluation setting. How are you supposed to determine which model is optimally trained without access to testing data? If you can determine whether a model is optimally trained, why do you need a generalization metric?\n\n- The paper only considers one dataset and quality metric (BLEU score). It would be valuable to consider a wider range of commonly used NLP benchmarks, such as the GLUE benchmark.\n- It's not entirely clear to me which weight matrices are being considered in these metrics, and how the scores of different weight matrices are aggregated. More generally, it would be valuable to discuss another difference from prior work, which is that the models considered here are based on the transformer architecture, while prior work has mainly studied CNNs (and most theoretical metrics have been developed for feed-forward networks, but see [3].)\n\n[1] Jiang et al., 2019. Fantastic generalization measures and where to find them.\n[2] Dziugaite et al., 2020. In search of robust measures of generalization.\n[3] Long and Sedghi, 2019. Size-free generalization bounds for convolutional neural networks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "01NOQIjMmYu", "reviewer": "Reviewer_zKbc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a very comprehensive empirical analysis of predicting the test set performance on NLP tasks, and reveals several interesting properties like shape based metrics are generally better than scale based ones, and some fitting strategies (for empirical spectral densities of a weight matric) like E-TPL could alleviate the issues of other typical fitting methods like pure PL based. ", "summary_of_the_review": "As stated above, despite that lacking of technical novelty, this paper is generally good given its solid empirical study which could be of good/common interest to NLP researchers. ", "strength_and_weaknesses": "**Strengths**\n\n1. The problem at hand is indeed meaningful: getting the full-set dev set performance for real NLP tasks, especially that web-scale ones, are very time and resource consuming, hence having a data-free surrogate metrics for model selection would be quite useful, especially for real industry use cases;\n\n2. The study here is quite solid which delivers convincing insights for the community. \n\n3. The paper is generally written in a clear way.\n\n**Weaknesses**\n\n1. The raw innovation of this paper seems limited given its natural of being an empirical study focused work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject