Towards Inferential Reproducibility of Machine Learning Research
Authors: Michael Hagmann
Philipp Meier
Stefan Riezler
Venue: ICLR 2023 poster (2023)
Keywords: reproducibility
variance component analysis
reliability
significance test

Abstract: Reliability of machine learning evaluation --- the consistency of observed evaluation scores across replicated model training runs --- is affected by several sources of nondeterminism which can be regarded as measurement noise. Current tendencies to remove noise in order to enforce reproducibility of research results neglect inherent nondeterminism at the implementation level and disregard crucial interaction effects between algorithmic noise factors and data properties. This limits the scope of conclusions that can be drawn from such experiments. Instead of removing noise, we propose to incorporate several sources of variance, including their interaction with data properties, into an analysis of significance and reliability of machine learning evaluation, with the aim to draw inferences beyond particular instances of trained models. We show how to use linear mixed effects models (LMEMs) to analyze performance evaluation scores, and to conduct statistical inference with a generalized likelihood ratio test (GLRT). This allows us to incorporate arbitrary sources of noise like meta-parameter variations into statistical significance testing, and to assess performance differences conditional on data properties. Furthermore, a variance component analysis (VCA) enables the analysis of the  contribution of noise sources to overall variance and the computation of a reliability coefficient by the ratio of substantial to total variance.

Published as a conference paper at ICLR 2023
TOWARDS INFERENTIAL REPRODUCIBILITY OF MACHINE LEARNING RESEARCH
Michael Hagmann', Philipp Meier Stefan Riezler1.2 Computational I Linguistics* & IWR? Heidelberg University, Germany
{hagmann, meier, riezler]ecl uni amii-m heide lberg.
ABSTRACT
Reliability of machine learning evaluation the consistency of observed evalua- tion scores across replicated model training runs - affected by several sources of nondeterminism which can be regarded as measurement noise. Current tenden- cies to remove noise in order to enforce reproducibility of research results neglect inherent nondeterminism the implementation level and disregard crucial inter- action effects between algorithmic noise factors and data properties. This limits the scope of conclusions that can be drawn from such experiments. Instead of removing noise, we propose to incorporate several sources of variance, including their interaction with data properties, into analysis of significance and relia- bility of machine learning evaluation, with the aim to draw inferences beyond particular instances of trained models. We show how use linear mixed effects models (LMEMs)) to analyze performance evaluation scores, and to conduct sta- tistical inference with a generalized likelihood ratio test (GLRT). This allows us incorporate arbitrary sources of noise like meta- petaa-arrmeter variations into statisti- cal significance testing, and to assess performance differences conditional on data properties. Furthermore, variance component analysis (VCA) enables the anal- ysis of the contribution noise sources to overall variance and the computation of reliability coefficient by the ratio of substantial to total variance.
INTRODUCTION
Training of deep learning models utilizes randomness to improve generalization and training effi- ciency, thus causing inherent nondeterminism that hampers the reliability of machine learning evaluation - the consistency of the measurement of evaluation scores across replicated training runs. [Gnndersen et al. (0222) list several sources of nondeterminism. e.g., implementation-level nondeterminism such as random ordering in floating-point accumulation in parallel GPU threads (Pham et al. 202T). 1 algorithmic factors such as variations in meta- pettaparameters and model architec- ture (Lucic et al. 2018 Henderson et al.) 2018) D Amour et al. 2020). or data-level factors such as variations in pre-processo precpsocessing and evaluation metrics (Post 2018
Chen al., 2022) or varying char- acteristics of data in different splits (Gorman & Bedrick 2019 Søgaard et al. 202Th. Zhuang et al.
(2022) show that implementation-lece in evaluation scores even for training runs on identical data, algorithmic settings and infrastructure. Furthermore, they point out strong effects of certain types of algorithm-level nondeterminism on certain subsets of the data.
nondeterminism is partly irreducible, leading to variability
Regarding the comparison of machine learning models, minor variations in these sources of non- determinism have huge impact the resulting evaluation scores and sometimes even reverse the relation between optimal results for baseline and state-of-the-art (SOTA) model (Reimers & Gurevych] 2017 Melis et al. 2018). This fact questions what be validly learned from typ- ical machine learning experiment. One current answer is to foster training reproducibility]] in the sense of an exact duplication of state-of-the-art (SOTA) training result under exactly the same conditions. In this view, all sources of nondeterminism are regarded as noise or nuisance factors
'The term was coined by Leventi- PPetti & Östreich 2022) and corresponds to Drummond 2009 's replica-
bility.

---

Published as a conference paper at ICLR 2023
(Forde & Paganini 2019) that are independent of the learning signal and need to be removed (or at least reduced, even if incurring a cost in efficiency (Ahn et al. 2022)). This goal is pursued by enforcing open-souree program code, publicly available data, and explicit descriptions of experi- mental settings, following reproducibility checklists (Heil et al.] 2021] Pineau et al.] 2021; Lucic et al. 2022). An unintended side effect of this approach is that the conclusions that can be drawn from such experiments are restricted to statements about a single training configuration on a single test set.
Another viewpoint is to embrace certain types of nondeterminism as inherent and irreducible con- ditions of measurement that contribute variance in performance evaluation in an interesting way. Instead of attempting to remove them, we propose to analyze the various components of measure- ment noise, and especially their interactions with certain properties of data. Such a study can be seen to fall under the umbrella of inferential reproducibilityy Goodman et al. (2016) define it to refer to the drawing of qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study. For the case of machine learning evaluation, we focus on algorithmic-level factors such as variations in meta-parameters and model architecture and on data-level factors the main sources of non-determinnism in training replications. These usually described independently of each other. Our goal is to answer the question whether a competitor model yields improvements over baseline across different meta-parameter settings and across dif- ferent characteristics of input data, and how variations in algorithmic settings interact with varying data characteristics.
The main contribution of our paper is to show how to apply well-known statistical methods to an- alyze machine learning evaluations under variability in meta-parameter settings and dependent on data characteristics, with a special focus on the detection of sources of variance and their interac- tion with data properties. These methods are based on linear mixed effects models (LMEMs) fitted to performance evaluation scores of machine learning algorithms. First, we conduct a generalized likelihood ratio test (GLRT) to assess statistical significance of performance differences between algorithms, while simultaneously acknowledging for variation in nondeterministic factors. While applicable to any source of nondeterminism. in this paper, we focus on meta-paramees variations as main source of variability. A key feature of our approach is the possibility to assess the significance performance differences under meta- parameter variation conditional on data properties. Second, we show how to variance component analysis (VCA) to facilitate nuanced quantitative assessment of the sources of variation in performance estimates. Lastly, we compute a reliability coefficient to assess the general robustness of the model by the ratio of substantial variance out of total variance. Reliability is also intimately related to the power of the significance test.
Code (R and Python) for the toolkit and sample applications are publicly available
2 LINEAR MIXED EFFECTS MODELS
A linear mixed effects model (LMEM) is extension of standard linear model that allows rich linear structure in the random component of the model, where effects other than those that can be observed exhaustively (so-called fixed effects) treated as a random samples from larger population of normally distributed random variables (so-called random effects).
The general form of LMEM is
Y = X,3 + Zb + E.
(I)
where X an (N k) matrix and Z an (N m) m))matrix, called model- or design-matricen which relate the unobserved vectors B and b to Y. B is k-vector of fixed effects and b is an m-dimennsioall random vector called the random effects vector. is an N-dimensional vector called the enror component. The random vectors are assumed to have the following distributions:
b A (0 vo), E Ao)
"This term corresponds to Drummond 2009 's reproducibility and was coined by Goodman al. 2016
Instead of contributing further the terminological confusion in this area, we refer the brief history of this discussion in Plesser (2018).
fhtpps:7 www. cl uni he de berg. de/statn pgt.pp/ wwtppp/epppriiie ca 1_mel___t___ methods_
tutorial/
2

---

Published as a conference paper at ICLR 2023
where the and Ag covariance matrices parameterized by the vector 8.
The most common application of LMEMs is to model complex covariance structures in the data when the usual i.id. assumptions fail to be applicable. This is the case for repeated grouped, and thus non-independent, measurements such as multiple ratings of same items by same subjects in psycho-linguistic experiments. LMEMs have become popular in this area due to their flexibil- ity (Baayen et al., 2008 Bates et al. 2015), and have even been credited as candidates to replace ANOVA (Barr al. 2013) to analyze experimental data. The price for this flexibility is an elabo- rate estimation methodology for which we refer the reader to Appendix A2 of Riezler & Hagmann
(2022) and to further literature (Pinheiro & Bates, 2000 McCulloch & Searle 2001]
West et al.
2007
Demidenko 2013:
Wood 2017).
3 GENERALIZED LIKELIHOOD RATIO TESTS w/ AND w/O MEASUREMENT VARIATION
Let us assume our goal is to test the statistical significance of an observed performance difference between a baseline and a SOTA system. Furthermore, let us assume we are comparing Natural Language Processing (NLP) models on benchmark test set of gold standard sentences. In order to conduct a generalized likelihood ratio test (GLRT) for this purpose, we need to fit two LMEMs on the performance evaluation data of bascline and SOTA system which analyze the data differently, and compare their likelihood ratio. Let us further assume an experimental design where variants of the baseline and SOTA models, corresponding to different meta- meta-parameter configurations during training, are evaluated on the benchmark data. Simple linear models are a suboptimal choice to analyze this experiment since they are based on the assumption that each system was evaluated once on disjoint set of sentences. This would force to average over variants, thereby losing useful information contained in the clusters of repeated measurements of the same test input.
LMEMs allow to better reflect this design and to leverage its statistical benefits by adding random effect 6g for each sentence in our evaluation model. Such a model decomposes the total variance of the evaluation score into three blocks: systematic variance due to the fixed effects of the model, variance due to sentence heterogeneity, and unexplained residual variance. This allows us reduce the as of yet unaccounted residual variance by attributing a variance component of to variance between sentences. If we think of the residual error as noise that masks the signal of measured performance scores, we can effectively perform a noise reduction that increases the power of our tests to detect significant differences.
straightforward technique to implement statistical significance tests using LMEMs the so-called nested models setup (Pinheiro & Bates 2000). First we train an LMEM that doesn't distinguish between systems. This restricted model
mo: = + b, + Eres
(3)
specifies common mean 3 for both systems fixed effect, and a sentence-specific deviation b. as random effect with variance of, and a residual error Eres with variance o² for the performance scores Y. It represents the null hypothesis that there is difference between systems. This model is compared to a more general model that allows different means for baseline and SOTA scores:
mi Y 3 + 8. I + bg + Cres
(4)
This model includes an indicator function L. to activate a fixed effect 3, that represents the deviation of the competing SOTA model from the baseline mean / when the data point was obtained by SOTA evaluation. The restricted model is a special case of this model (thus "nested" within the more general model) since it can be obtained by setting 3, to zero. Let 4n be the likelihood of the restricted model, and & be the likelihood of the more general model, the intuition of the likelihood ratio test is to reject the null hypothesis of no difference between systems if the statistic
to
(5)
l
yields values close zero.
The incorporation of a random sentence effect b, introduces a pairing of systems on the sentence level that corresponds to standard pairwise significance tests. However, clustering at the sentence
3

---

Published as a conference paper at ICLR 2023
level allows accounting for arbitrary kinds of uncertainty introduced by the random nature of the training process. This setup is thus not only suitable for pairwise comparisons of best baseline and best SOTA model in order to test training reproducibility, but it also allows incorporating broader variations induced by meta-parameter settings of baseline and SOTA systems, thus making it suitable to test inferential reproducibility.
A further distinctive advantage of GLRTs based on LMEMs is that this framework allows analyzing significance of system differences conditional on data properties. For example, we could extend models mo and my by fixed effect 3a modeling a test data property d like readability of an NLP input sequence, or rarity of the words in an input sequence, and by an interaction effect 3ed allowing to assess the expected system performance for different levels of d. The enhanced model
m² Y = 3 + Ba d (3e + Red d) Ic + b2 + Cres
(6)
would then be compared to a null hypothesis model of the form
my Y 3 Ba d d b, + Erea.
GLRTs belong to the oldest techniques in statistics, dating back to Neyman & Pearson 1933) ) Wilks (9338). For more information on extensions of GLRTs for multiple comparisons and on their as asymp- lotic statistics we refer the reader to Chapter 4 and Appendix A3 of Riezler & Hagmann (0022) and
to further literature (van der Vaart 1998
Pinheiro & Bates 2000 Pawitan 2001
Davison 2003
Larsen & Marx 2012).
4 VARIANCE COMPONENT ANALYSIS AND RELJABILITY COFFFICIENTS
The main goal of a reliability analysis in the context of reproducibility study is to quantify and analyze the sources of randomness and variability in performance evaluation, and to quantify the robustness of a model in a way that allows to draw conclusions beyond the concrete experiment. The first goal can be achieved by performing a variance component analysis (VCA). For example, let us assume we want to specify model for performance evaluation scores that besides a global mean ja specifies random effects to account for variations in the outcome Y specific to different sentences s and specific to different settings of a regularization parameter r. A tautological decomposition of the response variable into the following four components can be motivated by classical ANOVA theory (Searle et al., 1992, Brennan 2001):
== (He - (4) + (H, - p) + -Y -4, 14, μ))
(8)
The components of the observed score Y for particular regularization setting single sentence s are the grand mean of the observed evaluation score across all levels of regularization and sentences; the deviation V,- (14 (1) of the mean score μ, for sentence from the grand mean μa; the deviation U, (n, 104--10 (1) of the mean score f4, for a regularization setting r from the grand mean μe; and the residual error, reflecting the deviation of the observed score Y from what would be expected given the first three terms. Except for μu, each of the components of the observed score varies from one sentence another, from one regularization setting to another, and from one regularization-sentenee combination to another. Since these components are uncorrelated with each other, the total variance o² QY (1) can be decomposed into the following variance components:
d² - ()) = o² + o² + o
where of and of denote the variance due to sentences and regularization settings, and of denotes the residual variance component including the variance due to interaction of s and r.
Let 1/ U l denote a deviation from the mean for a face f whose contribution to variance we are interested in. Instead of estimating the corresponding variance components of by ANOVA expected mean square equations, we use LMEMs to model each v as component of the random
"In the psychometric approach Brennan 2001 conditions of measurement that contribute to variance in the measurement besides the objects of interest are called facets facets of measurement. Facets comprise what we called measurement noise above. In our running NLP example. the objects of interest in our measurement procedure are the sentences. They are the essential conditions of measurement. The only facet of measurement in this example the regularization settings, while the objects of interest are usually called facet.
4

---

Published as a conference paper at ICLR 2023
effects vector b in equation and model each corresponding variance component of an entry of the diagonal variance-covariance matrix the in equation 2
Besides greater flexibility in e LMEMs also allow analyzing the interaction of meta- parameters and data properties. This can be achieved, for example, by changing the random effect b, to a fixed effect By. and by adding a fixed effect Ba modeling test data characteristics, and an interaction effect 3ra modeling the interaction between data property d and meta meta-parameter
The final ingredient of a reliability analysis is the definition of a coefficient that relates variance components to each other, instead of inspecting them in isolation. The key concept is the so-called intra-class correlation coefficient (ICC), dating back to FisFehhr//// (925). A fundamental interpretation of the ICC is as measure of the proportion of variance that is attributable to substantial vari- ance, i.e., to variance between the objects of measurement. The name of the coefficient is derived from the goal of measuring how strongly objects in the same class are grouped together in mea- surement. Following [Brennan (2001), we can define a concrete reliability coefficient, denoted by pp for our application scenario. In our case, objects of interest are test sentences s, and substan- tial variance is variance o² between sentences. Assume facets f1, fe.... and selected interactions sf1, 8f2; J1J2. Then the reliability coefficient yp is computed by the ratio of substantial variance 0 to the total variance, i.e., to itself and the enror variance o that includes variance components for all random effects and selected interactions of random effects:
(10)
where o3 = =0 + + +.. o +... + f
oees. Based on this definition,
reliability of performance evaluation across replicated measurements is assessed as the ratio by which the amount of substantial variance outweighs the total error variance. That is, performance evaluation is deemed reliable if most of the variance is explained by variance between sentences and not by variance within sentence, such variance caused by random regularization settings or by residual variance due to unspecified facets of the measurement procedure. Naturally, different assumptions on thresholds on this ratio will lead to different assessments of reliability. A threshold of 80% is used, for example, Jiang (2018). Values less than 50%, between 50% and 75%, between 75% and 90%, and above 90%, are indicative of poor, moderate, good, and excellent reliability, respectively, according to Koo & Li (2016).
More information can be found to Chapter 3 and Appendix A2 Riezer & Hagmann 2022 and Shrout & Fleiss (1979); Searle et al. (1992); McGrra & Wong (966)):rrennn)) (2001); Webb et al. (2006).
VCA and ICCs date back to the works of Fisher 1925
A WORKED-THROUGH EXAMPLE
We exemplify the methods introduced above paperswithcode com open resource, namely the BART+R3F fine-tuning algorithm presented by Aghajanyan al. (202I) for the task of text summarization, evaluated on the CNN//ailyMaiil (Hermann et al. 2015) and RedditTIFU (Kim et al. 2019) datasets.
NLP example from the
BART+R3F was listed as SOTA for text summarization paperswithcode. com at the time of paper publication. It uses an approximate trust re- gion method constrain updates on embeddings ʃ and classifier g during fine-tuning in order not to forget the original pre-trained representations. This is done by minimizing a task loss L()) regularized by the Hallback-Leibler distance on normally or uniformly distributed parameters:
these datasets
[(0) + XK f=) f(r + 2)), where 2 ~ N(0, or u(-o,o))
(11)
The first question we want answer is that of training reproducibility is the result difference between baseline and new SOTA reproducible on the data and the code linked on the repository,
5 Among the many advantages of using LMEMs to estimate variance components is that the same model structure can be used for designs that are special cases of the fully crossed design, and the elegant handling of missing data situations. Sec Baayen et al. (2008) Barr et al.
(2033) Bates et al. (015 for further discussions on the advantages of LMEMs over model ANOVA estimators.
https: /tttps::/qqihuuii thub. ctm. www. seep//n-eeeeeeeeneeeeeeeeee 1yma ctttss///githhbb..c///aaoaoooooe
wttp:tttttw.c.../raaateeeeo /MMN
rxf
5

---

Published as a conference paper at ICLR 2023
Table 1: Text summarization results (Rouge-1/2/L)) for baseline (bl) (BART) and State-of-the-ar (SOTA) (BART+R3F) reported in Aghajanyan et al. (202T)
CNN/DailyMail
ReddiTTIFU
bl SOTA
29.1//2 44.38/21.53411177
24. 24.19/8.12/1 12/21.31
30.31/10.98/24.74
Table 2: Significance of result difference baseline-SOTA on CNN/DailyNews under Rouge-1 (R1),
Rouge-2 (R2) and Rouge-L (RL) evaluation.
bl
SOTA
p-value
effect size
R1
44.09
44.41
0.0001
-0.101
R2
21.13
21.44
< 0.0001
-0.080
RL
40.81
41.16
0.0001
-0.105
and under the meta- perampareeettr and preprocessing setup reported in the paper. As baseline we take pre-trained BART-large model (Lewis et al. 2020). The Rouge- (Lin & Hovy 2003) results for the text summarization task reported in Aghajanyan al. 2021 are shown in Table
Let us first look at the results on the CNN/DailyMail dataset. The paper gives detailed meta- parameter settings for the text summarization experiments, but reports final results maxima over training runs started from 10 unknown random sceds. Furthermore, the regularization parameter is specified as choice of X E [0.001, 0.01, 0.1], and the noise type choice from [, Using the given settings, we started the BART+R3F code from 5 new random seeds and the BART-large baseline from 18 random seeds on 4 Nvidia Tesla V100 GPUs each with 32 GB RAM and up- date frequency of 8. All models were trained for 20-30 epochs using loss-based stopping criterion. Searching over the given meta-parameter choices, obtained training reproducibility result given in Table 2 We find sigmificant improvements of the best SOTA model over the best baseline with respect to all Rouge-X metries (the difference baseline SOTA is negative). However, the effect sizes (standardized mean difference between evaluation scores) are small.
Let next inspect significance conditional on data properties. We We quantify properties of summariza- tion inputs by word rarity (Platanios et al.) 2019), i.e., the negative logarithm of the empirical proba- bilities of words in summary inputs, where higher values mean higher rarity. Furthermore, we caleu- late readability (Kincaid et al. 1975) of summary inputs by calculating the ratio of words/sentences and syllables/word. Readability scores are in principle unbounded, however, an interpretion scheme exists for the range from 0 (difficult) 100 (easy).
An analysis of significance conditional on data properties can be seen as first step of inferential reproducibility. The interaction plots given in Figure show significant difference in performance slope for Rouge-2 with respect to ease of readability, where the performance of the best SOTA system increases faster than that of the best baseline for easier inputs (left plot). Also, significant difference in Rouge-2 with respect to word rarity is seen where the best SOTA model is better than the best baseline for inputs with lower word rarity (right plot).
The next question of inferential reproducibility is whether the results given above are robust against meeta-parameter variations, and which metamparseeters are most important in order achieve the best result. We inspect the original grid of meta-parameter configurations of the SOTA model, given by crossing the given choices of parameaareeterss with each other, yielding 3 A x noise distributions x 5 random seeds = 30 configurations. As shown in Table 3] the relations between SOTA and baseline are turned around (the difference baseline SOTA is positive) showing significant wins of baseline over SOTA at medium effect size.
Since the performance variation of the baseline model over 18 random seeds was negligible (stan- dard deviations < 0.2% for Rouge-X scores), we conduct reliability analysis of the SOTA model in order to reveal the culprit for this performance loss. The variance component analysis in Table
https: http:r//dhheeeeooooeeeee/eeiiain

---

Published as a conference paper at ICLR 2023
Booel
Besefre
SXITA
50IA
521
ROUGE-2
ROUGE-2
0.2-
0.85-
0.12
Fiesch-Kincald Reading Esse
Word Rarly
Figure 1: Interaction of Rouge-2 of baseline (solid) and SOTA (dashed) with readability (left) and word rarity (right).
Table 3: Significance of baseline-SOTA on CNN/DailyNews under meta-parameter variation.
baseline
SOTA
p-value
effect size
R1
44.15
42.21
<
0.0001
0.390
R2
21.26
19.64
0.0001
0.301
RL
40.84
38.53
< 0.0001
0.531
Table 4: Variance component analysis for Rouge-1 (top), Rouge-2 (middle), and Rouge-L. (bottom)
estimates.
Variance component
Variance of
Percent
summary id lambda random_seed
0.00923
55.8
0.00254
15.0
0.00012
0.7
noise diissriistiittion
0.00005
0.3
residual
0.00464
27.1
summary _id
0.00992
62.7
lambda random_sed
0.00131
8.3
0.00008
0.5
noise distribution
0.00003
0.2
residual
0.00449
28.3
summary-id id lambda random_seed
0.00875
47.9
0.00519
28.4
0.00004
0.2
noise diissdddribbbttinn
0.00001
0.1
residual
0.00428
23.4
shows that the variance contributions due variation in random seeds choice of noise distri- bution are negligible. However, in all three cases the largest contribution to variance is due to the regularization parameter The percentage variance due to objects of interest, here summaries, readily be interpreted reliability coefficient ye, yiclding moderate reliability for performance evaluation under Rouge-1 and Rouge-2 (yp between 50% and 75%) and poor reliability for evaluation under Rouge-L (ip below 50%).
An inspection the interaction of data properties with the regularization parameter is given in Figure The interaction plots show significant difference in Rouge-2 performance of the SOTA model between regularization parameters, where performance for À = 0.1 is lower and decreases with increasing reading ease (top plot) and increasing word rarity (bottom plot).

---

Published as a conference paper at ICLR 2023
Lambde
Lambda
.01
025
001
0.20
ROUGE-2
ROUGE-2
0.20
0.18
0.18
56
100
200
Kiesch-Kincad
Word Rarry
Figure 2: Interaction of Rouge-2 of SOTA for different values of regularization parameter À with
readability (left) and word rarity (right).
Let inspect the results the ReddicTIFU dataset next. These data are interesting since they are much harder to read (mean readability score of -44899), however, reproducibility analysis on the RedditTIFU dataset was hampered by the fact that the train/dev/tsst split for RedditTIFU data (long version) was not given on paperswi hcode com. nor reported in the paper or the code. We used the spliç provided by ZZhong et al. (2020) instead. Under this data split, we found significant improvement of the best SOTA over the best baseline at a small effect size (-0.155) only for Rouge-2. If meta-pant parameter variation was taken into account, the effect size was even smaller (-0.0617). There were no significant interaction effects and neglible variance contributions from meta-parameters.
In sum, this small study allows nuanced assessment of the strengths and weaknesses of the BART+R3F model: Losing or winning new SOTA score strongly depends on finding the sweet spot of meta-parameter (here: A), while the paper's goal was explicitly reduce instability across meta-parameter settings. Performance improvements by fine-tuning are achieved mostly on easy-to- read and frequent-word inputs these comprise less than quarter of the CNN/Dailynews data. Furthermore, the optimal choice of main variance-introducing meta-parameter interacts strongly with the investigated data characteristics. Lastly, the model does seem be robust against vari- ations in data under a new random split on RedditTIFU the large gains reported for the split used the paper can no longer be achieved.
RELATED WORK
Our work must not be confused with approaches to automatic machine learning (AutoML.) Zimmer al.] (202T): Habelitz & Keuper (2020) or neural architecture search (NAS) (Zoph Lej 2017] Ebrahimi et al. 2017). While AutoML and NAS focus on efficient search of space of meta- parameter configurations for an optimal result on a particular dataset, our interest is analyzing the variance contributions of meta meta-parameters for given meta-parameter search experiment, and especially in the interactions of meta perea-err variations with characteristics of data, with the goal data. ANOVA-like techniques
of gaining insights about possible applications of model have been used to analyze meta-parameter importance in the context of AutoMI. (Hutter et al.) 2014), however, ignoring the crucial aspect of interactions of meta-parameter with data properties. Furthermore, or work is not restricted to meta peta-parameter variation as source of non-determinism, but can in principle be applied any source of randomness in machine learning training.
Discussions of reproducibility problems in research date back at least to Toannidis (2005), and for the area of machine learning least Hand (2006). Since then, multitude of papers has been published on various sources of interrccuciillity in various machine learning areas (see Gundersen al. 2022) for an overview), however, much less work has been invested concrete techniques to
zat
8

---

Published as a conference paper at ICLR 2023
quantify reproducibility. Recent works try to capture reproducibility of evaluations under replicated training runs by a single statistical measure (c.g., coefficient of variation in Belz et al. (2021) or
mean and confidence bounds in Lucie et al. (2018); Henderson et al. (2018). in difference to
goal of decomposing measurement noise into different components and analyzing the interaction of noise factors with data properties. Variance component analysis based on ANOVA techniques has been applied to information retrieval models (Ferro & Silvello, 2016: Voorhees et al., (017)) however, these approaches again ignore an incorporation of data variability into their analysis. We replace ANOVA methods by LMEMs for modeling and estimation (Wood) (017) and promote the ICC-based idea of quantifying reliability by the proportion of variance attributable to the objects of interest, which to our knowledge has not been applied to machine learning before.
Special-purpose significance tests have been proposed for particular evaluation metrics Dror et al. 2020), Chapter 3), for meta-parameter variations (Dror et al. 2019). and for multiple test data (Dror et al. 2017). One advantage of the proposed LMEM-based approach that it unifies these special-purpose techniques into single framework for hypothesis testing. Furthermore, extensions of bootstrap (Sellam et al.] 2022; Bouthillier et al.) 2021) or permutation (Clark et al., 201 1) tests have been proposed to incorporate meta- pera-par variation. The distinctive advantage of our ap- proach is that it enables analyzing significance of result differences conditional on data properties. These can be generic data properties like readability above, properties of combined datasets obtained from different sources like data splits, bootstrapped data, or different-domain data sets. The idea of treating test data as random effects and thus increasing the power of statistical significance testing has already proposed by Robertson & Kanoulas 2012). However, the general applicability of LMEMs and GLRTs as a unified framework for significance testing conditional on data properties and simultaneously incorporating arbitrary sources of variation has not yet been fully recognized in the wider community of machine learning research.
7 DISCUSSION
Widely recognized work by applied statisticians has proposed to abandon non-confirmatory statisti- cal significance testing, at least its role in screening of thresholds and as guarantor of reproducibility, but instead to report continuous p-values, along with other factors such as prior evidence (if avail- able) (Gelman & Loken, 2014; Colguhoun 2017 McShane et al. 2019). Our proposed use of GLRTs, VCA and ICCS aligns with these recommendations. Our focus is to use them as analysis tools of assess performance under different meta peramparrrettr settings, dependent on characteristics of data, and to detect important sources of variance and their interactions with data properties. This allows to address questions of genuine interest to researchers and users like "Will the SOTA algorithm's is stellar performance on the benchmark testdata lead to top performance on the kinds of datasets that my customers will bring?", more specifically "How will individual test example characteristics or particular meta- meta-parameter settings, and their interaction with data properties, affect performance?"
Like related work that analyzes model performance under given meta petamparr configurations (Dodge al. 2019; (ttrbbell et al., 2019), our work is limited by the lack of standardization in the notion of a mntaa-parameter search. This includes human factors regarding an unclear differenti- ation between meta-parameters and fixed design choices for particular models. Furthermore, formal criteria on how proper ranges across model families should be defined, are lacking. Thus our work should be seen as a contribution to interpretability of deep learning experiments, not the provision of a new decisive criterion to rank machine learning models with respect to reliability or p-value under variability of meta-pares neta-parameters and data. Nonetheless, our methods are readily applicable to performance evaluation data already obtained during meta- parameter optimization. They allow us to transform this usually unused data into new findings about algorithm behavior. We believe that they will be especially useful for large-scale experiments where manual inspection of variance due to interactions of large numbers of meta- parameerssssess and data properties is prohibitive.
ACKNOWLEDGEMENTS
This research has been conducted in project SCIDATOS (Scientific Computing for Improved De- tection and Therapy of Sepsis), funded by the Klaus Tschira Foundation, Germany (Grant number 00.0277.2015).

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The experiments reported in this paper are replications of published results and are not expected to raise any ethical concerns.
REPRODUCIBILITY STATEMENT
The data, code, and meta- petaaparameter settings for the experiments reported in this paper are docu- mented therein and are publicly available.
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations (ICLR), 2021. URL https: openreview net/ forum?ic
0Q08SN7 OMIV
Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I. Shamir. Repro- ducibility in optimization: Thcoretical framework and limits. In Proceedings of the 36th Confer- ence on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2022.
R.H. Baayen, D.J. Davidson, and D.M. Bates. Mixed-effects modeling with erossed random effects for subjects and items. Journal of Memory and Language, 59:390-412, 2008. doi: https://doi.
org/10.10166fj.mm mm0.
Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tilly. Random effects structure for confirmatory hypothesis testing: Keep maximal. Journal of Memory and Language, 68(3):
255-278, 2013. doi: https://doi.org/ 10.6006/j............ fttp:./m..2.22.... 201.1011..1.00 11.001.
Douglas Bates, Martin Mächler, Benjamin M. Bolker, and Steven C. Walker. Fitting linear mixed- effects models using lme4. Journal of Statistical Software, 67(1):1-48, 2015. doi: https://doi.or/ 10.18637/jss. v067.i01.
Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Online, 2021.
doi: http://dx.doi.org/
106655//1122221111 ma..-
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems (MLSys), 3, 2021. URL https: pttpe:///ooeeednngg mwsys....mmmmmmmmmmmmmmmmmmmm.mm www.oorg/pppeeeeee. 3tggpaapr/22021 file/cfecoe wtc.ecccee66eee355333eeee2222000
pdf
Robert L. Brennan. Generalizability theory. Springer, 2001. doi: http:///do.orr/110.0000 978-1-4757-3456-0.
Yanran Chen, Jonas Belouadi, and Steffen Eger. Reproducibility issues for bert-based evaluation metrics. CoRR, abs/2204.00004, 2022. doi: 1048550/AXXX2220..000
Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith. Better hypothesis testing for statisti- cal machine translation: Controlling for optimizer instability. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics (ACL), Portland, OR, 2011. URL httts:://ccaattae ort.ol////1110031330
David Colquhoun.
The reprodueibility of research and the misinterpretatton of p-values. Royal Society Open Science, 4(12), 2017. doi: 10. 10.1098/rsos. 1aess.771. 171085.
Alexander D' Amour, Katherine A. Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein. Matthew D. Hoffman, Farhad Hormoz- diari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi-An
10

---

Published as a conference paper at ICLR 2023
Ma, Cory Y. McLean, Diana Mineu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osbore,, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh. Victor Veitch, Max Vla- dymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. CoRR, abs/201 1.03395, 2020.
A. C. Davison. Statistical Models. Cambridge University Press, 2003. doi: http:://boi.org/10.10111 cbo978051 1815850.
Eugene Demidenko. Mixed Models: Theory and Applications with R. Wiley, 2013. doi: https:
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019. doi: http:/dx.doi.orgg
18653/v1/D19-1222
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. Replicability analysis for natural language processing: Testing significance with multiple datasets. In Transactions of the Association for Computational Linguistics (TACL), volume 5, pp. 471-486, 2017. doi: heapidxxdoo.oorgg.
http://dx. doi. wow..oog/....1111 1tt11116//acaa.0000000
Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance - how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019. doi: http://dx.doi.org// 186130.1./11///// 1266.
Rotem Dror, Lotem Peled, Segev Shlomov, and Roi Reichart. Statistical Significance Testing for Natural Language Processing. Morgan & Claypool, 2020. doi: https://doi.or//10.uuu 0090/ 978-3-031-02174-
Chris Drummond. Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, Montreal, Canada, 2009.
Sayna Ebrahimi, Anna Rohrbach, and Trevor Darrell. Gradient-free policy architecture search and adaptation. In Proceedings the Conference on Robot Learning (CoRL), Mountain View, CA, USA, 2017.
Nicola Ferro and Gianmaria Silvello. A general linear mixed models approach study system com- ponent effects. In Proceedings the the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy, 2016. doi: http://doo..r110.1114454.
2911530.
Ronald A. Fisher. Statistical Methods for Research Workers. Oliver and Boyd, 1925.
Jessica Zosa Forde and Michela Paganini. The scientific method in the science of machine learning. In Proceedingy of the ICLR 2019 Debugging Machine Learning Models Workshop, New Orleans, LA, USA, 2019.
Andrew Gelman and Eric Loken. The statistical crisis in science. American Scientist, 102(6):460- 465, 2014. doi: https://doi.org/0.. 151 1/2014.111.466.
Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. What does research reproducibility mean? Sci Transl Med, 8-341):1-6, 2016. doi: https://doi.org/10.. 12//ccitransmmeddaa5500
Kyle Gorman and Steven Bedrick. We need to talk about standard splits. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019. doi: http://dxx.do..org. wtt..ooo///....///1111111 0.18653/v1/P1..
Odd Erik Gundersen, Kevin Coakley, and Christine Kirkpatrick. Sources of irreproducibility in machine learning: A review. CoRR, abs/2204.07610, 2022. doi: 10.48550AARX1122244076671

---

Published as a conference paper at ICLR 2023
P. Habelitz and J. Keuper. PHS: A toolbox for parallel hyperparameter search.
CoRR,
1e//2002.11429, 2020.
doi: 10448550/ARXI.20022144.
David J. Hand. Classifier Technology and the Illusion of Progress. Statistical Science, 21(1):1 14, 2006. doi: https://doi.orr//000 http://ooo.ogg/111222/4448808000 1284400000044430000000
B.J. Heil, M.M. Hoffman, F. Markowetz, S. Lee, C.S. Greene, and S.C. Hicks. Reproducibility standards for machine learning in the life sciences. Nature Methods, 18:1122-1144, 2021. doi:
http:///oi.orr//II 10. 1038/ 5500004411-2--2-12255---6
Peter Henderson, Riashat Islam, Philip Bachmann, Joelle Pineau, Doina Precup. and David Meger. Deep reinforcement learning that matters. In Proceedings of the 32nd AAAI Conference on Arti- ficial Intelligence (AAAI), New Orleans, LA, USA, 2018. doi: http::/doii.ogge 1...bb./aa.//01..//aaaaiiai. v32il. 11694.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS), Mon- treal, Canada, 2015.
www.eeediiiinnueeeeeeeeeeeeeeeee neurips Pat.-oppppppppppp
Frank Hutter, Holger Hoss, and Kevin Leyton-Brown. An efficient approach for assessing hy- perparameter importance. In Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China, 2014. hRR/attps::/proccddlngg wtw..eedd..n.......rrssss/////
nutterl4.htm.
John P. A. loannidis. Why most published research findings are false. PLOS Medicine, 2(8), 2005.
doi: paa.37//oommall.pme...000 0020124.
Zhehan Jiang. Using the linear mixed-effect model framework to estimate generalizability variance
components in R. Methodology, 14(3):133-144 2018. doi: a000149.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technolo- gies (NAACL:HLT), Minneapolis, Minnesota, 2019. doi: http:/tpp//bb..ooooo//0......... 129-1260)..
J. P. Kincaid, R. P. Fishburn, R. L. Rogers, and B. S. Chissom. Derivation of new readability formulas for navy enlisted personnel. Technical report, Naval Air Station, Millington, TN, 1975.
Terry K. Koo and Mae Y. Li. A guideline of selecting and reporting intraclass correlations co- efficients for reliability research. Journal of Chiropratic Medicine, 15:155-163, 2016. doi: https://doi.or org/10. 10. jem. 2016.02.01..
Richard Larsen and Morris L. Marx. Mathematical Statistics and its Applications. Prentice Hall, 130.0001..3335001....6..6
fifth edition, 2012. doi: http::/ddo..org/100
A. M. Leventi-Peetz and T. Östreich. Deep learning reproducibility and explainable AI (XAD). CoRR, abs/2202. 1452, 2022. doi: 00d=(000005500AAAX2222221111111 14522
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-scquence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics (ACL), Online, 2020. doi: http://xx.de www.ooo...ooooooooooooo 11655//1/2220.ccclll..
Chin-Yeuu Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), Edmonton, Canada, 2003. URL http:///llaaaaoooo..o.ooooo
12

---

Published as a conference paper at ICLR 2023
Ana Lucic, Maurits Bleeker, Samarth Bhargav, Jessica Forde, Koustuv Sinha, Jesse Dodge, Sasha Luccioni, and Robert Stojnic. Towards reproducible machine learning research in natural lan- guage processing. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguisties: Tutorial Abstracts, Dublin, Ireland, 2022. doi: htpp//xx.dii.org (tgg/0.1163300 18.53v1//
2
Mario Lucie, Karol Kurach, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly- Are GANs created equal? A large-scale study. In Proceedings of the 32nd In- ternational Conference Neural Information Processing Systems (NIPS), Montréal, httpp/ccccpappee///////1111 #bberallccccssffeee0EEE00---pppp pdf
Canada, 2018. URL httpss:/pproceecinnss....r...s
Charles E. McCulloch and Shayle R. Searle. Generalized, Linear, and Mixed Models. Wiley, 2001.
doi: http::/doi.org/100 10. 1002/0471722073.
Kenneth O. McGraw and S. P. Wong. Forming inferences about some intraclass correlation coeffi- cients. Psychological Methods, 1(1):30-46, 1996. doi: https//dboorrr///..03//00999/999
Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L.. Tackett. Abandon statistical significance. The American Statistician, 73(6up1):235-245, 2019. doi:
https://doi.org/1000 http://0bb.oogg11100000000111110
Gabor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. Proceedings of the 6th Conference on Learning Representations (ICLR), Vancouver, BC, Canada, 2018 URL / openreview net forum? ? (tw.rrrummmmdiyyyyyyyyyye
J. Neyman and E. S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, 231:289 -37,, 1933, doi: https://oii.or/// 100. 1018///tt..933.0000.
Yudi Pawitan. In All Likelihood. Statistical Modelling and Inference Using Likelihood. Clarendon Press, 2001.
Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), Virtual, 2021. (145/3324884.341635
doi: https://doi.org//
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal of Machine Learning Research (JMLR), 22:1-20, 2021.
José C. Pinheiro and Douglas M. Bates. Mixed-Effects Models in S and S-PLUS. Springer, 2000.
doi: https://doi.org// 10. 1007/698882.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. Competence-bell based curriculum learning for neural machine translation. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Cachmologies(NAAA (NAACL:HLT)) Minneapolis, Minnesota, 2019. doi: 10. 10.18653/v1/119.111
Hans E. Plesser. Reproducibility replicability: A brief history of confused terminology. Fron- http://.....//...1.111...m
tiers in Neuroinformatics, 11(76): 2018. doi:
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the 3rd Conference Machine Translation (WMT), Brussels, Belgium, 2018. doi: http://dxxdoi..rr/1100 1.653/1v W18-6319.
Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Copenhagen, Denmark, 2017. doi: http:
//dx. trix.doi.org/10.11 1/171/111111111110005
13

---

Published as a conference paper at ICLR 2023
Stefan Riezler and Michael Hagmann. Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science. Springer, 2022. doi: http:://bl.or/110.0079973/ 10000/99788-00310118------
Stephen E. Robertson and Evangelos Kanoulas. On per-topie variance in IR evaluation. In Proceed- ings of the 35th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, Portland, OR, USA, 2012. doi: https://doi.org 11. 448114444442283233444444
Shayle R. Searle, George Casella, and Charles E. McCulloch. Variance Components. Wiley, 1992. doi: https://doi.or//00..u 10000007700077770011163333
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Ammur. Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. The multiBERTs: BERT reproductions for robustness analysis. In International Conference on Learning Representations (ICLR), 2022. URL https: httpsss//pppeeevveeee net/ forum?id=
KOE_ [0E_FOgFDGA)
Patrick E. Shrout and Joseph L.. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bullerin, 86(2):420-435 1979. doi: https://doi.org/10..00 1037000333229998860
Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. We need to talk about random splits. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Online, 2021. doi: wttp:/oo...oo/.....13//////// eacl-main. 156.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy, 2019. doi: http://dx.doi.org/ wttg//0..165.5 10..1853/v11PP 1/P19-135.
A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, 1998. doi: https://oi..rr// 10.1017/cbo9780511 1802256.
Ellen M. Voorhees, Daniel Samarov, and lan Soboroff. Using replicates in information retrieval evaluation. ACM Transactions on Information Systems, 36(2):1-31, 2017. doi: https://doi.org//00 1145/3086701.
Noreen M. Webb, Richard J. Shavelson, and Edward H. Haertel. Reliability coefficients and gen- eralizability theory. Handbook of Statistics, 26:81-214, 2006. doi: https://doi..or/1000 1016/
60169-7161106)220044-
Brady T. West, Kathleen B. Welch, and Andrzej T. Galecki. Linear Mixed Models: A Practical Guide Using Statistical Software. Chapman & Hall/CRC, 2007. doi: https://do..org/10 1201/ 1200/ 9781420010435.
Wilks. The large-sample distribution of the likelihood ratio for testing composite hypothe- ses. Annals of Mathematical Statistics, 19:60-92, 92, 1938. doi: https://do..or//1... 121....111//mm. 1177732360.
Simon N. Wood. Generalized Additive Models. An Introduction with R. Chapman & Hall/CRC, second edition, 2017. doi:
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Online, 2020. doi:
acl-main.552.
Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. In Proceedings of the 5th MLSys Confer- ence, Santa Clara, CA, USA, 2022. URL URLAttps:/proceedng d34
.ll.s.mls............... org /aper/
2022/ le 75 33..00eee333.c33533333e333eeee30
Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 (9):3079-3090, 2021. doi: 10.1 109/PPAM1..02..367776...
14

---

Published as a conference paper at ICLR 2023
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France, 2017. URL https: / www.review...eccceo.p. net/ Forummid-rr
15


Reviews:
[{"review_id": "yGQl6MnwJB", "reviewer": "Reviewer_V9qk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem experiments reproducibility with specific reference to inferential reproducibility which proposes to interpret the variation of performance values as due to the following factors; data characteristics, meta-parameter settings, includng also their interactions. The paper starts from the rationale that variance in performance values of different deep learning models and/or machine learning models in general is an intrinsic and interesting effect of non-determinism and a bug to be methodologically solved by researchers. A statistical model, i.e., linear mixed effects model, tailored to analyze performance data from paired experiments (models in this case) is developed to compare performances of state of the art models to that achieved by baseline models. The paper develops the argument of inferential reproucibility and the linear mixed effects model together with numerical experiments on natural language data that show how the proposed approach for inferential reproducibility can be effectively used to state whether the achieved performances by the baseline and the state of the art model are different or not.\n", "summary_of_the_review": "The paper is about a fundamental and urgent issue in scintific research, specially under the current setting where it seems that each new method is better and sometimes much better that existing baseline methods.\nThe paper also introduces and describes the use of linear mixed effect models to compare and futher analyze the perfoamnce achived by state of the art methods and that achieved by baseline methods. \nThe paper motivates the advantage of the proposed approach in theoretical terms and in a conving manner in my humble opinion.\nFurthermore, some numerical experiments are presented to support the statements made by the paper. However, I would have liked to see more numerical experiments to better explain the different aspects of the proposed approach.\n", "strength_and_weaknesses": "Strengths:\n- the problem addressed is a relevant one, I would say a fundamental problem for good scientific discussion in this times where almost each new pulished paper achieves better performances than well established existing baselines\n- the proposed model extends or let say replaces existing models aimed to the same goal\n- the proposed approach is elementary and can be easily interpreted and understood, it's assumptions and its' limitations\n- the paper is well organized and structured and it read easily and quite well\n\nWeaknesses:\n- numerical experiments are limited to only to two datasets, even if this is a weackness of the paper, I think it is a minor one\n- I personally do not like the strict statement that reproducibility is the wrong question to ask raher than inferential reproducibility, reproducibility is of great urgency and relevance while inferential reproducibility is a relatively new and fresh concept which deserves attention\n- stating that deep learning is non deterministic is a little misleading in my humble opinion, the outcome of learning deep learning models due to many hidded factors turns out to be stochastic but there are no theoretical reason that deep learning models are sthocastic.\n- at a give point of page 2, reading the paper, it seems that the authors could mention the no free lunch theorem while they did not, thus I would like them to check whether a link between that important theorem and what they wrote \"On the one hand, a successful duplication\nof a SOTA result on a benchmark does not guarantee generalization to new data. On the other hand, a model might generalize well to new data even if the exact SOTA result cannot be duplicated because of differences in computational budget.\"\n- maybe providing some more details about experimental design methodology for those who are not aware could improve the effectiveness of the paper, I personally know about about that but I do not think design of experiemnts (DoE) is a methodology that spread in the macine learning community.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lW0wMfAooqf", "reviewer": "Reviewer_994g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to reproducibility that focuses on assessing and reporting variability in performance rather than focusing on replicating precise performance values.  The authors propose using a linear mixed-effects model to analyze the factors that influence performance.", "summary_of_the_review": "An important topic, and lots of interesting exposition about the high-level problem and specific statistical methods. However, the novelty is less clear.", "strength_and_weaknesses": "\\+ The paper discusses an extremely important issue around generalization and statistical variability. This issue has a long history and a central role in science (see below), but that understanding appears to be missing form many current ML researchers. This paper is a useful remedy for that problem.\n\n– The authors make a high-level point about characterizing and comparing variability which is a longstanding issue in science generally and statistics specifically. Then they recommend the use of several known methods in statistics. Thus, it is unclear what is novel about the paper. The authors appear to contend that researchers in deep learning are not making sufficient use of these ideas and methods (and I would tend to agree). However, there is little evidence of that point in the current version of the paper.\n\n– The paper does not cite some relevant scholarship in this area, including Andrew Gelman's work on \"the garden of forking paths\" (Gelman & Loken 2014) and David Hand's work on \"the illusion of progress\" (Hand 2006) in the context of classification. Others (e.g., Clary et al. 2019; Cobbe et al. 2019; Agarwal 2021) have raised issues of variability in reporting of results of deep learning methods.\n\n– The authors present their view as \"heretical\" when a long tradition of work exists on how to make inferences in the face of statistical variability. That some researchers in machine learning appear to have missed these basic methods of science hardly make the authors' viewpoint heretical. \n\n– The authors claim to present \"...a new type of reproducibility called inferential reproducibility...\" They note that \"It asks the following question: Can qualitatively similar conclusions be drawn from an independent replication of a study?\" Here, the authors are identifying the difference between what is sometimes called \"reproducibility\" (different team; same experimental setup) and \"replicability\" (different team; different experimental setup) (ACM 2020). They, themselves, note some prior scholarship on this issue, so it is unclear why they refer to it as \"a new type of reproducibility.\"\n\n– The authors describe a general problem (a focus on replicating precise values of performance rather than assessing and understanding the effects of variability in performance), and then jump immediately to specific statistical methods. An intermediate topic appears to be entirely missing: What sorts of generalization are being evaluated in a given study, what sorts of generalization are to be reasonably expected in practical uses, and how do those two match up? For example, Hand (2006) explicitly identifies explicit differences of the joint distribution between training and test as likely in realistic cases, and he conjectures that newer methods for classification are likely to be more sensitive to such differences. Similarly, work in concept drift, transfer learning, and related fields have assessed the robustness of specific types of machine learning models to specific types of differences in the distributions and when those sorts of differences are to be expected in practice. The paper would be improved by discussing this sort of middle-ground issue that connects the high-level points of the introduction and the specific statistical methods proposed for use.\n\nReferences\n\nAssociation for Computing Machinery (2020). Artifact Review and Badging. Version 1.1.\n\nAgarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.\n\nClary, K., Tosch, E., Foley, J., & Jensen, D. (2019). Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments. arXiv preprint arXiv:1904.06312.\n\nCobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2019, May). Quantifying generalization in reinforcement learning. In International Conference on Machine Learning(pp. 1282-1289). PMLR.\n\nGelman, A., & Loken, E. (2014). The statistical crisis in science data-dependent analysis—a “garden of forking paths”—explains why many statistically significant comparisons don’t hold up. American scientist, 102(6), 460.\n\nHand, D. J. (2006). Classifier technology and the illusion of progress. Statistical science, 21(1), 1-14.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-_XzE8DtKvK", "reviewer": "Reviewer_yHxA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the use of nested linear mixed effects models to compare performance of machine learning models. This approach is proposed to incorporate the effects of random initialization, hyper-parameter values, and other non-deterministic elements of model evaluation in the comparison. The authors then detail a reliability coefficient that compares the amount of variance in model performance due to the above aspects to the total variance. This enables the quantification of the notion of reliability in machine learning model evaluation.", "summary_of_the_review": "A great start, but this paper needs more.", "strength_and_weaknesses": "The authors approach attempts to address criticisms in reproducibility in the machine learning field. By using standard statistical analyses that separate out sources of variation, the authors are able to better characterize in what way the performance of two models may differ. This can increase the understanding in the community of the strengths and weaknesses of different models.\n\nThe perspective of the authors is valuable in that it addresses overly simplistic understanding of performance of machine learning models. Namely, one model (SOTA) being categorically better than another (baseline), is an overly simplistic view. Further, the authors' perspective calls into direct question the role of hyper-parameter tuning, random weight initialization, and the availability of computational resources, most of which are not explicitly considered in performance comparisons.\n\nUnfortunately, I do not think the authors have presented enough study of their point of view for a viable paper. While the authors propose to improve upon the comparison of models, their enumeration of the many dimensions of flexibility in their approach makes it clear that such an approach could easily exacerbate issues of uncertainty rather than resolve them. While being flexible enough to include many different sources of variation is useful, a high degree of standardization of the comparison is needed. Further, the authors do not address the power of their proposed test. \n\nIn the end, one could argue that model comparison has more to do with the ability of a model to perform well on novel problems, beyond the test set used for the comparison. One chooses to use a SOTA model not because it performed well on some historical test set, but because one hopes that it will perform well on one's problem at hand. The method proposed by the authors does address this in some capacity by quantifying the variation due to observations (sentences) separately from that due to hyper-parameters and random initialization. \n\nMore standardization and demonstration of the proposed approach are needed.\nWhile the authors mention that code is available, they do not really specify what that code does. Libraries that make it easy for authors to use the proposed approach, would be welcome.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster