Modeling the Machine Learning Multiverse
Authors: Samuel Bell
Onno P. Kampman
Jesse Dodge
Neil D Lawrence
Venue: NeurIPS 2022 Accept (2022)
Keywords: reproducibility
transparency
replication
multiverse analysis
batch size
generalization gap
adaptive optimizers

Abstract: Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers.  Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, the multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

Modeling the Machine Learning Multiverse
Samuel J. Bell'
Onno P. Kampman2 Jesse Dodge² Neil D. Lawrence'
'Computer Laboratory, University Cambridge
*Department Psychology. University of Cambridge
AAlen Institute for Al {sjb326, opk20 mca21) ac. uk
jessedGallemai. .rg
Abstract
Amid mounting concern about the reliability and credibility of machine learning research, present principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis [|
introduced in response to psychology's own reproducibility crisis. To efficiently explore high oigh-dimensional and often continuous ML search spaces, we model the multiverse with Gaussian Process surrogate and apply Bayesian experimental de- sign. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthe- size conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, a multiverse analysis is simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.
1 Introduction
Machine learning research faces mounting concern about the reliability of our results and the credibility of our claims 20) The field of psychology has faced similar crisis [21]. and confrontation with its shortcomings has sparked practical innovations [ 25) One such innovation
of direct relevance the machine learning community: the nultiverse analysis [I]
Throughout any investigation, scientists make decisions about how to perform their work. In psychology, in other disciplines, there experiments. From just handful of choices, we reach such a large decision tree that researchers can repeatedly try different paths until they chance upon positive result (26 Even without conscious manipulation, these decision points pose a fundamental problem: what if the psychologist had chosen alternate-and perfectly reasonable- route through this garden of forking paths (7 Would their results and conclusions still stand?
plethora of different ways to conduct and analyze
In their replication of contentious study 28 on the effect of menstrual cycle and relationship status on women's 's political preferences, Steegen et al. [1] identify both decisions taken and reasonable alternatives They name the Cartesian product of alternatives multiverse,
of parallel universes each containing slightly different study. Introducing the multiverse analysis, (Steegen et al. re-run the study as if inside each universe, finding that the alleged effect of relationship status and fertility is highly sensitive to different choices, in particular the definition of "single".. Multiverse analyses have subsequently been used evaluate the robustness of claims across variety of psychological and neuroscientific settings (e.g. 29
'Fgg. outlier selection; variable discretization; and how to estimate menstrual onset.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In a machine learning context, we present the multiverse analysis as principled framework for analyzing robustness and generality. Consider an example of some modification to a model, say batch normalization [4]. To verify batch norm's 's efficacy, one needs a test bed including model architecture, optimization method, dataset, evaluation metric, and so on. Regardless of these specific choices, we would like that batch norm be effective in general. With multiverse analysis, we can systematically explore the effect of each choice, and understand the circumstances in which a claim holds true.
Our primary contribution is to introduce the multiverse analysis to ML, which we use to draw more robust conclusions about model performance. To efficiently explore the nigh-dimensional and often continuous ML search space, we model the multiverse with a Gaussian Process (GP) surrogate and use Bayesian experimental desigm ($ D.. We present motivating evidence that choosing exploration over optimization the essence of a mu tiverse analysis is essential when we seek proper understanding of our claims and their generality ($ D. In the first of two case studies, we use a multiverse analysis over hyperparameters to replicate an experiment on adaptive optimizers K5 finding that claims on the relative merits of optimizers are highly sensitive to learning rate ($ 4 Second, we perform a
more extensive multiverse analysis to synthesize divergent research on the large batch "generalization
gap" [36,, finding an interaction effect of batch size and learning rate ($ 5 We conclude by discussing
the limitations of our approach and directions for future work ($
Efficient multiverse exploration
Our approach to the multiverse analysis packages MI. researcher decisions into a simple framework, requiring only choice of search space X' and an evaluation function É, which together define our multiverse. This search space defines the set of reasonable choices. For example, we might consider only a few directly relevant hyperparameters ($ D.; include purportedly irrelevant choices such as dataset (8 : or conduct even more expansive analyses ($ D.. The evaluation function should pertain to the hypothesis being tested. Here, we define it model test accuracy ($ 5 or difference in test accuracy (S
A barrier to analyzing any ML multiverse is the size of the search space and the presence of continuous dimensions. This makes exhaustive search intractable. To overcome this, we use a GP surrogate- modeling& l as function of X-aas explore the space using Bayesian experimental design 377
as stand-in for the multiverse. Using the surrogate, we iteratively
1. Sample an initial design, Xx- ~ X, and evaluate é each point, Yo- = ((Xo);
2. Fit a GP model f to the sampled points Xo and corresponding results Yo:
3. Use an acquisition function a on f to sample and evaluate a new batch (X,, Y;);
4. Repeat steps 2-3 until an appropriate stopping criterion.
In step 1, our initial design is drawn from a Sobol sequence, low discrepancy sequence that achieves improved coverage over uniform random sampling higher dimensions (38). The search space and the evaluation function are specific to the analysis at hand and described later.
In step 2, we model the output of our evaluation function as noisy function of the inputs,
y = fxx)) + e e; ~ N (0, E)
Placing a GP prior over f with zero mean and positive definite kernel function k;
GP(O, k)
(2)
we obtain the posterior mean and variance d² 39 chap. 2]:
yi+1 ~ ((((((+)), ()(((x()))))
(3)
((x(+1) = kTk--yx
(4)
0² (x(+1)= k(x(4, x k" <T((K (K + -k,
(5)
where K is the kernel matrix and k = [k(x441, (04411,0):..0 xo), ..., k(x++1, x.)). We use a Matérn 5/2 kernel with automatic relevance determination 40
2

---

141
102
10h
Teat
10
22-4
10 23
307
321
331
107
3.
10
10
10
36
10
50
32 10
107
109
10'
(a) UCB
(b) IVR
(c) IVR
Figure 1: Contour plot of GP GP-predicted mean test accuracy over search space of C and (Gamma) as explored by (a) UCB and (b) IVR acquisition functions. (c) Secondary objectives, e.g. minimizing group-level outcome differences, may vary along the IVR-revealed plateau.
In step 3, we use integrated variance reduction (IVR) as our acquisition function A1 (motivated IVR calculates change total variance if we were to acquire candidate point X,+1
3
a(x: 1: X,, / (p: X Y++1) o² (p: X,, dp)
(6)
where X,, Y; excludes the candidate point and X,+1, Y441 includes it.
Using a Monte Carlo approximatior for the intractable integral, the most informative point x* to sample next is
X arg memminn X1+1
A key difference between standard hyperparameter optimization and our framework is that when optimizing, models except the best performing are discarded, whereas we use all available information to draw more robust conclusions. To do this, we continue to use our surrogate model. First, we visualize the multiverse using the surrogate's 's posterior predictive mean, and visualize the extent to which we have explored through the surrogate's 's posterior variance. Second, test for interaction effects by comparing a GP with a shared kernel Mahared against an additive kernel Madditive (see fig. ST. We compare their fit using the Bayes factor K,
P(X, MaMadiive)) K
P(X, M/Mahardd
where P(X, Y|M) is the marginal likelihood of the observations given model. K >> indicates that Makiivve better explains the data than M.hared indicating the absence of an interaction. This also provides appropriate termination condition for step 4: continue to sample until we reach conclusive Bayes factor. Finally, we perform a Monte Carlo sensitivity analysis how much change in one of the parameters would affect the model outputs.
2 to assess
For GP modeling we use GPy with EmuKit 45 for experimental design and sensitivity analysis. We use Torch Vision's 46 off-the-shelf deep learning model architectures.
3 Motivating example: SVM hyperparameters
Here we motivate the multiverse analysis and our focus on exploration with an analysis of hyper- parameter optimization. To both aid replication and assess the effect of hyperparameter tuning on final results, there are calls for increased transparency around search space and tuning method (e.g. Going one step further, we argue that premature optimization--ee even transparently reported hinders our understanding. We illustrate this with simple example, tuning two hyperpa- rameters of an SVM classifier show that optimizing leads distorted view of hyperparameter
II particularly high- dith-ddmmniinal search spaces, may be more appropriate to use quasi- quasi-Monte Carlo method to estimate the variance reduction over sample of points with improved coverage of the space.

---

space. We compare optimization and exploration using the upper confidence bound (UCB) IVR acquisition functions.
48 and
In this multiverse analysis, we define the search space as the SVM's regularization coefficient C' and the lengthscale 7 of its RBF kernel. Our evaluation function is the test accuracy of the SVM on the Wisconsin Breast Cancer Dataset [ of samples of suspected cancer to be classified as benign or malignant. Given the same initial sample, we evaluate 23 further configurations for each acquisition function.
If optimize, we would conclude from fig. D that there is a single region of strong performance. Conversely, exploring with IVR (fig. (T)) shows that this region is in fact a plateau, and we are free to any value of C' as long both cases, only by exploring do we learn about the full space of our options. This knowledge is of vital importance if properly account for additional real-world objectives, such minimizing disparity in outcomes across different social groups. We test this idea by assigning a synthetic majority/minorit group label, g ~ Bern(0.4), each datapoint, and show in fig. I how group-level outcomes can vary along the plateau revealed by IVR. Prematurely optimizing could easily result in selecting model that introduces group disparity.
scale y accordingly. While best test accuracy is similar in
There is, of course, an appropriate moment for optimization. As one moves along the spectrum from research to deployment, too should move from exploration exploitation. Our aim here is not to critique optimization per se, but to highlight that exploration is also of paramount importance. When conducting scientific research in particular, learn and understand as much possible, rather than eke out another marginal improvement. Here we multiverse analysis framework for systematic exploration.
argue is more appropriate to
Case study 1: When are adaptive optimizers helpful?
is popular adaptive optimizer for training deep neural networks. Questioning this practice, 355 suggest adaptive optimizers offer limited advantages over vanilla stochastic gradient descent (SGD) [SI]. They present experiments showing that SGD with momentum (52) outperforms all other optimizers across image recognition on CIFAR-10 53
Adam
600
Wilson et al.
language modeling, and constitency
parsing.
Under certain hyperparameter conditions, however, adaptive optimizers can be considered equivalent SGD with momentum, highlighting the crucial role of hyperparameter tuning [54]. replica- tions of [Wilson et a experiment with VGG 555 on CIFAR-10, additionally tuning Adam's 's c
hyperparameter introduced solely for numerical stability and typically ignored eliminates SGD's
advantage 6 [54].
can be considered proof by existence. In order refute Wilson et al. claim, it suffices to find any point in hyperparameter space where Adam beats SGD with momentum.
Both replications
6 54
1.0
1.0
Adam
amforence test accuracy
08
0.8
10
10
spsilen
30
Rpsiten
30
Varfance
Train incumacy
06
Test oc0urday
0.8
0.
0.4
10a
10*
16
16
02
02
139
10-
00
0.
(a) Mcan surface
(b) Variance surface
(c) Final accuracies
Figure 2: Contour plot of GP-predicted (a) mean difference test accuracy (SGD- Adam) and (b) variance over the search space of learning rate and c. Red regions indicate SGD with momentum outperforms Adam. White points are successful trials; black crosses failed. (c) Final train and test accuracies. Whiskers extend to min and max. Note SGD train accuracy has median, UQ and max 1.0.

---

However, this approach tells us little about relative optimizer performance in general. In this short case study, we perform multiverse analysis of Wilson et al. experiment with VGG on CIFAR- 10. We explore a reasonable and relevant search space and analyze how different hyperparameter choices lead to different optimizer recommendations.
4.1 Multiverse definition
Because the claim under scrutiny compares optimizers, we define the evaluation function as the difference between model test accuracy achieved by Adam and SGD with momentum. Let ge (((=-))) be the minibatch-estimated gradient of the loss function f w.r.t. the model parameters 0--1- For SGD with momentum, take a step of size in the direction of decaying sum of recent gradients.
8, = ad; = ud: ge
where momentum parameter controls decay. In contrast, Adam steps along the gradient normalized by an unbiased estimate of its first and second moments m, and V,
m
d Vi+t
my
m.
m. 3 + 31)g:
mg
V2 32V, + (1 B2)
Vo 0.
Like Wilson et al., we use the default of p 0.9 and use the Polyak implementation. For Adam,
we also use default parameters 31 = 0.9 and 8n 0.999. The model is VGG-16 with batch normalization 3 and dropout (56). trained for 300 epochs on CIFAR-I0.
We set our search space to learning rate a E [10-4, 1100 by C [10-", 100 100100000000 c is only applied to Adam. We evaluate 3 batches of 32 points.
4.2 Results
1.0
1.0
Epsiion
The contour plot fig. 2 shows large region 5 S 10+) in which SGD outperforms Adam. However, we also identify regions (approx. 10-3 or > 103) where the opposite is true, though we note the higher uncertainty (fig. 2b) in the high learning rate region.
Effect
0.5
0.5
See fig. 22 for raw results.
The change in relative performance is described almost entirely by learning rate. The sensitivity analysis (fig. 3) reveals main effects (i.e. sensitivity to each variable isolation) for learning rate and of 0.990 + 0.014 and -0.004 + 0.0002, and similarly total effects (i.e. sensitivity to each variable including its interaction with others) of 0.996 + 0.014 and 0.008 + 0.031. Testing for interaction effects 2635, thus rejecting the existence of an interaction between learning rate and e.
(a) Main
(b) Total
Figure 3: (a) Main and (b) to- tal effects learning rate and
LR drives almost all output variance. Bars are STD.
via Bayes factor (see eq. (8) ) we find
In summary, we find that SGD with momentum often, but not always, outperforms Adam when training VGG on CIFAR-10, 100, and that the conclusion we draw is highly sensitive to selected learning rate. Unlike others O 154). do find a significant effect of instead finding that learning rate determines which optimizer performs best. In practice, our results suggest optimizer choice may make little difference to final test error, as long as an appropriate learning rate used. If budget for hyperparameter search is limited, is unlikely to be the most efficient hyperparameter to include in search 5 Using a multiverse analysis, we have systematically explored how decisions about learning rate and epsilon impact conclusions about the optimal optimizer. In doing so, we have shown that reported results may vary according to researcher choices.
'hhat said, our results not preclude the possibility that with a much more extensive and optimization based search, the resulting estimate of the search space might reveal some (modest) room for improvement from tuning However, as indicated by the sensitivity analysis in fig. any effect is likely to be dwarfed by properly tuning learning rate, and is unlikely materially alter the best choice of optimizer.
5

---

27
21
21
Learning rabe
Learinng rabe
Leaminn rabe
(a) AlexNet on CIFAR-10
ResNet CIFAR-10
VGG CIFAR-10
20
20 Leaminn rate
10
Learning rate
Leominn rate
(d) AlexNet CIFAR 100
(c) ResNet CIFAR 100
(f) VGG CIFAR- 100
Figure 4: Contour plot of GP-predicted mean test accuracy over the search space of learning rate, batch size, dataset and model. White points are trials with training accuracy 0.99; black crosses were excluded. Overlayed translucent regions indicate high training error. For Tiny ImageNet see fig. S4 for variance see fig. SS The discrepancy between contours and data points (a) is due the coregionalized model sharing information across functions.
Case study 2: Is there a large-batch generalization gap?
Larger batch sizes prerequisite for distributed neural network training. However, many searchers have reported a "generalization gap". where generalization performance declines as batch
size increases 36 57 58 a phenomenon potentially (59-62 but not conclusively (33 linked the
sharpness of the resulting minima.
Subsequent work has sought ways mitigate the generalization gap by, in broad strokes, either scaling up the batch size throughout training [64| 65 pp. 262-265], or by scaling learning rate proportional to batch size (66-69), To make
of these results, must cut through myriad
researcher choices about datasets; model architectures; termination criteria [70;: linear 68] or
sublinear (e.g. square root) [66] scaling rules; high initial learning rate (66]; learning rate decay
warmup [68]; layer-specific learning rates 6%%; batch norm variants [66]; regularization techniques e.g. label smoothing (70); and
In second case study, synthesize existing research on the relationship between batch size, learning rate and generalization error.
multiverse analysis to
5.1 Multiverse definition
Our evaluation function the test accuracy of model trained according the sampled config- uration.
The search space includes learning rate a E [10-1, 10-1), -j, batch size E (2 ....... 2
model E [AlexNet [7]. VGG (55), ResNet [22 and dataset [CIFAR-10, CIFAR-100 533 Tiny
ImageNet (3)) Specifically, we use VGG- 16 with batch norm and ResNet-18 Tiny ImageNet was
selected as substitute for ImageNet [7]] to limit compute expenditure.

---

For categorical parameters, we treat each pair of model and dataset as its own function with an intrinsic coregionalization model 75 76. Given base kernel k, our multi-output kernel matrix is:
Bm Wm W + diag(Ning (Km)).
Ba = Waw d + (Ka)
XXX, = mm @ Ba @ xx,,))
where Ba and Bm represent dataset and model respectively. In interpreting these parameters, the outer product of w defines how related each output is, whereas allows them to vary independently.
We evaluate 6 batches of points. Before analysis, we discard all model runs with training accuracy < 0.99 and re-fit the model. In addition to previous methods we also analyze the coregion- alization parameters investigate how much dataset and model impact our results.
Tiny Imagehet
5.2 Results
CHARR 100
CIPAR110
Of 224 trials, 9 failed to converge and further 115 did not reach training accuracy 2 0.99. preliminary analysis the raw results (see fig. (33) shows test accuracy is barely positively correlated with learning rate (p = 0.08) and slightly negatively correlated with batch size (p -.22). In fig, however, a consistent plateau emerges across all models on both CIFAR-10 and CIFAR- 100: test accuracy is maintained as long batch size scales up with learning rate. On both sides of this plateau, if either learning rate or batch size error will increase.
CIFAR-10
Correlation
CIFAR-100
Thny ImageNer
(a) Dataset
large, generalization
AlexNet
ResNet
VGG
We were unable to successfully train any model Tiny Im- ageNet. Of the 42 models we trained on Tiny ImageNet, 22 reached the 0.99 training accuracy threshold, but none of these obtained a test accuracy higher than 0.01, indicating significant overfitting, See fig. SS for Tiny ImageNet contours.
AlexNet
Correlation
ResNet
Testing for interaction effects via Bayes factor (see eq. ( we find K 0.58, suggesting that additive kernel is not sufficient to explain the data and indicating an interaction effect. This implies that batch size alone does not explain generalization, batch size and learning rate together. Our modeling approach gracefully handles the failures on Tiny ImageNet, such that even with Tiny ImageNet trials removed, the interaction effect remains present (K 0.34).
VGG
(b) Model
Figure 5: Multi-output GP func- tion correlations. (a) Tiny Im- ageNet is moderately negatively correlated with CIFAR-10/100 (b) All three model outputs are highly correlated.
The coregionalization kernels explain how model and dataset affect our results. In fig. 55 all model outputs are highly cor- related, indicating that choice of model does not impact the relationship between batch size, learning rate, and generalization error. contrast fig. 5 shows highly correlated outputs across CIFAR-10 10 and CIFAR-100, and moderate negative correlation with Tiny ImageNet. A strength of our chosen coregionalized model is that it flexibly captures the batch size and learning rate relationship in CIFAR- 10 and 100, despite the failures on Tiny ImageNet. Our results CIFAR-10 and CIFAR-100 100 indicate that the relationship may hold across datasets where training was successful, including across different dataset complexities, in contrast to previous work (57].
1.0
1,0
Batch
Detaset
Eftect
0.5
0.5
(a) Main
(b) Total
Figure 6: (a) Main and (b) total ef- fects of all parameters. LR drives more output variance than batch size, but dataset is most important. Increased total effect for LR and batch size confirms interaction ef- fect. Bars are STD.
Finally, our sensitivity analysis in fig. reveals next to no main effect for either learning rate (0.03 + 0.06) or batch size (0.02 0.06). However, the larger total effects for both learning rate (0.19 0.1 and batch size (0.15 0.11) again support the existence of interaction effect. Both learning rate and batch
7

---

size are dominated by the main effect of dataset (0.83 + 0.03), though this is likely due to Tiny ImageNet.
In conclusion, our framework reveals a complex interaction between learning rate, batch size, and generalization error that is consistent across all models and two out of three datasets. These results support scaling learning rate linearly with batch size [1 batch size (up to 2 after which learning rate is no longer corrective (57 70) After accounting
57 68
but do not support threshold
for learning rate, we find no consistent evidence of the proposed [36] [58] large-batch generalization
gap. Here, a multiverse analysis has proven a useful tool pull together disparate research on the generalization gap and to surface underlying trends.
6 Discussion
In the three applications of the multiverse analysis presented here, we have demonstrated that decisions taken by the researcher can have significant effect on the final conclusions drawn. In our SVM hyperparameter example, we highlight the important role of systematic exploration and show that premature optimization may both limit our understanding and present only a partial truth to downstream practitioners. Our first case study optimization demonstrated how varying the learning rate results in fundamentally different conclusions about whether to use SGD with momentum or Adam for optimization. Conversely, our second case study showed a simple relationship between batch size, learning rate and generalization error irrespective of model and dataset.
6.1 Choice of search space
Across these examples we have tried illustrate differently complex analyses by way of differently sized search spaces. In case study 1 we inflated a rather small multiverse focusing only on directly relevant hyperparameters, though in case study 2 we built larger multiverse including parameters (model and dataset) that vary freely between existing research. An expected critique of multiverse analyses is that choice of search space is itself choice However, choosing dimensions and bounds for search space is more principled than choosing specific points, and that declaring each dimension makes assumptions of relevance or irrelevance explicit. That said, it always possible for subsequent work to critique the choice of search space and to add new dimensions. Indeed, our first case study should be expanded to include each of Wilson et al. ss experiments with different model architectures, datasets, deep learning frameworks and additional adaptive optimizer variants, though we reinforce that we chose a limited analysis provide simple case study. Our second case study also presents a number of interesting avenues for expansion in subsequent analyses, including additional datasets and model architectures, learning rate schedules and optimizers. Most interesting would inclusion of termination criterion and evaluation metric, both previously highlighted as key drivers of divergent findings 70
6.2 Compute cost
Our main contribution is present the multiverse analysis framework and show how it can be used to draw robust conclusions about model performance, so we allocated our compute budget to showcase illustrative examples. Our framework is equally appropriate for research pushing with a larger budget, though such experiments aren't necessary to demonstrate our framework's value. In this project, we used 1138 hours of GPU time, at rough cost of $775 with 140 kg of CO22 We expect costs to scale with multiverse size, requiring a pause for consideration in light of recent critiques of the environmental impacts of ML [77 79 In response, we first note that least some of the required compute is already taking place as part of existing trial-and- -and-rroor workflows. Second, through the introduction of surrogate model for the multiverse, one can substantially reduce the amount of exploration required. Finally, we suggest that all resources consumed in producing non-robust results, and those of subsequent work that builds upon them, are wasted by definition.
of-the--rrtt
'The same applies to GP surrogate setup. e.g. kemel choice, though suggest our settings used in this work will be a suitable default for most purposes. See figs. 56 to 510 comparison of different kernels. "Calculated using https://mlco2. github................mm........ 1 et/iimmm/o impact [i assuming A100 GPUs on the University of Cambridge HPC cluster with carbon efficiency 0.307 kgCO, kWh.
8

---

6.3 Future work
is common practice when reporting model performance train a number of runs with different random seeds and report the mean and standard deviation. Number of runs varies according to researcher budget and time. Appropriate sample size to enable robust inference in the face noise is rarely considered. Our approach to the multiverse analysis presents an elegant potential solution to this issue. our examples thus far we train only a single model for each point and assume homoscedasticity for simplicity. However, research in experimental design suggests that accounting for heteroscedasticity by separately modeling the variance the surrogate 80 ST could allow for principled trade-off between conducting another run and sampling new configuration
S
handful of studies in human computer interaction H In an exciting area for development, the authors also develop interactive visualizations to help the reader explore the effects each choice, up and including re-rendering written conclusions in an online version of the paper. Given that the largest multiverse in our work is 4-dimensional, our faceted contour plots are sufficient to communicate key trends. However, in more expansive multiverses novel visualization techniques will become essential.
While we introduce the multiverse analysis to ML, has previously been applied
34 promise enforce distinction between Committing to an experimentation and analysis plan in advance can be helpful foil for questionable practices such tweaking the parameters until one reaches a positive result. However, this commitment is limited to single analysis- particular instantiation of decision set et-while the other possible analyses, the remainder of the multiverse, remain unexplored. We see the multiverse analysis as complementary to pre-registration, in that registrist a multiverse analysis both codifies auxiliary hypotheses and allows room for exploration.
Recent experiments with pre-registration in ML (e.g.
exploratory and confirmatory research 2
35
7 Conclusion
For continued progress in ML, we depend on reproducible results and conclusions that generalize to new settings. We have introduced the multiverse analysis as a principled framework to explore the impact of researcher decisions and facilitate the drawing of more robust conclusions. Our first case study unifies conflicting work on the benefit adaptive optimizers and reinforces that optimizer merit is driven primarily by learning rate. In our second case study we identified complex interaction between batch size, learning rate and generalization error, and we dispute the existence of generalization gap. We have also presented evidence supporting the practice of scaling learning rate linearly with batch size.
By using multiverse analysis, both researchers and practitioners gain more robust claims, better un- derstanding of how decisions impact results, and helpful insight into the generality and reproducibility of conclusions.
Acknowledgments and Disclosure of Funding
SB is supported by the Biotechnology and Biological Sciences Research Council [grant number BBSRC #B/M011194/1 NL is supported by Senior Turing AI Fellowship funded by the UK government's 's Office for AI, through UK Research and Innovation (grant reference EP/V030302/1). and delivered by the Alan Turing Institute. NL's chair is endowed by DeepMind.
References
[1] Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5):702-712, 2016.
[2] Odd Erik Gundersen and Sigbjøm Kjensmo. State of the art: Reproducibility in artificial intelligence. In 32nd AAAI Conference Artificial Intelligence, AAAI 2018, pages
2018.
[3] Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research reproducible, In Proceedings of the 36th International Conference on Machine Learning, ICML, pages 1150-1159, 2019.
[4] Zachary C. Lipton and Jacob Steinhardt. Troubling trends in machine- learning scholarship. Queue, 17(1): 45-77. 2019.

---

[5] D. Sculley, Jasper Snoek, Ali Rahimi, and Alex Wiltschko. Winner's curse? On pace, progress, and empirical rigor. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, 2018.
[6] A. Feder Cooper. Yucheng Lu, Jessica Forde, and Christopher M. De Sa. Hyperparameter optimization is deceiving us, and how to stop it. In Advances in Neural Information Processing Systems, volume 34, 2021.
[7] Edward Raff. A step toward quantifying independently reproducible machine learning research. In Advances in Neural Information Processing Systems, volume 32, 2019.
[8] Jessica Forde and Michela Paganini. The scientific method in the science of machine learning. 2019, 190.:1994..9.......
[9] Ali Rahimi and Benjamin Recht. Reflections random kitchen sinks, 2017. URL URLhttpp::/wwwt argmin het/2017/12205//itchhrr sinks/ Accessed 2022-04-23
[10] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforvement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[11] Rishabh Agarwal, Max Sehwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, volume 34, 2021.
[12] Mario Lučié, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? large-scale study. In Advances in Neural Information Processing Systems, volume 31, 2018.
[13] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress?
A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys, page 101-109, 2019.
[14] Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through crowded valley- benchmarking deep learning optimizers. Proceedings of the 38th International Conference on Machine Learning, ICML, pages 9367-9376. PMLR, 2021.
[15] Luke Dakden-Raynen Jared Dunmmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. 2019, arXiv:1909. 12475.
[16] Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation neural language models. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, 2018, stXivv17777 .0558..
[17] Nils Reimers and Iryna Gurevych. Reporting score distributions makes difference: Performance study of for sequence tagging. 2017, MtXXvv1107.09985
[18] Kyle Gorman and Steven Bedrick. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786-270 1281-2791,26 2019.
[19] Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. 2021, arXiv:2106.15195.
[20] Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, and Zhenzhong Lan. Do transformer modifications transfer across implementations and applications? 2021, 1tXiv:2102.1197.
[21] Samuel J. Bell and Onno Kampman. Perspectives on machine learning from psychology's reproducibility crisis. In Science and Engineering of Deep Learning Workshop, ICLR, 2021, arXiv:2104.08878.
[22] Leslie K. John, George Loewenstein, and Drazen Prelec. Measuring the prevalence of questionable research practices with incentives for truth telling, Psychological Science, 23(5):524-532, 2012.
[23] Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, Han L. J. van der Maas, and Rogier A. Kievit. An agenda for purely confirmatory research. Perspectives on Psychological Science 7(6)632 638, 2012.
[24] Christopher D. Chambers. Registered reports: A new publishing initiative at Cortex. Cortex, 49(3): 609-610, 2013.
10

---

[25] Richard A. Klein, Kate A. Ratliff, Michelangelo Vianello, Reginald B. Adams, Stěpán Bahník, Michael J. Bernstein, Konrad Bocian. Mark J. Brandt, Beach Brooks, Claudia Chloe Brumbaugh. Zeynep Cemalcilar, Jesse Chandler. Winnee Cheong, William E. Davis, Thierry Devos, Malthew Eisner, Natalia Frankowska, David Furrow, Elisa Maria Galliani, Fred Hasselman, Joshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, Hans Ijzerman, Mclissa Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jaime Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Troy G. Steiner, Justin Storbeck, Lyn M. Van Swol, Donna Thompson, A. E. Van "T Veer, Leigh Ann Vaughn, Marek Vranka, Aaron I.. Wichman, Julie A. Woodzicka, and Brian A. Nosek. Investigating variation in replicability: A "Many Labs" replication project. Social Psychology. 45 (3):142-152, 2014.
[26] Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything significant. Psychological Science, 22(11): 1359 1366, 2011.
[27] Andrew Gelman and Eric Loken. The statistical crisis in science: data-dependent analysis-a "garden of forking paths" explains why many statistically significant comparisons don't hold up. American Scientist, 102(6):460-466) 2014.
[28] Kristina M Durante, Ashley Rae, and Vladas Griskevicius. The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24(6):0007-1016, 2013.
[29] Scamus Donnelly, Patricia J Brooks, and Bruce D Homer. Is there a bilingual advantage on interference- control tasks? A multiverse meta- meta-analysis of global reaction time and interference cost. Psychonomic bulletin & review, 11111122114177 2019.
[30] Elise K. Kalokerinos, Yasemin Erbas, Eva Ceulemans. and Peter Kuppens. Differentiate regulate: Low negative emotion differentiation is associated with ineffective use but not selection of emotion-regulation strategies. Psychological Science, 30(6):863-879, 2019.
[31] Kathryn L. Modecki. Samantha Low-Chop Low-Choy, Bep N. Uink. Lynette Vernon, Helen Correia, and Kylie Andrews. Tuning into real effect of smartphone use on parenting: multiverse analysis. Journal of Child Psychology and Psychiatry, (1))8855 865, 2020.
[32] Tina B. Lonsdorf, Anna Gerlicher, Maren Klingelhüffr-Jens, and Angelos-Miltiadis Krypotos. Multiverse analyses in fear conditioning research. Behaviour Research and Therupy, 153: 104072, 2022.
[33] Jessica Dafflon, Pedro F Da Costa, František Váša, Ricardo Pio Monti, Danilo Bzdok, Peter Hellyer, Federico Turkheimer, Jonathan Smallwood, Emily Jones, and Rohert Leech. A guided multiverse study of neuroimaging analyses. Nature Communications. 13(7758), 2022.
[34] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference Machine Learning, ICML., pages 448-456, 2015.
[35] Ashia C. Wilson, Rebecca Roclofs, Mitchell Stemn, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[36] Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. large-batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the Sth International Conference on Learning Representations, ICLR 2017, 2017, arXiv:1669.00489
[37] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review, Statistical Science, pages 273-304, 1995.
[38] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Muchine Learning Research. 13:281-305, 2012.
[39] Christopher K. Williams and Carl Edward Rasmussen. Gaussian Processes for Machine Learning. The MIT Press, 2006. ISBN 978-0-262- 18253-9.
[40] Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine lcarning algorithms. In Advances in Neural Information Processing Systems, volume 25, 2012.
[41] Jerome Sacks, William J Welch, Toby J Mitchell, and Henry P Wynn. Design and analysis of computer experiments. Statistical Science, 4(4):409 423, 1989.

---

[42] I. M. Sobol. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation, 55(1)2771-280, 2001.
[43] Andrea Saltelli. Making best use of model evaluations to compute sensitivity indices. Computer Physics Communications, 145(2):280 297. 2002.
[44] GPy: A Gaussian Process framework in python. http://github..com///fffffeiiill
2012.
[45] Andrei Paleyes, Mark Pullin, Maren Mahsereci, Cliff McCollum, Neil Lawrence, and Javier González. Emulation of physical processes with Emukit. In 2nd Workshop on Machine Learning and the Physical Sciences, NewrlPS, 2019.
[46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Autodiff Workshop, NeurIPS, 2017.
[47] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer. Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility machine learning research (a report from the NeurIPS 2019 reproducibility program). 2020, arXiv:2003.12200
[48] D. D. Cox and S. John. A statistical method for global optimization. In Proceedings of the 1992 IEEE International Conference on Systems, Man, and Cybernetics, volume 2, pages 1241-1246, 1992.
[49] William Wolherg, W. Street, and Olvi Mangasarian. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995.
[50] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2014, arXiv:1422.66880
[51] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951
[52] Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR computational mathematics and mathematical physicx, 4(5):1-17, 1964.
[53] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[54] Dami Choi, Christopher Shalluc, Zachary Nado, Jachoon Lee, Chris J. Maddison, and George E. Dahl. On empirical comparisons of optimizers for deep learning, 2019, arXiv:1910.05446.
[55] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion, 2014, 13XX::1449915555
[56] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
[57] Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael W. Mahoney, and Joseph Gonzalez. On the computational inefficiency of large batch sizes for stochastic gradient descent. 2018. arXiv:1811.1299)
[58] Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. 2018, 100.62180..0761..
[59] Sepp Hochreiter and Jürgen Schmidhuher. Flat minima. Neural Computation, 9(1):1-42, 1997.
[60] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, volume 31, 2018.
[61] Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima SGD. 2017, 1rXiv:1711..04 1,04623.
[62] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGII SGDD: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124115 2019.
[63] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
12

---

[64] Aditya Devarakonda, Maxim Naumov. and Michael Garland. AdaBatch: Adaptive batch sizes for training deep neural networks. 2017, arXiv:171...2029.
[65]
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, (W2)):22-311, 2018.
[66] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Procesing Systems, volume 30, 2017.
[67] Samuel L.. Smith, Picter-Jan Kindermans, Chris Ying, and Quae V. Le. Don't decay the learning rate, increase the batch size. 2017, arXiv:1711.00489.
[68] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in I hour. 2017, (iXx:1106.00277
[69] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. 2017, 1.Xiv1100.00380
[70] Christopher J. Shallue, Jachoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20:1-49: 2019.
[71] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Syxtems, volume 25, 2012.
[72] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[73] Tiny ImageNet. https://tiny: imagennt. hww.huapp..c.o com/
[74] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, and Michael Bernstein. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211225 252, 2015.
[75] Jeffrey D. Helterbrand and Noel Cressie Universal cokriging under intrinsic coregionalization. Mathemat- ical Geology, 26(2):205-22 226, 1994.
[76] Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrenee. Kernlss for vector-valued functions: A review. Foundations and Trends in Machine Learning, 4(3):15--66, 2012.
[77] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. 2019, arXiv:1910.0970
[78] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. 2019, arXiv:1906.02243.
[79] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AL Communications of the ACM, 63(12):54 63, 2020.
[80] Paul Goldberg, Christopher Williams, and Christopher Bishop. Regression with input-dependent noise: A Gaussian Process treatment. volume 10, 1997.
[81] Mickaël Binois, Robert B. Gramacy, and Mike Ludkovski. Practical heteroscedastic Gaussian Process modeling for large simulation experiments. Journal of Computational and Graphical Statistics, 24))
808-821, 2018
[82] Mickaël Binois, Jiangeng Huang, Robert B. Gramacy, and Mike Ludkovski. Replication or exploration? Sequential design for stochastic simulation experiments. Technometrics, 61(1):7-23, 2019.
[83] Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. Increasing the transparency of research papers with explorable multiverse analyses. In Proceedings of the 2019 Conference on Human Factors in Computing Systems, pages 1-15, 2019.
[84] The pre-registration workshop. httpss://preregister science/
[85] Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHlaven, and David Mellor. The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11): 2600-2606, 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See 6.2 for compute cost/climate impact discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] To be included in supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See 6.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All datasets and models cited throughout.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jmncEMMr7LN", "reviewer": "Reviewer_yyG7", "summary": "This paper proposes an ML multiverse analysis framework to draw robust scientific conclusions about model performance by focusing on the model exploration using Gaussian process surrogate models and Bayesian experimental design. The paper shows two motivating case studies: \n1) presented a multiverse analysis demonstrating that different hyperparameter choices lead to different optimizer recommendations; \n2) used the framework to investigate the conflicting research on the effect of learning rate on the large batch training generalization gap. \n\nThe core idea of the ML multiverse analysis is to use integrated variance reduction (IVR) as the acquisition function and keep all available information to draw more bust conclusions. The analysis is carried out by visualizing the posterior predictive mean to the extent to which the surrogate's posterior variance has been adequately explored. Also, the analysis tests for the interaction effects of the different dimensions of the search space by calculating the Bayes factor of a GP with shared kernel against an additive kernel. Finally, a Monte Carlo sensitivity analysis is used to test how much a change in one of the parameters would influence the model outputs. ", "questions": "-  As pointed out by the authors, the largest dimension of the multiverse in the two case studies is 4, therefore, faceted contour plots are sufficient for visualization purpose. In case of high-dimensional search space, besides visualization, are there any other quantitative techniques can be used to analyze the result? ", "limitations": "The author did addressed the potential environmental concern of the framework due to computation scaling up with the search space. On the other, scalability of the proposed method with respect to the search space will probably limit the practicality of the proposed multiverse analysis framework. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength:\n- The paper is well written and structured \n- The paper provides a principled and practical way to to achieve reproducible results and draw robust conclusions, which will help the continued progress in the ML community. \n- The effectiveness of the proposed framework is well-demonstrated by two comprehensive case studies. \n\n## Weakness: \n- As pointed out by the author, the computation time of the framework will scale with the multiverse size, given the recent critiques of environmental impacts on ML, needs to be carefully considered. One possible way to reduce the computation time is to make use of all the existing experiments results through efficient surrogate models. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XLjcdH_i-gy", "reviewer": "Reviewer_dGQ4", "summary": "The paper adopts a concept of psychology, called ‘Multiverse Analysis’, to systematically study credibility of claims and robustness of conclusions. Multiverse are universes each with a slightly different layout. Based on this concept, GP model is employed to quantify and emulate the relationship between interest of study and variance in layouts. The whole experiment is conduct in a Bayesian way: construct initial design; collect data; find next experiment point that reduce the variance. The existence of interaction effect is determined by comparison of probability of the model considering interaction effect and the model not given current data.", "questions": "How to verify the significance of model comparison (interaction effects)? If the Bayes factor is close to 1, say 0.998. ", "limitations": "The authors have discussed the limitations. I have one more comment. \n\nFor the IVR acquisition function in Eq (6), the authors just simply use a Monte Carlo approximation to deal with multivariate integral. It cannot be accurate when for large-dimension problem.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n(1) It is very novel to consider investigating the machine learning multiverse as the design of experiments problem with the steps of initial design, surrogate model, acquisition function, and sequential design.\n\n(2) The proposed framework is capable of comprehensively verify or study an interested claim of effect. Bayes factor is a fantastic idea to determine if interaction effect exists. \n\nWeakness: \n(1). The current method in the manuscript is not able to quantify the magnitude of interaction effects. When model the categorical factor, kernel function is a production. Therefore, the effects are kind of interaction effect among all categorical factors and continuous factors.\n\n(2). The current manuscript has not sufficiently described how to choose the initial design, and how the choice of the initial design affects the performance of the proposed method. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Gg0_hPI8yF0", "reviewer": "Reviewer_wcgM", "summary": "The paper proposes to use active learning with Gaussian Processes (GPs) as a way to perform sample-efficient \"multiverse analysis\" of machine learning models, i.e. a way to understand the full space of hyperparameter settings and their influence on model performance rather than simply optimizing them. It then applies this analysis to three examples, one with a motivating (but artificial) goal related to inter-group variability, and two motivated by real recent debates in the literature. ", "questions": "I hope that the rebuttal can help clarify and justify the impact of the findings as presented, for example by more precisely discussing ways in which they drive new conclusions not present in the cited prior work. Alternatively, to the extent that the conclusions are similar in broad strokes, I hope the rebuttal can defend the ways in which the systematicity of the present experiments shores up some aspect of the two case studies that remained previously unclear or murky. A third way for the paper to have greater impact would be to demonstrate that additional conclusions can be drawn from the meta-models that extend beyond what has been hypothesized or shown previously. \n\nFinally, I also hope that the rebuttal will clarify the following puzzle regarding the adaptive optimizer study: on lines 167-170, the paper discusses how Adam's epsilon parameter seems to not matter for the purposes of determining whether it outperforms SGD, and grants that epsilon could be tuned to improve Adam's performance. If that is true, then one can take a vertical slice through figure 2a and move along it in some dimensions in which Adam performance improves but SGD performance does not (since SGD performance should be invariant to movement in that dimension of the space). To the extent that we do not see such a thing, the present contribution seems incompatible with the claim that epsilon can be used to improve Adam's performance, or be used to close the Adam-SGD gap. What am I missing here? \n", "limitations": "Limitations are discussed, even alongside the claims / conclusions rather than being shunted to a separate section -- this is good. Benefits of using sample-efficient active learning are discussed. Negative societal implications are not discussed explicitly, though I don't think there's anything major to discuss here. One thing not mentioned that is possibly worth engaging with is that for very large-scale models that take thousands of hours to train on thousands of GPUs, even regular hyperparameter search may be out of reach, never mind a multiverse analysis, so the impact of this sort of work will likely be in smaller settings potentially farther from state of the art. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nI enjoyed reading the paper: it is well-written such that I have essentially no quibbles related to typos, confusing wording, or the like, and more importantly the motivation and core ideas are presented so clearly as to seem obvious post hoc. The idea of using a meta-model to improve efficiency of multiverse analysis is likewise useful (though I believe there's a similar idea in a biorxiv preprint from a few years ago, doi:10.1101/2020.10.29.359778, and in older work from psychology doi:10.1037/0033-295X.113.1.57). The technical approach is reasonable given the setup (flexible GP surrogates, global active learning with IVR by taking an expectation over the lookahead posterior variance), and the conclusions aren't overstated given the relatively modest studies undertaken. \n\n## Weaknesses \nThe essence of a good paper is an insightful idea or contribution taken to new payoffs, and both seem relatively modest in the paper as it stands. \n\nWith regards to the core idea, the paper imports a pre-existing conceptual framework (in the sense that multiverse analysis is not new, though it seems new to ML), and uses standard tools in only a slightly novel application (in the sense that GP active learning is not new in ML, though the typical application is to pure optimization). \n\nWith respect to the new payoffs, the analysis of adaptive optimizers comes to a similar high-level conclusion to what is in prior work (that both SGD and Adam can perform equally well depending on hyperparameter tuning), and the paper does not go beyond exploring the two parameters that have been explored previously. At the same time, when it comes to the details, the paper does not seem to replicate the effect of epsilon on the SGD-Adam gap. This is potentially interesting and while the paper claims this result is not incompatible with the potential to improve Adam performance by tuning epsilon, I'm not sure I see how both can be compatible (more on this in \"questions\" below). \n\nWith the large batch generalization gap study, the paper explores a space substantially smaller than the multiverse of reasonable choices previously explored in the literature, and the relationship it does find matches prior work regarding batch size and learning rate, \n\nWithout disputing that the present work undertakes both studies in an elegant and efficient way relative to prior work, I think the latter still serves as a \"warmup\" demonstration of shoring up existing results, and not a strong new thing that couldn't have been done without the proposed approach, while the former adds another datapoint to the space without providing a definitive understanding of the benefits of Adam over SGD. \n\nFurthermore, the small search spaces and large batches that the paper considers don't really showcase the benefit of active learning where, especially for a pure exploration setting, a quasi-random search baseline is likely to be quite strong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9BqNInpAF7S", "reviewer": "Reviewer_fyXL", "summary": "The paper proposes the Multiverse analysis as a step toward more transparent, reliable and credible results when we are facing large number of choices in our research design even before running the actual experiments. Speficically the authors leverage Bayesian optimization idea to develop a framework to search over large search space, which is used to guide honest research designs. ", "questions": "1. As the authors stated in the paper, someone would question the fact that search space is also a choice. I agree with the explanations given by the author, however, I think the choice of kernel (beyond the two in the paper) and the prior (e.g. $\\Sigma$) would also make a difference, especially when the budget is limited as we cannot run many steps. Do you have results showing robustness of your conclusion to these choices.\n2. I am not sure what the \"appropriate stopping criterion\" in step 4 is. Is it just a visualization as stated in line 82? ", "limitations": "The authors addressed their limitations and negative societal impact. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The problem itself is really important and the authors put a lot of effort in reproducibility, providing the code for running the experiments. Also, the presentation is very clear and the three examples chosen clearly convey the message the authors want to share. \n\nWeaknesses: My main concern is whether this is just a straightforward application of Bayesian Optimization in parameter tuning when the parameters are replaced by different research designs in this setup. Some other concerns are listed in the questions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept