Title: Generalizability of experimental studies
Venue: NeurIPS 2024
Accepted: no
Keywords: Generalizability

Replicability

Experiment

Experimental Study

Benchmark
Abstract: Experimental studies are a cornerstone of machine learning (ML) research. A common, but often implicit, assumption is that the results of a study will generalize beyond the study itself, e.g. to new data. That is, there is a high probability that repeating the study under different conditions will yield similar results. Despite the importance of the concept, the problem of measuring generalizability remains open. This is probably due to the lack of a mathematical formalization of experimental studies. In this paper, we propose such a formalization and develop a quantifiable notion of generalizability. This notion allows to explore the generalizability of existing studies and to estimate the number of experiments needed to achieve the generalizability of new studies. To demonstrate its usefulness, we apply it to two recently published benchmarks to discern generalizable and non-generalizable results. We also publish a Python module that allows our analysis to be repeated for other experimental studies.

Abstract

Experimental studies are a cornerstone of machine learning (ML) research. A common, but often implicit, assumption is that the results of a study will generalize beyond the study itself, e.g. to new data. That is, there is a high probability that repeating the study under different conditions will yield similar results. Despite the importance of the concept, the problem of measuring generalizability remains open. This is probably due to the lack of a mathematical formalization of experimental studies. In this paper, we propose such a formalization and develop a quantifiable notion of generalizability. This notion allows to explore the generalizability of existing studies and to estimate the number of experiments needed to achieve the generalizability of new studies. To demonstrate its usefulness, we apply it to two recently published benchmarks to discern generalizable and non-generalizable results. We also publish a Python module that allows our analysis to be repeated for other experimental studies.

1 Introduction

Due to the importance of experimental studies, the machine learning (ML) community advocates for 15 high methodological standards [20, 12, 13, 17, 8, 31, 32, 44]. Failure to meet these standards can 16 have significant consequences, such as the ongoing reproducibility crisis [6, 47, 50, 51, 30]. 17

18

19

20

21

22

23

24

25

26

27

Reproducibility is not the only desirable property of a study. For example, the reader expects that the best encoders of categorical features identified in [41] will not only remain the best when the study is reproduced, but will also outperform their competitors on new datasets. This property of getting the same results from different data is known as replicability [46, 48]. Replicability is a special case of generalizability , the property of obtaining the same results with any change in the inputs. The assumption of generalizability is arguably the main motivation for extensive experimental studies and benchmarks. However, existing definitions of generalizability do not quantify how well the results of a study can be transferred to other contexts. This hinders the usefulness of such studies and leads to confusion. For example, articles [38, 41, 49, 42] and [19, 8, 11, 13, 29, 43] report that the results of experimental studies are often contradictory.

28

29

30

31

32

Quantifying generalizability can also help determine the appropriate size of experimental studies. For example, one dataset is unlikely to be sufficient to draw far-reaching conclusions, but 10 6 datasets are likely enough. Of course, such large studies are usually not practical: it is crucial to determine the minimum amount of data needed to achieve generalizability. This principle also applies to decisions other than the number of datasets, such as the choice of quality metric and the initialization seed.

A notion similar to generalizability is model replicability [1, 22, 23, 24, 33, 36, 37]. A model is 33 ρ -replicable if, given i.i.d. samples from the same data distribution, the trained models are the same 34 with probability 1 -ρ [33]. Adapting this definition to quantify generalizability is not trivial, as 35 it requires formalizing experimental studies. The latter must take into account several aspects: the 36 research question, the results of a study, and how to compare the results. Regarding the problem of 37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

defining the size of experimental studies, the current literature addresses the (crucial, but orthogonal) problem of choosing appropriate experimental factors [20, 12, 13, 17, 8, 31, 32, 44]. While these studies recommend varying the factors, they do not help decide how many of the factor levels are enough.

Our contributions are as follows:

we formalize experimental studies and their results;

we develop an algorithm to estimate the size of a study to obtain generalizable results;

we propose a quantifiable definition of the generalizability of experimental studies;

we consider two recent experimental studies on categorical encoders [41] and Large Language Models [55] and show how their results may or may not be generalizable.

we will publish the GENEXPY 1 Python module to repeat our analysis in other studies.

Paper outline: Section 2 is related work, Section 3 formalizes experimental studies, Section 4 defines generalizability and provides the algorithm to estimate the required size of a study for generalizability, Section 5 contains the case studies, Section 6 describes limitations and concludes.

2 Related work

We first discuss the literature related to the motivation we are tackling, i.e., why experimental studies may not generalize. Second, we overview the existing concept of model replicability, closely related to our work. Finally, we show other meanings that these words can assume in other domains.

Non-generalizable results. It is well known that experimental results can significantly vary based on design choices [38, 41, 49, 42]. Possible reasons include an insufficient number of datasets [19, 41, 3, 12] as well as differences in hyperparameter tuning [13, 41], initialization seed [30], and hardware [56]. As a result, the statistical benchmarking literature advocates for experimenters to motivate their design choices [7, 43, 11, 13, 44] and clearly state the conclusions they are attempting to draw from their study [7, 45].

Replicability and generalizability in ML. Our work formalizes the definitions of replicability and generalizability given in [48, 46]. Intuitively, replicable work consists of repeating an experiment on different data, while generalizable work varies other factors as well - e.g., quality metric, implementation. A recent line of work, initiated by [33], has linked replicability to model stability: a ρ -replicable model learns (with probability 1 -ρ ) the same parameters from different i.i.d. samples. This definition has later been adapted and applied to other learning algorithms [23], clustering [24], reinforcement learning [22, 37], convex optimization [1], and learning rules [36]. Recent efforts have been bridging the gap between replicability, differential privacy, generalization error, and global stability [15, 16, 26, 45, 21]. However, these applications remain limited to model replicability.

Replicability and generalizability in Science. In other fields of Science, generalizability and replicability take different meanings. In social sciences, generalizability theory is a tool to quantify the effect of different factors on numerical responses [14]. In medicine, the replicability proposed in [34] is the probability of observing a positive treatment effect in a meta-study. Although these concepts are related to generalizability of experimental studies, they are limited to purely numerical responses or specific study designs.

3 Experiments and experimental studies

An experimental study is a set of experiments comparing the same alternatives under different 78 experimental conditions . An experimental condition is a tuple of levels of experimental factors , the 79 parameters defining the experiments. Different factors play different roles in the study: the design 80 and held-constant factors are fixed by design, while the generalizability of a study is defined in terms 81 of the allowed-to-vary factors. The study aims at answering a research question , which defines its 82

Universe of valid

experimental conditions

C

LLM1

LLM2

LLM3

Alternatives

A

pos

1

pos

2

pos

3

pos

4

pos

5

pos

6

pos

7

pos

8

S

S

E

(pos

5

), E(pos

7

)

,

E(pos

8

)

E

(pos

3

), E(pos

4

)

,

E(pos

6

)

Results of

experiments

Study

Similar?

Generalizability

Example 3.1 . (The 'checkmate-in-one' task, cf. Figure 1) An experimenter wants to compare three 84

The assignment is to find the unique checkmating move from a position of pieces on a chessboard: an 86

Large Language Models (LLMs), the alternatives , on the 'checkmate-in-one' task [55, 2, 5, 4, 18]. 85

LLMsucceeds if and only if it outputs the correct move. The experimenter considers two experimental 87

88

90

91

factors

: the number of shots, find if LLM

1

n

, and the initial position on the chessboard, pos

l

. The number of shots ranks consistently against the other two LLMs when changing the initial position, for a

fixed number of shots.

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

3.1 Experiments

An experiment evaluates all the considered alternatives under a valid experimental condition .

Alternatives. An alternative a ∈ A is an object compared in the study, like an LLM in Example 3.1. Here, A is the set of alternatives considered in the study, with cardinality n a .

Experimental factors. An experimental factor is anything that could, in principle, affect the result of an experiment. i denotes a factor, C i the (possibly infinite) set of levels i can take, c ∈ C i a level of i , and I the set of all factors. We adapt Montgomery's classification of experimental factors [44, Chapter 1] and discern between design factors , held-constant factors , and allowed-to-vary factors .

Design factors , e.g., whether and how to tune the hyperparameters, quality metrics, number of shots, are chosen by the experimenter.

Allowed-to-vary factors , e.g., 'dataset' or 'chessboard position' in Example 3.1, may affect the outcome but cannot be held constant: the experimenter expects results to generalize w.r.t. these factors; I atv denotes them.

Held-constant factors , e.g., implementation, initialization seed, number of cross-validated folds, may affect the outcome but are not in the scope of the experiment and are fixed by the experimenter.

Experimental conditions. An experimental condition c is a tuple of levels of experimental factors, c = ( c i ) i ∈ I ∈ C ⊆ ∏ i ∈ I C i . We endow C with a probability µ , as we will need to sample from it to define the result of a study in Section ?? . The probability space ( C, F , µ ) is the universe of valid experimental conditions . C may not coincide with ∏ i ∈ I C i as some experimental conditions may be invalid , i.e., illegal or not of interest. Validity has to be assessed on a case-by-case basis. For instance, in Example 3.1, C = { ( pos l , n ) } l,n , where pos l is a legal configuration of pieces on a chessboard and m is the non-negative number of shots.

Experimental results. The experiment function E evaluates the alternatives A under a valid experimental condition c ∈ C . Unless necessary, we consider A fixed and omit it in our notation. We require that E : C →R n a is a measurable function, for some fixed A . Finally, the result of an experiment E ( A, c ) is a ranking on A .

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

Definition 3.1 (Ranking (with ties)) . Aranking r on A is a transitive and reflexive binary endorelation on A . Equivalently, r is a totally ordered partition of A into tiers of equivalent alternatives. r ( a ) denotes the rank of a ∈ A , i.e., the position of the tier of a in the ordering. W.l.o.g. ( R n a , P ( R n a )) denotes the measure space of all rankings of n a objects, where P indicates the power set.

Example 3.1 (Continued) . The result of an experiment on ( pos l , n ) is a ranking of the three LLMs, according to whether or not they output the checkmating move. Suppose that only LLM 1 and LLM 2 output the correct move. Then E ( pos l , n ) ranks LLM 1 and LLM 2 tied as best and LLM 3 as worst.

3.2 Experimental studies

A study is defined by its research question Q , i.e., its scope and goals . The scope consists of the alternatives A , the valid experimental conditions C , and the allowed-to-vary factors I atv . The goal is the kind of conclusions one is attempting to draw from the study. For now, the goal is a statement of interests, i.e., a set of strings.

Definition 3.2 (Research question) . The research question Q = ( A,C,I atv , goals ) is a tuple containing the set of alternatives A , the experimental conditions C , the set of allowed-to-vary-factors I atv , and the goals of the study.

Example 3.1 (Continued) . The research question of the 'checkmate-in-one' study is as follows. The scope is ( A = { LLM a } a =1 , 2 , 3 , C = { ( pos l , n ) } l,n , I atv = { 'position' } ) ) . The goal is 'Does LLM 1 rank consistently against the other LLMs?'

A crucial element of our formalization is the distinction between ideal and empirical studies. An ideal study exhausts its research question; however, its result is not observable. An empirical study is an observable sample of an ideal study.

3.2.1 Ideal studies

The ideal study on a research question Q = ( A,C,I atv , goals ) is the experimental study consisting of an experiment for each valid experimental condition c ∈ C . We say that such a study exhausts Q . Hence, there exists exactly one ideal study on Q . The result of an ideal study is the probability distribution of the results of its experiments. Recall that the experiment function E : ( C, F , µ ) → ( R n a , P ( R n a )) is measurable.

Definition 3.3 (Result of an ideal study) . The result of an ideal study with research question Q = ( A,C,I atv , goals ) is

where E -1 ( r ) = { c : E ( c ) = r } ⊆ C is the preimage of r through E .

In general, multiple experiments of a study may yield identical results. Definition 3.3 supports this by assigning a higher probability mass to results that occur more often.

3.2.2 Empirical studies

Consider again a research question Q = ( A,C,I atv , goals ) . In practice, as C might be infinite or too large, one can only run experiments on a sample of valid experimental conditions { c j } N j =1 iid ∼ ( C, µ ) .

The study performed on { c j } N j =1 is an empirical study on Q , of size N . As for ideal studies, the result of an empirical study is the probability distribution of the results of its experiments.

Definition 3.4 (Result of an empirical study) . The result of an empirical study on Q is

Where Q , { c j } N j =1 is a research question and a set of valid experimental conditions as above. 158

The result of an empirical study can be thought of as the empirical distribution of a sample following 159 the distribution of the result of the corresponding ideal study. With a slight abuse of notation, 160 indicating both the sample and its empirical distribution as ˆ S ( ) , we write 161

4 Generalizability of experimental studies

The currently accepted definition of generalizability is the property of two independent studies with the same research question to yield similar results [46, 48]. Although intuitive, this notion is not directly applicable as it does not provide a way to measure the generalizability of a study. We now introduce a quantifiable notion of generalizability of experimental studies, as the probability that any two empirical studies approximating the same ideal study yield similar results.

Definition 4.1 (Generalizability) . Let Q = ( A,C,I atv , κ ) be the research question of an ideal study, let P = S ( Q ) be the result of that study, and let d be some distance between probability distributions. The generalizability of the ideal study on Q is

where ε ∈ R + is a similarity threshold .

As the result of an ideal study is usually unobservable (cf. Section 3.2), we do not know the true distribution P . However, we can observe the result of an empirical study, ˆ P N = ˆ S N ( Q ) , which approximates P under the assumption that the experimental conditions are i.i.d. samples from C . As the sample size N increases (the empirical study becomes larger), ˆ P N converges in distribution to P .

Definition (1) requires a distance d between probability distributions. In the next sections, we propose to use a generalizability based on kernels and Maximum Mean Discrepancy (MMD) [27], as it allows to compute generalizability w.r.t. different research questions. The underlying idea is that we can capture the goal of a study with an appropriate kernel. We conclude this section with an algorithm to estimate the number of experimental conditions required to obtain generalizable results.

4.1 Similarity between rankings - kernels

Whether two experimental results (i.e., rankings) are similar or not ultimately depends on the goal of the study. For instance, consider two rankings on A = { a 1 , a 2 , a 3 } , r = (1 , 2 , 3) and r ′ = (1 , 3 , 2) , where r i is the tier of alternative a i . The conclusions drawn from r and r ′ are identical if one's goal is to find the best alternative, but very different if one's goal is to obtain an ordering of the alternatives. One can use kernels to quantify the similarity between experimental results. Kernels are suitable to formalize the aspects of the result of a study one wants to generalize, i.e., the goals of the study. For instance, one kernel is suitable to identify the best tier while another kernel focuses on the position of a specific alternative. In the following, we describe three representative kernels that cover a wide spectrum of possible goals.

Borda kernel. The Borda kernel is suitable for goals in the form 'Is the alternative a ∗ consistently ranked the same?'. It uses the Borda count: the number of alternatives (weakly) dominated by a given one [9]. For a pair of rankings, we compute the Borda counts of a ∗ , and then take their difference.

where b l = { a ∈ A : r l ( a ) ≥ r l ( a ∗ ) } is the number of alternatives dominated by a ∗ in r l and ν ∈ R is the kernel bandwidth. The Borda kernel takes values in [ e ( -νn a ) , 1 ] . If ν is too large compared to 1 / | b 1 -b 2 | , the kernel is oversensitive and will penalize every deviation too much. On the contrary, if ν is too small, the kernel is undersensitive and will not penalize deviations unless they are very large. As | b 1 -b 2 | ∈ [0 , n a ] , we recommend ν = 1 / n a .

Jaccard kernel. The Jaccard kernel is suitable for goals in the form 'Are the best alternatives consistently the same ones?'. As it measures the similarity between sets [25, 10], we use it to compare

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

N Q

the topk tiers of two rankings. 201

202

203

where r -1 ([ k ]) = { a ∈ A : r 1 ( a ) ≤ k } is the set of alternatives whose rank is better than or equal to k . The Jaccard kernel takes values in [0 , 1] .

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

Mallows kernel. The Mallows kernel is suitable for goals in the form 'Are the alternatives ranked consistently?'. It measures the overall similarity between rankings [35, 40, 39]. We adapt the original definition in [39] for ties,

where n d = ∑ a 1 ,a 2 ∈ A | sign ( r 1 ( a 1 ) -r 1 ( a 2 )) -sign ( r 2 ( a 1 ) -r 2 ( a 2 )) | is the number of discordant pairs and ν ∈ R is the kernel bandwidth. If a pair is tied in one ranking but not in the other, one counts it as half a discordant pair. The Mallows kernel takes values in [ exp ( -2 ν ( n a 2 )) , 1 ] . If ν is too large compared to 1 / n d , the kernel is oversensitive and it will penalize every deviation too much. On the contrary, if ν is too small, the kernel is undersensitive and will not penalize deviations unless they are very large. As n d ∈ [ 0 , ( n a 2 )] , we recommend ν = 1 / ( na 2 ) .

4.2 Distance between distributions - Maximum Mean Discrepancy

Having sorted out how to measure the similarity between the results of experiments, we now discuss how to measure the distance between the results of studies. We chose the Maximum Mean Discrepancy (MMD) [27], for the following reasons. First, MMD is compatible with the kernels described in Section 4.1, i.e., it takes into consideration the goal of the studies. Second, it handles sparse distributions well; this is needed as empirical studies are typically small compared to the number of all possible rankings, which grows exponentially in the number of alternatives. 2 Finally, it comes with bounds and theoretical guarantees, which we will use in Section 4.3.

Definition 4.2 (MMD (empirical distributions)) . Let X be a set with a kernel κ , and let Q 1 and Q 2 be two probability distributions on R n a . Let x = ( x i ) n i =1 , y = ( y i ) m i =1 be two i.i.d. samples from Q 1 and Q 2 respectively. Then,

Proposition 4.1. MMDtakes values in [ 0 , √ 2 · ( κ sup -κ inf ) ] , where κ sup = sup x,y ∈ X κ ( x, y ) and κ inf = inf x,y ∈ X κ ( x, y ) .

4.3 How many experiments ensure generalizability?

When designing a study, an experimenter has to decide how many experiments to run in order to obtain generalizable results. In other words, they need to choose a (minimum) sample size n ∗ that achieves the desired generalizability α ∗ and the desired similarity ε ∗ .

To estimate n ∗ we make use of a linear dependency between the logarithms of the sample size n and the logarithm of the α ∗ -quantile of MMD ε α ∗ n that we have observed in our experiments.

Proposition 4.2. ∀ α ∗ , there exist β 0 ≥ 0 , β 1 ≤ 0 s.t.

Appendix A.3.2 provides a proof for a simplified case. Proposition 4.2 suggests that one can use a small set of N preliminary experiments to estimate n ∗ . One can then iteratively improve that estimate with the results of additional experiments.

Our algorithm, shown in detail in Appendix A.3.3, requires specifying the desired generalizability, α ∗ , and the similarity threshold between the studies results, ε ∗ . Then, it performs the following steps:

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

0.7

0.8

0.9

0.95

0.99

a

∗

10

20

30

40

n

∗

0.01

0.05

0.1

0.2

0.3

d

∗

g

1

g

2

g

3

it estimates the α ∗ -quantile of MMD for all n less than some budget n max . If there exists an n less than n max that satisfies the condition in (2), we return it as n ∗ ;

finally, it outputs n ∗ = exp ( β 1 log ( ε α ∗ n ) + β 0 ) , which satisfies the condition in (2) thanks to Proposition 4.2.

it then fits the linear model in (3), computing the coefficients β 0 and β 1 ;

In practice, choosing ε ∗ is hardly interpretable as it is a threshold on MMD. To solve this, we propose choosing ε ∗ as a function of another parameter δ ∗ , such that

Here, δ ∗ represents the distance between two rankings as computed by the kernel and f κ is the function linking the distance to the kernel value. For instance, for the Jaccard kernel, δ ∗ is simply the Jaccard coefficient between the topk tiers of two rankings, f κ ( δ ∗ ) = 1 -δ ∗ , and ε ∗ ( δ ∗ ) = √ 2(1 -(1 -δ ∗ )) . For the Mallows kernel (with our recommendation for ν ), δ ∗ is the fraction of discordant pairs, f κ ( x ) = e -x , and ε ∗ ( δ ∗ ) = √ 2(1 -e -δ ∗ ) . As a concrete example, achieving ( α ∗ = 0 . 99 , δ ∗ = 0 . 05) -generalizable results for the Jaccard kernel means that, with probability 0 . 99 , the average Jaccard coefficient between two rankings drawn from the results is 0 . 95 .

5 Case studies

5.1 Case Study 1: A benchmark of categorical encoders

We now evaluate the generalizability of a recent study [41] that analyzes the performance of encoders for categorical data. The performance of an encoder is approximated by the quality of a model trained on the encoded data. The design factors are the model, the tuning strategy for the pipeline, and the quality metric for the model, while the only allowed-to-vary factor is the dataset. We impute missing values in the results of the study by assigning the worst rank. We evaluate how well the results of the study generalize w.r.t. three goals:

( g 1 ) Find out if One-Hot encoder (a popular encoder) ranks consistently amongst its competitors, using the Borda kernel with ν = 1 / n a .

( g 3 ) Evaluate whether the encoders are typically ranked in a similar order, using the Mallows kernel with ν = 1 / ( na 2 ) .

( g 2 ) Investigate if some encoders outperform all the others using the Jaccard kernel with k = 1 .

Figure 2 shows the predicted n ∗ for different choices of α ∗ and δ ∗ , the other one fixed at 0 . 95 and 0 . 05 respectively. The variance in the boxes comes from variance in the design factors. For example, the results for the design factors 'decision tree, full tuning, accuracy' have a different ( α ∗ , δ ∗ ) -generalizability than the results for 'SVM, no tuning, accuracy'. We observe on the left that - as expected - obtaining generalizable results requires more experiments as the desired

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

0.7

0.8

0.9

0.95

0.99

a

∗

0

20

40

60

n

∗

0.01

0.05

0.1

0.2

0.3

d

∗

g

1

g

2

g

3

generalizability α ∗ increases. We can also see that the variance of the boxes increases with α ∗ . This means that the choice of the design factors has a larger influence on the achieved generalizability. We observe the same when decreasing δ ∗ , as it corresponds to a stricter similarity condition on the rankings. In the rather extreme cases of α ∗ = 0 . 7 or δ ∗ = 0 . 3 , even less than 10 datasets are enough to achieve ( α ∗ , δ ∗ ) -generalizability.

Consider now goal g 2 for two different choices of design factors: (A): 'decision tree, full tuning, accuracy' and (B): 'SVM, full tuning, balanced accuracy'. Furthermore, let ( α ∗ , δ ∗ ) = (0 . 95 , 0 . 05) : we estimate n ∗ = 28 for (A) and n ∗ = 34 for (B), corresponding to the bottom and top whiskers of the corresponding box in Figure 2. As both (A) and (B) were evaluated using n = 30 experiments, we conclude that the results of (A) are (barely) (0 . 95 , 0 . 05) -generalizable, while those of (B) are not. Hence, one should run more experiments with fixed factors (B) to make the study generalizable.

5.2 Case study 2: BIG-bench - A benchmark of Large Language Models

We now evaluate the generalizability of BIG-bench [55], a collaborative benchmark of Large Language Models (LLMs). The benchmark compares LLMs on different tasks, such as the checkmatein-one task (cf. Example 3.1), and for different numbers of shots. Task and number of shots are the design factors . Every task has a number of subtasks, which is the allowed-to-vary factor . We stick to the preferred scoring for each subtask. As the results have too many missing values to impute them, we only consider the experimental conditions where at least 80% of the LLMs had results, and to the LLMs whose results cover at least 80% of the conditions.

Similar to before, we define the three goals as follows:

( g 1 ) Find out if GPT3 (to date, one of the most popular LLMs) ranks consistently amongst its competitors, using the Borda kernel with ν = 1 / n a .

( g 3 ) Evaluate whether the LLMs are typically ranked in a similar order, using the Mallows kernel with ν = 1 / ( na 2 ) .

( g 2 ) Investigate if some encoders outperform all the others using the Jaccard kernel with k = 1 .

Figure 3 shows the predicted n ∗ for different choices of α ∗ and δ ∗ , the other one fixed at 0 . 95 and 0 . 05 respectively. Again, the variance in the boxes comes from variance in the design factors, i.e., the task and the number of shots. As before, increasing α ∗ or decreasing δ ∗ leads to higher n ∗ . Unlike in the previous section, n ∗ for g 2 greatly depends on the combination of fixed factors, as we now detail.

Consider now goal g 2 for two different choices of design factors: (A): 'conlang_translation, 0 shots', and (B): 'arithmetic, 2 shots'. Furthermore, let ( α ∗ , δ ∗ ) = (0 . 95 , 0 . 05) . For this choice of of parameters, we estimate n ∗ = 44 for (A), corresponding to the top whisker of the corresponding box in Figure 2. As the study evaluates (A) on 10 subtasks, it is therefore not (0 . 95 , 0 . 05) -generalizable. In fact, we estimate that this would require 34 more subtasks. For (B), on the other hand, we estimate n ∗ = 1 : the best 2-shot LLM for the observed subtasks is always PALM 535B. Hence, the result of a single experiment is enough to achieve (0 . 95 , 0 . 05) -generalizability.

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

10

20

30

40

N

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

relative error

(a) Categorical encoders

10

20

30

40

N

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

relative error

g

1

g

2

g

3

(b) LLMs

Note that, although we correctly estimated n ∗ = 1 for (B), this estimate relies on 10 preliminary experiments. In other words, our algorithm was able to quantify in hindsight that a single experiment would have been enough to obtain generalizable results. Of course, however, one cannot trust an estimate of n ∗ based on only one experiment. The next section thus investigates how the number of preliminary experiments influences the estimate of n ∗ .

5.3 How many preliminary experiments?

This section evaluates the influence of the number of preliminary experiments N on n ∗ . For each study, we consider the design factor combinations for which we have at least 50 experiments. This results in 23 out of 48 combinations for the categorical encoders and 9 out of 24 combinations for the LLMs. For each of those combinations, we consider the estimate n ∗ 50 made at N = 50 as the ground truth and observe how the estimates of n ∗ for N < 50 differ. Figure 4 shows the relative error | n ∗ N -n ∗ 50 | / n ∗ 50 , for different goals: the relative errors behave very differently. For goal g 3 (Mallows kernel), even n ∗ 10 is close to n ∗ 50 for a majority of the design factor combinations. On the contrary, one needs 20 to 30 preliminary experiments for goal g 1 (Borda kernel). This means that knowing the goals of a study when performing preliminary experiments can help understand how trustworthy the estimate of n ∗ is.

6 Conclusion

Limitations & future work. First, we dealt with experimental results as rankings. Other forms of results, e.g., the absolute performance of alternatives according to some quality measure, will require the development of appropriate kernels. Second, our approach uses kernels to compute the similarity of experimental results and MMD the distance between the results of studies. There are, however. other possible choices. Third, we processed missing evaluations by either dropping them or imputing them. One could analyze different solutions, for instance by adapting the kernels to missing values. Fourth, we estimate the distribution of the MMD by sampling multiple times from the results. A non-asymptotic theory of MMD, at least for some kernels, might yield more insights in improving this procedure. Fifth, we plan to investigate the possibility of actively selecting experiments to obtain good estimates of the required size n ∗ with less preliminary experiments. Sixth and related to the previous one, we intend to obtain some guarantees on the convergence of n ∗ to the true value.

Conclusions. An experimental study is generalizable if, with high probability, its findings will hold under different experimental conditions, e.g., on unseen datasets. Non-generalizable studies might be of limited use or even misleading. This study is, to our knowledge, the first to develop a quantifiable notion for the generalizability of experimental studies. To achieve this, we formalize experiments, experimental studies and their results - rankings and distributions over rankings. Our approach allows us to estimate the number of experiments needed to achieve a desired level of generalizability in new experimental studies. We demonstrate its utility showing generalizable and non-generalizable results in two recent experimental studies.

Acknowledgments 342

343

...

References 344

Kwangjun Ahn et al. 'Reproducibility in Optimization: Theoretical Framework and Limits'. 345 In: NeurIPS . 2022. 346

Scott Alexander. 'A very unlikely chess game, 2020'. In: URL https://slatestarcodex. 347 com/2020/01/06/a-very-unlikely-chessgame/.(cited on pp. 29 and 30) (). 348

Maxime Alvarez et al. 'A Revealing Large-Scale Evaluation of Unsupervised Anomaly 349 Detection Algorithms'. In: CoRR abs/2204.09825 (2022). 350

Prithviraj Ammanabrolu et al. 'Bringing stories alive: Generating interactive fiction worlds'. 351 In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital 352 Entertainment . Vol. 16. 1. 2020, pp. 3-9. 353

Prithviraj Ammanabrolu et al. 'Toward automated quest generation in text-adventure games'. 354 In: arXiv preprint arXiv:1909.06283 (2019). 355

Monya Baker. '1,500 scientists lift the lid on reproducibility'. In: Nature 533.7604 (2016). 356

Thomas Bartz-Beielstein et al. 'Benchmarking in Optimization: Best Practice and Open 357 Issues'. In: CoRR abs/2007.03488 (2020). 358

Alessio Benavoli et al. 'Time for a Change: a Tutorial for Comparing Multiple Classifiers 359 Through Bayesian Analysis'. In: J. Mach. Learn. Res. 18 (2017), 77:1-77:36. 360

JC de Borda. 'M'emoire sur les' elections au scrutin'. In: Histoire de l'Acad'emie Royale des 361 Sciences (1781). 362

Mathieu Bouchard, Anne-Laure Jousselme, and Pierre-Emmanuel Doré. 'A proof for the 363 positive definiteness of the Jaccard index matrix'. In: International Journal of Approximate 364 Reasoning 54.5 (2013), pp. 615-626. 365

Anne-Laure Boulesteix, Rory Wilson, and Alexander Hapfelmeier. 'Towards evidence-based 366 computational statistics: lessons from clinical research on the role and design of real-data 367 benchmark studies'. In: BMC Medical Research Methodology 17 (2017), pp. 1-12. 368

Anne-Laure Boulesteix et al. 'A statistical framework for hypothesis testing in real data 369 comparison studies'. In: The American Statistician 69.3 (2015), pp. 201-212. 370

Xavier Bouthillier et al. 'Accounting for Variance in Machine Learning Benchmarks'. In: 371 MLSys . mlsys.org, 2021. 372

Robert L Brennan. 'Generalizability theory'. In: Educational Measurement: Issues and Prac373 tice 11.4 (1992), pp. 27-34. 374

Mark Bun et al. 'Stability Is Stable: Connections between Replicability, Privacy, and Adaptive 375 Generalization'. In: STOC . ACM, 2023, pp. 520-527. 376

Zachary Chase, Shay Moran, and Amir Yehudayoff. 'Stability and Replicability in Learning'. 377 In: FOCS . IEEE, 2023, pp. 2430-2439. 378

Giorgio Corani et al. 'Statistical comparison of classifiers through Bayesian hierarchical 379 modelling'. In: Mach. Learn. 106.11 (2017), pp. 1817-1837. 380

Sahith Dambekodi et al. 'Playing text-based games with common sense'. In: arXiv preprint 381 arXiv:2012.02757 (2020). 382

Mostafa Dehghani et al. 'The Benchmark Lottery'. In: CoRR abs/2107.07002 (2021). 383

Janez Demsar. 'Statistical Comparisons of Classifiers over Multiple Data Sets'. In: J. Mach. 384 Learn. Res. 7 (2006), pp. 1-30. 385

Peter Dixon et al. 'List and Certificate Complexities in Replicable Learning'. In: NeurIPS . 386 2023. 387

Eric Eaton et al. 'Replicable Reinforcement Learning'. In: NeurIPS . 2023. 388

Hossein Esfandiari et al. 'Replicable Bandits'. In: ICLR . OpenReview.net, 2023. 389

Hossein Esfandiari et al. 'Replicable Clustering'. In: NeurIPS . 2023. 390

391

392

Thomas Gärtner, Quoc Viet Le, and Alex J Smola. 'A short tour of kernel methods for graphs'. In: Under Preparation (2006).

Badih Ghazi et al. 'On User-Level Private Convex Optimization'. In: ICML . Vol. 202. Pro393 ceedings of Machine Learning Research. PMLR, 2023, pp. 11283-11299. 394

Arthur Gretton et al. 'A Kernel Method for the Two-Sample-Problem'. In: NIPS . MIT Press, 395 2006, pp. 513-520. 396

Arthur Gretton et al. 'A Kernel Two-Sample Test'. In: J. Mach. Learn. Res. 13 (2012), pp. 723397 773. 398

Odd Erik Gundersen, Kevin L. Coakley, and Christine R. Kirkpatrick. 'Sources of Irrepro399 ducibility in Machine Learning: A Review'. In: CoRR abs/2204.07610 (2022). 400

Odd Erik Gundersen et al. 'On Reporting Robust and Trustworthy Conclusions from Model 401 Comparison Studies Involving Neural Networks and Randomness'. In: ACM-REP . ACM, 402 2023, pp. 37-61. 403

Torsten Hothorn et al. 'The design and analysis of benchmark experiments'. In: Journal of 404 Computational and Graphical Statistics 14.3 (2005), pp. 675-699. 405

Karl Huppler. 'The Art of Building a Good Benchmark'. In: TPCTC . Vol. 5895. Lecture Notes 406 in Computer Science. Springer, 2009, pp. 18-30. 407

Russell Impagliazzo et al. 'Reproducibility in learning'. In: STOC . ACM, 2022, pp. 818-831. 408

Iman Jaljuli et al. 'Quantifying replicability and consistency in systematic reviews'. In: 409 Statistics in Biopharmaceutical Research 15.2 (2023), pp. 372-385. 410

Yunlong Jiao and Jean-Philippe Vert. 'The Kendall and Mallows Kernels for Permutations'. 411 In: IEEE Trans. Pattern Anal. Mach. Intell. 40.7 (2018), pp. 1755-1769. 412

Alkis Kalavasis et al. 'Statistical Indistinguishability of Learning Algorithms'. In: ICML . 413 Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 15586-15622. 414

Amin Karbasi et al. 'Replicability in Reinforcement Learning'. In: NeurIPS . 2023. 415

Fred Lu, Edward Raff, and James Holt. 'A Coreset Learning Reality Check'. In: AAAI . AAAI 416 Press, 2023, pp. 8940-8948. 417

Colin L Mallows. 'Non-null ranking models. I'. In: Biometrika 44.1/2 (1957), pp. 114-130. 418

Horia Mania et al. 'On kernel methods for covariates that are rankings'. In: (2018). 419

Federico Matteucci, Vadim Arzamasov, and Klemens Böhm. 'A benchmark of categorical 420 encoders for binary classification'. In: NeurIPS . 2023. 421

Duncan C. McElfresh et al. 'On the Generalizability and Predictability of Recommender 422 Systems'. In: NeurIPS . 2022. 423

Iven Van Mechelen et al. 'A white paper on good research practices in benchmarking: The 424 case of cluster analysis'. In: WIREs Data. Mining. Knowl. Discov. 13.6 (2023). 425

Douglas C Montgomery. Design and analysis of experiments . John wiley & sons, 2017. 426

Shay Moran, Hilla Schefler, and Jonathan Shafer. 'The Bayesian Stability Zoo'. In: NeurIPS . 427 2023. 428

Engineering National Academies of Sciences, Medicine, et al. Reproducibility and replicability 429 in science . National Academies Press, 2019. 430

Roger D Peng. 'Reproducible research in computational science'. In: Science 334.6060 (2011), 431 pp. 1226-1227. 432

Joelle Pineau et al. 'Improving Reproducibility in Machine Learning Research(A Report from 433 the NeurIPS 2019 Reproducibility Program)'. In: J. Mach. Learn. Res. 22 (2021), 164:1434 164:20. 435

Zhen Qin et al. 'RD-Suite: A Benchmark for Ranking Distillation'. In: NeurIPS . 2023. 436

Edward Raff. 'Does the Market of Citations Reward Reproducible Work?' In: ACM-REP . 437 ACM, 2023, pp. 89-96. 438

Edward Raff. 'Research Reproducibility as a Survival Analysis'. In: AAAI . AAAI Press, 2021, 439 pp. 469-478. 440

Isaac J Schoenberg. 'Metric spaces and positive definite functions'. In: Transactions of the 441 American Mathematical Society 44.3 (1938), pp. 522-536. 442

Bernhard Schölkopf. 'The kernel trick for distances'. In: Advances in neural information 443 processing systems 13 (2000). 444

J Laurie Snell and John G Kemeny. 'Mathematical models in the social sciences'. In: (No 445 Title) (1962). 446

Aarohi Srivastava et al. 'Beyond the Imitation Game: Quantifying and extrapolating the 447 capabilities of language models'. In: CoRR abs/2206.04615 (2022). 448

Donglin Zhuang et al. 'Randomness in Neural Network Training: Characterizing the Impact 449 of Tooling'. In: MLSys . mlsys.org, 2022. 450

451

452

453

454

455

456

457

458

459

460

461

462

where d l = { a ∈ A : r l ( a ) ≥ r l ( a ∗ ) } is the number of alternatives dominated by a ∗ in r l and ν ∈ R .

Proposition A.1. The Borda kernel as defined in (4) is a kernel.

Proof. Define a distance where d l = { a ∈ A : r l ( a ) ≥ r l ( a ∗ ) } is the number of alternatives dominated by a ∗ in r l . Now, 463 ( R n a , d ) is isometric to ( R , ∥·∥ 2 ) via the map r l ↦→ d l . Hence, d is c.p.d. and κ b is a kernel. 464

Definition A.2 (Jaccard kernel) .

where r -1 ([ k ]) = { a ∈ A : r 1 ( a ) ≤ k } is the set of alternatives whose rank is better than or equal to 465 k . 466

467

468

469

470

471

472

473

474

475

Then, 476

where ∥·∥ p indicates the entry-wise matrix p -norm and the equality holds because the entries of the 477 matrices are either 0 or 1. As a consequence, ( R n a , n d ) is isometric to ( R n a × n a , ∥·∥ 2 ) via Φ . Hence, 478 n d is c.p.d. and κ m is a kernel. 479

Proposition A.2. The Jaccard kernel as defined in (5) is a kernel.

Proof. It is already know that the Jaccard coefficients for sets is a kernel [25, 10]. As the Jaccard kernel for rankings is equivalent to the Jaccard coefficient for the k -best tiers of said rankings, the former is also a kernel.

Definition A.3 (Mallows kernel) .

A Details for Section 4

A.1 Details for Section 4.1

This section contains the proofs to show that the similarities introduced in Section 4.1 are kernels, i.e., symmetric and positive definite functions. As symmetry is a clear property of all of them, we only discuss their positive definiteness. Our proofs for the Borda and Mallows kernels follow that in [35]: we define a distance d on the set of rankings R n a and show that ( R n a , d ) is isometric to an L 2 space. This ensures that d is a conditionally positive definite (c.p.d.) function and, thus, that e -νd is positive definite [52, 53]. Our proof for the Jaccard kernel, instead, follows without much effort from previous results. For ease of reading, we restate the definitions as well.

Definition A.1 (Borda kernel) .

where n d = ∑ a 1 ,a 2 ∈ A | sign ( r 1 ( a 1 ) -r 1 ( a 2 )) -sign ( r 2 ( a 1 ) -r 2 ( a 2 )) | is the number of discordant pairs and ν ∈ R is the kernel bandwidth.

Proposition A.3. The Mallows kernel as defined in (6) is a kernel.

Proof. The number of discordant pairs n d is a distance on R n a [54]. Consider now the mapping of a ranking into its adjacency matrix,

A.2 Details for Section 4.2 480

Proof.

A.3 Details for Section 4.3

A.3.1 Choice of α ∗ , ε ∗ , and δ ∗

Consider a research question Q = ( A,C,I atv , κ ) and the corresponding ideal study with result P . The algorithm introduced in Section 4.3 aims at finding the minimum n ∗ such that, given two independent empirical studies on Q , they achieve similar results. It has two hyperparameters, α ∗ and ε ∗ . α ∗ ∈ [0 , 1] is the generalizability that one wants to achieve from the study, i.e., the probability that two independent realizations of the same ideal study will yield similar results. ε ∗ ∈ R + is a similarity threshold: the results of two empirical studies x , y iid ∼ P are similar if MMD κ ( x , y ) ≤ ε ∗ . However, as it is, ε ∗ is not interpretable. Instead, adapting the proof of Proposition 4.1, we can bound MMDby imposing a condition on the kernel, as we'll now illustrate. The key remark is that we are looking for a condition in the form

where δ ′ ∈ [0 , κ sup ] replaces the third summatory in (7). In other terms, we can interpret δ ′ as the minimum acceptable value for the average of the kernel, E P 2 [ κ ( x, y )] . We now go a step further and compute δ ′ (a condition on the kernel) from δ ∗ ∈ [0 , 1] (a condition on the rankings). The relation between δ ′ and δ ∗ changes with the kernel, and so does the interpretation of δ ∗ . For the three kernels we discuss in Section 4.1:

Mallows kernel with ν = 1 / ( n 2 ) : δ ∗ is the fraction of discordant pairs, δ ′ = e -δ ∗ .

Jaccard kernel : δ ∗ is the intersection over union of the top k tiers, δ ′ = 1 -δ ∗ .

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

Borda kernel with ν = 1 / n a : δ ∗ is the difference in relative position of a ∗ in the rankings, normalized to the length of the rankings, δ ′ = e -δ ∗

A.3.2 Proof of proposition 4.2 504

Proposition 4.2. ∀ α ∗ , there exist β 0 ≥ 0 , β 1 ≤ 0 s.t. 505

Proof. We provide a proof replacing MMD with the distribution-free bound defined in [28]. 506

where: 507

508

509

(1)

ε

′

=

ε

+

√

2

κ

sup

/

n

.

(3) Take the complementary event. 510

Now, 511

concluding the proof. 512

Remark. Altohugh theoretically sound, using the abovementioned bound instead of MMD leads to excessively conservative estimates for n ∗ , roughly one order of magnitude greater than the empirical estimate.

A.3.3 Pseudocode for the algorithm 516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In order, our claims are: the formalization in Section 3; the definition generalizability in Section 4; the algorithm for study size in Section 4.3, the case studies in Section 5, and we provide a link to the anonymized GitHub repository for the module.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs are in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: On GitHub.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: On GitHub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The boxplots in Section 5 show the variability for the choice of fixed factors. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The analysis we showcase in Section 5 executes very fast, requiring in total less than 4 hours on a standard office laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: To the best of our knowledge, we referenced all sources in the appropriate way. Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our Python module is documented on GitHub.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to formalize the concept of generalizability in experimental studies in machine learning research. It relies on three different types of kernels in order to quantify difference between the rankings in an experiment output. A core contribution is the development of an algorithm for estimating the number of experimental studies needed in order to generalize the results at a desired level.\n\nSTRENGTHS:\n- The topic is interesting and worthwhile.\n- The paper is clearly written.\n- The formalization of generalizability is well-defined and nicely parameterized through the use of kernels.\n- The practical usefulness of the algorithm is somewhat unclear to me.\n\nWEAKNESSES:\n- There is no discussion on the computational costs of the algorithm (except for a vague statement that it is very fast in the checklist).\n- The empirical evidence for the algorithm's effectiveness appears somewhat weak to me (see respective item in _Questions_).\n- The python package is not properly configured I think. I see this after having installed the package into a virtual environment with the correct python version and\n  using `pip install . -r requirements.txt`:\n\n  ```python\n  In [1]: import genexpy\n  ---------------------------------------------------------------------------\n  ImportError                               Traceback (most recent call last)\n  Cell In[1], line 1\n  ----> 1 import genexpy\n\n  File ~/.pyenv/versions/genexpy/lib/python3.11/site-packages/genexpy/__init__.py:4\n        2 from .src import lower_bounds\n        3 from .src import mmd\n  ----> 4 from .src import probability_distributions\n        5 from .src import rankings_utils\n        6 from .src import relation_utils\n\n  File ~/.pyenv/versions/genexpy/lib/python3.11/site-packages/genexpy/src/probability_distributions.py\n  :11\n        8 from typing import Literal\n      10 from genexpy import kernels as ku\n  ---> 11 from genexpy import rankings_utils as ru\n      12 from genexpy import relation_utils as rlu\n      15 def sample_from_sphere(na: int, n: int, rng: np.random.Generator) -> np.ndarray[float]:\n\n  ImportError: cannot import name 'rankings_utils' from partially initialized module 'genexpy' (most l\n  ikely due to a circular import) (/home/<anonymous_reviewer>/.pyenv/versions/genexpy/lib/python3.11/site-packages\n  /genexpy/__init__.py)\n\nQUESTIONS:\n### Questions\n\n- To me it feels like the evidence for why the algorithm works is somewhat circular. If I am not mistaken, the case studies in 5.1 and 5.2 just show the output of the algorithm, which is really just an example of what your method outputs and doesn't really help show how well your method performs. And as far as the result in section 5.3 goes, how can this can be generalized to other problems? As far as I am able to tell, there is no reason to assume that even $N=50$ should be sufficient for $g_1$ in the general case? How do I know how to trust $n^*$? Maybe I'm missing something here.\n\n### Suggestions\n\n- The captions for your figures are too brief. They should at least describe what is shown in the plots. Please be more detailed. At least describe the variables shown.\n- Put conclusions before limitations and future work in section 6.\n- Please include a discussion on the computational costs of the algorithm.\n- L19: It's hard to tell what this is a reference too. Be more explicit.\n- L46: I'm nitpicking, but large language models shouldn't be in title case, I think.\n- L53: Consider replacing \"motivation\" by \"problem\".\n- L60: Consider replacing \"conclusions\" by \"hypotheses\".\n- L101: This sentence is phrased oddly. Please revise it. Maybe \"are\" should be removed?\n- L111: Incorrect hyper-reference (??).\n- L215: Remove \"the\" in front of \"Maximum\".\n- L232: \"exist\" should be \"exists\".\n- L260: It should be \"the one-hot encoder\".\n- L513: \"Altohugh\" should be \"Although\".\n\nLIMITATIONS:\nSome of the limitations are discussed but I still think the paper could be more self-critical of for instance $n^*$. Possible computational costs are also not discussed.\n\nThere are no potential negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper deals with experimental studies. After providing a mathematical formalization, it focuses on the generalizability of these studies. The main contribution is a quantitative estimate of the the size of the study to obtain generalizable results. Experiments on LLMs are conducted.

STRENGTHS:
- [mathematical formulation] It is nice to have a solid formulation of experimental studies, this is quite relevant to the community.

WEAKNESSES:
- [train / test split] A concrete problem in machine learning practical experimentation is that of train / test split, and more particularly its absence (that is, training on the test). I do not see this issue discussed in the paper. Can it be incorporated in the setting? Is it possible to clarify whether the paper assumes that the training is done on a training set without calibration on a validation set or is this hidden somewhere? What would then be the influence on the number of experiments?
- [testing between rankings] If I understand correctly, the paper proposes (in Section 4.1) to check whether rankings are consistent by performing kernel two-sample test, with adapted kernels. This does not seem standard to me: there exists some ad-hoc statistical tests (e.g., Kendall's \tau, Spearman's \rho, etc.). Why not use them directly? Is there an advantage to using MMD?  

- [minor comments]:
  - missing ref line 111
  - repeated word ('of') line 300

QUESTIONS:
see weaknesses

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper tries to formalise the notion of an experimental study by considering the sampling process of acquiring a dataset.  It then uses this notion to argue about generalisability.

STRENGTHS:
The problem of understanding the performance of machine learning when tested on new data is a very important problem.  The authors use some technically sophisticate methods to tackle this problem.

WEAKNESSES:
For me the authors model of an experiment is too simplistic and does not capture the problems faced by machine learning.  If we collect medical data then that data is likely to vary depending on the equipment used, the clinicians running the equipment and population where the data comes from.  These kinds of variations are the bugbear for machine learning, but not captured at all by the model.  Another issue is that a lot of data is non-stationary.  Even in the much used example of checkmate in one.  If a machine learns this very well, then players against the machine are likely to learn their mistake and alter their play.  Thus, I am not convinced that the model being proposed is particularly interesting.

QUESTIONS:
What kind of variations in datasets does your model capture?
Where does this push the field beyond the current statistical learning theory?

LIMITATIONS:
This is fine.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a formalism for the generalizability of experimental studies in ML.\n\nSTRENGTHS:\nAnything pushing to get better practices in evaluation of ML is very important.\n\nWEAKNESSES:\nI could quibble with some of the setup, which is a bit confusing to me: design factors being properties of the context rather than of the alternative, for example, is kind of odd, but I don't think this is very important.\n\nThe big problem is that there is a huge literature on a very similar problem and I have very little sense of how this work connects to it: starting with the Neyman-Pearson lemma and going down through the standard corpus of decision theory, we have a lot of statistical tools for thinking about this problem in very broad strokes. After reading this paper, I have a sense that you're trying to solve a very similar problem (given a sample from some population, what can I say about the reliability of my estimate? How many samples would I need to be sure that it's reliable?)\n\nSection 4.3 seems to be rederiving some form of power analysis.\n\nLooking at A.3.3, it appears that the procedure is essentially the following:\n(1) [the inner loop from 1...n_rep] construct a null hypothesis at a given sample size, find the upper alpha quantile of that null hypothesis\n(2) Repeat this at a variety of sample sizes\n(3) Estimate a power-law relationship between sample size and the upper-alpha quantiles\n(4) Predict the sample size which would have such an upper-alpha quantile\n\nThis procedure is an empirical version of power analysis where the null distribution is not known but simulated and extrapolated. If I know the type-I error rate, type-II error rate and a distribution under the null and under the alternative, deriving the required sample size is straightforward. Indeed, we have a CLT for MMD (at least under some kernels) [1], so these distributions are known asymptotically, which is likely plenty for the purposes of sample size determination. Do \\alpha^* and \\delta^* map onto concepts from Neyman-Pearson? It's entirely possible.\n\nThis is an important question because decision theory has very well established results on things like uniformly most powerful tests. When we just invent a new framework rather than relying on well-trod ones, we are likely to derive suboptimal procedures unless we compare very carefully to these existing procedures. There's no similarly sophisticated discussion of error properties in this paper, which would be reasonable if this were truly the first paper in its vein, but I don't think that's the case.\n\nFurther, it's not clear to me why these similarities between rankings should be the target of inference. Rather, shouldn't I care about whether, based on the sample of allowed-to-vary factors I've used, alternative A is preferred to alternative B? This is an extremely standard matter of decision theory as far as I can tell. By moving to these more complicated research questions about rankings it clouds this fact, but I'm not sure it needs to. If the target of inference were instead to be a rank of K alternatives, I believe a decision theorist would take a somewhat similar approach to what you've done here: define a similarity metric based on the research question. An example solution to a problem like this would be [2], [3]. I just don't see why we need this new framework to accomplish a task I think we already have the tools for.\n\nIt's entirely possible that there's a contribution here, but it can't just be \"this is a new task\". We have methods from decision theory that have been designed for a wide range of decision tasks, and its incumbent upon the authors to demonstrate why those existing tools do not fit the task in front of them.\n\n[1] https://www.jmlr.org/papers/volume24/22-1136/22-1136.pdf\n[2] https://onlinelibrary.wiley.com/doi/abs/10.1002/mcda.313\n[3] https://www.sciencedirect.com/science/article/abs/pii/S0377221715008048\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new mathematical framework and a corresponding new algorithm to evaluate the generalizability of published experimental studies, by adapting Montgomery's classification of experimental factors [44].\nThey demonstrate the efficacy of this framework in evaluating the generalizability of two popular published experimental studies.\n\nSTRENGTHS:\n1. The paper appears to be theoretical strong. \nIt goes beyond the standard notion of reproducibility of an experimental study and comes up a definition of generalizability of an experimental study.\nMorever, Proposition 4.2 provides a theoretical result on the sample size $n$ necessary to obtained a desired generalizability $\\alpha^*$ for a desired similarity $\\epsilon^*$, and the authors also provide an algorithm (A.3.3) to compute this sample size.\nSince the similarity $\\epsilon^*$ is hard to specify, they make it a function of the kernelized distance between rankings $\\delta^*$.\n\n2.\nThe empirical evaluation in Fig. 2 and Fig. 3, on the categorical encoder comparison from [41] or the BIG-bench framework for LLM comparison from [55], respectively, demonstrates the practical utility of the proposed approach in determining sample sizes to guarantee generalizability.\n\nWEAKNESSES:\n1. The clarity in the writing can be significantly improved:\n\n1.A. Symbols are used before defining them, typos exist, and symbols are not used consistently:\n\n1.A.a. On line 118, the symbol $\\mathcal{R}_{n_a}$ is mentioned, but the relation of this symbol to the ranking on alternatives only becomes clear later in Definition 3.1.\n\n1.A.b. The Section number is missing on line 111.\n\n1.A.c. The symbol $m$, is defined as the number of shots, on line 115, whereas line 114 uses the symbol $n$ rather than $m$. Moreover, on line 88, $n$ is defined as the number of shots.\n\n1.A.d. In contrast to 1.A.c, in eq. (1), after line 170, the symbol $n$ is now used without providing a definition. It now appears to be the size of any study, in a general definition, rather than the number of shots, as defined on line 88. \n\n1.A.e. On Sec. 5.3, line 315, $N$ is defined as the number of preliminary experiments, whereas on line 154, it is defined as the size of the sample of valid experimental conditions. Do these mean the same thing ?\n\n1.B. Sec. 3.1 defines a ranking of alternatives as the primary result of an experimental study.\nHowever, the effect size, i.e., the magnitude and sign of the difference between two alternatives, can be important in certain experiments.\nThe MMD kernel, used in Sec. 4.2, actually allows measuring this effect size, as discussed in [27], but the limitations imposed by the usage of this MMD kernel within the author's generalizability framework, are not clear despite the somewhat cryptic discussion in Sec. 6.\n\n2. \nThe experimental evaluation is limited to a comparison of ranking differences between alternatives, and does not include a measurement of the practical differences between alternatives, or the significance of these differences.\n\nQUESTIONS:\n1.\nThe experimental evaluation seems to depend strongly on the choice of kernel, and its parameters. In Sec. 5.1, (g2), or Sec. 5.2, (g2), for example, what would happen if the Jaccard kernel hyper-parameter $k$ were varied ?\n\n2. Please refer to the question under weakness 1.A.e.\n\nLIMITATIONS:\nPlease refer to the potential limitation underlying weakness #2. Is it possible to quantify the magnitude of differences between alternatives using the generalizability framework provided by the authors ? The authors mention this limitation in Sec. 6, but it is not clear why the MMD kernel cannot quantify magnitude of differences.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}