We Need to Talk About Reproducibility in NLP Model Comparison
Authors: Yan Xue
Xuefei Cao
Xingli Yang
Yu Wang
Ruibo Wang
Jihong Li
Venue: EMNLP 2023 Main (2023)
Keywords: reproducibility
NLP model comparison
corpus splitting strategy

Abstract: NLPers frequently face reproducibility crisis in a comparison of various models of a real-world NLP task. Many studies have empirically showed that the standard splits tend to produce low reproducible and unreliable conclusions, and they attempted to improve the splits by using more random repetitions. However, the improvement on the reproducibility in a comparison of NLP models is limited attributed to a lack of investigation on the relationship between the reproducibility and the estimator induced by a splitting strategy. In this paper, we formulate the reproducibility in a model comparison into a probabilistic function with regard to a conclusion. Furthermore, we theoretically illustrate that the reproducibility is qualitatively dominated by the signal-to-noise ratio (SNR) of a model performance estimator obtained on a corpus splitting strategy. Specifically, a higher value of the SNR of an estimator probably indicates a better reproducibility. On the basis of the theoretical motivations, we develop a novel mixture estimator of the performance of an NLP model with a regularized corpus splitting strategy based on a blocked $3\times 2$ cross-validation. We conduct numerical experiments on multiple NLP tasks to show that the proposed estimator achieves a high SNR, and it substantially increases the reproducibility. Therefore, we recommend the NLP practitioners to use the proposed method to compare NLP models instead of the methods based on the widely-used standard splits and the random splits with multiple repetitions.

We Need to Talk About Reproducibility in NLP Model Comparison
Yan Xue', Xuefei Cao², Xingli Yang', Yu Wang', Ruibo Wang', Jihong Li
1. School of Computer and Information Technology, Shanxi University, Taiyuan, 030006
2. School of Automation and Software Engineering, Shanxi University, Taiyuan, 030006
3. School of Mathematical Sciences, Shanxi University, Taiyuan, 030006
4. School of Modern Education Technology, Shanxi University, Taiyuan, 030006
20201 240701 10email. sxu. edu. cn
{caoxuefei, yangxingli, wangyu, wangruibo, lijh]@sxu. edu. cn
Abstract
researchers found that the majority of the conclu- sions model comparisons can not be well repro- duced in similar or slightly different experimen- tal settings involving many real-world NLP tasks Beerg-Kirkpatriee et al., 2012: Søgaard et al., 2014; Belz et al., 2023), including POS tagging, parsing, text summarization, machine translation and so on.
NLPers frequently face reproducibility crisis in comparison of various models of real- world NLP task. Many studies have empiri- cally showed that the standard splits tend to produce low reproducible and unreliable con- clusions, and they attempted to improve the s- plits by using more random repetitions. How- ever, the improvement a comparison of NLP models is limited at- tributed to a lack of investigation on the rela- tionship between the reproducibility and the estimator induced by a splitting strategy. In this paper, formulate the reproducibility in a model comparison into probabilistic func- tion with regard to a conclusion. Furthermore, we theoretically illustrate that the reproducibil- ity is qualitatively dominated by the signal-to- noise ratio (SNR) of a model performance es- timator obtained on corpus splitting strate- gy. Specifically, a higher value of the SNR of an estimator probably indicates a better repro- ducibility. On the basis of the theoretical moti- vations, we develop a novel mixture estimator of the performance of NI.P model with regularized corpus splitting strategy based on a blocked 2 vooss-validation. We conduct numerical experiments multiple NLP tasks to show that the proposed estimator achieves a high SNR, and substantially increases the reproducibility. Therefore, we recommend the NLP practitioners to use the proposed method compare NL.P models instead of the method- s based on the widely-used standard splits and the random splits with multiple repetitions.
Although checklist methodology has been rec- ommended in many NLP conferences, the checklist could merely address the reproducibility problem- s in the levels of methods and results rather than in the level of inferential conclusions (Bouthillier et al., 2019). Considering that about 70% of pa- pers had an unpromising low reproducibility (Belz et al., 2021), whether a conclusion in an NLP mod- el comparison is reproducible if one draw it from different experimental setup is still an important problem that is not sufficiently addressed. In actu- al, the reproducibility of a conclusion depends on multiple variates or factors in an NLP experiment, such as source code, text corpus and so on.
the reproducibility
An important factor that affects the reproducibil- ity is the splitting strategy of a text corpus, which has been well recognized in recent years. As a thumb rule, standard splits on a text corpus are frequently used to ensure fair comparison. How- ever, Gorman and Bedrick (2019) found that the conclusion obtained the standard splits is less reproducible and instable, and thereby they rec- ommended random splits for NLP practitioners. Different from their views, Søgaard et al. (2021) pointed out that the random splits lead to overly optimistic performance estimators and harm the NLP model comparison. Thus, they recommended multiple biased splits for assessing NLP models. Morcover, Rob (2021) proposed novel splitting strategy and introduced a tune set to select an opti- mal neural model in hundreds of epochs for fairness in subsequent comparisons. These studies provided sufficient empirical evidences to show the impor- tance of a splitting strategy on the reproducibility
1 Introduction
In NLP domain, the reproducibility of empirical experimental conclusions has attracted most re- searchers' attention because the reproducibility cri- sis becomes more and more serious along with the increasing complexity of the proposed model architectures for a specific NLP task (Belz et al., 2021). In particular, in the past few decades, many

---

in a comparison but lacking of necessary theoret- ical investigation on the relationship between the reproducibility and the estimator induced from a splitting strategy. Thus, the improvement on the standard splits is limited. Essentially, various split- ting strategies lead to different estimators of the performance of an NLP model. The variance and the expectation of a performance estimator are crit- ical to the reproducibility of the comparison of two NLP models. Therefore, in order to select an opti- mal splitting strategy and a reasonable estimation method, it necessary to establish the theoreti- cal relationship between the reproducibility and a performance estimator in a model comparison.
from population D. Let A and B be two NLP mod- els built on the corpus D,, and the corresponding model instances are denoted as A(x))) and (DD). Furthermore, let V be a commonly-used perfor- mance metric, such as accuracy, F score and so on. Without loss of generality, we assume that higher values of indicate better performance. Further- more, if a previous empirical study compared the performance of models ((D,) and ((D,) based on particular splitting strategy S and published a conclusion of (A))),)) > v()().)) in paper, then we have to ask a question: How the splitting strategy affects the reproducibility of the con- clusion v(A(Da)) > v(B(Dn))? For example, assume the conclusion is obtained based on stan- dard splits. If another research group uses slightly different splits to perform similar comparison, to what extent the conclusion is reproduced?
In this paper, we formulate the reproducibility in NLP model comparison from a probabilistic per- spective and theoretically establish the relationship between the reproducibility and the signal-to-non noise ratio (SNR) of an estimator of model performance. We illustrate that an estimator with a higher SNR probably leads to better reproducibility in a com- parison. Therefore, problem to maximize the SNR of an estimator in model comparison. We further summarize that the solution to the problem should satisfy two criteri- a with regard to the variance and the expectation of the estimator, respectively. Motivated by the criteria, we develop novel method that uses a blocked 3 x 2 cross-validation (BCV) coupled with a mixture estimator of NL. model performance and provide theoretical explamations of the method. We conduct numerical experiments multiple NLP tasks to show that the proposed method produces an estimator with high SNR and substantially in- creases the reproducibility of a conclusion in an NLP model comparison.
Formally, conditioned on a published model comparison conclusion v(A(D.)) > v(B(D.)), the reproducibility in a comparison can be formal- ized into the following probabilistic form.
establish an optimization
P(is(A(D.)) > is (B(D))))) P( 0),
(1)
where ig 2 denotes the estimator of the difference between the performance of models A(D,) and ( (D,) based on randomly-generatee splits in S,
and the randomness in Eq. (1) comes from S.
An optimal splitting strategy S should make the reproducibility in comparison as large as possible. Thus, we purse the optimal splitting strategy and estimator by solving the following optimization problem.
(S', D*) argmazs, P(( A 00, (2)
2 Reproducibility in Model Comparison
where the splitting strategy and estimator should be optimized simultancously because different esti- mators can be constructed an identical splitting strategy.
In this paper, we merely consider the reproducibil- ity in a comparison of two NLP models from the perspectives of splitting strategy and estimation method. When the number of models is larger than two, the comparison can be decomposed in- to a collection of comparisons of any two NLP models among the candidates, and the comparison results on multiple pairs of models can be further reduced toward the final result with a proper correc- tion, such as the Bonferroni procedure (Dror et al., 2017).
It is hard to solve the optimization problem in Eq. (2) since the distribution of iz- is unknown and varies over different NLP corpora. Therefore, to develop an appropriate surrogation of Eq. (2), conditioned on the conclusion of v(A(D,)) > (B(D..)), we apply the one-side Chebyshev in- equality on Eq. (1) and obtain the following lower bound.
SNR
We denote a text corpus as D, where n is the cor- pus size, and all the samples in D,, are IID-sampled
P(
0)
(3)
+ SNR? 3 (B

---

where (N is the SNR of the estimator is B with the following form.
Repetition Index
Fold
Fold 2
B() B(2)
B(3), B(a)
2
B(2): (()))
B() B(a)
E
B(n) ((4)
B B(y)
SNR(U (DAD
(4)
Varl [1
Table Splitting rules a x BCV.
where E|-] and Varl-] stand for the expectation and variance of an estimator, respectively.
in i/ Dg- and enlarges the variance. As illustrat- ed in the plots in the first row of Figure 2, when a splitting ratio 8:1 used, the variance can not be efficiently reduced even using more splits. On the basis of similar observation, Dietterich (1998) decided to use a splitting ratio of 5:5 and recommended a usage of repeated two-fold CV in model comparison because the expectation of the number of overlapping samples between any two-fold CVs only n/4 (Markatou al., 2005).
Eq. (3) provides a lower bound of the repro- ducibility. The lower bound is a monotonically increasing function with regard to the SNR. The lower bound further illustrates that small variance and slightly large expectation of ing lead to high value of the SNR, which probably indicates a better reproducibility. Therefore, instead directly addressing the problem in Eq. (2), we aim to figure out a proper solution to the next best optimization problem as follows.
,*) = argmaags.SSN((C (-AB
(5)
The effectiveness of repeated two-fold CV on model comparison has been showed in series of studies, including but not limited to the work of (Alpaydin, 1999; Yildiz, 2013; Wang al., 2014; Wang and Li, 2019). Moreover, novel version of the repeated two-fold CV, named m BCV, that leads to a smaller variance in 1 (Wang et al., 2017a), achieves better performance in model comparison (Wang et al., 2017b).
Furthermore, according to the definition of SNR in Eq. (4), the solution Eq. (5) should satisfy the following two criteria.
Criterion I. The optimal splitting strategy and es- timation method should reduce the variance Varlig A B1 as much as possible.
Criterion II. The optimal splitting strategy and estimation method should maximize the ex- pectation ETDA as much as possible.
A x 2 BCV, recommended in this paper, is specific version of m x 2 BCV with m = 3. The construction of 3 x 2 BCV is straightforward: divide a corpus D, into four equal-sized block- s, namely B(1), B(2): B(3), B(a)= and combine the blocks form the splits according to the rules in Table I.
3 Our Approach
Although the optimization problem Eq. (2) is relaxed into second best problem in Eq. (5), the latter is still not solved through an analytic method- ology. Despite this, the two criteria obtained from Eq. (5) may shed light heuristic design of better splitting strategy and estimation method.
A 3 x 2 BCV possesses at least four advantages for ensuring the reproducibility in comparison of NLP models: (1) 3 2 BCV leads smaller variance of an estimator than randomly generated splits with size of 6 and ratio of 5:5 (Wang al., 2019); (2) a x BCV is straightforward to use thanks to its simple combination rules in Table I; (3) when the sizes of a training set and validation set is equal, certain frequency distributions over linguistic units of the training and validation sets consistent with relatively high probability (Wang and Li, 2019); and (4) each sample in D.. occurs with the same counts in all training sets of a BCV, which facilitates us to design an aggregated estimator based on majority voting and is firstly proposed in this paper.
3.1 2 BCV
According to Criterion I, a reasonable splitting strategy to reduce the variance of ig- is using more splits in S, which has been empirically veri- fied by many researchers (Moss et al., 2018; Gor- man and Bedrick, 2019). Nevertheless, the variance depends not only the number of splits but also its splitting ratio. Specifically, if a large portion of D, is used in training, then any two training sets in two splits of S possess a large overlapping samples that introduces unnecessary correlations

---

We demonstrate the computation of vote esti- mator DA-B in supervised NL.P model. Denote supervised text corpus as D, {(x), yy where Xj is the input text and y corresponds to the supervised labels. For example, for a semantic role labeling task, x is a sentence with a pre-defined target word, and y is the gold IOB2 labels for rep- resenting the semantic role chunks in the sentence. On the basis of a 3 2 BCV, six basis models ob- tained on D,., denoted as A(1)(2)- A(3)(4)- A(2)(4)= A(1)33) A(1)(4), A(2) of which the subscripts Validated Block
Basis Models
Basis Predictions
Vote Predictions
Ba))
Ap)(3) A(2(4)) A (3)(4)
(0 h. (. (13))))))
J.
B())
A())a) (1)aa)-An(())) A()(()) (3)
(0(xa))))) J. (00)())22) J. (0())())(2))
(9%)
B(3)
A
(000)))0)) (8010)..)) ((12)().03)
((4) Anjee A()())
(0(x)))11)) (0())3).(4)) ((er:))))))((((e))))).)))))
(8(a))
Table 2: Aggregation rules in a vote estimation on 3 x BCV.
3.2 Mixture Estimation for 1
not satisfy Criterion II in Section 3.2. In contrast, an aggregation estimation method based on majori- ty voting leads to promising estimator of which the expectation is more closer to Un than that of the hold-out estimators (Yang et al., 2023). We abbre- viate the aggregation estimation as vote estimator and denote it as P
Criterion II, the expectation Elog mcans the expectation of the difference of the estimators in two models A(Da) and (D))). In theory, the expectation in single model is frequently regard- ed as monotonically inereasing with regard to corpus size of n. However, for the expectation of the difference, i.e., Elpg- it not reasonable to assume that it is increasing with regard to n. In order to express clearly, we abbreviate E[igA-0 as Vn of which the subscript the size of a training set. Furthermore, we consider the following two cases about V/--
On the basis of the theoretical results in (Yang et al., 2023), we obtain that the vote estimator sat- isfies the following two properties.
1//22 S Eli vote 2).
(6)
EE vote Un
|/ 1n/2l-
(7)
Case I. Vn is an increasing function with regard to n. It shows that increases, model A(Dr) is substantially better than (D))))
Thus, the vote estimator is more suitable to Cri- terion II than the hold-out estimators and their average.
Case II Vn is decreasing with regard to It mean- that when becomes large, the performance superiority of model A(D,)) gradually dis- appearing.
Although the relationship between in and n in a real-world NLP task may be more complex, the assumptions in the above- meetionmet two cases are more loose than that in previous studies which fre- quently assumed that the value of v,, is almost un- changed with regard to n (Dietterich, 1998; Nadeau and Bengio, 2003; Wang and Li, 2019). More- over, we assume that sgn(Vn/2) = sg)( where sgn(-) is the sign function. We consider that the assumption is natural when n is slightly large.
correspond the indices of the blocks Table 1. Then, we use three of the basis models to validate the samples in each block and obtain three sets of basis predictions. Furthermore, on the basis of ma- jority voting, we aggregate the three sets into set of vote predictions. The aggregation rules are given in Table 2. Finally, we compare the vote predic- tions (ūp}y= (0)i,) U (1))) U (i)s)) U (()) and the gold predictions {uj}リ=n to obtain the vote estimator of model AA, denoted as 2-1 mote In similar way, we can compute the vote estimator iB for model B. Then, the vote estimator for Case I is A vote = D-A 17 vote-
We adopt two different estimation methods for the two cases respectively. In actual, in a 3 x 2 BCV, researchers frequently obtain six hold-out estimators of v. The six estimators correspond to the six folds in Table 1. In theory, each hold-out estimator is unbiased to Pn/2-
For Case I, because Pn increasing with regard to n, we obtain that 2n/2 S Vy- Therefore, the hold-out estimators and the average of the estima- tors is an unpromising estimation method that can

---

For Case II, because Vn is decreasing with n, we obtain that 1//2 2 Vn- Thus, we can prefer the average of the six hold-out estimators because the averaged estimator unbiased to 2///2- and the averaged estimator is more stable and more suitable to Criteria I and II than the hold-out estimators and the vote estimator. We denote the averaged estimator as iAーBB
3 2 BCV with an averaged estimator (Avg).
We adopt the averaged estimator ang v based on a x 2 BCV as a baseline all experiments. The 3 x 2 BCV averaged estimator has been studied in a comparison task of supervised classifiers (Wang et al., 2015).
3 BCV with vote estimator (Vote). The vote estimator sate based on a 3 x 2 BCV is used as baseline over all NL.P model comparisons. This baseline is first developed this paper.
Unfortunately, for real-world NLP model, whether its Vn satisfies Case or Case is un- known. Thus, we have construct a novel mixture estimator to comprehensively consider the two cas- es. The mixture estimator has a form as follows.
Morcover, the training and validation sets pro- duced by the splitting strategy proposed in (Sø- gaard et al., 2021) possess biased and different distributions. This characteristic differs from the fundamental assumption our paper that the train- ing and validation sets should be drawn from an identical distribution D. Therefore, we don't 't con- sider the method in (Søgaard et al., 2021) as our baseline. The method in (Rob, 2021) focuses on situation of tuning a neural NLP model that is out of our consideration. Thus, we don't 't use the method in (Rob, 2021) as baseline.
DA-B = A "mii
DAAB vote
+ A)
DAB
(8)
avg
where A is an indicator function such that A - 1 when sate D and A 0 otherwise. In comparison, we consider that when A = 1, Case I has a higher probability to occur than Case II, and vice versa. Therefore, we use A switcher to adaptively select an estimator between A-B and oate.
DA--B ang
We named our proposed method, i.e., the mix- ture estimator based on a 3 x 2 BCV, as Mixture.
3.3 Baselines
Four baselines are considered this paper. Standard splits (ST). The conventional stan- dard splits on a corpus is static and designed in advance. The standard splits frequently use ration of 8:1 to split a text corpus into a training set and a validation set (Gorman and Bedrick, 2019). The static property in standard splits tend to produce an obvious over-fitting phenomenon attributing to the well-known "file-drawer" problem (Scargle, 2000) and the adaptive usage problem of fixed hold-out (Dwork et al., 2015). Therefore, to eliminate the static property, we randomly generate splits with a ratio of 8:1 in a hold-out manner and use them as our first baseline. We also named them as standard splits.
Experiments
We compare our proposed method with the base- lines in Scction 3.3 on the following NLP tasks. Semantic role labeling task (SRL): Identify the boundaries of all semantic role chunks in a Chinese sentence with gold word segmentation and pre-defined target word based on the Chi- nese FrameNet annotation convention. A Chinese FrameNet training corpus used as D,, that con- tains 35,473 Chinese sentences and 78,749 seman- tic chunks. A Bi-LSTM algorithm coupled with linguistic features of word, POS, target word, and frame name, is used to train and validate an SRL. model. Considering the embedding size of a target word is important parpee,rrrmeter, we compare the two SRL models obtained from the settings of the embedding size being 10 and 20, respectively.
Random splits (RS). The random splits for NLP model comparison are recommended in (Gorman and Bedrick, 2019). They proposed to use multiple repetitions of hold-out validation with a splitting ratio of 8:1 to compare NLP models. The repetition count is set to an integer in [3, 10] in this paper, which is the same with (Søgaard et al., 2021). It is noted that vote estimator is hard to construct in random splits because the occurrence counts of the samples in all training sets of random splits are different. Moreover, for fair comparison, we mainly compare our 3 x 2 BCV with the random splits with repetitions.
Named entity recognition task (NER): Identi- fy the boundaries of all NER chunks without recog- nizing their type labels. We use CONLL 2003 En- glish NER training set D,, that contains 14,987 sentences and 23,499 chunks. The corpus contains four types of named entities, including PER, LOC, ORG, and MISC. We consider the NER task as a sequence labeling problem and use CRFs to train and validate an NER model. Two labeling sets of

---

For the three NLP tasks, we consider the follow- ing empirical conclusions in model comparisons. Sample Means (%)
Standard Errors (x 10-
Confidence Intervals (%)
Task
A(D.)
B (D.))
A(D.)
B(D,)
A(D..)
B
SRL 61.98
61.60
5.90
5.94
[61.87, 62.10]
[61.48, 61.71]
NER
89.38
89.20
1.86
2.07
[89.34, 89.42]
[89.16, 89.24]
Chunking
82.25
82.20
1.16
1.10
[82.23, 82.27]
[82.18, 82.22]
Table 3: Confidence intervals model performance on three NLP tasks.
IOB2 and IOBES are used to induce two different NER models that are compared in our experiments. Chinese base-phrases chunking task (Chunk- ing): Identify the boundaries of all pase-phrases in
standard errors and the confidence intervals with 0.05. The results listed in Table 3 illustrate that for each of the three NLP tasks, the confidence intervals between models A(D,) and B B(D,) are not overlapped. Therefore, the empirical conclu- sions are considered be correct.
a Chinese sentence with gold word segmentation. The Chinese sase-phrases corpus developed by Zhang and Zhou (2002). The corpus used as D, that contains 14,248 sentences and 186,319 chunks. We use CRFs based on different labeling sets of IOB2 and IOBES to obtain two different chunking models that are compared in our experiments.
On the basis of a text corpus and a splitting s- trategy, we randomly generate the corresponding splits and use the splits to train and validate the compared NLP models and obtain the correspond- ing estimators of the difference of the performance of the two compared models. This process is inde- pendently repeated in 1,000 times, and we obtain 1, 1,000 values of an estimator that is used to compute the corresponding SNR and reproducibility.
The above three tasks usually adopt precision, recall, and F1 score evaluation metrics. Due to space limitation, we merely report the results about F1 score.
For the computation of the SNR, we approximate the expectation and the variance of estimator with the sample mean and the sample variance of the 1,000 values of an estimator. Then, we use Eq. (4) to compute the numerical value of the SNR.
For the SRL task, we consider that the embed- ding size of 20 (model A) is better than that of 10 (model B) as a conclusion.
For the computation of the reproducibility, we compute the proportion of the 1,000 values greater than zero. Then, we obtain the numerical value of the reproducibility by replacing the probability in Eq. (1) with the empirical proportion.
For the NER task, IOBES (model A) empir- ically better than 1OB2 (model 13).
For the chunking task, the conclusion of IOB2 (model A) outperforming IOBES (model B) is considered because a base-phrase chunk is frequently short, and IOBES too delicate.
We organize experiments into three groups to illustrate (1) the relationship between the SNR and the reproducibility; and (2) the distribution of an estimator in each of different methods; and (3) the reproducibility over different baselines and our method.
To check the correctness of the above empir- ical conclusions, i.c.. v(A(D,)) > v(B(D,)). we conduct numerical simulations to obtain the
performance of models A and B. Specifically, we combine the original training and test sets and randomly sample a training set and a test set in without--eplacement manner from the com- bined data set. The sampling ratio is the same the original splitting ratio. Then, inde- pendently perform the process 1,000 times and use the sample mean of the 1,000 values of es- timators (AA()))) and (B)),)) to approximate v(A(D,)) and (B)()))) Considering that the 1,000 estimators are IID, we further compute the
4.1 Experiments on SNR vs. Reproducibility
We first perform experiments to illustrate the rela- tionship between the SNR and the reproducibility based on random splits. Specifically, we vary the repetition count of random splits from 3 to 10 and depict the corresponding SNR and reproducibility in Figure 1.
Intuitively, when the repetition count increas- es, the SNR becomes large. From Figure 1, it is obtained that with an increasing SNR, the re-

---

Chunking Task
1.6
0.750
0.725
0.6
0.725
0.30
SNR
0.45
0.90
0.5
0,700
Reproddud
0.675
0.675
0.40
0.650
0.85
0.4
1.0
0.30
-0.625
0.625
10
10
RS
RS
RS
SNR
Reproducibilit
Figure I: Illustration of the relationship between reproducibility and SNR.
SRL
Task
0.02
0.01
SWR
0.00
0.000
0.000
0.01
-0.002
-0.005
10
5
9
10
R.5
R5
R5
SRL Task
NER Task
Chunking Task
0.04
0.01
0.005
0.00
0.000
00.02
-0.005
avg
misture
agg
vote
mixtur
avg
vale
mixtute
Figure 2: Box plots of the distributions of the estimators in different methods.
producibility increases qualitatively. The increase doesn't strictly hold in a quantitative manner be- cause the analytic form in the right-hand side of Eq. (3) with regard to the SNR is a lower bound rather than an equivalent form of the reproducibility. As showed in the third plot in Figure 1, when the repe- tition count changes from 4 to 5, the SNR increases but the reproducibility slightly decreases. A few also be found in Table 4.
estimator in random splits cannot be efficiently re- duced even using more splits. An important reason that the overlapping samples between any two training sets random splits account for large proportion (about 70%) of D..
In the second row of Figure 2, the distributions of the estimators in different methods are compared.
From the perspective of Criterion I in Section 2, common observation can be found over three NLP tasks. Standard splits and random splits pos- sess larger variance than the other three methods based on 3 BCV that verifies the minimal variance property an estimator in 3 BCV, as described in Section 3.1.
similar observations Despite this, the qualitative relationship still allow to maximize the SNR the second best choice to find a better splitting strategy and estimation method for ensuring high reproducibility.
4.2 Experiments on Distribution of ig 1
We further provide the distributions of the values of the estimators in different baselines and our method in box-plots showed in Figure 2. The horizontal dotted line indicates zero value.
Moreover, with the perspective of Criterion II, we concentrate on the expectations of the estima- tors. Several interesting observations are found over the three NLP tasks.
The three plots in the first row of Figure 2 pro- vide the distributions of the estimators in random splits over the three NLP tasks with increasing repetition count. From the three plots, we obtain that when the repetition count is greater than the width of the box-plot keeps almost unchanged. other words, the variance and the SNR of the
For the SRL task, considering the plot in the left bottom of Figure 2, we find that the expectations of the estimators in different methods are almost identical, i.c.. Pa/2 Vy. That corresponds the assumption adopted in previous studies. In this case, the estimator that owns the smallest variance performs better in the model comparison.

---

ST
RS
Avg
Vote
Mixture
Task
Splitring Strategy
8:1 hold-out
6 repetitions of 8:1
x 2 BCV
SNR
0.181
0.414
0.465
0.645
0.684
SRL
Reproducibility
0.636
0.665
0.659
0.694
0.714
SNR 0.510
1.258
0.889
1.179
1.421
NER
Reproducibility
0.678
0.893
0.830
0.900
0.940
SNR 0.195
0.483
3.200
1.818
3.063
Chunking
Reproducibility 0.590
0.665
1.000
0.960
1.000
Table 4: SNR and reproducibility of different methods on the three NI.P tasks.
For the NER task, the estimator in random splits has a higher expectation than the averaged estima- tor in a that Case in Section 3.2 occurs, i.e., Mn/2 S Pn- In this casc, even the averaged estimator has the smallest variance, its SNR is not high (as shown in Table 4), because the expectation of the averaged estimator is small.
reproducibility that indicates the optimality of the selected NLP models based on standard splits can- not be guaranteed. This conclusion is similar with that in (Gorman and Bedrick, 2019; Rob, 2021).
BCV. The phenomenon indicates
The random splits have a higher SNR and re- producibility than the standard splits because the former uses more repetitions to make the estimator more stable. Thus, consider the improvements in (Gorman and Bedrick, 2019) are effective and significant. However, the room for improvement on the reproducibility in random splits by using more splits is limited attributing to the large portion of the overlapping samples between the training sets. Compared with the standard splits and the ran- dom splits, the estimators in 3 x 2 BCV achieve better SNR and reproducibility. Therefore, a 3 x 2 BCV better splitting strategy in an NLP mod- el comparison. Furthermore, comparing the three estimators a BCV, we obtain that in most situations, the mixture estimator achieves the high- est SNR and reproducibility. Thus, the mixture estimator in 3 x 2 BCV, proposed in this paper, should be preferred in an NLP model comparison. In summary, we recommend three guidelines:
For the chunking task, the plot in the right bot- tom of Figure 2 shows that the expectation of ran- dom splits is lower than that of the averaged esti- mator. In other words, Case II in Section 3.2 holds, i.c., 1n/2 2 Vn. and the averaged estimator has higher SNR.
From the above observations with regard to the expectation, we have to re-state that the relationship between Vn and n varies over different NLP tasks and is unknown in advance. This is important motivation for to develop mixture estimation based on 3 BCV.
Moreover, it is noted that we concentrate on the experimental settings of model A being slightly better than model 13 in Figure 2 because when mod- el A is substantially better than model B (such as BERT vs. CRFs), all splitting strategies, including ST, RS, and 3 x BCV, tend to achieve the best re- producibility (=1.0). Thus, it hard observe the difference between the effects the different split- ting strategies on the reproducibility even though our proposed method still owns higher SNR.
(1) Standard splits should be used with caution in comparison of two NLP models because tandard splits may lead to low reproducibility and unreliable conclusion in the comparison.
(2) A 2 BCV could be preferred when assess- ing and comparing NLP models. The aver- aged estimator and vote estimator a 3 x 2 BCV can be considered because they achieve promising reproducibility in the task of model comparison.
4.3 Experiments on Reproducibility
Several experiments are performed to compute the reproducibility of different splitting strategies and estimation methods over the three NLP tasks. The results are given in Table 4. Several observations are obtained on Table 4.
(3) NLP practitioners could give priority to the mixture estimator coupled with a 3 x 2 BCV in a comparison of two NLP models.
The standard splits possess the lowest SNR and

---

5 Related Work
Conclusions
In this paper, we theoretically analyze the relation- ship between the reproducibility in an NLP model comparison and the SNR of an estimator of the difference of the performance of two NLP mod- els. We further show that higher value of the SNR probably indicates a better reproducibility. To increase the reproducibility, mization problem maximize the SNR and further establish two criteria for figuring out an optimal splitting strategy and a better estimation method. We develop a novel method that uses 3 x BCV coupled with a mixture estimator and illustrate the superiority of the proposed method through multi- ple real-world NLP tasks. We recommend the NLP practitioners to use a 3 x 2 BCV and the mixture estimator to compare their NLP models.
recent decades, a task of reproducing and/or replicating results and conclusions in published NLP papers has attracted many researchers' atten- tion. Many studies have illustrated that the repro- ducibility in NLP models has close relationship with many factors, including availabilliy of source code (Arvan et al., 2022), randomization manner of instances in text corpus (Bestgen, 2020), na- ture of an NLP task and under-specified or miss- ing details in a published paper (Rim et al., 2020), readability of a published paper and number of hy- perlinks (Akella et al., 2022), splitting strategy of a corpus (Gorman and Bedrick, 2019), statistical test used in NLP task (Søgaard, 2013; Søgaard al., 201 4), tradition of publishing reproduction attempts (Cohen et al., 2016), and on. There- fore, several systematic reviews and meta analy- sis related to the reproducibility in NLP models are conducted in recent years, including but not limited to (Cohen et al., 2018; Arvan et al., 2022; Belz et al., 2021, 2022, 2023). Furthermore, many effective methods are recommended to ensure high reproducibility in NLP study. For example, Dror et al. (2017) proposed novel replicability analysis framework based on partial conjunction testing. Dror al. (2019) developed a deep dom- inance method to properly compare neural NLP models. Belz (2022) recommended a quantitative measure of reproducibility from a metrological per- spective. Magnusson et al. (2023) investigated the commonly-used checklist method.
formulate opti-
In future, we aim investigate comparison of multiple NLP models and try our best to figure out better method to improve the reproducibility of the comparison based on the splitting strategy an m x 2 BCV.
Limitations
Despite the novelty and benefits of our method for reproducibility of conclusion in an NLP mod- el comparison, it does include some drawbacks. In this paper, the theoretical analysis and experi- mental results illustrate that our proposed method can substantially improve the reproducibility of a conclusion a comparison of two NLP models. However, we have not obtain an equivalent form of the reproducibility with regard to the SNR that indicates there is still room for improvement from theoretical perspective. In addition, our proposed method can be improved through adopting m x BCV instead of a 3 x 2 BCV and regularizing the divergence between the empirical distributions of the training and validation sets in an m BCV. These would form our future directions.
Besides the above-mentionn mentioned studies, an attrac- tive direction is to investigate the relationship be- tween a splitting strategy of a text corpus and the reproducibility in comparison of two NLP mod- els. Moss et al. (2018) showed that the optimized values of hypee--parameter in an NLP model are highly sensitive how data set is partitioned, Gorman and Bedrick (2019) illustrated that the pop- ular standard splits are insufficient to reproduce a conclusion in the task of NL.P model comparison. They recommended multiple random splits of a text corpus. Søgaard et al. (2021) further made an im- provement on the splitting strategy of text corpus and proposed biased splits for NLP practitioners. Rob (2021) introduced tune set to make the op- timization process of a neural NLP model more reliable. These studies shed lights on the direction and inspired us to further optimize the splitting strategy from a different perspective.
Acknowledgement
We thank the anonymous reviewers for their helpful and constructive comments. This paper is support- ed by the National Natural Science Foundation of China (62076156, 61806115 15) and the Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi (2021L004). The experiments are supported by High Performance Computing System of Shanxi University. The cor- responding author is Jihong Li.

---

References
Kevin Bretonnel Cohen, Jingbo Xia, Christophe Roed- er, and Lawrence Hunter. 2016. Reproducibility in natural language processing: A case study of t- wo r libraries for mining pubmed/medline. In 4RE- AL Workshop: Workshop on Research Results Re- producibility and Resources Citation in Science and Technology of Language, volume 2016, pages 6-12.
Akhil Pandey Akella, Hamed Alhoori, and David Koop. 2022. Reproducibility signals in science: A prelim- imary analysis. In Proceedings of the first Workshop on Information Extraction from Scientific Publica- rions, pages 140-144, Online. Association for Com- putational Linguistics.
Kevin Bretonnel Cohen, Jingbo Xia, Pierre Zweigen- baum, Tiffany Callahan, Orin Hargraves, Foster Goss. Nancy Ide, Aurélie Névéol, Cyril Grouin, and Lawrence E. Hunter. 2018. Three dimensions of reproducibility in natural language processing. In Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (I.- REC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ethem Alpaydin. 1999. Combined 2 CV F-test for comparing supervised classification learning al- gorithms. Neural Comput., 1 11(8): 1 185-1892.
Mohammad Arvan, Luís Pina, and Natalie Parde. 2022. Reproducibility in computational linguisties: Is source code enough? In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2350-233 2361, Abu Dhabi, U- nited Arab Emirates. Association for Computational Linguistics.
Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification Iearn- ing algorithms. Neural Comput., 10(7):1895-1923,
Anya Belz. 2022. A Metrological Perspective on Re- producibility in NLP", Computational Linguistics. 18(4))1125-1133 1335.
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing: Testing significance with mul- tiple datasets. Trans. Assoc. Comput. Linguisties. 1.6.1
Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. A systematic review of re- producibility research in natural language process- ing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 381-393, Online. Association for Computational Linguistics.
Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance how properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2773-2785, Florence, Italy. Associa- tion for Computational Linguistics.
Anya Belz, Maja Popovic, and Simon Mille. 2022. Quantified reproducibility assessment of NLP result- s. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toni- ann Pitassi, Omer Reingold, and Aaron Roth. 2015. The reusable holdout: Preserving validity in adap- tive data analysis. SCIENCE, 349(648))636 638.
1: Long Papers), pages 16-28, Dublin, Ireland. As- sociation for Computational Linguistics.
Anya Belz, Craig Thomson, and Ehud Reiter. 2023. Missing information, unresponsive authors, experi- mental flaws: The impossibility of assessing the re- producibility of previous human evaluations NLP. In The Fourth Workshop on insights from Negative Results in NLP, pages 1-10, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Kyle Gorman and Steven Bedrick. 2019, We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computation- al Linguistics, pages 2786-2791, Florence, Italy. As- sociation for Computational Linguistics.
Ian Magnusson, Noah A. Smith, and Jesse Dodge. 2023. Reproducibility in NLP: What have we learned from the checklist? In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 12789-12811, I, Toronto, Canada. Association for Computational Linguistics.
Taylor Berg-Kirkpatron
David Burkett, and Dan K- lein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995-1005, Jeju Island, Korea, Association for Computational Linguisties.
Marianthi Markatou, Hong Tian, Shameek Biswas, and George Hripesak. 2005. Analysis of variance of cross-validation estimators of the generalization ror. J. Mach. Learn. Res., 6:1 117-1868.
Yves Bestgen. 2020. Reproducing monolingual. multi- lingual and cross-lingual CEFR predictions. In Pro- ceedings of the Twelfth Language Resources and E- valuation Conference, pages 5595-5602, Marseille, France. European Language Resources Association.
Henry Moss, David Leslie, and Paul Rayson. 2018. Using J-K-fold cross validation to reduce variance when tuning NLP models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2978-2989, Santa Fe, New Mexi- co, USA. Association for Computational Linguistic- 8.
Xavier Bouthillier, César Laurent, and Pascal Vincen- t. 2019. Unreprodueible research is reproducible. In International Conference on Machine Learning, pages 725-734. PMLR.

---

Claude Nadeau and Yoshua Bengio. 2003. Infer- ence for the generalization error. Mach. Learn., 22(3):239-281.
Yu Wang, Jihong Li, and Yanfang Li. 2017b. Choos- ing between two classification learning algorithms based calibrated balanced 5 x 2 cross-validated f-test. Neural Process. Lett., 46(1):1-13.
Kyeongmin Rim, Jingxuan Tu, Kelley Lynch, and James Pustejovsky. 2020. Reproducing neural en- semble classifier for semantic relation extraction in- Scientific papers. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 5569-5578, Marseille, France. European Language Resources Association.
Yu Wang, Jihong Li, Yanfang Li, Ruibo Wang, and X- ingli Yang. 2015. Confidence interval for f1 mea- sure of algorithm performance hased on blocked 3 x volsiaaali,...n.. IEEE Trans. Knowl. Data Eng.. 27(3):651-655
Yu Wang, Ruibo Wang, Huichen Jia, and Jihong Li. 2014. Blocked 3 x 2 pross-valddate r-test for com- paring supervised classification learning algorithms. Neural Comput., 26(1 (6())200--25.
van der Goot Rob. 2021. We need to talk about train- dev-test splits. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 4485 4494, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Xingli Yang, Yu Wang, Ruibo Wang, and Jihong Li. 2023. Ensemble feature selection with block- regularized m x cross-validation. IEEE Trans- actions on Neural Networks and Learning Systems, 34(9))6628-666 6641.
Jeffrey D Scargle. 2000.
Publication bias: The "file- drawer" problem in scientific inference. Journal of Scientific Exploration, 14(1):91-106.
Olcay Taner Yildiz. 2013. Omnivariate rule induction using a novel pairwise statistical test. IEEE Trans. Knowl. Data Eng.. 25(9):2105-211
Anders Søgaard. 2013. Estimating effect size across
datasets. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 607-611, 1, Atlanta, Georgia. Associa- tion for Computational Linguistics.
Yugi Zhang and Qiang Zhou. 2002. Chinese base- phrases chunking. In The First Workshop on Chinexe Language Processing, SIGHAN@COLING 2002, Taipei, Taiwan, August 24- September 1, 2002.
Anders Søgaard, Sebastian Ebert, Jasmiju Bastings, and Katja Filippova. 2021. We need to talk about random splits. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics; Main Volume, EACL 2021, Online, April 19 23, 2021, pages 1823-1832. Association for Computational Linguistics.
Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Hector Martínez Alonso. 2014. What's in a p-value NLP? Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 1-10, Ann Arbor, Michi- gan, Association for Computational Linguisties.
Ruibo Wang and Jihong Li. 2019. Bayes test of preci- sion, recall, and Fl measure for comparison of two natural language processing models. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4135 4145. Association for Computational Linguis- tics.
Ruibo Wang, Jihong Li, Xingli Yang, and Jing Yang. 2019. Block-regularized repeated learning- testing for estimating generalization error. Inf. Sci., 477:246-264.
Ruibo Wang, Yu Wang, Jihong Li, Xingli Yang, and Jing Yang. 2017a. Block-regularized m x 2 cross- validated estimator the generalization error. Neu- ral Comput., 19955511--554


Reviews:
[{"review_id": "XxEZX562lf", "reviewer": "Reviewer_39w5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. In our paper, we mainly consider inferential reproducibility in a comparison of two NLP models. Inferential reproducibility is about the conclusions drawn, and it indicates to what extent a conclusion is reproducible if one can draw it from a different experimental setup [1].\n\nMany empirical studies [2-5] have achieved a consensus that the variability in the empirical results is critical to the reproducibility of a model comparison besides the reported numerical results of the performance of two NLP models, and the former is even more important than the latter. Qualitatively speaking, smaller variability and a large difference between the observed results of the performance of two NLP models lead to a highly reproducible conclusion.\n\nFrom a statistical perspective, our paper illustrates that the signal-to-noise ratio (SNR), which combines the above-mentioned variability and the expected difference, can be used to characterize a theoretical lower bound of the reproducibility (refer to Eq. (3) in our paper). A larger SNR leads to higher reproducibility. When the SNR tends to infinity, the reproducibility tends to one, correspondingly.\n\nSpecifically, the performance of two NLP models can be regarded as two random variables, namely, $\\hat{\\nu}_A$ and $\\hat{\\nu}_B$, attributing to the randomness in the NLP modeling process. The reported numerical results in an empirical study are considered as the observations of the two random variables. Furthermore, the conclusion in a comparison of two NLP models can be introduced as $\\nu_A > \\nu_B$ where $\\nu_A = E[\\hat{\\nu}_A]$ and $\\nu_B = E[\\hat{\\nu}_B]$. In other words, the expected performance of model $A$ is superior to that of model $B$. On the basis of these notations, we further define reproducibility as the following conditional probability.\n$$P(\\hat{\\nu}_A > \\hat{\\nu}_B  | \\nu_A > \\nu_B).  \\tag{1}$$\n\nThe reproducibility in Eq. (1) is interpreted as the probability of reproducing the conclusion $\\nu_A > \\nu_B$ from the observed results $\\hat{\\nu}_A$ and $\\hat{\\nu}_B$. By defining a difference of $\\hat{\\nu}_d = \\hat{\\nu}_A-\\hat{\\nu}_B$, we rewrite the reproducibility as the following form.\n$$P(\\hat{\\nu}_d > 0 | \\nu_A > \\nu_B). \\tag{2}$$\n\nFrom Eq. (2), we obtain that the reproducibility heavily depends on the shape of the conditional density $P(\\hat{\\nu}_d| \\nu_A > \\nu_B)$. If the shape is very thin and far from zero, indicating the variability of the reported results is small and their difference is large, the reproducibility is high. Because the location and scale of the density can be characterized by the expectation and the variance of $\\hat{\\nu}_d$, the shape of the density can be characterized by the SNR of $\\hat{\\nu}_d$. According to the one-side Chebyshev inequality, we can obtain a theoretical lower bound of the reproducibility as follows.\n$$P(\\hat{\\nu}_d > 0 | \\nu_A > \\nu_B)\\ge\\frac{SNR^2}{1+SNR^2}  \\tag{3}$$\nwhere the SNR is the signal-to-noise ratio of $\\hat{\\nu}_d$ (refer to Eq. (4) in our paper). From Eq. (3), when the variance of $\\hat{\\nu}_d$ decreases and the expectation of $\\hat{\\nu}_d$ is slightly large, then the SNR of $\\hat{\\nu}_d$ becomes large, and the reproducibility would correspondingly become high. This theoretical interpretation is consistent with the empirical observations in [2-5].\n\nIn sum, the signal-to-noise ratio is very useful in determining reproducibility and facilitates us to find a better estimation method of model performance to produce a highly reproducible conclusion in a comparison of NLP models.\n\n[1] Xavier Bouthillier, C$\\acute{e}$sar Laurent, and Pascal Vincent. 2019. Unreproducible Research is Reproducible. In ICML, pages 725-734.\n\n[2] Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. A Systematic Review of Reproducibility Research in Natural Language Processing. In EACL, pages 381-393.\n\n[3] Taylor Berg-Kirkpatrick, David Burkett, and Klein Dan. 2012. An Empirical Investigation of Statistical Significance in nlp. In EMNLP, pages 995-1005.\n\n[4] Henry Moss, David Leslie, and Paul Rayson. 2018. Using J-K-fold Cross Validation To Reduce Variance When Tuning NLP Models. In ICCL, pages 2978-2989.\n\n[5] Kyle Gorman and Steven Bedrick. 2019. We Need to Talk about Standard Splits. In ACL, pages 2786-2791.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper contains great motivation by exploring the reproducibility crisis in NLP. \n2. The paper is well organized. \n3. Experiments seem to be solid and comprehensive. \n4. This paper may have contributions to the NLP community. ", "reasons_to_reject": "1. It is unclear why the Signal-to-Noise Ratio is useful in determining reproducibility from a theoretical perspective.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addressed the reproducibility crisis in NLP model comparison, where standard splits often lead to unreliable conclusions. They proposed a new approach based on a probabilistic function and demonstrated theoretically that reproducibility is influenced by the signal-to-noise ratio (SNR) of the model performance estimator obtained from a corpus splitting strategy. Building on this, they developed a novel mixture estimator with a regularized corpus-splitting strategy and conducted experiments on multiple NLP tasks to show its effectiveness in achieving high SNR and increasing reproducibility. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u5YGlb7RS4", "reviewer": "Reviewer_wxeB", "summary": "", "questions": "A.  Following your observations from equations (1) to (5), did you consider exploring other splitting strategies than 3x2 BCV? \nB. How does 3x2 BCV compare to mx2 BCV, for various m, in terms of the observations laid out in this paper?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. In the submitted paper, the main contribution is to theoretically investigate the relationship between reproducibility and a splitting strategy. As a novel splitting strategy, a 3$\\times$2 BCV has indeed been investigated from the perspective of statistical estimation in previous studies [1-2]. However, the previous perspective is different from our perspective of reproducibility (refer to Eq. (2) in our paper) considered in this paper. Moreover, the fourth advantage of a 3$\\times$2 BCV (mentioned at the end of Section 3.1 in our paper), i.e., facilitating us to design an aggregated estimator, is firstly proposed in this paper and plays an important role in ensuring the high reproducibility of a conclusion. \n\nFor your question (A), there are several other available splitting strategies in our consideration [3-4], including Monte Carlo cross-validation (CV), repeated learning-testing, K-fold CV, etc. Nevertheless, due to the limited response time and the expensive simulation-based evaluation method in our experiments, we could merely conduct a fraction of the additional experiments over the other splitting strategies. Here, considering that a 5-fold CV is widely used, we mainly conduct experiments on the reproducibility of a 5-fold CV in the NER and chunking tasks. The results are shown in the following table. \n\n|           | ST | RS |  5-fold CV  |  3x2 BCV avg  |  3x2 BCV vote  |  3x2 BCV mixture |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|   |  |  |   |  **NER**  |  |  |\n |  Expectation ($\\times$10$^{-4}$ ) | 17.33  |  17.33  |  15.63  |  5.94  |  10.50  |  11.43  |\n |  Variance ($\\times$10$^{-7}$ ) | 115.30 | 18.98 | 8.13 | 4.46 |7.93 | 6.47 |\n  |  SNR |  0.51  |  1.26  |  **1.73** | 0.89 | 1.18  |  1.42 |\n   |  Reproducibility | 0.68  |  0.89 | **0.98** | 0.83 | 0.90 |  0.94 |\n|   |  |  |   |  **Chunking**  |  |  |\n   | Expectation  ($\\times$10$^{-4}$ ) | 4.82  |  4.54  |  4.20  |  11.81  |  10.12  |  12.62 |\n | Variance ($\\times$10$^{-7}$ )  | 61.19 | 8.83  | 3.47  |  1.36  |  3.10  |  1.70  |\n |  SNR  |  0.20  |  0.48  |  0.71  | **3.20**  | 1.82  |  3.06  |\n |  Reproducibility  |  0.59  |  0.67  |  0.70  |  **1.00**| 0.96  | **1.00** | \n\nFrom the above table, we obtain that our proposed mixture estimation based on a 3$\\times$2 BCV owns a comparable reproducibility with a 5-fold CV in the NER task and achieves a substantially better reproducibility than a 5-fold CV in the chunking task. Specifically, on the two tasks, the three types of the 3$\\times$2 BCV estimators constantly possess smaller variances than the 5-fold CV estimator. Nevertheless, in the NER task, the expectation of a 5-fold CV estimator is larger than that of 3$\\times$2 BCV estimators. This observation is consistent with the assumption (Case I) mentioned in lines 272-274 in our paper. Thus, the 5-fold CV owns a slightly higher SNR and reproducibility than our proposed method. In contrast, in the chunking task, because the expectation decreases w.r.t the training set size (as mentioned in Case II in lines 275-278 in our paper), our proposed method has a higher SNR than the 5-fold CV. Thus, the proposed method achieves better reproducibility than the 5-fold CV in the chunking task. In sum, we consider the proposed method to be a better choice than a 5-fold CV.\n\nFor your question (B), we further consider the m$\\times$2 BCV with m=5, 7, and 9, and compare them with a 3$\\times$2 BCV. Due to the limited time, we merely conducted the experiments in the NER task. The experimental results are shown in the following table. \n|  | 3$\\times$2 BCV | 5$\\times$2 BCV | 7$\\times$2 BCV | 9$\\times$2 BCV | \n|:-:|:-:|:-:|:-:|:-:|\n|  |  | **Averaged** | **estimation** |  |  \n  |  Expectation  ($\\times$10$^{-4}$ )  |  5.94  |  6.94  |  5.54  |  6.89  |\n |  Variance ($\\times$10$^{-7}$ )    |  4.46  |  3.07  |  2.30  |  2.05  |\n |  SNR  |  0.89  |  1.25  |  1.16  |  **1.52**  |\n |  Reproducibility  |  0.83  |  0.88  |  0.88  |  **0.94**  |\n|  |    | **Vote** | **estimation** |  |  \n |  Expectation  ($\\times$10$^{-4}$ )  |  10.50  |  9.77  |  11.16  |  12.07  |\n |  Variance ($\\times$10$^{-7}$ )  |  7.93  |  9.68  |  9.41 |  8.51 |\n |  SNR  |  1.18  |  0.99  |  1.15  |  **1.31**  |\n |  Reproducibility  |  0.90  |  0.77  |  0.88  |  **0.92**  |\n|  |  | **Mixture** |  **estimation** |  |  \n |  Expectation  ($\\times$10$^{-4}$ ) |  11.43  |  12.00  |  12.32  |  13.28  |\n |  Variance ($\\times$10$^{-7}$ )  |  6.47  | 6.33  |  7.16  |  6.00 |\n |  SNR  |  1.42 | 1.51 | 1.46 | **1.71** |\n |  Reproducibility  |  0.94  |  0.93  |  0.94  |  **0.99**|\n  \nFrom the table, we obtain that (a) for each of the three estimators, with an increasing m, the variance of the averaged estimator decreases; (b) when m increases, the expectation of the averaged estimator remains almost unchanged, and the expectations of the vote and mixture estimation increase. In total, a larger m in m$\\times$2 BCV frequently leads to a larger SNR and a higher reproducibility. Nevertheless, in contrast to a 3$\\times$2 BCV, an mx2 BCV with m>3 has an expensive computational cost and a complex splitting manner, and thus it is not easy to use. Therefore, we recommend a 3$\\times$2 BCV in our paper instead of a general m$\\times$2 BCV.\n\nMoreover, thanks for your warm reminder about the typos and the presentation improvements. We will carefully revise the errors in the future version of our paper. \n\n[1] Wang, Y., et al. 2014. Blocked 3x2 Cross-Validated t-test for Comparing Supervised Classification Learning Algorithms. Neural Computation 26(1): 208-235.\n\n[2] Ruibo Wang and Jihong Li. 2019. Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models. In ACL, 4135-4145.\n\n[3] Arlot, S. and A. Celisse. 2010. A survey of cross-validation procedures for model selection. Statistics surveys 4: 40-79.\n\n[4] Henry Moss, David Leslie, and Paul Rayson. 2018. Using J-K-fold Cross Validation To Reduce Variance When Tuning NLP Models. In ICCL, 2978-2989.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors nicely lay out the motivation for the 3x2 BCV over alternative splitting strategies such as standard or random splits. The results also seem to back up the theory.", "reasons_to_reject": "Most of the observations regarding 3x2 BCV have been previously made known, as the authors acknowledge at the end of Section 3.1.", "questions_to_authors": "A.  Following your observations from equations (1) to (5), did you consider exploring other splitting strategies than 3x2 BCV? \nB. How does 3x2 BCV compare to mx2 BCV, for various m, in terms of the observations laid out in this paper?", "paper_topic_and_main_contributions": "The authors address the issue of reproducibility of the results of any two models under different data splitting strategies. They relate the reproducibility estimate to that of a 'signal-to-noise' ratio. Following the definition of the aforementioned quantity, they argue for a blocked 3x2 cross validation splitting strategy for which, in addition to a standard average estimator, they introduce two more, a majority vote estimator and a mixture estimator combining the previous two.  As as application for their experiments they target 3 NLP tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 136: remove the 'and' before etc\nline 523: in other words\nline 631: remove deeply"}, "extra_text": {}}, {"review_id": "GnJ29zLt6C", "reviewer": "Reviewer_zPdM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. The evaluation method in our paper is different from the typical evaluation method in many empirical NLP studies because our aim in the submitted paper is not the same as the aims in typical NLP model evaluation and comparison tasks.\n\nOur aim is to investigate the relationship between different splitting strategies and the reproducibility of a published conclusion in a model comparison (as mentioned in line 143 in our paper). An effective and direct evaluation method for the relationship is to numerically compute the true values of the reproducibility over different splitting strategies based on computationally intensive statistical simulations. Consider that the reproducibility of a conclusion is formulated into the following conditional probability (as same as Eq. (1) in our paper).\n$$P(\\hat{\\nu}_S(A(D_n))> \\hat{\\nu}_S(B(D_n))|\\nu_S(A(D_n))> \\nu_S(B(D_n))). \\tag{1}$$\n\nTo compute the probability, we should introduce a conclusion in advance as the condition of the probability in our evaluation experiments and then perform statistical simulations. The computation frequently executes thousands of numerical simulations, and thus our evaluation is substantially different from the typical evaluation methods in the NLP model evaluation and comparison tasks.\n\nThe simulation-based evaluation method in our paper is very similar to the evaluation method on the comparisons of type I and II errors and replicabilities of different significance tests in a comparison task of machine learning algorithms (including but not related to [1-4]). In these studies, because type I and II errors are probabilities conditioned on a hypothesis, their evaluation methods assumed that a hypothesis holds and numerically simulated the values of type I and II errors and optimized a significance test to satisfy the assumed hypothesis. In the machine learning domain, the simulation-based evaluation method in these studies helps the researchers find several useful and interesting results. For example, a K-fold cross-validated t-test tends to produce false positive conclusions in an algorithm comparison, and repeated two-fold cross-validation achieves a better performance in a t-test than a hold-out validation and a K-fold cross-validation in the task of algorithm comparison. In the NLP domain, the simulation-based evaluation method makes it possible to explore the theoretical properties of new NLP models and methods. As a typical example, our paper uses the simulation-based evaluation method to investigate the relationship between a splitting strategy and reproducibility.\n\nYou pointed out that keeping the splits static could ensure that all comparisons are performed in the same conditions. Despite this, the static splits frequently make the sequentially reported results have a tendency to overfit the static test set, and thus the subsequently published conclusions are probably false positive. This disadvantage of static splits is well known as the ''file-drawer'' problem in scientific inference [6] and is investigated in adaptive data analysis [7]. In particular, static splits possibly make the conclusions less reproducible and unstable [8]. Therefore, we try our best in the paper to find another splitting strategy for highly reproducible conclusions.\n\nWe thank you for your constructive suggestion for using a significance test. We consider that a significance test can produce the significance of a conclusion rather than the reproducibility of the conclusion. Therefore, we carefully think that it may not be very necessary to use a significance test before our simulation-based evaluation. Moreover, in the NLP domain, how to select a suitable significance test for an NLP task and a specific evaluation metric (such as $F_1$-score) is still an open challenge [9]. As an important future direction, we will carefully consider how to construct an appropriate significance test in a comparison of NLP models to reduce the false positive conclusions.\n\nFinally, thanks for your warm reminder about the presentation improvements. We will carefully revise our presentation in the paper and add a related work section in the published version.\n\n[1] Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7): 1895-1923.\n\n[2] Alpaydin, E. (1999). Combined $5\\times2$ cv F-test for comparing supervised classification learning algorithms. Neural Computation 11(8): 1885-1892. \n\n[3] Nadeau, C. and Y. Bengio (2003). Inference for the generalization error. Machine Learning 52(3): 239-281. \n\n[4] Bouckaert, R. R. and E. Frank (2004). Evaluating the replicability of significance tests for comparing learning algorithms. Advances in knowledge discovery and data mining, Springer: 3-12. \n\n[5] Wang, Y., et al. (2014). Blocked 3x2 cross-validated t-test for comparing supervised classification learning algorithms. Neural Computation 26(1): 208-235. \n\n[6] Jeffrey D Scargle. 2000. Publication bias: The ''file- drawer'' problem in scientific inference. Journal of Scientific Exploration, 14(1):91–106. \n\n[7] Dwork, C., et al. (2015). The reusable holdout: Preserving validity in adaptive data analysis. Science 349(6248): 636-638. \n\n[8] Kyle Gorman and Steven Bedrick. 2019. We Need to Talk about Standard Splits. In ACL, pages 2786–2791. \n\n[9] Søgaard, A., et al. (2014). What's in a p-value in NLP? In the Eighteenth Conference on Computational Natural Language Learning, pages 1-10.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper addresses the important fact of reproducibility in NLP. \n* Their proposed approach outperforms the rest of the candidates based on their evaluation criteria.", "reasons_to_reject": "* I am a bit concerned about the evaluation. Typically, when you evaluate a model, you compare the learnt distribution against the test dataset which represents the true distribution. And then, see which models performed better. However, in this paper is the other way around, you assume that model A is better than B and optimize how to split the dataset in order to satisfy the hypothesis. I understand that, in fact, dataset splitting could be a relevant factor in the model performance. But keeping the splits static makes sure that both models are compared in the same conditions (train/test splits). Additionally, if we just consider the estimations of standard splits, despite the NER task, it is not clear to me whether model A is significantly better than B. I think a significant test is required here before further analysis is performed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel method to split a dataset in order to improve the reproducibility of the results, i.e., the same conclusions are drawn from different splits. The paper compares different splitting strategies on 3 different benchmarks and evaluates how many times the conclusion of system A is better than system B occurs (reproducibility). As a conclusion, their proposed method outperforms the rest of the candidates.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* {line 248-250} “3 x 2 BCV is easy-to-use and has a low computational cost in training that is important in the study of reproducibility” This is not true, the computational cost depends on both model and data, and in cases of large models and large datasets, spending 3 times more training time could be very expensive. I do not say it is not necessary but is not cheap.\n* The Related Work section is merged in the Introduction, I would prefer to have it in another different section."}}]

Author rebuttal:

Meta-review: The paper addresses the impact of splitting strategies on reproducibility. It provides theoretical motivations for using blocked 3x2 cross-validation and presents experiments showing that this indeed provides a higher signal-to-noise ratio. The paper is presented nicely with theoretical motivations that are backed up by results.

There were concerns about the appropriateness of the signal-to-noise ratio as a measure for reproducibility and that the evaluation is optimised to show a specific result. The rebuttal provides extensive explanations of why signal-to-noise is suitable and convincingly explains why the experimental setup is different from the regular case where we want to test whether something is it case. 
It would be good if a future version provides these explanations (in main text of appendix, depending on space) so that readers understand it at first hand.
Decision: Accept-Main