PAPER

Statistical Reform in Information Retrieval?
Tetsuya Sakai
Waseda University, Tokyo, Japantetsuyasakai@acm.org

Abstract
IR revolves around evaluation. Therefore, IR researchers should employ sound evaluation
practices. Nowadays many of us know that statistical significance testing is not enough,but not all of us know exactly what to do about it. This paper provides suggestions on
how to report eﬀect sizes and confidence intervals along with p-values, in the context ofcomparing IR systems using test collections. Hopefully, these practices will make IR papers
more informative, and help researchers form more reliable conclusions that “add up.” Finally,
I pose a specific question for the IR community: should IR journal editors and SIGIR PC
chairs require (rather than encourage) reporting of eﬀect sizes and confidence intervals?

1

Introduction

The objective of this paper is to initiate a discussion on better practices in reporting experimental results in IR, especially in the context of laboratory experiments using test collections.
Early IR researchers were rather reluctant to use statistical significance tests (especially parametric tests); that has changed over the past decades, but many modern IR researchers who
use test collections report only the p-values (or worse, we often just say “the diﬀerence isstatistically significant at α = 0.05”1 .
It is well-known that reporting p-values is not enough, because they reflect both thesample size (number of topics in our case) and the eﬀect size (magnitude of the diﬀerence
between systems) (e.g., [4, 7, 25]). For example, suppose that an IR researcher comparedsystems X and Y using a test collection with n topics, and obtained per-topic performance
diﬀerences (d1 , . . . , dn ) = (x1 − y1 , . . . , xn − yn ) in terms of some evaluation measure. The
test statistic for a paired t-test in this case would be

"

√ d¯
d¯
= n√
t0 = !
V /nV(1)

where d¯ = nj=1 dj /n (the sample mean, which is the unbiased estimate of the population"¯ 2 /(n − 1) (the unbiased estimate of the population variance).
mean) and V = nj=1 (dj − d)Now, as t0 becomes larger (i.e., more extreme), the corresponding p-value becomes smaller
and therefore the result is more likely to be considered statistically significant. However, it
1

Or even worse: “the diﬀerence is significant.”

ACM SIGIR Forum

3

Vol. 48 No. 1 June 2014

is clear from Eq. 1 that√ a large t0 may mean either (a) the sample size n is large; or (b) the¯ V , the diﬀerence between X and Y measured in standard deviation
sample eﬀect size d/units, is large. A p-value does not tell us which is the case. In short, discussing eﬀect sizes
along with p-values means trying to isolate the real diﬀerence from the sample size eﬀect.
Outside the IR community, some top-tier journals now require reporting of eﬀect sizes
and confidence intervals along with p-values [4, 5, 6, 7, 13, 14]. Whereas, take the ACM TOIS
Information for Authors page2 as an example: as of May 14, 2014, it only mentions classificalsignificance testing: “Statistical tests should be included to support empirical claims. Whenreporting statistics, the name of the statistic, the degrees of freedom, the value obtained, andthe p-value should be reported, e.g., F (3, 65) = 4.83, p < 0.01.” Thus, while eﬀect sizes
and confidence intervals may be nothing new to many IR researchers3 , it appears that thereporting practice has not really prevailed, at least in test collection-based studies. Hence,
this paper is primarily intended for those who routinely use IR test collections to compare two
or more systems and conduct statistical significance tests. I shall try to provide a tentative
guideline for reporting eﬀect sizes and confidence intervals in the above contexts. If the
IR community standardise such reporting practices to some extent, this may help us form
reliable and practically significant4 conclusions through accumulation of informative resultsfrom diﬀerent studies.
The remainder of this paper is organised as follows. Section 2 is a collection of quotations
from past IR research, which reflects how the IR community slowly adopted (parametric)
statistical significance testing, and how it has been questioned recently. Readers not interested in the history can skip this section and go directly to Section 3, which suggests exactly
how to report eﬀect sizes and confidence intervals for researchers trying to compare two or
more IR systems using a test collection with n topics. Section 4 discusses whether and how a“statistical reform” [5, 6] in IR should be brought about, as well as a few other possibilities.

2

Looking BackIn the early days, IR researchers were very cautious about the assumptions underlying (parametric) significance testing.
“An attractive feature of the sign test is that normality of the input data is not required,and since this normality is generally hard to prove for statistics derived from a requestdocument correlation process, the sign-test probabilities may provide a better indicator ofsystem performance than the t test.” (Salton and Lesk, 1968 [18], p.15)“Parametric tests are inappropriate because we do not know the form of the underlying
distribution.” (Van Rijsbergen, 1979 [23], p.136)“Since the form of the population distributions underlying the observed performance values
is not known, only weak tests can be applied; for example, the sign test.” (Sparck Jones andWillet, 1997 [21], p.170)
If researchers are worried about the underlying assumptions such as normality, there are2

http://tois.acm.org/authors.html
See, for example, Peter Bailey’s SIGIR paper writing tip No.5: http://research.microsoft.com/en-us/
people/pbailey/sigir-paper-writing-tips.aspx.
4“A statistically significant result is one that is unlikely to be the result of chance. But a practically significant
result is meaningful in the real world. It is quite possible, and unfortunately quite common, for a result to bestatistically significant and trivial. It is also possible for a result to be statistically nonsignificant and important.” [4]3

ACM SIGIR Forum

4

Vol. 48 No. 1 June 2014

distribution-free, computer-based alternatives to parametric significance testing, namely, thebootstrap test [16, 19] and the randomisation test [20, 22]. On the other hand, the modern
view is that parametric tests such as the t-test and ANOVA are in fact applicable to many
situations in IR, as these tests are known to be robust to assumption violations. The t-test,for example, is known to be highly consistent in practice with the distribution-free bootstrap
and randomisation tests.
“While the errors may not be normal, the t-test is relatively robust to many violations ofnormality. Only heavy skewness (lack of symmetry) or large outliers (observations very far
from the mean) will seriously compromise its validity.” (Hull, 1993 [8], p.334)“These simulation experiments suggest that if we are going to worry about assumptions,homoscedasticity and linearity rather than normality are the ones we should worry about,though even then the errors are small and may very well cancel out.” (Carterette, 2012 [3],p.23)
But while the IR community began to accept parametric significance tests, the aforementioned limitations of significance testing began to receive more attention than before, and statistical reforms were introduced in several research disciplines outside IR [5, 6]. Carterette’s
recent remarks on significance testing in IR is in line with these developments:“With all of these questions, it is hard to escape the feeling that statistical inference isultimately subjective, only providing a thin veneer of objectivity that makes us feel a little more
comfortable about experimental rigor. That does not mean we should throw out the entire
edifice — on the contrary, though we believe our analysis conclusively shows that a p-value
cannot have any objective meaning, we still believe p-values from paired t-tests provide a goodheuristic that is useful for many of the purposes they are currently used for. We only argue
that p-values and significance test results in general should be taken with a very large grainof salt, and in particular have a limited eﬀect on publication decisions and community-widedecisions about “interesting” research directions.” (Carterette, 2012 [3], p.27)
As was mentioned earlier, p-values are considered uninformative (or even harmful) insome research disciplines [5, 6]. So what exactly can we do about it? This paper attempts
to provide a tentative answer, by means of a draft guideline for reporting eﬀect sizes andconfidence intervals in IR experiments using test collections.
Before concluding this section, it is worth recalling a well-known fact that some early IR
researchers did try to distinguish between statistical significance and practical significance.
Discussing eﬀect sizes is a more theoretically-grounded approach to measuring the latter.
“It must nevertheless be admitted that the basis for applying significance tests to retrieval
results is not well established, and it should also be noted that statistically significant performance diﬀerences may be too small to be of much operational interest.” (Sparck Jones, 1981,
p.243)“In a broad way, I shall characterise performance diﬀerences, assumed statistically significant, as interesting if they are at least noticeable, i.e. of the order of 5-10% diﬀerent, andas rather more interesting if they are material, i.e. more than 10%.” (Sparck Jones, 1974,
p.5)

3

Looking Forward

This section provides a draft guideline for reporting eﬀect sizes (ESs) and confidence intervals(CIs) along with p-values. We consider two typical cases: comparing two systems with a

ACM SIGIR Forum

5

Vol. 48 No. 1 June 2014

paired t-test, and comparing m(> 2) systems with a two-way ANOVA without replication5 .
The former test should be applied when we have two systems X and Y and their pertopic performance diﬀerences (d1 , . . . , dn ) = (x1 − y1 , . . . , xn − yn ); the latter test may be
applied when we have m(> 2) systems with their per-topic performance scores xij (i =1, . . . , m, j = 1, . . . , n). Note that when m > 2, conducting a t-test with a significancecriterion α independently for every system pair results in a familywise error rate of 1 − (1 −α)m(m−1)/2 [3, 4]: for example, if m = 10 and α = 0.05, the probability that there is at least
one Type I error (finding a nonexistent diﬀerence) amounts to 1 − 0.9545 = 0.90. A better
approach in this case would be to conduct ANOVA first to test the hypothesis that all of them systems are equally eﬀective; if this null hypothesis is rejected, then a multiple comparison
test such as the randomised Tukey HSD test [3, 17] can be applied, as discussed later6 .

3.1

Comparing Two Systems

3.1.1

Two-sided Paired t-test

Given {xj } and {yj }, the per-topic performance scores for systems X and Y (j = 1, . . . , n),2 ) and y ∼ N (µ , σ 2 ).
we assume that the scores are independent and that xj ∼ N (µX , σXjYY
2 +σ 2 , andUnder these !
assumptions, dj = xj −yj ∼ N (µ, σ 2 ) where µ = µX −µY and σ 2 = σXYt = (d¯ − µ)/ V /n ∼ t(n − 1) (i.e., t distribution with n −!
1 degrees of freedom). Therefore,¯ V /n ∼ t(n − 1).
under the null hypothesis H0 (namely, µX = µY ), t0 = d/Given a significance criterion α, we reject H0 if |t0 | ≥ t(n − 1; α) where t(φ; α) is thetwo-sided critical t value with φ degrees of freedom for probability α7 . As was mentioned
earlier, the actual p-value should be reported8 .

3.1.2

Reporting Eﬀect Sizes

For the paired t-test (See Eq. 1), a useful eﬀect size to report would be the sample eﬀect size
given by¯|d|ES pairedt = √ .
(2)V"Recall that d¯ = nj=1 dj /n (the sample mean, which is the unbiased estimate of the popu"¯ 2 /(n − 1) (the unbiased estimate of the population
lation mean µ) and V = n (dj − d)j=1variance σ 2 ).
It should be noted that Eq. 2 is not an estimate of the population eﬀect size !
given by µ/σ:
rather, it happens to be an estimate of a population eﬀect size of the form µ/σ 2(1 − ρXY ),where ρXY denotes the population correlation coeﬃcient between X and Y [14]. Hence,eﬀect sizes for paired t-tests as defined by Eq. 2 should not be compared directly with eﬀect
sizes for unpaired tests such as Cohen’s d [4]. If all IR test collection users stick to the paired
test and Eq. 2 wherever appropriate, there should be no problem.

5“without replication” means that, for every system-topic pair, there is exactly one measurement, which isusually true in experiments based on test collections.
6
Ellis [4] remarks that the approach of adjusting α (e.g., Bonferroni correction) “may be a bit like spending$1,000 to buy insurance for a $500 watch”.
7
With Microsoft Excel, TINV(α, φ) or T.INV.2T(α, φ).
8With Microsoft Excel, the p-value may be obtained by TDIST(t0 , φ, 2) or T.DIST.2T(t0 , φ).

ACM SIGIR Forum

6

Vol. 48 No. 1 June 2014

3.1.3

Reporting Confidence Intervals!

Since we have assumed that t = (d¯ − µ)/ V /n ∼ t(n − 1), it follows thatPr [−t(n − 1; α) ≤ t ≤ t(n − 1; α)] = 1 − α ,(3)which can be rewritten as
Pr [d¯ − M E ≤ µ ≤ d¯ + M E] = 1 − α ,(4)where the margin of error ME is given by:#

ME = t(n − 1; α) V /n .

(5)Thus, using Eq. 5, a 100(1 − α)% CI for µ is given by [d¯ − ME , d¯ + ME ].

3.2

Comparing m(> 2) Systems

3.2.1

Two-way ANOVA without replication

Suppose we have m systems evaluated with n topics; we have the performance scores xij(i = 1, . . . , m and j = 1, . . . , n) which we assume to be independent. We furthermore
assume that xij can be modeled as xij = µ + ai + bj + εij , where εij ∼ N (0, σ 2 ) and"m"ni=1 ai = 0,
j=1 bi = 0. Here, µ is the grand mean of the population, ai is the systemeﬀect, bi is the topic eﬀect, and εij is the residual9 . Note that we assume a common varianceσ 2 across the m systems (just like when we conduct an unpaired t-test where the variance isunknown): this is called the homoscedasticity assumption.
From the observed values {xij }, we first compute the grand mean of the sample x̄ =1 "m "n
1 "n
1 "m
i=1
j=1 xij , the system means x̄i• = n
j=1 xij , and the topic means x̄•j = m
i=1 xij .
mn
Then we compute the following sum of squares:
ST =

m $
n
$

i=1 j=1

SA = n

m
$
i=1

SE =

m $
n
$

i=1 j=1(x̄i• − x̄)2 ,(xij − x̄)2 ,SB = m

n
$

j=1

(6)

(x̄•j − x̄)2 ,(xij − x̄i• − x̄•j + x̄)2 = ST − SA − SB .

(7)

(8)

Note that ST measures how the observed values diﬀer from the grand mean, while SA measures how each system mean diﬀers from the grand mean; SE represents what is left after
removing the between-system variation SA and the between-topic variation SB from the total
variation.
Our null hypothesis H0 for the system eﬀect is that a1 = . . . = am = 0, i.e., that all
systems are equivalent. Let the mean squares be VA = SA /φA , VB = SB /φB , VE = SE /φE
where φA = m − 1, φB = n − 1, φE = (m − 1)(n − 1). Then under H0 , it is known that
9F0 = VA /VE = (n − 1)SA /SE ∼ F (φA , φE )(9)

In Two-way ANOVA without replication, the system-topic interaction cannot be separated from the residual.

ACM SIGIR Forum

7

Vol. 48 No. 1 June 2014

where F (φA , φE ) is the F distribution with (φA , φE ) degrees of freedom. Hence, we reject
H0 if F0 ≥ F (φA , φE ; α), where F (φA , φE ; α) is the critical F value (with φA and φE ) for
probability α10 . It is clear from Eq. 9 that a large F0 may imply either (a) a large sample
size n; or (b) that the between-system variation SA is substantially higher than SE , whichsuggests that not all systems are equal11 .
If H0 (a1 = . . . = am = 0) is rejected, then a Tukey HSD (Honestly Significant Diﬀerences)
test or its computer-based randomised version [3, 17] can be applied to find exactly whichpairs are statistically significantly diﬀerent, while keeping down the familywise error rate
to α12 . For systems X and Y whose diﬀerence in means is given by d¯XY , the!classicalTukey HSD test rejects the null hypothesis that X and Y are equivalent if |d¯XY |/ VE /n ≥q(m, (m − 1)(n − 1); α), where q(m, (m − 1)(n − 1); α) denotes the critical value of thestudentised range distribution with (m, (m − 1)(n − 1)) degrees of freedom at α, which needs
to be obtained by a table lookup. Note that VE can be obtained from the ANOVA. Whereas,
the randomised Tukey HSD is distribution-free and can be automatically computed for any
degrees of freedom.

3.2.2

Reporting Eﬀect Sizes

With ANOVA, the population eﬀect sizes often discussed are:
η2 =

2
2
σA
σA
=
2 + σ2 + σ2 ,
σT2
σA
B
E

(10)

2
σA
2
2
σA + σ E

(11)

ηp2 =

2 , σ 2 , σ 2 , are the unknown population variances that correspond to the sample
where σT2 , σABEvariations ST , SA , SB , SE , respectively. Thus, η 2 represents “how much of the total variancecan be explained by the between-system variance,” while the partial eﬀect size ηp2 represents
the same quantify after removing the between-topic variance. The above eﬀect sizes are often
estimated from samples as η̂ 2 = SA /ST and η̂p2 = SA /(SA + SE ), but it is known that η̂ 2
and η̂p2 are positively biased: they heavily overestimate η 2 and ηp2 especially when the sample
size is small [14]. (Weren’t we trying to isolate the eﬀect size from the sample size?) For
two-way ANOVA without replication, more accurate estimates of η 2 and ηp2 can be obtainedas [7, 14, 15]:
φA (VA − VE )
ω2 =
,
(12)
S T + VBωp2 =φA (VA − VE ).
SA + (n − φA )VE(13)

The above eﬀect sizes measure the system eﬀect as a whole in the context of ANOVA.
However, what IR researchers are often more interested in is the Tukey HSD result whichshows exactly which of the systems are statistically significantly diﬀerent. When researchers
10

With Microsoft Excel, FINV(α, φA , φE ) or F.INV.RT(α, φA , φE ). The p-value for F0 can be obtained as
FDIST(F0 , φA , φE ) or F.DIST.RT(F0 , φA , φE ).
11
The topic eﬀect may also be tested in a similar way: the null hypothesis in this case is that b1 = . . . = bn = 0,i.e., that all systems are equivalent, and F0 = (m − 1)SB /SE ∼ F (φB , φE ).
12
An implementation of Carterette’s randomised Tukey HSD test is available from http://www.f.waseda.jp/
tetsuya/tools.html.

ACM SIGIR Forum

8

Vol. 48 No. 1 June 2014

report on the p-values of the Tukey HSD test (or its randomised version), one is tempted toreport eﬀect sizes of the form similar to Eq. 2, that is not directly aﬀected by the sample
size and measures the diﬀerence in standard! deviation units. Since the test statistic for theclassical Tukey HSD test is given by d¯XY / VE /n, a useful sample eﬀect size might be:!

ES HSD = |d¯XY |/ VE .

(14)

Recall that VE is the remainder mean square, obtained after removing the between-systemand between-topic variations.
As there are several eﬀect size estimates available for ANOVA [7, 15], it is important for
us to clarify exactly which eﬀect sizes we are reporting in our papers. Moreover, we should
provide the basic statistics such as SA and SE in our papers so that other researchers cancompute their favourite eﬀect sizes from our data. Of course, making the raw data publicly
and permanently available to the community would solve a lot of problems.

3.2.3

Reporting Confidence Intervals

In the context of ANOVA where we accept the homoscedasticiy assumption, the following
margin of error can be used to obtain a CI for every system [9]:#ME = t(φE ; α) VE /n .

(15)Thus, for the i-th system, its 100(1 − α)% CI is given by [x̄i• − M E, x̄i• + M E]. Note thatthe above gives a tigher margin compared to Eq. 5.
Finally, it is important to visualise CIs by means of error bars on graphs. Every graphthat shows mean statistics should contain error bars (that represent CIs)13 . With Microsoft
Excel, it is very easy to add error bars to graphs, with user-specified ME values14 .

3.3

Examples3.3.1

Example: Comparing Two Systems

Table 1: Example: per-topic performances and diﬀerences for systems X and Y (n = 10).
Topic ID
System X
System Y

01
0.39
0.27

02
0.28
0.04

03
0.31
0.18

04
0.21
0.08

05
0.19
0.19

06
0.64
0.54

07
0.75
0.57

08
0.36
0.28

09
0.66
0.20

10
0.54
0.40

mean
0.43
0.28

¯

!Suppose we have obtained the results shown in Table 1. Then d = 0.16, V = 0.015, t0 =¯d/ V /n = 4.1√ so the p-value is TDIST(4.1, 10 − 1, 2) < 0.0029. The sample! eﬀect size is¯ V = 1.3. Also, if α = 0.05, then ME = TINV(0.05, 10 − 1) ∗ V /n = 0.088,
ES pairedt = d/so the 95% CI for the diﬀerence is 0.16 ± 0.088. These results may be reported as follows:“According to a two-sided paired t-test for the diﬀerence in means d¯ = 0.16 (with theunbiased estimate of the population variance V = 0.015), System X statistically significantly
outperforms Y (t(9) = 4.1, p < 0.0029, ES pairedt = 1.3, 95% CI [0.07, 0.25]).”

13

Error bars are sometimes used to represent standard deviations or standard errors, so it is important to clearly
indicate what the bars represent.
14
See, for example, http://office.microsoft.com/en-us/excel-help/add-error-bars-to-a-chart-HA102840044.
aspx .

ACM SIGIR Forum

9

Vol. 48 No. 1 June 2014

3.3.2

Example: Comparing m(> 2) Systems

Table 2: Example: per-topic performances for systems X, Y, Z (m = 3, n = 10).
Topic ID
System X
System Y
System Z

01
0.40
0.35
0.35

02
0.44
0.40
0.40

03
0.42
0.40
0.37

04
0.40
0.39
0.38

05
0.39
0.40
0.39

x̄i•
0.41
0.39
0.38

Table 3: Two-way ANOVA (without replication).
Between-system
Between-topic
WithinSum of squares
SA = 0.0027
SB = 0.0034
SE = 0.0016

Degrees of freedom
φA = 2
φB = 4φE = 8Mean squares
VA = 0.0013
VB = 0.00084
VE = 0.00020

F0
6.8
4.3
–

Suppose we have obtained the results shown in Table 2. The grand mean is then""x̄ = 0.39, SA = n m(x̄i• − x̄)2 = 0.0027, SB = m nj=1 (x̄•j − x̄)2 = 0.0034, ST =
i=1
"m "n
2
i=1
j=1 (x̄ij − x̄) = 0.0076, SE = ST −SA −SB = 0.0016, VA = SA /φA = 0.0027/(3−1) =0.0013, VB = SB /φB = 0.0034/(10 − 1) = 0.00084, VE = SE /φE = 0.0016/(3 − 1)(10 − 1) =0.00020. Therefore, F0 = (n − 1)SA /SE = 6.76, and the p-value is FDIST(6.76, φA , φE ) <0.020. (The topic eﬀect can be tested in a similar way.) The population eﬀect sizes for
the ANOVA can be estimated as ω 2 = 0.27, ωp2 = 0.70 (Eqs. 10 and 11). According to arandomised Tukey HSD test, only the diﬀerence √between X and√ Z is found to be statistically significant (p = 0.029), and ES HSD = d¯XZ / VE = 0.022/! 0.00020 = 1.6. As for thecommon margin of error, if α = 0.05, ME = TINV(0.05, φE ) ∗ VE /n = 0.015. This is thenapplied to the system means 0.41, 0.39, 0.38 to obtain the CIs. These results may be reported
as follows:“Table 3 shows the result of a two-way ANOVA (without replication) applied to thecomparison of m = 3 systems with n = 5 topics. The system eﬀect is statistically significant(F (2, 8) = 6.8, p < 0.020)15 . The population eﬀect size and the partial population eﬀect sizefor the ANOVA can be estimated from Table 3 as ω 2 = φA (VA − VE )/(ST + VB ) = 0.27
and ωp2 = φA (VA − VE )/(SA + (n − φA )VE ) = 0.70. A randomised Tukey HSD test showsthat only√ the diﬀerence between X and Z is statistically significant (p = 0.029, ES HSD =0.022/ VE = 1.6). Figure 1 shows the mean performances of the three systems with 95%
CIs using the same VE from Table 3.”

4

Summary

Following the practices in some research disciplines outside IR, this paper provided a draft
guideline for reporting experimental results in IR with p-values, eﬀect sizes and confidence
intervals. I do not claim that this is the right way to report results, but I argue that it
is important to (a) make papers as informative as possible, for example, by separating theeﬀect size from the sample size and by reporting basic statistics such as V (Section 3.1.2),
SA , SE (Section 3.2.1); and (b) share some basic reporting practices across IR researchers tofacilitate meta-analysis [4] and to form conclusions that “add up” [1].
15

The topic eﬀect is also statistically significant (F (4, 8) = 4.3, p < 0.039).

ACM SIGIR Forum

10

Vol. 48 No. 1 June 2014

0.44

SomeEvaluationMeasure

Error bars represent 95% CIs(margin of error=0.15).

0.42
0.40
0.38

0.36

0.41

0.34

0.39

0.38

System Y

System Z

0.32
0.30
System X

Figure 1: Comparison of systems X, Y and Z.
According to Fidler et al. [5, 6] who have observed the (slow progress of) statistical
reforms in medicine, psychology and ecology over the past decades, it is extremely hard tobring about statistical reforms. They conclude that “requirements may be more eﬀective
than recommendations.” To IR researchers, this translates to: “Maybe IR journal editors
and SIGIR PC chairs should require (rather than encourage) reporting of eﬀect sizes andconfidence intervals?” I think they should.
Another important point to remember is that reporting eﬀect sizes and confidence intervals is not an end in itself. Based on the eﬀect size estimates, researchers should compare
the impact of their research with others, and wherever possible discuss whether their results
may be practically significant.
Finally, there are alternatives to classical hypothesis testing. In psychology, Killeen [12]recently proposed prep to replace p-values: it represents the probability that a replicationof a study would give a result in the same direction as the original study, which to someof us may sound reminiscent of the topic-splitting approach of Zobel [26] and Voorhees andBuckley [24]. Another alternative is the Bayesian approach to hypothesis testing [2, 10, 11],which was first conceptualised in 1935. Will “new” approaches such as these eventuallyreplace classical significance testing in IR?

References[1] T. G. Armstrong, A. Moﬀat, W. Webber, and J. Zobel. Improvements that don’t add
up: Ad-hoc retrieval results since 19998. In Proceedings of ACM CIKM 2009, pages
601–610, 2009.
[2] B. Carterette. Model-based inference about IR systems. In ICTIR 2011 (LNCS 6931),
pages 101–112, 2011.
[3] B. Carterette. Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM TOIS, 30(1), 2012.
[4] P. D. Ellis. The Essential Guide to Eﬀect Sizes. Cambridge University Press, 2010.
[5] F. Fidler and G. Cumming. Lessons learned from statistical reform eﬀorts in otherdisciplines. Psychology in the Schools, 44(5):441–449, 2007.
[6] F. Fidler, C. Geoﬀ, B. Mark, and T. Neil. Statistical reform in medicine, psychology
and ecology. The Journal of Socio-Economics, 33:615–630, 2004.

ACM SIGIR Forum

11

Vol. 48 No. 1 June 2014

[7] R. J. Grissom and J. K. John. Eﬀect Sizes for Research: Univariate and Multivariate
Applications (Second Edition). Routledge, 2012.
[8] D. Hull. Using statistical testing in the evaluation of retrieval experiments. In Proceedings of ACM SIGIR 1993, pages 329–338, 1993.
[9] S. Ishimura. Analysis of Variance (Ninth Edition) (in Japanese). Tokyo Tosho, 1999.
[10] V. E. Johnson. Revised standards for statistical evidence. Proceedings of the National
Academy of Sciences of the United States of America, 110(48):19313–19317, 2013.
[11] R. E. Kass and A. E. Raftery. Bayes factors. Journal of the American Statistical
Association, 90(430):773–795, 1995.
[12] P. R. Killeen. An alternative to null hypothesis significance tests. Psychological Science,
16:345–353, 2005.
[13] R. Nuzzo. Statistical errors. Nature, 506(13):150–152, 2014.
[14] M. Okubo and K. Okada. Psychological Statistics to Tell Your Story: Eﬀect Size,
Confidence Interval (in Japanese). Keiso Shobo, 2012.
[15] S. Olejnik and J. Algina. Measures of eﬀect size for comparative studies: Applications,interpretations, and limitations. Comtemporary Educational Psychology, 25:241–286,
2000.
[16] T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM
SIGIR 2006, pages 525–532, 2006.
[17] T. Sakai. Metrics, statistics, tests. In PROMISE Winter School 2013: Bridging between
Information Retrieval and Databases (LNCS 8173), 2014.
[18] G. Salton and M. Lesk. Computer evaluation of indexing and text processing. Journalof the ACM, 15:8–36, 1968.
[19] J. Savoy. Statistical inference in retrieval eﬀectiveness evaluation. Information Processing and Management, 33(4):495–512, 1997.
[20] M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significancetests for information retrieval evaluation. In Proceedings of ACM CIKM 2007, pages
623–632, 2007.
[21] K. Sparck Jones and P. Willet, editors. Readings in Information Retrieval. Morgan
Kaufmann, 1997.
[22] J. Urbano, M. Marrero, and D. Martı́n. A comparison of the optimality of statisticalsignificance tests for information retrieval evaluation. In Proceedings of ACM SIGIR
2013, pages 925–928, 2013.
[23] C. J. van Rijsbergen. Information Retrieval (Second Edition). Butterworths, 1979.
[24] E. M. Voorhees and C. Buckley. The eﬀect of topic set size on retrieval experimenterror. In Proceedings of ACM SIGIR 2002, pages 316–323, 2002.
[25] W. Webber, A. Moﬀat, and J. Zobel. Statistical power in retrieval experimentation. In
Proceedings of ACM CIKM 2008, pages 571–580, 2008.
[26] J. Zobel. How reliable are the results of large-scale information retrieval experiments?
In Proceedings of ACM SIGIR 1998, pages 307–314, 1998.

ACM SIGIR Forum

12

Vol. 48 No. 1 June 2014