The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

Simple Evaluation Metrics for Diversiﬁed Search Results
Tetsuya Sakai†
Nick Craswell‡
Ruihua Song†∗†Stephen Robertson
Zhicheng Dou
Chin-Yew Lin†
†Microsoft Research Asia, PRC‡Microsoft, USA∗Microsoft Research Cambridge, UK
tetsuyasakai@acm.org

ABSTRACT
Traditional information retrieval research has mostly focussed
on satisfying clearly speciﬁed information needs. However,in reality, queries are often ambiguous and/or underspeciﬁed. In light of this, evaluating search result diversity is
beginning to receive attention. We propose simple evaluation metrics for diversiﬁed Web search results. Our presumptions are that one or more interpretations (or intents)
are possible for each given query, and that graded relevanceassessments are available for intent-document pairs (as opposed to query-document pairs). Our goals are (a) to retrieve documents that cover as many intents as possible;and (b) to rank documents that are highly relevant to more
popular intents higher than those that are marginally relevant to less popular intents. Unlike the Intent-Aware (IA)metrics proposed by Agrawal et al., our metrics successfully
avoid ignoring minor intents. Unlike α-nDCG proposed by
Clarke et al., our metrics can accomodate (i) which intents
are more likely than others for a given query; and (ii) graded
relevance within each intent. Furthermore, unlike these existing metrics, our metrics do not require approximation,and they range between 0 and 1. Experiments with thebinary-relevance Diversity Task data from the TREC 2009
Web Track suggest that our metrics corrrelate well with existing metrics but can be more intuitive. Hence, we argue
that our metrics are suitable for diversity evaluation given
either the intent likelihood information or per-intent graded
relevance, or preferably both.

Categories and Subject Descriptors
H.3.3 [Information Storage and Retrieval]: Information
Search and Retrieval

General Terms
Experimentation

Keywords
ambiguity, diversity, evaluation, novelty, test collection

1.

INTRODUCTION

Traditional information retrieval research has mostly focussed on satisfying clearly speciﬁed information needs. However, in reality, queries are often ambiguous and/or underspeciﬁed [7]. When the retrieval system has no or little

EVIA 2010 June 15, Tokyo, Japan.
Copyright held by National Institute of Informatics

knowledge of the user, the best it can do is to produce output that reﬂect several interpretations (or intents) of suchqueries. In light of this, evaluating search result diversity is
beginning to receive attention [1, 6, 5, 11, 18, 20]. We areparticularly interested in evaluating diversiﬁed Web searchengine results, where each result is assumed to be a single
ranked list of documents. While richer forms of output suchas those involving document clustering and aggregating results from diﬀerent media and information sources are possible today, a ﬂat ranked list remains a simple and eﬀective
method for presenting retrieved material to the user.
We propose simple evaluation metrics for diversiﬁed Web
search results. Our presumptions are that one or more interpretations (or intents) are possible for each given query, andthat graded relevance assessments are available for intentdocument pairs (as opposed to query-document pairs). Our
goals are (a) to retrieve documents that cover as many intents as possible; and (b) to rank documents that are highly
relevant to more popular intents higher than those that are
marginally relevant to less popular intents. Our approach is
to evaluate Properties (a) and (b) separately ﬁrst, and then
combine the outcome.
Unlike the Intent-Aware (IA) metrics proposed by Agrawal
et al. [1] our metrics successfully avoid ignoring minor intents. Unlike α-nDCG proposed by Clarke et al. [6], ourmetrics can accomodate (i) which intents are more likely
than others for a given query; and (ii) graded relevancewithin each intent. Furthermore, unlike these existing metrics, our metrics do not require approximation, and are guaranteed to range between 0 and 1 and are therefore suitable for comparison and averaging across topics. Experiments with the binary-relevance Diversity Task data fromthe TREC 2009 Web Track suggest that our metrics corrrelate well with existing metrics but can be more intuitive.
Hence, we argue that our metrics are suitable for diversity
evaluation given either the intent likelihood information or
per-intent graded relevance, or preferably both. We expectthat these two types of information will become available
with diversity test collections in the very near future because (1) methods exist to estimate the likelihood of eachintent [1, 19]; and (2) search engine companies already use
graded relevance for evaluation, and it is natural for themto adopt graded relevance also for multiple-intent queries.
The remainder of this paper is organised as follows. Section 2 describes previous work related to the present study,
with a focus on evaluation metrics. Section 3 proposes new
metrics for evaluating diversiﬁed search results, and clariﬁes
their advantages over exising metrics. Section 4 describes an― 42 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

experiment using the Diversity Task runs from the TREC
2009 Web Track to support our claims. Section 5 concludes
this paper.

2.

RELATED WORK

2.1 S-recall
Zhai, Cohen and Laﬀerty [22] proposed a simple metriccalled S-recall (subtopic recall) for evaluating subtopic retrieval. Suppose that a topic consists of n subtopics, andthat a document can either be relevant or nonrelevant toeach subtopic. Let dr denote the document at rank r, and
let I(dr ) denote the set of subtopics to which dr is relevant.
Then, S-recall at document cutoﬀ l is deﬁned as| lr=1 I(dr )|/n, i.e. the proportion of subtopics that arecovered by the top l documents. When used on its own,
it is a rather crude metric, since it disregards the positionsof relevant documents within top l and does not accomodate graded relevance. The only diﬀerence between S-recall
and instance recall used earlier at the TREC interactive
track [10] appears to be that the former is used in ranked
retrieval while the latter was used in set retrieval.
Based on S-recall, Zhai, Cohen and Jaﬀerty further deﬁned S-precision and WS-precision at a given S-recall level [22].
Both of them involve computation of an NP-hard problem,
and therefore approximation is required.
Carterette and Chandar [2] used S-recall for a task similar to subtopic retrieval, which they called faceted topic retrieval. They argued that for their task, S-recall at lmin ,where lmin is the minimum rank at which perfect S-recall
can be achieved, is the most natural evaluation measure.
However, they also point out that ﬁnding lmin is NP-hard.
Our proposed metrics for evaluating diversiﬁed results use
S-recall as one of its components. However, our methods do
not involve NP-hard problems.

to ignore long tail users. In contrast, although our proposed
metrics also utilise P (i|q), they pay attention even to minor
intents, as we shall demonstrate in Section 4 with TREC
Diversity Task runs.
Another problem with nDCG-IA is that its value is always
less than one (unless a single ranked list is ideal for all intents– which is highly unlikely). Hence, comparing and averaging
IA metric values across topics are not recommended.
The Diversity Task of the TREC 2009 Web Track used
the IA version of precision (precision-IA) as a secondary
metric. In addition to the general problems of IA metrics
discussed above, precision-IA has a few more weaknesses:(i) its per-intent precision itself is undernormalised, when
the document cutoﬀ (e.g. l = 10) is larger than the numberof relevant documents for the intent2 ; (ii) it disregards the
positions of relevant documents. Hence we prefer to use
nDCG-IA as one of our “baselines.”

2.3

α-nDCG
Clarke et al. [6] proposed α-nDCG to evaluate diversity
and novelty in search results. They view information needs
and documents as sets of nuggets, based on the ideas fromquestion answering and summarisation evaluation. For example, suppose a query has two intents (or “informationneeds”) i1 and i2 . Then they represent them in the form
i1 , i2 ⊆ {n1 , . . . , nm } where nk represents a nugget and mis the total number of nuggets involved for this particular
topic. For example, it may be that m = 4, i1 = {n1 } andi2 = {n2 , n3 , n4 }. We shall come back to this example later.
at
Let Ji (k) be a ﬂag indicating whether the document
rank k is relevant to intent i or not, and let Ci (r) = rk=1 Ji (k),i.e. the number of relevant documents found within top r
for intent i. Moreover, for convenience let Ci (0) = 0. For
each document at rank r, Clarke et al. deﬁned what we callnovelty-biased gain N G(r) as follows:

2.2 Intent-Aware Metrics

N G(r) =Agrawal et al. [1] proposed a family of Intent-Aware (IA)metrics, and primarily examined nDCG-IA. Let i be an intent (called “category” in [1]) and suppose that for eachquery q, the probabilities of diﬀerent intents P (i|q) are given.
Then nDCG-IA for q at cutoﬀ l is given bynDCG-IA@l =P (i|q) nDCGi @l(1)iwhere nDCGi @l is the “standard” version of nDCG at l1
computed by assuming that the sole intent of query q is i.
That is, for each intent i, we separately imagine an ideal
ranked output, which lists up documents that are relevant
to intent i in decreasing order of relevance, and compare thesystem output against the i-th ideal ranked output.
The IA metrics assume that every user has a single intent(i.e. category), and aim to satisfy the “average” user. Suppose that a query has two intents i1 and i2 with P (i1 |q) =0.9 and P (i2 |q) = 0.1. Then, whether a document relevant
to intent i2 is retrieved or not has very little impact on theoverall nDCG-IA value. Hence, a Web search engine tuned
with nDCG-IA may fail to include a document relevant to i2(say) on the ﬁrst page. In other words, the IA metrics tend1

For every rank r, the gain at rank r is discounted by dividing it by log(r + 1). Note that this diﬀers from the original
discounting method that used log b (r) with log base b [8].

mJi (r)(1 − α)Ci (r−1)

(2)

i=1

where α is a parameter, which we shall discuss later.
Clarke et al. [6] computed nDCG based on N G(r), insteadof the traditional gain that directly reﬂects the graded relevance value of a document [8]. The resultant metric iscalled α-nDCG. When α = 0, α-nDCG is reduced to standard nDCG with the number of matching nuggets as thegraded relevance value. The key feature of α-nDCG is thatit discounts the value of each retrieved relevant document for
intent i based on the number of relevant documents alreadyseen for the same intent.
Although α-nDCG is a theoretically-derived metric, thetheory builds on a series of assumptions. Firstly, α-nDCG
assumes that the relevance of one nugget to the user’s intent is independent of other nuggets, that a nugget can either be relevant or nonrelevant to the need, and therefore
that only the number of matching nuggets determines theimportance of an intent. Hence, unlike the IA metrics, itdoes not accomodate P (i|q). Secondly, it assumes that therelevance of one nugget to a document is independent ofother nuggets, that a nugget can either be relevant or non2
Of the 199 intents from the TREC 2009 Web Track Diversity Task data (See Table 1), as many as 95 intents havefewer than 10 relevant documents. With l = 10, it is impossible to achieve perfect precision for these intents.

― 43 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

relevant to a document, and therefore that only the numberof matching nuggets determines the importance of a document. Consider the aforementioned example with i1 and i2 ,having one nugget and three nuggets, respectively. According to α-nDCG, a document that covers both n2 and n3 is
more important than one that covers n1 , even though theformer only partially covers intent i2 while the latter completely covers i1 . Moreover, consider an ambiguous query
such as “java” [20] run against a Wikipedia document collection, and suppose that each of its intents (or senses) ij
corresponds to a single nugget nj . According to α-nDCG, asingle disambiguation page that lists up m diﬀerent sensesof “java” retrieved at rank 1 is more valuable than a set of mpages that actually describe each sense in detail, retrievedat ranks 1-m. Thirdly, α-nDCG assumes that relevance assessor’s positive judgments (i.e. judging that a document
contains a nugget) are erroneous with probability 1 − α,while negative judgments (i.e. judging that a document does
not contain a nugget) are always correct (See Section 4.1
in [6]). This ﬂat probability α is exactly the parameter used
for discounting “redundant” documents, as shown in Eq. 2.
However, we are inclined to believe that erroneous negative
judgments (i.e. missing an existing nugget in a document)
are at least as likely as erroneous positive judgments (i.e.
reporting a nonexistent nugget in a document).
The fact that α-nDCG has two distinct discount mechanisms also deserves discussions. α-nDCG ﬁrst discounts
a relevant document based on the number of relevant documents seen so far for the same intent, using α. This ismeant to penalise redundancy (i.e. lack of novelty) in theranked list. The metric then discounts the same relevant
document based on the number of total documents seen sofar (i.e. the absolute document rank), using the logarithmic discount. It is not clear whether this explicit penalty
on redundancy through “double discount” is necessary atleast for Web search, where returning a minimal set of documents that together cover all intents is not an absolute requirement: we can present (say) 10 documents regardless ofwhether there are (say) two intents or ﬁve intents in a query.
Moreover, even if two retrieved documents are relevant tothe same intent, the second relevant document may still be
informative to the user, unless the two documents are (near) duplicates. We also point out that one of the motivations
for introducing the original logarithmic discount in nDCG
was that the user may have “cumulated information fromdocuments already seen” [8]. If this is the case, α-nDCG is
discounting “redundant” documents twice, in diﬀerent ways.
Another potential weakness of α-nDCG is that computing
its ideal gain vector is NP-complete, and therefore an approximation is required. Hence, in theory, a system output
may outperform the approximated suboptimal ideal output.
We believe that evaluation metrics should be easy to understand and easy to compute.
As α-nDCG was the primary metric used in the recent
Diversity Task of the TREC 2009 Web Track, we use it as
our primary “baseline” in our experiments.

2.4 NRBP

The most basic version of NRBP is deﬁned as:
N RBP =∞m
1 − (1 − α)p  r−1 p
Ji (r)(1 − α)Ci (r) (3)
m
r=1
i=1

where p is the persistence parameter of RBP, i.e. the probability that the user will move from one document to theone beneath it, irrespective of document relevance. Hence,just like α-nDCG, NRBP has two discount mechanisms: one
from the viewpoint of redundancy based on α, and anotherfrom the viewpoint of going down the ranks based on p.
Just like α-nDCG, NRBP relies on the novelty-biased gain
(Eq. 2), and therefore all of the above potential weaknessesof α-nDCG also apply to NRBP. In addition, it inherits apotential weakness of RBP: RBP is heavily undernormalised
for topics with few relevant documents and does not averagewell across topics [16]3 . Similarly, NRBP relies on an “ideal
ideal vector” [7], where it is assumed that there are inﬁnite
number of highly relevant documents. This implies that the
maximum NRBP value any system can achieve for a topic
with few relevant document is well below 1. Averaging such
undernomalised values across topics can be problematic.
Clarke, Kolla and Vechtomova [7] further discuss the integration of the above form of NRBP with the IA approachof Agrawal et al. [1]. They envision a topic with mutually
exclusive categories, and a set of subtopics for each category.
They propose to use the probabilities P (i|q) at the category
level, and use the above form of NRBP at the subtopic level.
While such an evaluation protocol is novel and interesting, itremains to be seen whether it is feasible and to what extent
can that somewhat complex metric reﬂect user’s perceptionof the search quality.

2.5 Other Related Metrics
Chapelle et al. [3] brieﬂy discussed an extension of their
Expected Reciprocal Rank (ERR) metric for handling diversity, following the IA approach [1]. Hence we argue thatsuch a metric inherits the potential disadvantages of the IA
metrics discussed in Section 2.2. As Chapelle et al. note,
ERR is a graded-relevance extension of Reciprocal Rank
(RR), and a special case of the Normalised Cumulative Utility (NCU) [17], discussed below.
Sakai and Robertson [17] examined a family of metrics
called NCU, based on the probability of the user abandoning a ranked list at each document rank and the utility of
the ranked list given that rank. When it is assumed that the“stopping” probability is uniform across all relevant documents, NCU is reduced to existing simpler metrics such as
Average Precision (AP), or its graded-relevance extension Qmeasure which we will use later to design our proposed metrics. In addition to this ﬂat probability distribution, Sakai
and Robertson also examined a rank-biased probability distribution, based on the assumptions that users abandon theranked list at a relevant document, and that they are more
likely to abandon it near the top ranks. The rank-biased
probability distribution is deﬁned based on the number of
relevant documents seen so far and resembles the noveltybased discount of α-nDCG, although they did not discuss
diversity/novelty. They showed that the ﬂat probability
distribution (i.e. emphasising long-tail users who dig deep

3
Very recently, Clarke, Kolla and Vechtomova [7] have proFor example, in a binary relevance environment with p =posed Novelty- and Rank-Biased Precision (NRBP), by com0.95, the best-possible RBP for a query with 5 known relebining the ideas of α-nDCG and Rank-Biased Precision (RBP) [9]. vant documents is only .226 [16].

― 44 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

down the ranked list) leads to more stable experimental results than the rank-biased one, even if the latter is closer tothe reality.
In his ﬁrst proposal of Q-measure, the aforementioned
graded-relevance IR metric, Sakai [13] applied it to factoid
question answering evaluation, given correct answer strings
that form several equivalence classes. He proposed that,within a system output (i.e. ranked list of answer strings),
only one answer string from each equivalance class should becounted as relevant. This is similar to the idea behind theparameter α of α-nDCG. However, unlike α-nDCG, Sakai’s
Q-measure with answer equivalence classes used graded relevance for each answer string.
Reciprocal Rank is another metric that is somewhat related to diversity evaluation, as it does not reward retrievalof multiple relevant documents. So are its graded-relevancevariants [15], and “1-call at l”, which requires systems toreturn at least one relevant document within top l [4].

3.

PROPOSED METRICS

Our goals are (a) to retrieve documents that cover as many
intents as possible; and (b) to rank documents that arehighly relevant to more popular intents higher than thosethat are marginally relevant to less popular intents. In Sections 3.1 and 3.2 we propose how to evaluate Properties (a)and (b), respectively, and in Section 3.3 we combine theseevaluation metrics.

3.1 I-recall
Our ﬁrst “proposal” is to use S-recall (see Section 2.1) toevaluate Property (a). We prefer to call it I-recall (intent
recall) because we are interested in handling queries withmultiple possible intents or interpretations. The intents may
well be subtopics of an underspeciﬁed query, or they may
correspond to diﬀerent senses of an ambiguous query. Thus,
let Iq denote the complete set of intents for query q. and let
n = |Iq |. A document may be relevant to n (0 ≤ n ≤ n)intents. Let dr denote the document at rank r, and let I(dr )denote the set of intents to which document dr is relevant.
For a document cutoﬀ l, deﬁne:| lr=1 I(dr )|(4)
I-rec@l =nwhich is just the proportion of intents covered by the top l
documents. Note that if l < n, an I-recall of 1 may not be
achievable even if some retrieved relevant documents cover
multiple intents. In the experiment we report later, themaximum number of intents per topic in our data is n = 6,
while we use l = 10 to follow the Diversity Task at TREC
2009 Web Track. So I-recall ranges between 0 and 1 for all
topics.
Suppose that n = l = 10, and that System x has ten
documents at ranks 1-10, each relevant to exactly one newintent, while System y has exactly one relevant document at
rank 1, which covers all ten intents. Note that, in contrast
to α-nDCG, I-recall does not rate y higher than x. Returning a minimal number of documents that together cover all
intents [2] is not of our concern, and no NP-hard problem isinvolved here. This is because Web search can return (say)ten documents within the ﬁrst search result page, regardlessof whether it is possible to cover all the intents with just one
document. Moreover, as was discussed in 2.3, a single document that covers multiple intents is not necessarily better

than a set of documents that each highly satisﬁes a single
intent, at least for our purpose.
I-recall is a binary-relevance metric – for each intent, adocument can either be relevant or nonrelevant – and it
assumes that each intent is equally important. The idea
is to satisfy every user (each corresponding to a diﬀerent
set of intents) at least to some extent within top l. Below,
we discuss simple graded-relevance metrics that can reﬂect
the importance of each intent and examine the positions of
relevant documents, to complement I-recall.

3.2 div-nDCG and div-Q
Consider a query q with its set of possible intents Iq , and adocument d. Let rel be a random binary variable, and deﬁne
rel = 1 for (q, d) iﬀ ∃i ∈ Iq s.t. rel = 1 for (i, d). That is,
we say that d is relevant to q if d is relevant to at least oneof q’s intents, and otherwise it is not relevant. According
to the probability ranking principle [12, 20], systems shouldrank documents by P (rel = 1|q, d).
Like the IA metrics, our presumption is that P (i|q) values
are available. Moreover, just like the IA metrics, we assume
that, for any pair of intents i, i ∈ Iq , i and i are exclusive:that is, a user searching on q has only one of the possible
intents. Under this assumption, we obtain:
P (rel = 1|q, d) =P (i|q)P (rel = 1|i, d) .

(5)

i∈Iq

We note that this is exactly what Spärck-Jones, Robertsonand Sanderson [20] have casually discussed, though not in
the context of how to evaluate systems.
We estimate P (rel = 1|i, d) in Eq. 5 based on manual relevance assessments as follows. Suppose, for example, we havefour relevance levels so that highly relevant, relevant, partially relevant and judged nonrelevant documents are manually obtained for each intent i (rather than for each queryq). Then we may assign a gain value [8], e.g. 3,2,1,0, to eachtype of the above judged documents, respectively. Let gi (d)denote the gain value of a document d with respect to intent
i. Now, let us further assume that P (rel = 1|i, d) ∝ gi (d).
That is, we interpret the gain values as statistics that directly reﬂect the probability of (binary) relevance of a document to an intent4 . Then from Eq. 5, the probability ranking
principle reduces to ranking documents by

P (i|q) gi (d)

(6)

i∈Iq

which we call the global gain (GG) of document d given
query q.
Let GG(r) denote the global gain of the document
at rankr, and let the cumulative GG be CGG(r) = rk=1 GG(k).
Moreover, let GG∗ (r) and CGG∗ (r) denote the GG and the
CGG at rank r for an ideal ranked output (i.e., one thatexhaustively lists up the relevant documents in decreasingorder of the global gain as deﬁned in Eq. 6). Note that,
unlike nDCG-IA that require n = |Iq | distinct ideal ranked
lists, we require a single ideal list.
Based on global gains, existing graded-relevance metrics
4
This is one way to interpret graded relevance assessments.
Note that the probability ranking principle is still based on
binary relevance.

― 45 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japansuch as nDCG and Q-measure [17] can be computed5 :lGG(r)/ log(r + 1)
div-nDCG@l = lr=1∗ (r)/ log(r + 1)
GG
r=1
r
r
k=1 J(k) + β
k=1 GG(k)rdivBR(r) =r + β k=1 GG∗ (k)1 J(r)divBR(r)R r=1

3.3 Idiv-nDCG and Idiv-Q
(7)

(8)

L

div-Q =

(9)

where l is a document cutoﬀ; J(k) is a ﬂag indicating whether
the document at rank k is (at least partially) relevant toat least one intent; β is the persistence parameterfor the
blended ratio [17], which combines precision ( rk=1 J(k)/r)
and(global) cumulative gain
 normalised( rk=1 GG(k)/ rk=1 GG∗ (k));6 R is the number of documents that are (at least partially) relevant to at least one
intent; and L is the size of the system output.
The only diﬀerence between traditional nDCG/Q-measureand div-nDCG/Q is that the latter metrics use the global
gains (Eq. 6) instead of the raw gains. Just like α-nDCG,
div-nDCG and div-Q assume that documents that cover
many intents are important. However, unlike α-nDCG, theimportance is weighted according to how important each intent is (i.e. P (i|q)), and how relevant each document is toan intent (i.e. gi (d)). These metrics reward systems thatsatisfy Property (b) discussed earlier.
Note that Q-measure and nDCG rely on the number of
relevant documents R (the former directly, and the latter
indirectly through the use of an ideal ranked list). Some researchers argue that using R for evaluation no longer makes
sense in this Web search era as the true R is diﬃcult/impossible
to obtain [23]. However, we argue that interpreting R as
the number of known relevant documents is one reasonableapproach: if relevance assessors have already given us 5 relevant documents for Topic 1, and 50 relevant documents
for Topic 2, utilising this information for normalisation isindeed useful. For example, metrics that rely on R suchas Q-measure are known to be more discriminative than
precision-oriented metrics like RBP, in that they are morerobust to variance across topics (See Section 4.4) [16].
Unlike metrics like α-nDCG and nDCG-IA, the maximum
value of div-nDCG is exacly one regardless of the document
cutoﬀ value l. Whereas, the maximum value of div-Q is exactly one provided that the system output size L is larger
than or equal to R (because if L < R, obviously the systemcannot list up all relevant documents). In traditional ad hoc
tasks at evaluation forums like TREC and NTCIR, usually
L >> R holds, so this should not be a problem. However,for Web search evaluation environments in which a small
cutoﬀ is often used (e.g. l = 10 so that the system output isin eﬀect truncated to size L = 10), the above undernomalisation problem may occur. Hence, to avoid this, we use thefollowing cutoﬀ-based version of div-Q in this paper:
div-Q@l =5l1
J(r)divBR(r)min(l, R) r=1(10)“div” obviously stands for “diversity.”
A small β (e.g. β = 0.1) makes the blended ratio resemble
precision; a large β (e.g. β = 1000) makes it resemble normalised cumulative gain. The former implies heavier penalties for relevant documents retrieved later in the ranked list.
We let β = 1 in this paper.
6

I-recall is a simple, binary-relevance, intent-level metricthat rewards wide coverage of diﬀerent intents in the top
ranks. div-nDCG and div-Q are simple, graded-relevance,
document-level metrics that reward early retrieval of documents that are highly relevant (or more precisely: “highly
likely to be relevant”) to major intents. As we want both
Properties (a) and (b), we simply combine the metrics asfollows:
Idiv-nDCG@l = γI-rec@l + (1 − γ)div-nDCG@l
Idiv-Q@l = γI-rec@l + (1 − γ)div-Q@l

(11)
(12)

where γ is a parameter. In this paper, we let γ = 0.5 bydefault. We use linear combination because we would liketo reward systems even when they satisfy only one of thetwo properties mentioned above. (A harmonic mean of two
metrics, for example, would be zero if either of them is zero.)
The Idiv metrics are rather ad hoc, in that they lack a uniﬁed user model7 . Nevertheless, they oﬀer several strengths:
1. They are easy to interpret. It is easy to imagine thekind of ranked output that the Idiv metrics are designed to pursue – all the possible intents should belisted up in the early ranks, and highly relevant documents for the major intents should be ranked above
partially relevant documents for the minor intents.
2. They are easy to compute – unlike α-nDCG, S-recall@lmin ,S-precision and WS-precision (See Section 2.1), no NPhard problem is involved.
3. Unlike IA metrics, even minor intents are considered
important, especially with a large γ (0 ≤ γ ≤ 1).
4. Unlike α-nDCG, it accomodates P (i|q) and gradedrelevance of documents to each intent.
5. Unlike α-nDCG and IA metrics, they are guaranteed
to range fully between 0 and 1, provided that the document cutoﬀ l is chosen for I-recall so that it is nosmaller than the maximum number of intents acrossthe query set (See Section 3.1).

4. EXPERIMENTS
4.1 Data
We have already clariﬁed the advantages of our Idiv metrics over existing metrics. We now demonstrate how theywork in practice compared to nDCG-IA and α-nDCG, using the recent Diversity Task (Category A) data from the
TREC 2009 Web Track [5]. The statistics of the test collection are summarised in Table 1. As it shows, the topic set
includes 243 subtopics, but only 199 of them have at least
one relevant document. These are treated as our intents. As
many as 23 intents have only one relevant document, andwe point out that this is not good news for nDCG-IA, since
its per-intent nDCG values for such intents will rely solely
on whether one particular document is in top 10 or not.
7
The original Q-measure, as an instance of NCU, does havea user model: There is a population of users abandoning theranked list at diﬀerent ranks, and the “stopping probability”
is uniform across all relevant documents [17].

― 46 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

Table 1: Statistics of the TREC 2009 Web Track Diversity Task (Category A) test collection.
Documents
TopicsIntents
Relevance data

ClueWeb09 (approx. one billion Web pages; 25TB of uncompressed data in multiple languages) [5]50 topics (12 ambiguous; 38 faceted) with subtopics. 243 subtopics (177 informational; 66 navigational)
199 intents (i.e. subtopics with at least one relevant document).
Max. #intents per topic: 6. Max. #intents per document: 5.
Total relevant across 50 topics: 4942. Total relevant across 199 intents: 6501.

Table 2: τ and τap rank correlation: Idiv-nDCG@10 run rankings vs. other rankings as baselines.
α-nDCG@10
nDCG-IA@10
Idiv-Q@10 (same γ)

γ=0.880/.778.960/.931.880/.882

γ = .2.913/.807.940/.908.947/.931

γ = .5.893/.747.867/.774.980/.977

γ = 0.8.873/.747.807/.709.993/.990

γ = 1 (I-rec@10).860/.729.793/.692
1/1

Table 3: τ and τap rank correlation: Idiv-Q@10 run rankings vs. other rankings as baselines.
α-nDCG@10
nDCG-IA@10
Idiv-nDCG@10 (same γ)γ=0.773/.670.880/.846.880/.885

γ = .2.860/.742.953/.910.947/.934

γ = .5.887/.733.873/.775.980/.977

γ = 0.8.867/.736.800/.700.993/.990

γ = 1 (I-rec@10).860/.729.793/.692
1/1

Table 4: τ and τap rank correlation: eﬀect of γ on the Idiv-nDCG@10 (Idiv-Q@10) run ranking, with γ = .5
as the baseline.
Idiv-nDCG@10 (γ = .5)
Idiv-Q@10 (γ = .5)

γ=0.867/.774.807/.722

γ = .2.913/.812.893/.792

Unfortunately, this test collection is not ideal for our purpose, because it has neither the information on P (i|q) for
each topic nor graded-relevance assessments for each intent.
We overcome the ﬁrst problem by using simulated P (i|q)
values: for a query with n intents, weassume that the jk
th intent has the probability 2n−j+1 / n
k=1 2 . As for thesecond problem, we chose to use the existing binary relevance assessments “as is” even though this means that thegraded-relevance capability of the Idiv metrics cannot bedemonstrated. Thus, in our experiment, the global gain of
a document (Eq. 6) at rank r reduces to:GGbinary (r) =P (i|q)Ji (r) .
(13)

4.2 System Ranking Comparisons
We quantify the similarity between two run rankings (basedon two diﬀerent evaluation metrics) by Kendall’s τ rank correlation and Yilmaz/Aslam/Robertson τap [21]. τ is a monotonic function of the probability that a randomly chosen pairof ranked items is ordered identically in the two rankings.
Hence a swap near the top of a ranked list and that near thebottom of the same list has equal impact. Whereas, τap is“top-heavy,” in that it is a monotonic function of the probability that a randomly chosen item and one ranked above it
are ordered identically in the two rankings. While τ is symmetrical, τap is not: it treats one ranked list as the groundtruth.

γ = 1 (I-rec@10).927/.898.907/.877

Table 2 shows τ and τap rank correlations between a run
ranking based on Idiv-nDCG (with γ = 0, .2, .5, .8, 1) andone based on another metric. As Eq. 11 shows, Idiv-nDCG
with γ = 0 equals div-nDCG, and Idiv-nDCG with γ =1 equals I-recall. Note also that Idiv-nDCG is comparedwith Idiv-Q using the same value of γ. Values higher than
0.9 are shown in bold for convenience. For example, the τ
between Idiv-nDCG (γ = .5) and α-nDCG is .893, and thecorresponding τap value (with α-nDCG taken as the groundtruth) is .747. It can be observed that:• Idiv-nDCG with diﬀerent γ’s rank runs similarly to
α-nDCG, but there are diﬀerences;i∈Iq

We plan to construct and use graded-relevance diversity testcollections in our future work.
The TREC 2009 Web diversity task received 25 run submissions. Below, we evaluate all of these runs with our proposed metrics as well as α-nDCG and nDCG-IA. As was
mentioned in Section 3.1, we use l = 10 as the document
cutoﬀ for all metrics, because diversifying the ﬁrst result
page is especially important for Web search engines. We
used the oﬃcial α-nDCG values released by the Web trackorganisers (hence α = .5); as for all other metrics, we used a
set of scripts that we developed ourselves for diversity evalution in general. The scripts are publicly available at
http://research.nii.ac.jp/ntcir/tools/ir4qa eval2.tar.
gz.

γ = 0.8.940/.916.927/.905• Idiv-nDCG with diﬀerent γ’s rank runs similarly tonDCG-IA, and the similarity increases as γ becomes
smaller (i.e. as Idiv-nDCG moves away from I-recall
and towards div-nDCG);• Idiv-nDCG rank runs similarly to Idiv-Q, especiallywith a large γ (Note that when γ = 1, they both equal
I-recall).
Table 3 shows a similar table for Idiv-Q. It can be observedthat Idiv-Q with diﬀerent γ’s also rank runs similarly to αnDCG and to nDCG-IA, but that there are diﬀerences.
Table 4 shows τ and τap rank correlations between two
rankings, both based on Idiv-nDCG but with diﬀerent γ’s.
The table also contains similar information for Idiv-Q. It canbe observed that diﬀerent γ’s do not change the run ranking
dramatically.
Figure 1 also shows the robustness of Idiv-nDCG to the
choice of γ. The two axes represent I-recall and div-nDCG,
and the solid slant lines represent contour lines for IdivnDCG with γ = .5; the dotted slant lines represent thosewith γ = .2 and γ = .8. Each circle in the graph represents
a run. For example, it can be observed that runs MSDiv3
and MSRACS lie close to the same solid contour line, andtherefore that they are almost equally eﬀective in terms of
Idiv-nDCG with γ = .5. MSDiv2 is the top performer according to Idiv-nDCG with γ = .5 and with γ = .8; MSDiv3
is the top performer according to Idiv-nDCG with γ = .2
(which is in agreement with α-nDCG as we shall see later).
However, the “top ﬁve” runs in Figure 1 lie very close to each― 47 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

0.5

I-rec@10

MSRAACSF

0.4

MSDiv2Top
according toIdiv-nDCG
and Idiv-Q:
MSDiv2MSRACSɶс͘ϴMSRABASE

0.35

MSDiv30.4

ɲ-nDCG@10

0.3
MSDiv1

ɶс͘ϱ

0.25

0.3

nDCG-IA@10

Top
according toɲ-nDCG:MSRAACSF

Idiv-nDCG@10
Idiv-Q@10

0.2

ɶс͘Ϯ
0.15
0.2

Top
according to
nDCG-IA:
MSDiv3

0.1

0.05
0.1

Figure 1: TREC 2009 diversity runs evaluated with
Idiv-nDCG@10, with contour lines for γ = .2, .5, .8.

uvaaolWatSklfu

WatSklfb

spc

WatSklq

watd1

watd3

watd5

Sab9wtBDiv1

mudvibm5

Sab9wtBDiv2

Sab9wtBfDiv

MSDiv1mudvimp

MSDiv2MSRABASE

twCSodpRBB

0.5

twCSodpRNB

0.4

uogTrDYScdA

0.3

THUIR09QeDiv

0.2

THUIR09AbClu

0.1

THUIR09FuClu

0

MSDiv3

div-nDCG@100

MSRACSMSRAACSF

0

Figure 2: TREC 2009 diversity runs evaluated with
diﬀerent metrics, sorted by the oﬃcial α-nDCG values.

other and the diﬀerences are not statistically signiﬁcant (See
Section 4.4). The main observations from this ﬁgure are:• I-recall (an intent-based recall metric) and div-nDCG(a document-based precision metric) are highly correlated with each other8 ;• Because of the high correlation between I-recall anddiv-nDCG, Idiv-nDCG is quite robust to the choice ofγ for ranking runs.
We have obtained very similar results for Idiv-Q as well.
Since Idiv-nDCG and Idiv-Q are robust to the choice of γ
for ranking runs, and because we do not have a systematic
method for determining γ, we focus our attention on γ = .5
henceforth.
Figure 2 visualises the run rankings by Idiv-nDCG, Idiv-Q(both with γ = .5), nDCG-IA and α-nDCG. The runs havebeen sorted by the oﬃcial α-nDCG performance, so the αnDCG curve is monotonically decreasing; each increase in
the other curves represents a disagreement with α-nDCG. As
indicated in the ﬁgure, Idiv-nDCG and Idiv-Q rank MSDiv2
at the top; nDCG-IA ranks MSDiv3 at the top; and α-nDCG
ranks MSRAACSF at the top. However, none of these runs
is statistically signiﬁcantly better than the others in any ofthe metrics.
We have demonstrated that, given the intent probabilities
P (i|q), Idiv-nDCG and Idiv-Q produce run rankings thatare similar to, but diﬀerent from those based on α-nDCG
and nDCG-IA. A more important question is whether these
metrics are measuring what we want to measure. Below, we
examine the actual ranked lists of MSDiv2 and MSRAACSF
to demonstrate how our proposed metrics reward diversiﬁed
systems.

8
τ = .793. τap = .686 with I-recall as the ground truth;τap = .749 with div-nDCG as the ground truth.

4.3 A Closer Look at Disagreements between
Metrics
Figure 3 shows, for Idiv-nDCG, nDCG-IA and α-nDCG,
the performance of MSDiv2 minus that of MSRAACSF per
topic. Thus, dots above the horizontal axis represent “wins”
by MSDiv2, and those below represent wins by MSRAACSF.
For 38 topics out of 50, the three metrics agree with one
another as to which run is better. Idiv-nDCG disagrees
with nDCG-IA for 7 topics, and disagrees with α-nDCG for
6 topics; nDCG-IA disagrees with α-nDCG for 9 topics. The
trend is similar for Idiv-Q. Below, we take a close look at
three topics highted in Figure 3.
Topic 21 has n = 5 intents, with P (i1 |q) = 32/62, P (i1 |q) =16/62, . . . , P (i5 |q) = 2/62. The actual top-10 ranked lists,
with relevant intents for each document, are shown below.
(Document ID preﬁxes “clueweb09-en” are omitted throughout this paper to save space.)
rank
1
2
3
4
5
6
7
8
9
10

MSDiv20005-48-03496
0013-96-08199
0013-66-18211
0013-66-18147
wp01-79-16252
0003-59-13504
0041-68-19894
0004-64-17952
0080-91-29662
0004-51-16201

i1

MSRAACSF
0010-85-03735
0012-66-27529
0012-46-18076
0013-27-24714
wp02-04-16080
0037-28-21832
0062-72-09421
0047-54-35373
0099-86-05525
0060-08-24587

i5
i5

Because Idiv-nDCG and nDCG-IA utilise the fact that i1
is a much more popular intent than i5 (in our experimental setting), they prefer MSDiv2 to MSRAACSF. Whereas,
α-nDCG prefers MSRAACSF, as it can only treat each intent equally. This example demonstrates that, given theintent probabilities, Idiv-nDCG and nDCG-IA can be moreintuitive than α-nDCG.
Topic 28 also has n = 5 intents. The ranked lists areshown below:― 48 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

0.7
0.6

Topic38:
nDCGͲIAsays
MSDiv2>MSRAACSF;
IdivͲnDCG says
MSDiv2<MSRAACSF;
ɲͲnDCG says
MSDiv2Ҹ MSRAACSF

Topic21:
nDCGͲIAandIdivͲnDCG
sayMSDiv2>MSRAACSF;
ɲͲnDCGsaysMSDiv2<MSRAACSF

0.5
0.4

diffinɲͲnDCG@10
diffinnDCGͲIA@10
diffinIdivͲnDCG@10

0.3
0.2
0.1
0
1

3

5

7

9

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Ͳ0.1
Ͳ0.2
Ͳ0.3
Ͳ0.4
Ͳ0.5

Topic28:IdivͲnDCG saysMSDiv2>MSRAACSF;
ɲͲnDCG saysMSDiv2<MSRAACSF;
nDCGͲIAsays MSDiv2Ҹ MSRAACSF

Figure 3: Per-topic performance of MSDiv2 minus that of MSRAACSF.
rank
1
2
3
4
5
6
7
8
9
10

MSDiv2wp00-88-23480
0007-19-33611
0002-47-07470
0009-64-18412
wp02-24-00069
wp01-05-00069
0127-32-28190
0004-42-16367
0100-95-14715
0030-22-33316

i4
i4

i5
i2

MSRAACSF
0022-00-34499
0007-19-33611
wp00-88-23480
0037-28-16212
0009-64-18412
0029-92-16034
0064-52-31354
0115-25-00480
0034-67-36999
0020-05-14160• Given the intent probabilities, Idiv-nDCG can be moreintuitive than α-nDCG, as the latter metric does nottake the probabilities into account. (The same is true
for Idiv-Q.)

i4
i4
i2
i4

Idiv-nDCG prefers MSDiv2; α-nDCG prefers MSRAACSF;
nDCG-IA is undecided, but leans towards MSRAACSF. Since
MSDiv2 covers three intents while MSRAACSF covers only
two, Idiv-nDCG may be the most intuitive for the purposeof diversiﬁcation. Note that having a document relevant to
i5 in MSDiv2 has very little impact on the computation ofnDCG-IA as this is a “very minor” intent (See Eq. 1).
Topic 38 has n = 3 intents, with P (i1 |q) = 8/14, P (i2 |q) =4/14, P (i5 |q) = 2/14. The ranked lists are shown below:
rank
1
2
3
4
5
6
7
8
9
10

MSDiv20011-83-04353
0022-89-07369
0009-04-14265
0006-15-25073
0009-49-19901
0078-29-22726
0026-63-30995
0052-82-45572
0008-62-07163
0004-07-33887

i1
i1
i1
i1
i1
i1

MSRAACSF
0022-32-01173
0011-83-04353
0009-04-13849
0008-94-08357
0039-80-34409
0004-30-16737
0009-70-21188
0024-34-31519
0035-17-17820
0009-12-14653

i1
i1 , i3
i1
i1

Idiv-nDCG prefers MSRAACSF, as it covers not only i1 but
also i3 . α-nDCG is undecided, but leans towards MSRAACSF.
Again, nDCG-IA prefers MSDiv2 as it virtually ignores theminor intent i3 .
The above examples demonstrate that:• nDCG-IA indeed tends to ignore minor intents andmay be counterintuitive for the purpose of evaluating
diversity;

We also remind the reader that our proposed metrics (a) canhandle per-intent graded relevance (unlike α-nDCG), and(b) do not require approximation and range fully between 0
and 1 (unlike α-nDCG and nDCG-IA).

4.4 Discriminative Power
We ﬁnally compare our proposed metrics with α-nDCG
and nDCG-IA in terms of discriminative power [14]. We
want metrics that are robust to variation across topics, sothat the same conclusion can be reached as to which of
two given systems is better, regardless of the choice of the
topic set. More precisely, we measure discriminative powerby conducting a statistical signiﬁcance test for every pairof runs, and counting the number of signiﬁcantly diﬀerent
pairs. We have 25 runs, so 25*24/2=300 run pairs are tested.
For signiﬁcance testing, we use the two-tailed paired bootstrap test, with α = .05 and B = 1000 bootstrap samples [14].
Note that this experiment is not about whether the metrics
are right or wrong; it is about how metrics can be consistent across experiments and as a result how often diﬀerences
between systems can be detected with high conﬁdence. We
regard high discriminative power as a necessary condition
for a good evaluation metric, not as a suﬃcient condition.
The discriminative power method also provides a natural estimate of the performance delta between two systems
required to achieve statistical signiﬁcance. This is done byrecording, for every run pair, the performance diﬀerence that
corresponds to the borderline between signiﬁcance and nonsigniﬁcance among the B = 1000 trials, and then by selecting the largest value among all run pairs (i.e. a conservative
estimate). This is one of the advantages of using the boot-― 49 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

Table 5: Discriminative power and estimated diﬀerence between two mean performances required for
α = .05.
metricIdiv-Q@10
Idiv-nDCG@10
α-nDCG
I-rec@10
nDCG-IA
div-nDCG@10
div-Q@10

disc. power
192/300=64.0%
188/300=62.7%
185/300=61.7%
181/300=60.3%
173/300=57.7%
171/300=57.0%
148/300=49.3%

diﬀ. required
0.12
0.13
0.10
0.20
0.08
0.10
0.12

strap test. Details can be found in [14].
Table 5 summarises the results of our discriminative power
experiments. For example, Idiv-Q is the most discriminative
metric of the ones we examined, managing to detest statistical signiﬁcance for 192 run pairs; Given 50 topics, when theperformance diﬀerence between to systems is 0.12 or higher
in Idiv-Q, this diﬀerence is usually statistically signiﬁcant.
It can be observed that9 :• Idiv-Q (γ = .5) is more discriminative than its components, namely I-recall and div-Q; similarly, Idiv-nDCG(γ = .5) is more discriminative than I-recall and divnDCG.
• Idiv-Q and Idiv-nDCG (γ = .5) are at least as discriminative as α-nDCG (α = .5) and possibly more discriminative than nDCG-IA. For example, Idiv-Q manages
to obtain seven more signiﬁcantly diﬀerent run pairs
compared to α-nDCG.
Hence Idiv-Q and Idiv-nDCG are at least as good as existing metrics from the viewpoint of robustness to variation
across topics and hence consistency across experiments, andour approach of combing I-recall and div-Q/div-nDCG isprobably beneﬁcial.

5.

CONCLUSIONSThis paper proposed simple, intuitive metrics called IdivnDCG and Idiv-Q for evaluating diversiﬁed search results.
They are designed to (a) retrieve documents that cover as
many intents as possible; and (b) rank documents that arehighly relevant to more popular intents higher than thosethat are marginally relevant to less popular intents. We
showed that they oﬀer several advantages over existing metrics: They do not require approximation and range fully between 0 and 1; Unlike the IA metrics, they pay attention tominor intents; Unlike α-nDCG, they can accomodate P (i|q)
and per-intent graded relevance. We have also demonstrated
that they correlate well with the existing metrics, and canbe more intuitive. We therefore argue that they are suitablefor evaluation with diversity test collections that accomodate either P (i|q) or per-intent graded relevance.
Our future work includes setting parameters such as γbased on clickthrough data (possibly per topic); explicitly
handling informational vs. navigational intents; and designing new diversity metrics that are directly motivated by auser model. However, we maintain that evaluation metrics
should be easy to understand and easy to compute.
9
According to these bootstrap tests, the “top ﬁve” runs inFigure 1 are not signiﬁcantly diﬀerent from one another interms of any of the metrics shown in Table 5. Whereas,some of these runs are signiﬁcantly better than MSDiv1 also
shown in Figure 1 in terms of several diﬀerent metrics.

6. REFERENCES[1] R. Agrawal, G. Sreenivas, A. Halverson, and S. Leong.
Diversifying search results. In Proceedings of ACM WSDM
2009, pages 5–14, 2009.
[2] B. Carterette and P. Chandar. Probabilistic models ofnovel document rankings for faceted topic retrieval. In
Proceedings of ACM CIKM 2009, pages 1287–1296, 2009.
[3] O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan.
Expected reciprocal rank for graded relevance. In
Proceedings of ACM CIKM 2009, pages 621–630, 2009.
[4] H. Chen and D. R. Karger. Less is more. In Proceedings of
ACM SIGIR 2006, pages 429–436, 2006.
[5] C. L. Clarke, N. Craswell, and I. Soboroﬀ. Preliminary
report on the TREC 2009 web track. 2009.
[6] C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova,
A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty anddiversity in information retrieval evaluation. In Proceedingsof ACM SIGIR 2008, pages 659–666, 2009.
[7] C. L. Clarke, M. Kolla, and O. Vechtomova. An
eﬀectiveness measure for ambiguous and underspeciﬁed
queries. In Advances in Information Retrieval Theory(ICTIR 2009), LNCS 5766, pages 188–199, 2009.
[8] K. Järvelin and J. Kekäläinen. Cumulated gain-based
evaluation of IR techniques. ACM Transactions on
Information Systems, 20(4):422–446, 2002.
[9] A. Moﬀat and J. Zobel. Rank-biased precision formeasurement of retrieval eﬀectiveness. ACM Transactions
on Information Systems, 27(1):Article No.2, 2008.
[10] P. Over. TREC-7 interactive track report. In Proceedings of
TREC-7, 1999.
[11] F. Radlinski, P. N. Bennett, B. Carterette, and
T. Joachims. Redundancy, diversity and interdependent
document relevance. SIGIR Forum, 43(2):46–52, 2009.
[12] S. E. Robertson. The probability ranking principle in IR.
Journal of Documentation, 33:130–137, 1977.
[13] T. Sakai. New performance metrics based on multigrade
relevance: Their application to question answering. In
NTCIR-4 Proceedings Open Submission Session, 2004.
[14] T. Sakai. Evaluating information retrieval metrics based onbootstrap hypothesis tests. IPSJ Transactions on
Databases, 48(SIG9 (TOD35)):11–28, 2007.
[15] T. Sakai. On the properties of evaluation metrics for ﬁnding
one highly relevant document. IPSJ Transactions on
Databases, 48(SIG9 (TOD35)):29–46, 2007.
[16] T. Sakai and N. Kando. On information retrieval metrics
designed for evaluation with incomplete relevanceassessments. Information Retrieval, 11:447–470, 2008.
[17] T. Sakai and S. Robertson. Modelling a user population for
designing information retrieval metrics. In Proceedings of
EVIA 2008, pages 30–41, 2008.
[18] M. Sanderson. Ambiguous queries: Test collections needmore sense. In Proceedings of ACM SIGIR 2008, pages
499–506, 2008.
[19] R. Song, D. Qi, H. Liu, T. Sakai, J.-Y. Nie, H.-W. Hon,and Y. Yu. Constructing a test collection with multi-intent
queries. In Proceedings of EVIA 2010, 2010.
[20] K. Spärck-Jones, S. E. Robertson, and M. Sanderson.
Ambiguous requests: Implications for retrieval tests,
systems and theories. SIGIR Forum, 41(2):8–17, 2007.
[21] E. Yilmaz, J. Aslam, and S. Robertson. A new rankcorrelation coeﬃcient for information retrieval. In
Proceedings of ACM SIGIR 2008, pages 587–594, 2008.
[22] C. Zhai, W. W. Cohen, and J. Laﬀerty. Beyond
independent relevance: Methods and evaluation metrics for
subtopic retrieval. In Proceedings of ACM SIGIR 2003,
pages 10–17, 2003.
[23] J. Zobel, A. Moﬀat, and L. A. Park. Against recall: Is it
persistence, cardinality, density, coverage, or totality?
SIGIR Forum, 43(1):3–15, 2009.

― 50 ―