III-I

III.

Relevance Assessments and Retrieval System Evaluation
M. E. Lesk and G. Salton

Abstract
Two widely used criteria for evaluating the effectiveness of
information retrieval systems are, respectively, the recall and theprecision.

Since the determination of these measures is dependent

on a distinction between documents which are relevant on the one hand,and documents which are not relevant on the other to a given query
set, it has sometimes been claimed that an accurate, generally valid
evaluation cannot be based on recall and precision.
A study was made to determine the effect of variations inrelevance assessments on the average recall and precision values used
to measure retrieval effectiveness.

Using a collection of 1200 documents

in information science for test purposes, it is found that large scaledifferences in the relevance assessments do not produce significant
variations in average recall and precision.

It thus appears thatproperly computed recall and precision data may represent effectivenessindicators which are generally valid for many distinct user classes.

1.

Introduction
Over the last few years, the interest in the design of automatic

information handling systems has steadily increased.

At the same time,it has become necessary to devote a good deal of attention to theevaluation of information systems in an attempt to identify thoseIII-2

factors which contribute to system effectiveness. Many criteria canbe used in such an evaluation process; furthermore, the factors which
may be most appropriate in one circumstance may not be in another. In
particular, different effectiveness indicators might be generated depending on whether one's viewpoint is the user's, the manager's, or theoperator's.

The manager, for example, may be most concerned aboutsystem costs, whereas the operator may be interested primarily in thecharacteristics of the equipment used in the process. The user, however,is not normally interested in the equipment, and may be only peripherally
concerned with costs. He does, however, want to make certain that thesystem is responsive to user needs.
Many recent efforts at retrieval system evaluation have been
based mainly on user criteria, and while several possible criteria are
available - including, for example, the type of presentation of theoutput, the amount of user effort needed during a search, the time lag
between submission of a query and the presentation of search results,
and the coverage of the collection being searched - it is generally
agreed that the two most important user-oriented measures are the abilityof the system to retrieve wanted and at the same time to rejectunwanted information.

As a result, several of the more recent evaluation

studies have used a test methodology based mainly on the computationof the recall and precision values applicable to a set of test queries[1,2,3].
Recall and precision are defined, respectively, as the proportion of relevant material actually retrieved, and the proportion of
retrieved material actually relevant.

In an ideal system, it may beIII-3assumed that everything relevant to a user's query is in fact retrieved(thus producing a recall of 1) while everything not relevant is rejected(producing a precision of 1 ) , In real life, conditions are not so
perfect, and it is generally not possible to achieve at the same time
both a high recall and a high precision.
In order to generate recall and precision values, it isnecessary first to differentiate retrieved from non-retrieved documents,and second to separate documents termed relevant to a query from thosetermed nonrelevant.

The second of these partitions must obviously depend

on a personal judgment either by the author of a given query, or by asystem operator, or an outside expert.

In any case, once a decision isreached about the relevance of each document to each query, it ispossible by examining the set of retrieved and nonretrieved documents tocompute unique recall and precision values.

Unfortunately, relevanceassessments tend to vary depending on who renders the judgment, and therecall and precision values obtained by using these assessments may
then turn out to be inherently unstable.

This question is further

investigated in the remainder of this study.

2.

The Relevance Problem
In a recent study of the relevance judging process, Cuadra andKatter recognize four main types of variables which potentially affectthe outcome of a relevance judgment [4]: First, the type of document
being judged, including its subject matter, level of difficulty, levelof condensation, style, and so on; next the conditions under which thejudgments must be rendered, that is the time available, the order ofIII-4

presentation and size of the document set, the type of task specification,and so on; then, the statement specifying the information requirement
which determines relevance; and, finally, the type of judge used torender the judgments, that is, his experience, background, attitude,
and so on. These variables are summarized in the chart of Fig. 1.
Additional variables may enter into the process if the judgment to berendered is not expressible as a simple yes/no decision.
Because of the obvious complexity of the judgment process,numerous authors have stated that stable relevance judgments cannot
possibly be obtained from individual informants. Fairthorne, forexample, has suggested that individual relevance judgments should be
replaced by global judgments representing a consensus of ideas byseveral independent judges [5]. O'Connor and Doyle have pointed outthat the expression of a user's information need can take manydifferent forms, and that it is not possible in consequence simply toclaim that "document A is relevant to query B" without appropriatequalifying statements [6,7,8].

Taube has drawn the conclusion thatrecall and precision are not concepts which can be properly definedor used in retrieval systems evaluation [9].
A number of studies have also been conducted to show thatdifferent

sets of relevance judgments are actually obtained under

different judgmental conditions. Thus, distinctions are madebetween "motivated" and "unmotivated" judges [10], and between judgments
based on an examination of full compared with partial document excerpts[11].

Furthermore, in the two most extensive studies of the judgmentprocess by Cuadra and Katter [4, 12] and Rees [13, 14] respectively, aIII-5Document

Judgment Conditions(subject matter, content,style, level of condensation)(time available, orderof presentation, definition of relevance)

Information Requirement
StatementJudge(knowledge, intelligence,experience, attitude,
use orientation, error
preference)(content, specificity of
information, textual
attributes)

Variables Related to Relevance Judgments

Fig. 1(adapted from Cuadra and Katter [4])III-6large number of factors are varied and the effect on the resulting
relevance judgments is observed.
The conclusion is sometimes drawn from studies such as thepreceding that the existing methodology in systems evaluation must be
revised, and that evaluation results based on recall and precision areunreliable and must be viewed with great caution. Cuadra and Katter statein particular [4]:"the first and most obvious implication is that one cannot
legitimately view 'precision' and 'recall' scores as precise
and stable bases for comparison between systems or systems
components, unless ... (appropriatecontrols are introduced)11
Rees voices similar misgivings in a somewhat different context [13]:"the lack of replication (that is experimental control
permitting duplication of the experiments) of the resultsof either the SMART [3] or the Cranfield studies [1] mustnecessarily introduce a note of caution to the existenceof 'rules' and generalizability of results".
While these sentiments appear at first to be perfectly justified,since the subjectiveness and variability of individual relevance judgments
cannot obviously be contested, the jump which is necessary to reach theconclusion that recall and precision results are unreliable because
relevance judgments are unstable has never been adequately proved or
substantiated.

Indeed, there exists some evidence that such a conclusion

cannot be drawn from the available evidence.

Giuliano and Jones, forexample, made a small study using a panel of three relevance judges.

III-7

Their findings are summarized as follows [2]:"for purposes of comparing retrieval performance curves
for two or more search options, it does not appear
to matter much whether the curves are for any one of thesingle judges, or whether they are the averaged curves
for a panel of three judges; the differences areprimarily ones of scale, and the relative positions of
the curves for the different search options tend to be
the same in all cases".

Rees and Schultz also find that the judgmental groups used in theirstudy agree substantially as to the relative positioning (I.e. ordering
in decreasing order of relevance to a search request) of the documents,
although the judges tend to assign to the documents different numericalratings [14].
The experimental evidence cited above may indicate that, contrary
to expectations, recall and precision results do not vary as widely as
the relevance judgments used generate them.

Several reasons can be citedfurther to support such an opinion:a)recall and precision data are normally given as averages
over many search requests; these averages may not besensitive to small variations in the results for individual
queries;b)

recall and precision data depend mainly on the relative
positions of relevant and nonrelevant documents, when the
documents are arranged in decreasing or increasing relevanceorder; individual changes in the composition of therelevant and nonrelevant document sets may have only aminor effect on the ordering of the sets as a whole;III-8c)disagreements among relevance judges may affect mostly theborderline cases, while preserving a general consensus
for a large set of items difinitely termed either relevant
or nonrelevantj such borderline cases normally receivea low position in the relevance ordering, and their effecton the recall and precision values may be expected to be
negligible;d)recall-precision results are often given as relative
differences between sets of different search and retrieval methods; the recall and precision results mayvary in such a way that differences between methods arepreserved even though the values for the individual methods
may change.

These questions are further examined in an experiment to be
described in the remaining sections of this study.

3.

The Experiment
The evaluation procedures incorporated into the SMART document

retrieval system lend themselves to a pairwise comparison of theeffectiveness of two or more processing methods.

Specifically, a numberof evaluation parameters are computed for each of the processing methods
under investigation.

A comparison of the corresponding measures for

two or more methods can then be used to produce a ranking of the methods
in decreasing order of retrieval effectiveness.
The following evaluation measures are generated by the SMART
system for each processing run [3]:a)a recall-precision graph reflecting the average
precision value at ten discrete recall points(from a recall of 0.1 to a recall of 1.0 in intervalsof 0.1) ;III-9b)

two global measures, known as normalized recall andnormalized precision, which together reflect theoverall performance level of the system;c)two simplified global measures, known as rank recall
and log precision, respectively.

The experiments described in the present report were designed to determine the degree of sensitivity of the SMART evaluation output to variations in the relevance assessments and used to compute the evaluation
measure.

If the recall-precision output obtained by SMART turns out tobe unstable because of the instability of the relevance judgments used,then an extrapolation of the results to other user populations anddifferent retrieval environments may not be possible.

On the other

hand, if the evaluation output remains stable, then the significanceof the results appears to be confirmed.
A collection of 1268 abstracts in the field of documentationand library science comprising about 131,500 English text words (thefIspraf collection) was used for experimental purposes.

The collection

includes most of the articles published in 1963 and 1964 in American
Documentation and several other journals in the information retrieval
area.

Eight different persons were used to generate a total of 48

different search requests in the documentation field; each person wasfamiliar with the library science field, either being a librarian himself or a student in library science, and each one was asked to produce six requests that might actually be asked by library science students.

To aid in the query generation, a detailed and carefully drawn

set of instructions was distributed to the group of query authors.

The111-10main criteria proposed for the query formulation are reproduced inTable I.
Each query was expected to represent a real information need,and had to be expressed in grammatically correct, and hopefully unambiguous English.

As usual for queries processed by the SMART system,

positive formulations were required, and the queries were to begenerated independently from the document collection;in particular, no"source" document was to be used for the formulation of any of the queries.
Following receipt of the query formulations from each of theeight authors, the texts of the document abstracts comprising thecollection were distributed, and each author was asked to assess therelevance of each document abstract with respect to each of his six
queries.

Dichotomous relevance judgments were to be used, asserting

either the relevance or the nonrelevance of each item for each, query.
Furthermore, the relevance criterion to be used was a strict one, inthe sense that relevance of a document was to be specified only
"if it is directly stated in the abstract as printed, or
can be directly deduced from the printed abstract, that the
document contains information on the topic asked for in thequery".
Since each query presumably represented an information need, an abstract
would thus be called relevant if the author felt that given the abstract
he would with great probability wish to consult the complete document.
After receipt of the relevance judgments from each of theauthors (the A judgments), a second, independent set of relevance judgments(the B judgments) was obtained by asking each person in the test group toIII-ll

judge for relevance six additional queries originated by six different
people, not including himself.

The same relevance criteria were used

for the second relevance judgments as for the original ones, the only
difference being that the A judgments were rendered by query authors,
whereas the B judgments arenonauthor judgments.

In order to preserve

independence, the B judges were not informed of the A judgments previously
obtained, nor was there any interaction between assessors either before
or during the judging process.
For each of the 48 queries, a set of four different document
sets thus became available, each consisting of the items termed relevant by
a different set of people as follows:

A set:

relevance assessed by query author;

B set:

relevance assessed by outside subject expert;

C set:

relevance asserted by either A or B assessor;

D set:

relevance asserted by both A and B.

The situation is summarized in Table 2.
A measure of agreement in the relevance judgments can be obtained for the query set from the material of Table 3.

For each query, thenumber of items is given for sets A and B, respectively, as well as thetotal number of distinct items (set C ) , and the total number of items
common to both sets A and B (set D ) .

Each query number listed in Table 3

is coded in such a way that the number ij is assigned to the query
authored by person i, with the second (B) relevance judgment being obtained
from person j.
The agreement among the relevance sets is measured as usualPositive Criteria for
Query Formulation

Negative Criteria for
Query Formulation

1.

Generate queries of realinterest to a potential
researcher or student

Avoid "exotic" topics anddoubtful subject matter2.

Formulate queries in clear,coherent, properly punctuated, grammatically correctsentences

Avoid metaphors, jokes, andallusions3.

Use from 50 to 100 words andup to 3 sentences to formulate queries

Do not submit queries corresponding to the contents of a
specific document; do not
rephrase specific document
contents

4.

Use positive formulations
stating what subject areas
are actually wanted

Avoid negative formulations
and clauses introduced by"exceptn, "other than","not", etc.

5.

Use homogeneous query
formulations representing
a single topic

6.

Use only common abbreviations

Principal Criteria for Query Formulation
Table 1

111-13

Group of Judges |ExplanationA

Original group of query authors. Each
person in A group made relevancejudgments for his six queries

B

Nonauthor judges. Each person in B
group made relevance judgments for six
queries corresponding to six different
authors from A group

C

Document is relevant to a given query
if either the A judge or the B judge
termed it relevantDDocument is relevant to a given query
if both A and B judges termed it
relevant

Relevance Judgment Groupings

Table 2

111-14by dividing the total number of common items by the total number ofdistinct items (A0B)/(AUB).
of Table 3.

The numeric values are given in column 6

An average agreement score is given for each author incolumn 7 of Table 3,This score is seen to vary from a high of 0.53

for author 6 to a low of 0.11 for author 8.

The overall agreement

for all 8 authors is seen to be slightly higher than thirty percent(0.3074).

This figure is believed to be typical of the consistency

obtainable under independent conditions from different assessors.
It also agrees with comparable figures contained elsewhere in theliterature.
It remains to show how such a relatively low consistency
level is reflected in the evaluation output.

This is described further

in the next section.

4.

Experimental Results
Three of the principal automatic language analysis procedures

incorporated into the SMART system are used with the Ispra collection
under study.

The methods known as word form, word stem, and thesaurus,

respectively, may be described as follows:a) word form:

texts of document abstracts and queries are
reduced by removal of common words and final
1 fs

endings, and weights are assigned to theremaining word forms; the reduced texts arethen matched to obtain document-query correlation coefficients;b) word stem:

texts are treated as above, except that com-

111-15
Query
NumberFirst
Judge

1 Number <Df Relevant

Second
Judge

A 1B

1

J D

"
(AUBl (AnB),J Agreement

Average
Agreementwith Author

J AoB _ D
j AUB =S C
T

12
13
14
15
16
17
21
23
25
26
27
28
31
32
34
35
36
37
41
42
45
46
47
48
51
52
53
54
56
58
61
63
64
65
67
68
71
72
73
74
75
78
81
82
83
84
86
87
Totals

17
18 26
1 9
4
3
4
3
19
6 21
4
22
25 37
10
7
9 14
2
9
13 15
7
20
25 ' 37
8
32
8 39
1
20
7 ! 252
19
8 1 23
4
14
17 23
8
16 19
7
4
3
4
5
2
18 j 226
2
27 | 20 45
2
27
27 35
IS
34
8 41
1
6 11
5
0
14
8 17
5
19
20 26
13
41 42
8
7
10
8 12
6
10 10
8
8
25
33 44
14
10
11 16
5
72
16 78 j 10
31 ! 14 42
3
13
9 19
3
34
25 47
12
33
22 40
15
10 11
6
5
23
49 61
11
12
12 17
7
6
9
7
4
11
10 11
ia
7
9
9
7
14
28 31
11
14 15
9
8
19
11 23
7
9
22 24
7
6
5
6
5
12
22 24
10
22
6 24
4
37
32 54
15
34 1 4 35
3
21
10 28
3
18
0 18
0
17
8 25
Q

853

713 1260306

0.34620.75000.19050.27030.14290.46670.21620.02560.08000.17390.3478.,... — |/> Author 1
0.3611/\
1 Author 2f0.17570.2105U

0.40000.09090.04440.54290.0244|]I Author 3[
0.183800.2941.0.5000
0.16670.5000Q.8000Q.3182

1 Author 4[0.42981/0.31250.12820.07140.15790.25530.37500.45450.18030.41180.44440.9091

1[

Author 5
0.2167(1
1

Author 6f

0.52970.77771j

0.35480.53330.30430.29170.8333,0.41671

0.16670.27780.08570.1071P

1

Author 7f0.4557f

Author 8
0.106700uOverall
Average|

Agreement of Relevance Judgments
Table 3

- . . . , . , i... •0.30741

111-16

o
o
0)

E E „
kO
U-

0)
•CO

D
w
D
O

T)
w.

TJ
w

««
0>

o

o

x:

CKO-2

-1
/4V
/

£ £o H
<]
o

CD

10

c

E

< H CD

03

/

< /

•*"

E *-*

<

5

2

c
o
"o

</>

-L
00

<D

o

« "° t>

H CVJ
"35

«2

o.
«o

X
OJ

»-•

a>
CD
CD
CM

>
a

.-

.2? =1
CN

c
cr o
c
o

C/>

c
a>

E
3

<

"3

00 o
* -<D

a)

O O

i
—

o co
w o

O

o

cr

en
CM

111-17

E

£ »

o 2 2
LL en D

(0

c
0)

E
•D

3

a
o
i/>

<

"c ^-*
a> a)

E £

3
a o
o >
ex X)
00 .2
CD

»-

3

^
0)

a>

a) C
0)

^

o

IM
E E

o

C

t.

°

o

o

o

°

a:

o
a>

<*> c
«> m

w

l/>

s

D0<I

3
O

</>

ii

0)

E

o*
J<0

< o a

iV

A*
^<\J

c
o
o
a>

«•—
c
a>

00

CP

CM

"O
3

=
>
o o
a -~*
0)

cr

Cn o

111-18i Recall

1 0.1
0.3
0.5
0.7
! 0.9

A

B

.405
.196
.102
.039
.023

.467
.206
.081
.028
.018

Average Precision Values
A-B
C
D
-.062
-.010
+ .021+ .011

D-C

.664
.322
.165
.07Q
.023

.216
.087
.037
.012
-.006

Average Precision Values
A-B
C
D

D-C

+ .CLQ5

.448
.235
.128
.0-58
.029

a). Word Form Process

Recall
0.1
0.3
0.5
0.7
0.9

A

B

.514
.363
.243
.162
.095

.524
.375
.283
.182
.093

-.010
-.012
-.040
-.020
+ .002

.503
.376
.266
.167
.056

.693
.434
.308
.196
.111

.190
[ | .058
.042
.029
.055

b) Word Stem Process

Recall

0.1
0.3
0.5
0.7
0.9

A
.612
.406
.293
.204
.112

B

Average Precision Values
A-B
C
D

D-r

.627
.398
.307
.199
.106

.801
.485
.319
.227
.119

.197
.052
.004
.014
.006

-.015
+ .008-.014+ .005+ .006

.604
.433
.315
.213
.113

c) Thesaurus Dictionary
Average Standard Recall and Precision Values
for 3 Analysis Methods and 4 Types of Relevance Judgments
Table 41

111-19

plete suffixes are removed from text words toreduce the texts to weighted word stems ? the querydocument matching process remains the same as inprocess a ) ;c}thesaurus:

each word stem produced by procedure b) is looked
up in a thesaurus providing synonym recognition,and the resulting weighted concept identifiers
assigned to queries and documents are compared(instead of word forms or word stemsl.

The output produced by the SMART system consists of superimposed
recall-precision graphs exhibiting averages over a complete query set
for several processing methods {3]. The method which generates the curve
closest to the upper right-hand corner of the graph (where recall andprecision are equal to 11 exhibits the best performance.

Under normal

circumstances, an evaluation of performance for a variety of processing
methods does not require a detailed comparison of the actual recall andprecision values, but only an examination of the ranking of the corresponding recall-precision curves.

Thus, to show that the performancemeasures are insensitive to changes in the relevance judgments, it issufficient to observe a consistent ranking of the recall-precision graphs
obtained from the several processing methods.

The data for the four typesof relevance judgments (A, B, C, and D) are shown averaged over the 48
queries in Fig. 2.
The following conclusions may be drawn from the output of Fig. 2:a)

all four sets of relevance judgments produce the sameranking of the processing methods; in particular, theword form process is always much less powerful than the111-20other two procedures, and the thesaurus process isslightly better than the word stem match;b)

the main difference in the output produced by the A
and B judgments is the somewhat closer agreement
between word stem and thesaurus runs for the B
judgments than for A;c)the best results in terms of recall and precision
are obtained for the D judgments which represent
the agreement between both A and B relevance judges;
for low recall, the precision is about 20 percent
higher for D than for A, B, or C.

While it is clear from the output of Fig. 2 that the SMART
evaluation output does not vary witlx variations in the relevance judgments,
it may be of interest to examine the data in somewhat more detail.
Table 4 contains the numeric values corresponding to the curves of Fig. 2,
The average precision is given for each of five recall points for thefour curves of Fig. 2.

In addition, the numeric precision difference isgiven at these same recall points between the A and B curves (in column
4 of Table 4), and between the C and D curves (in column 7 of the table).
It may be seen that the maximum difference between the averaged A and B
output occurs for the word form process at very low recall (precisiondifference of 0.06).

The normal precision difference for the two setsof relevance judgments is about 1 to 2 percent. For the thesaurus run,which exhibits the best performance, the maximum precision difference isonly 0.015 at low recall, with a normal difference of less than one percent.
From the output of Table 4 and Fig. 2, it appears reasonable
to conclude not only that the performance methods are ranked in the111-21

Query Numbers
41-48

12-17

21-28

31-37

.028
.016
.026
.015
.001 , .002

Word Form

A
B
A-B

.011
.013
-.002

.023
.014
.009

Word Stem

A
B
A-B

.077
.085
-.008

.056
.03 7
.019 -.028

.162
.165
-.003

A
B
A-B

.133
.122
.011

.078
.079
-.001

.254
.236
.0.18

Thesaurus

a)

-.013

71-78

51-58 '61-68

81-87

Ail

.018 1 .021
.017 |.010.004 1
.008

.034
.016
.018

.021
.013
.023
.019
-.006 ,-.002

(05)
/S»45

.063
.065
-.002

.108
.038
.104
.048 |
.004
-.010

.134
.162
-.028

.067
.074
-.007

(op)

/5l5

_§_

.120 1.118.002

Average Rank Recall Differences (3 Analysis Methods)

Query Numbers
12-17

21-28

31-37

41-48

51-58

61-68

71-78

81-87

All

.363
.499
-.136

.437
.307
.130

.215
.236
-.021

.209
.300
-.091

-1
1

.663
.566
.097

.256
.299
-.043

.469
.518
-.049

.764
.720
.044

.366
.533
-.167

.557
.590
-.033 |! Word Form

A
B
A-B

.172
.289
-.117

.243
.228
.015

.279
.176
.103

.481
.406
.075

.342
.258
.084

Word Stem

A
B
A-B

.511
.489
.022

.416
.432
-.016

.358
(52)

.659
.632
.027

.385
.432
-.047

A

.673
.641
.032

.507
.562
-.055

.321

.717
.659
.058

.472
.445
.027

1
Thesaurus

b).

B
A-B

(|o)

Average Normalized Precision Differences (3 Analysis Methods)

Average Rank Recall and Normalized Precision for Each of 8 Query Authors

Table 5

111-22same order, no matter which of the four sets of relevance judgments
is used, but also that the actual performance differences resulting fromdifferences between author and nonauthor judgments are negligible. This
point can also be made by looking at the individual performance differences
for each of the 8 query authors as shown in Table 5.
Table 5 exhibits the average rank recall (in Table 5(a)) andaverage normalized precision (Table 5 (b)) for the six queries originatedby each of the eight authors.

In each case, the average obtained byusing the author relevance judgments is shown (case A), followed by
the average for the same six queries using the nonauthor judgments,
followed finally by the difference of the measures between A and B.

It

may be seen once again that the processing methods are ranked identicallyby 7 out of 8 authors from the best method tthesaurus) to the worst(word form).

Only for the queries of author 6 (nos. 61-68X does theword stem process produce slightly better results than the thesaurus
method; however, the word form process is inferior even for that author.
When the B relevance judgments are used, the same ranking is again
obtained for 7 out of 8 query sets.

For queries 61-68, the word stem

process is again superior to the thesaurus, while for queries 31-37 and
51-58, the B judgments produce approximately equal performance for word
stem and thesaurus.

The differences in rank recall and normalized pre-cision obtained for the two sets of relevance judgments (A and B) are
shown in row 3 of Table 5 for each dictionary.

The differences areagain exceedingly small.
In the next section, performance results are given for individual
queries, and an attempt is made to explain why the relatively large111-23

differences in the relevance judgments do not lead to substantial
differences in the performance parameters.

5.

Judgment Consistency and Performance Measures
In order to explain why the average recall and precision datapreviously exhibited are relatively insensitive to differences in therelevance assessments, it is necessary to look at the performance
characteristics for some individual queries.
Consider first the data of Table 6 giving normalized recall andnormalized precision figures averaged over the 48 queries for the four
sets of relevance assessments.

It may be seen that with the sole ex-

ception of the word form normalized recall, the highest performance isobtained in each case using the D judgments followed by B, A, andfinally C.

The D judgments, however, represent the agreement in therelevance assessments between authors and nonauthors, and the corresponding relevance sets are therefore produced under reasonably stringent
conditions (at least two independent people must agree before an item
is termed relevant).

On the other hand, the C judgments are produced

by relatively free criteria, since an item is called relevant if either
one of two independent judges calls it relevant.

It appears then fromthe output of Table 6, that the D judgments which are designed to selectthose documents most certainly relevant to each query, also selectthose documents most efficiently retrieved by the computer system.

Thatis, the query-document pairs which are most closely and unarguably
related are exactly the pairs on which the retrieval performance is best.
This result is confirmed by the output of Table 7, where thoseGroup

Normalized Recall
Word
Stem
Form
Thes

Normalized Precision
1 WordJ Form
Stem
Thes

A

0.52890.83400.88580.20900.46900.5570B

0.52490.84520.89040.24730.51040.5850C

0.62340.82490.87770.18040.36550.4885

D

0.54930.87590.92120.34920.57770.6403

Normalized Recall and Precision Averages(3 Analysis Methods and 4 Types of Relevance Judgments)(Cosine Correlation, Numeric Vectors)

Table 6

111-25

Performance Measure
Top 12B
A
betterbetterRelevance
Judgment
Consistency

Middle 24A
Bbetterbetter

|Bottom 12
BA
better better

Top 12

4

5

1

2

0

0

Middle 24

1

2

6

11

4

0

i Bottom 12

0

0

0

4

5

3

Correlation of Judgment Consistency with Performance

Table 7

Performance Measure:

Relevance Judgment Consistency:(NP. for A) + (JSIP for B) + (NR for A) + (NR for B)

No. of relevant in D
(No. of relevant in A)(No. relevant in B)

111-26

queries are selected which exhibit the best agreement between therelevance assessors.

Such queries represent relatively unambiguous,

closely related query-document sets.

It is found, just as with the

D judgments, that these closely related pairs produce the best
retrieval performance.
Table 7 contains a plot of performance effectiveness versus
consistency in the relevance judgments.

Specifically, a single per-formance measure and a single measure of relevance consistency are computed for each query as follows:(NP for A) + (NP for B). + (NR for A) + (NR for B)
performance measure = — —
T
and
. „
.
Number of relevant items in D
judgment consistency =•(No. of relevant in A)(No. of relevant in B),

where NP and NR are normalized precision and normalized recall, respectively.
The 48 queries are then arranged into three groups for performance Cthe
top 12, the middle 24 and the bottom 12), and into three groups forrelevance judgment consistency.

Table 7 shows how relevance consistency

correlates with performance.
It may be seen that performance is best for those queries withthe best relevance consistency.

Indeed, 9 of the 12 queries in each

top group are also in the other top group. Contrariwise, not a single
query from the bottom 12 in judgment consistency is in the top 12 forperformance, and vice versa, not a single query from the bottom 12 inperformance is in the top 12 for judgment consistency.

111-27

The performance indicators of Table 7 are further subdivided
into queries for which the B judgments provide the better performanceand those for which the A judgments are superior.

In the former case,the nonauthor judgments proved more useful than the author judgments,
indicating possibly that these queries are ambiguous or poorly formulated.
It may be seen from the table that this is the case for a total of
5+7+9 = 21 queries out of 48, of which 9 are ranked in the bottom 12
for performance.
It is now possible to explain why the recall-precision output
is basically invariant for the collection under study, even though theagreement among relevance judgments is relatively low:a).

on the one hand, the performance is best for thosequeries with the best consistency in the relevancejudgments;b)on the other hand, the recall and precision measures
are most sensitive to documents (both, relevant andirrelevant) retrieved early in the search, that is,
documents with low rank.

The conclusion is then obvious that although there may be a considerable
difference in the document sets termed relevant by different judges,
there is in fact a considerable amount of agreement for those documents
which appear most similar to the queries and which are retrieved earlyin the search process, (assuming retrieval is in decreasing correlation
order with the queries).

Since it is precisely these documents whichlargely determine retrieval performance, it is not surprising to findthat the evaluation output is substantially invariant for the different

111-28

sets of relevance judgments.
The situation is illustrated by a typical query (number 12) inFig. 3.

The first row of Table 3 shows that for this particular query,the number of relevant items identified by A was 17, while the B judge
identified 18 relevant documents. The total number of distinct relevant
items was 26 of which 9 were chosen in common by the A and B judges.
The agreement score is 0.3462.

The ranks of all 26 relevant documents

are given in Fig. 3(a), with the common items being shown underlined.

It

may be seen that of the 8 relevant items with the lowest rank (fromrank 1 to rank 25) there was agreement between the judges for 6 items;on the other hand, of the 8 relevant items retrieved with highest rank
(ranks 178 to 832) there was not a single agreement between the A and
B judges. The two recall-precision graphs for query 12 are shown in
Fig. 3(b); they are seen to be remarkably similar, reflecting the factthat for the top 25 documents retrieved, the differences in relevancejudgments between the A and B judges are very small indeed.
In conclusion, it can be stated that, if the relevance assessments
obtained from the query authors used in the present study are typical ofwhat can be expected from general user populations of retrieval systems,
then the resulting average recall-precision figures appear to be stableindicators of system performance which do in fact reflect actual retrieval
effectiveness.

6.

Machine Search Effectiveness
It has been said elsewhere [17] that the retrieval effectivenessobtained with the automatic text processing methods incorporated into the111-29

SMART system appears to be roughly equivalent to the effectivenessobtainable with presently operating manual, or semi-automatic retrieval
systems.

It may be of interest to ask how this presently achievable

performance compares with the performance of the best possible imaginable
delegated search system.

The differences between present performanceand such an optimum delegated search system may then give an indicationof the amount of improvement in performance which may eventually result
from future developments.
It is not completely unreasonable to assume that the best
possible delegated search system is one where a subject expert completely
reads through an entire document collection and ranks each document
in decreasing similarity order with a given search query.

Such a system,

which for obvious reasons is not operationally implementable, should intheory be superior to any search system based on indexing or on otherreduced document representation.

The set of B searchers used in thepresent experiment can then be assumed to constitute such an ideal
search system, since they in fact were asked to search through the complete
document collection for each query.
A comparison has been made between the amount of material"retrieved" by the B searcher (that is the number of documents termed
relevant by B ) , and the number of relevant items retrieved by themachine search using the same cut-off as the B searcher.

In both cases,

relevance is determined by using the author (A) judgments as criteria.
Specifically, "optimum recall" and "optimum precision" figures arecomputed for each query by evaluating the performance of the B searcher(in comparison with the A relevance judgments) as follows:

o
o

X

<D

E

X

13

X

CO
X
X

</>

X

E
X

c
o

0>
X

O

X

CD

ml
O
ro
mi

CVJI

°l

J *

X

X

O

a

CD
c
<Q o

O

tfl

u

</»

in

0)

o

L.

t

0i

0)
o cr

X

fO .Q

X

o

X
X

X

X

X

X

X

CxJl
GO

CVJ

C\J

d

X

X

X

=1

X

X

X

o>

X

CxJ|

X

X

x

-1

X

X

X

RANK

X

!2|

<

CD

.

c
o

.2
o
a>

GO

c

<

CM

° m°

* - 0)o 3 a>EXCo « 0)oao o a)
N a

rr >* • - '

X

i-

CD

*l

X

3

a

<\J

>

in"

O

+.
X

3

<o
*
ro

Q

X

(NJ

col

CM

o
o

X

0)

o» o»
•o -a

0)
X

tt)

E E

c
X

c

col

X

c

cor

00
CO

X

lev

105 133 178 179 196 199 277 298 322 832

111-30

J_

*.
O

fO

CVJ

d

d

CM

a,
c
O

rf !
— o
O w
—

a>

w
a>

CL
»

t

E

o

u

Q.

o

en

•H

111-31,,
no. of relevant in (AfiB)/r,v
optimum (B)recall =no. of relevant in A,„*
. .
no. of relevant in (AOB)
optimum (B) precision =:;—:no. of relevant in B
These optimum recall and precision figures are then compared with themachine performance, using the thesaurus dictionary for analysis purposes.

To permit a fair comparison, the number of items retrieved in

the machine search must be the same as the number of items "retrieved"by the B searcher; the cutoff in the machine search is therefore set atthe number of relevant items identified by B.

The machine recall andprecision are defined as follows:,.
.,.,
no. of relevant in A retrieved before cutoff
machine recall =no. of relevant in A

machine precision =no. of relevant in A retrieved before cutoff
total no. of relevant in B (.cutoff)

The output of Table 8 shows that the overall machine search
results are about 25 percent lower on the average than the ' B 1 results.
For some query sets (for example 41-48, and 71-78), the results areapproximately equivalent, and for five queries out of the set of 48,
the machine performance is in fact better than that of the B searcher.
It is seen in Table 8 that the average recall and precision for the B
searcher are about 0.46, whereas the comparable machine figures are
0.32.

These figures once again demonstrate that the improvements

obtainable by refinements in the search and analysis techniques (from0.32 to 0.46) are relatively modest, in comparison with the desirable
perfect system where recall and precision are close to 1.

The gap

111-32

between the complete human search and a perfect search (from 0.46 to 1)
appears to be due to ambiguities inherent in the query formulation process
and to the difficulties of reconciling the user's view of a subject areawith the subject analysis provided for a given document collection. Thislatter gap may well never be bridged by any search and retrieval system
likely to come into existence in the foreseeable future.
To summarize, several sets of relevance judgments are used in
the present study in conjunction with a document collection of over1200 items in library science and documentation.

The retrieval resultsin terms of recall and precision obtained for the various relevance
sets are substantially identical, even though the overall agreement
among the relevance assessments is only about thirty percent.

This fact

is explained and the conclusion is drawn that there appears to be noreason to reject previously published evaluation results for manual or
automatic searches, because of uncertainties and instabilities in thecomputation of the performance measures.

It is pointed out again thatthe absolute performance achievable under present conditions, or likely
to be achieved in the future, is much lower than the theoreticallydesirable optimum.

111-33

12-17

21-28

31-37

41-48

51-58

61-68

71-78

81-87

All

j Optimum Search[ Machine Search(Thesaurus}

0.51

0.50

0.35

0.34

0.59

0.25

0.54

0.68

0.46

0.36

0.19

0.17

0.43

0.16

0.40

0.48

0.11

0.32

i Difference

Q.15

Q.31

0.18

-0.09

0.43

-0.15

0.06

0.57

0.14

aI

Optimum Search
1 Machine Search
1 (Thesaurus}_
Difference

Average Optimum and Machine Recall

12-17

21-28

31-37

41-48

51-58

61-68

71-78

81-87

All

0.52

0.49

0.45

0.38

0.54

0.35

0.47

0.52

0.46

0.40

0.24

0.17

0.33

0.23

0.31

0.35

0.27

0.32

0.12

0.25

0.28

0.05

0.31

0.04

0.12

0.25

0.14

b).

Average Optimum and Machine Precision

Optimum (B) Recall

Relevant in D
Relevant in A

Optimum (B) Precision

Relevant in D
Relevant in B

Machine Recall

Relevant in A before cutoff
Relevant in A

Machine Precision

Relevant in A before cutoff
Relevant in B (cutoff)

Comparison of Optimum with Machine (Thesaurus) Performance

Table 8References[1]

C. W. Cleverdon, J. Mills, and E., M. Keen, Factors
Determining the Performance of Indexing Systems:
Vol. 2, Test Results, Cranfield, England 1966.

[2]

V. E. Giuliano and P. E. Jones, Study and Test of a
Methodology for Laboratory Evaluation of Message
Retrieval Systems, Report ESD-TR-66-405, A. D. Little,
Inc., Cambridge, August 1966.

[3]

G. Salton and M. E. Lesk, Computer Evaluation ofIndexing and Text Processing, Journal of the ACM,
Vol. 15, No. 1, January 1968, pp. 8-36.

[4]

C. A. Cuadra and R. V. Katter, Experimental Studies
of Relevance Judgments: Final Report, Report TM-3520,
Vol. 1: Project Summary, Vol. 2: Description ofIndividual Studies, System Development Corp., Santa
Monica, June 30, 1967.

[5]

R. A. Fairthorne, Implications of Test Procedures,in Information Retrieval in Action, Western Reserve
University Press, Cleveland, 1963, pp. 109-113.

[6]

J. O'Connor, Relevance Disagreements and Unclear Request
Forms, American Documentation, Vol. 18, No. 3, July 1967.

[7]

J. O'Connor, Some Questions Concerning Information Need,
American Documentation, Vol. 19, No. 2, April 1968.

[8]

L. B. Doyle, Is Relevance an Adequate Criterion in
Retrieval System Evaluation, Proceedings of the 26th
Annual Meeting, American Documentation Institute,
Chicago, October 1963.

[9]

M. Taube, A Note on the Pseudomathematics of Relevance,
American Documentation, Vol. 16, No. 2, April 1965,
pp. 69-72.

[10]E. D. Dym, Relevance Predictability - Investigation,
Background and Procedures, in Electronic Handling ofInformation: Testing and Evaluation, A. Kent, et.al.
editors, Thompson Book Co., Washington, p. 175-185.

[11]

D. L. Shirey and M. Kurfeerst, Relevance Predictability Data Reduction, in Electronic Handling of Information:
Testing and Evaluation, A. Kent et.al. editors, Thompson
Book Co., Washington, pp. 187-198.

111-35

References(contd)[12]

C. A Cuadra and R. V. Katter, Opening the Black Box
of Relevance, Journal of Documentation, Vol. 23,
No. 4, December 1967.

[13]

A. M. Rees, Evaluation of Information Systems andServices, in Annual Review of Information Science andTechnology, Vol. 2, C. Cuadra, editor, Interscience
Publishers, New York, 1967.

[14]

A. M. Rees and D. G. Schultz, A Field Experimental
Approach to the Study of Relevance Assessments in
Relation to Document Searching, Final Report to the
National Science Foundation, Center for Documentation
and Communication Research, Case Western Reserve
University, October 1967.

[15]

G. Salton, et. al., Scientific Reports on the SMART
System to the National Science Foundation, Nos. ISR-11,
ISR-12, ISR-13, Department of Computer Science, Cornell
University, Ithaca, New York, June 1966, June 1967, andJanuary 1968.

[16]

G. Salton, Search and Retrieval Experiments in Real-Time
Information Retrieval, Proceedings IFIP Congress '68,
Edinburgh, August 1968.

[17]

G. Salton, A Comparison Between Manual and Automatic
Indexing Methods, Technical Report No. 68-11,
Computer Science Department, Cornell University, March 1968.

111-36

Appendix

It is shown in this appendix that the ranking in decreasingorder of performance for several processing methods stays constant underconditions of considerable generality, assuming that the performanceorder is defined by the usual recall and precision measurements.
A perfect relevance judge can be characterized by the factthat he will call a relevant document in fact "relevant" with a probability equal to 1.0, while calling a nonrelevant item "relevant"with a probability equal to 0.0. A somewhat slipshod judge who makes
random errors in judgment can be characterized by probabilities p
and p, where p is the probability that he will call a relevant

document "relevant", and pis the probability that he will call anonrelevant document "relevant". As p decreases from 1.0, and pr*rnr
increases from 0.0, the judge becomes increasingly inaccurate.
Consider a retrieval system, operating with procedure a, at
a recall R and precision P , measured by the perfect judge. Callaathe number of documents retrieved n, the total number of documentsin the collection N, and the total number of relevant items G.
The performance of this system can be evaluated using judgments madeby a slipshod

judge. The total number of relevant retrieved in P »n;aof these P »n*p are called relevant. Furthermore, (1-P )-n nonrelevantaraare retrieved; of these p • (1-P )*n are called relevant. The apparentrnra
precision using the slipshod judge for evaluation is therefore:
(P np
T,I

P" =
a

a

+ (1-P )np

r

a

n

^nr

)
_

,

N

= P (p -p ) + P a r nr
nr

Ill

Similarly, the apparent recall turns out to be(R G(p-p ) + np )a r r ^nr*nr
G(p -p ) + Np
r nr
nr

Ra

It is obvious that if p >p , P1 is monotonically increasing with P ;r nraaalso R1 is linearly increasing with R .

That is, any judge with p >pranks retrieval methods in the same order of performance as does theideal judge; and any two inaccurate judges for whom this criterion issatisfied therefore produce identical performance rankings.
if p

Furthermore,<<p , the above transformations are equivalent to changes ofscale, and even percentage changes in the measures will be accurately
reproduced.

In practice, p

is expected to be very small, so that

this condition is likely to be fulfilled.
Of course, the statistical expectations derived here, do not
imply that random variations may not produce individual queries forwhich an inaccurate order is apparently indicated.

If a reasonable

number of queries is averaged, however, the above results are expected
to hold.
It should be noted that the constraints placed on the relevancejudgments are very weak:

it is necessary only that a judge be more likely

to label relevant material as "relevant" than he is to label nonrelevantmaterial as "relevant".

It does not matter if only a small fraction of therelevant material is identified as relevant, so long as less than thatfraction of the nonrelevant material is called relevant.

It does notmatter either if the total number of documents called relevant is greaterthan the true number relevant, less than the true number, or equal to it;nor does it matter if the majority of the documents labeled relevant are
actually nonrelevant.

Because of the weakness of the constraints; it istherefore most unlikely that any misranking of the performance of retrieval methods can result from inferior relevance judgments.