ISR-9
August, 1965

XXI-1

XXI. EVALUATION VIEWPOINTS IN DOCUMENT RETRIEVAL
J. Rocchio

Summary
The fundamental notions involved in measuring the performanceof a document retrieval system are examined, and it is concluded that
there are two different viewpoints which may be taken depending on one's
basis assumptions.

The rationale for each of these is examined and thedifferences between them is discussed.

Some illustrations are given withrespect to the effect of the evaluation viewpoint on precision vs. recallmeasures.

1.

Introduction
A document retrieval system operating on a retrieval request pro-duces a. partition of a source document collection into two disjoint subsets, the retrieved subset and thenonretrieved subset. The basis for

judging the performance of the system is a comparision between this partition and the one which the originator of the query would produce if hewere to examine every source document.

In most cases such a direct com-parision is not feasible and thus various artifact^ (such as sampling, useof contrived queries, etc.) are used to approximate the ideal case. However in all instances the underlying basis for system evaluation remains
the same*XXI-2

In some retrieval systems a retrieval operation may produce aranking or even a metric ordering of source documents with respect tothe input query,. In these cases questions of "degree of relevance"
can be avoided by inducing a partition from the ordering using a cutoff criterion, or by requiring that each relevant document (one which
the user would retrieve) be ranked above every member of the nonrelevant seto

In this latter case then the basis for evaluation is a com-parison between the ordering induced by the system and the ordering induced by the user rather than between the respective partitions.
In either case, if we assume that the set of documents relevant
to each input query is both known and fixed, any performance evaluation
is a measure of (a) the indexing transformation which produces the images
manipulated by the system; (b) the query image (since we assume that thequery structure for a fixed relevant set may vary), and (c) the searchor matching function which operates on these images to produce an effective partition of the source document collection.
While the above outlines the basic mechanics of retrieval systemevaluation, it neglects an important aspect of evaluation which has often
been overlooked, namely the evaluation viewpoint.

Under the above as-sumptions about the system model there still remains an important choice
to be made by the evaluator. Since the notion of relevance is so complex and so dependent on personal factors, any system evaluation mustclearly be statistically based.

Thus, in effect, any evaluation of adocument retrieval system must in fact be an estimate of the degree towhich the systems ability to detect relevance matches that of humans.

XXI-3The validity of such an estimate based on the system performance withrespect to some finite sample set of queries depends on how representative the sample set is of the real life environment of the system*Itis not however this question to which we address ourselves but rather
to a related one^ namely whether the evaluation viewpoint considers retrieval queries as atomic, or whether the relation between a relevant
document and the set of nonrelevant documents (implicitly defined byeach query for all members of the document set relevant to that query)is considered atomic0

These alternative evaluation viewpoints called

macro and micro evaluation respectively are both tenable$ and in fact
have been compared one against the other without the comparers* appreciating the distinction,

20

Macro Evaluation
Macro evaluation is a query oriented viewpoint, pnd as suchany performance index formulated, on this basis would be query distributed or averaged on a per-query basis*

The justification for con-sidering queries as atomic is simply that this corresponds to the viewpoint of the system user*

The user interacts with the retrieval system

via his retrieval request^ and would thus judge the usefulness of thesystem on the basis of its performance with respect to his request,. The
sample mean of an evaluation parameter obtained by averaging over thetotal number of queries represents an estimate of the worth of the system
to the average user*

An underlying assumption of this approach is ofcourse that the distribution of results over the sample set of queries
allows a meaningful average performance per-query to be produced*

XXI-U
If the performance distribution is too wide, one might try to categorize requests in some fashion so as to produce a meaningful performance
estimate for each category. This of course would still be done on a query
average basis.

3. Micro Evaluation
Micro evaluation stems from a document oriented viewpoint, in which
one assumes that the determining element of system behavior is the relation
between a relevant document and the set of nonrelevant documents. With thisviewpoint each query provides a set of samples of this relation, and thus
any performance index is document distributed or averaged on a per document
basis.

This approach makes it necessary to justify that the set of samples

provided by a single request are statistically independent and reflectiveof the relevance relation which pertains in general. If this is not true,
the micro evaluation viewpoint will weight any performance index so derived
heavily towards the systems behavior for those requests having a large number of relevant documents, thus distorting the statistical validity of theestimate.

It is not however, the primary purpose of this discussion to es-

tablish which of these evaluation orientations has more merit, but rather
to point out the differences so that the implications of each are clearly
understood#xxi-5k•

ExampleAs an illustration of the effect of the evaluation viewpoint on

performance indices we consider now the precision-recall curve of Cleverdon which has been perhaps the most widely used evaluation tool for document retrieval systems*

As Cleverdon has conceived and used this mea-

sure it is a. micro evaluation performance index*
Define recall as the number of relevant documents retrieved divided by the total number of relevant documents, and precision as thenumber of relevant documents retrieved divided by the total number ofdocuments retrieved.

Clearly these parameters of a retrieval operation

assume that the matching function of the system induces a dichotomus
partition of the source document set. Each ratio is indicative of adifferent aspect of how this partition induced by the system compares
with, that induced from the users' relevance judgements. The recall ratio measures the inclusiveness of the retrieved subset with respect tothe relevant set, and the precision measures the exclusiveness of theretrieved set with respect to thenonrelevant set. Cleverdon!s ap-proach was to plot these ratios, one versus the other with a parameterof the matching function (cutoff criterion) providing different points.
The evaluation of a single request by this means is clearly unambiguous o However, the evaluation of system behavior with respect toa sample set of N
the macro viewpoint.

queries can be produced from either the micro orXXI-6
Assume that for a particular cutoff (which specifies a partitionof the source collection), the total number of retrieved documents forthe ith query is t., and the number both relevant and retrieved is r. .
Further let the total number of documents which are relevant to request
i be n, .
Under these conditions a single point on the precision-recall curveis defined according to the micro viewpoint astry.
in
micro recall = I— i=l.

..

micro precision
micro precision£—1=1y. 'r.
= ^— i=l= *— i=lThus the micro recall is the number of relevant documents retrieved
per relevant document, and micro precision is the number of relevant documents retrieved per retrieved document.
iation of bothnandt

Unless there is only small var-over all sample requests these parameter do notreflect the behavior of an average request.
From a query oriented point of view the above conditions definerather than one point in the precision-recall plane.

NThis set of points canbe represented by a single average point defined bysN
macro recall=« ) —i-1 XN;

macro precision=r= ) -—
i=l X

Thus the macro recall is just the average recall over the query sample, while the macro precision %s the corresponding average precisionc

XXI-7

It is interesting to note that if the number of relevant documents per
query is constant or the number of retrieved documents per query is constant then the micro and macro recall or precision are respectively identical.

This of course is expected since in this event the micro statistic

merely weights the performance of each query equally and. is therefore
identical to the macro statistic.
A numerical example may serve to illustrate the differences inthese approaches more concretlyc

Consider a set of two queries for which

the following table describes the hypothetical system behavior.

nCutoff 1
tr

Cutoff 2
tr

1

103

2

20

623

32

60

2

Query

Hypothetical Retrieval Results for
A Sample of two Queries
TABLE 1For the above retrieval results the following recall-precision
figures indicate the micro and macro parameter values.

XXI-8

Cutoff 2

Cutoff 1
micro

macro

micro

mzr-rorecall,n.M.6?

.63

precision.66.66.1.17

Jomparison of Micro and Macro Precision-Recall Points
TABLE 2
From Table 2 it can be noted that at cutoff 1 micro recall issignificantly lower which reflects the fact that the request having thelarge number of relevant documents (l) has much lower recall than request (2)oSimilarly at cutoff 2 micro precision is lower since in thiscase the request having more retrieved documents (2) has lower precision,
A comparision between query-averaged and document-averaged precision vs. recall curves for a sample set of 2\\ queries is shown in Fig.
loThe results were obtained from experiments run on the SMART system.

Since the matching function in SMART produces a ranked output, cutoff
points were chosen by the following methods

let the user examine theretrieved list in order until he encounters

nconsecutive nonrelevant

document^'; this defines a cutoff point; by varying

n5

a series of points

on £he precision vs. recall curve can be produced.

Information Storage and Retrieval, Report No. ISR-8 to the National
Science foundation, Computation Laboratory of Harvard University(December 1961*)*

XXI-9ries

T3UJCT32 Gh o>P •HCD •P0fctf.;
IDCD>

O-JiHUCD+3CDcuto r-Ha)
X CD
6 OVHCD C\J
H i-iw w fnMo (U a) CD O
H oP-. -P•Hs£40\<Dfn • H pu, <y o-P
U EO
ctl o o - d 2
K O ^(MgJGO

O<Hb(,cdCD«

C> o<*; x j- P CDivOC >
CDCD

HoCD

•1-^tO

CO'vO'<N

UOTST0 9ac[XXI-10

Note that although this method of specifying a cutoff is not ideal,
since it ignores any metric information contained in the retrievedordering, it might correspond quite closely to the way a user would
react in practice to the system*

5.

Conclusion
The only generalization which can be made with respect to theeffect of these alternative viewpoints on a precision vs« recall curveis that recall results derived from the micro viewpoint will be more
representative of the system behavior on queries having a large numberof relevant documents, and that micro precision will be more representative of the system behavior on queries having a large number of retrieved documents*

Without evidence that this in fact is not biasing

the performance estimate for the retrieval system it would seem thatthe macro viewpoint has more merit.