III. jrTior Investigations
Of The Relevance Feedback Retrieval Algorithm
Rocchio l f 7 fJ

suggests an algorithm for relevance

feedback baaed on the properties of the distance iunction
used*

If the set of relevant documents is known, the querythat will be "closest" to this set of documents and furthestfrom the set of non-relevant documents can be formed*If

the cosine correlation is used as a distance function, thisideal query iswhere each r

is the vector describing a document relevantto the user1a queryf and each aa document not relevant*is the vector describing

Thus fl is the number of documentsrin the collection that ere relevant to the request, and N eis the number of non-relevant documents, or the remainderof the collection*
Tnia ideal query is useless for retrieval« because if
the documents relevant to each request were known, a retrieval operation would not be needed*

Rocchio suggeststhat the ideal query might be approached by iteration*The

user is asiced to make relevance judgments on a small re*trieved set of documents9 and this set is used to update
the former query aa followsi<i< \-a^no.

•n >

r

14-Snr^r-(A)15where nr and n8 arc the numbers of relevant and non-relevant
mery,
documents retrieved by the previous query,r^ •Rocchio investigated relevance feedback uaina; formula A
end the ZUAB1 retrieval systemk J'.

A set of seventeen nat-ural language search requests and a collection of 405 abstracts of articles published in IKS Transactions on
Electronic Computers (March-Septemberf 1958) ^ere indexedusing a SMART regular thesaurus (Section II)• Relevancejudgments for the sample queries were constructed by a manual search of the entire document collection*

Averageretrieval results for the collection described are improvedby two iterations of the relevance feedback process describedin formula A*

Hocchio suggests construction of multiple

queries when the documents desired are not clustered in thedocument vector space0
Another investigation of a relevance feedback system
was based on the ADI collection", a collection of 82 documents presented at a conference on documentation*

Thirty-five queries were constructed for this collection, and thedocuments considered relevant to those requests were specified by the two originators of the queries*

The investigationof relevance feedback in the ADI collection was conducted by
Kiddie, Uorwitz, and Diets ^llK

They used 22 of the 35

queries and studied a slightly different algorithm for modifying the search query*

Their formula 1st1

16
Three differences from Rocchiofa formula are immediatelyapparent:a)

The descriptor vectors are not nor^ali^ed by theirlength.

I ; RoGChio's formula, the change to the weight ofconcept /a1* in the query depends not only on the weightassigned to concept *a* in a retrieved document vector but
also on the length of that document vector; that is, on thenumber of other concepts and on the magnitudes of weights
in the document vector•

This is not the case in the Riddle9

Horwitasf and Diet* formula*

When the latter formula la used,

for instancei a document with generally highly weighted
concepts changes the query more than does a document withgenerally lower weighted conceptsf the number of concepts
toeing equal•

Weight magnitudes being roughly equal % adocument with more concepts changes the query more th^none with fewer*

Rocchio's formula compensates for theseeffects*b)

The parameter oe% which ia the one variable in theabove formula9 ia constant for all queries•

Hocchio's for-mula uses a different multiplier for each queryt the multiplier being dependent on the numbers of relevant and nonrelevant documents retrieved (n^ and n }•rsc)

The non-relevant documents retrieved on the previous

iterations are not used to update the query*

However,Kiddie* Korwitz, and Oietz tested a Hne ative heuristic
strategy" which uses the two non-relevant documents Jtrut
retrieved (the two which the system falsely judges most17relevant to the query) to update those queries that retrieve
no further relevant documents on the first feedback iteration*
For such queries the formula becomesi

1roc >_i

X>_8iCO)1

The feedback algorithm of fiiddlef Horwit»faad Dietz
produces an improvement in performance on most of the queriestested*The three experimenters recommend that the variable oL

in their formula be set to 1 for the first iteration andthen increased by 1 for each subsequent iteration (called* increasing alpha strategy")•

They also recommend their

negative heuristic strategy (formula C)«
Crawford and Melzer JJ12] have tested a relevance
feedback strategy that ignores the original query after theinitial search if a relevant documentlfi found*.

Theiralgorithm is:
If at least one relevant document is retrieved within
the first n documents, the original query is ignored andone additional relevant document is used for each iteration:

iBut if no relevant documents are retrieved within the first n

On iterations after the firstf the second formula of theirstrategy is not used*18Using the Cranfield 200 document collection (SectionIV), Crawford and Helz JT found their strategy superior toformula B wheno*~ • 1*

(131Steinbuhler and Aleta l yj have tested Rocchiofs
algorithm in the ADI collection, for the fworst cose1 when
only non-relevant documents are available for feedback
after the first search operation*

The information from

non-relevant documents alone, when used to modify the query
according to formula A, gives better retrieval than theinitial query*
114*1
Kelly k J proposes an addition to the relevance
feedback algorithm when no relevant documents are retrieved*
He points out that in these cases no new concepts are added
to the query, and recommends adding concepts that occur
frequently in the document collection*

He tested thisrecommendation on sets of artificially constructed •query1and •document1 vectors with success*However, Steinbuhler

and Aleta L *•* found that adding frequent concepts to thequery degraded performance in the ADI collection*
The results reported in Section VI of this study givefurther insight into the problems investigated by the fiveearlier studies cited* Section VII uses both the earlier
studies and the present report to support recommendations
for document retrieval systems*