VI

VI.

Relevance Feedback in an Information
Retrieval System

W. Riddle, T. Horwitz, and R. Dietz

1.

Introduction
An information retrieval system must be judged in the end by the user

population, and therefore it should be designed primarily to satisfy theuser's needs.

However, the implementation of an efficient system, measured

in terms of the amount of relevant material retrieved for the average user,
is difficult to implement with presently available computing equipment
because of their batch processing mode of operation.

Typical turn-aroundtimes preclude efficient man-machine interaction, and a user must generally
be satisfied with the results of a single search*

However, the use of onlya single search may not produce adequate results. The reduction of searchtime by the use of two-level searches, which match the query first against
the centroid vectors of document clusters and then against the individual
documents in highly correlated clusters, may cause same relevant documents
to be lost. [2] Moreover, some indexing terms may be interpreted differently
by users with different fields of interest.

In fact, users unfamiliar with

the indexing terms employed may formulate queries which,-after translation
to indexing concepts, may not adequately represent the user's requirements.
In addition, the result of most correlation procedures presently used tomatch documents.and search requests depends on the relative positions of
the queries and documents in the n-space determined by the indexing tenns;
but the resulting correlations do not necessarily reflect the relevance ofVI-2
the documents to a user's need.
Attempts have been made to improve the results obtained in a single
search through a document collection by improving the query before a search
is made, or by using a correlation function which better reflects
relevance.

Improvement of a query by an expansion done by the system

prior to a search of the document set has been suggested.[U]This

expansion is done either on the basis of statistically determined concept
relations, or on the basis of a concept hierarchy, and causes concepts
to be added to the query vector if they do not originally appear but are
statistically correlated with or hierarchically related to concepts which
do appear.

It has also been suggested that the user himself reformulate

his query prior to the search, and tests using the SMART System [3]indicate that improved results, in terms of the number of relevant documents retrieved, are obtained by this method.

The reformulation is done

before the query is processed, on the basis of a statistical analysis of
the document set with respect to the index terms present in the original
query. The improvement is effected by the elimination of those terms
which have a high frequency in the document set (and are therefore notadequate differentiators), and reinforcement of those terms appearing
infrequently in the document set (i.e. good differentiators). Maron andKuhns have suggested a correlation technique using relevance numbers.
These numbers are determined by probabilistic indexing, a method in which
the indexer assigns a numerical value indicating the probabilistic valueof that term to the document being indexed, [l] These methods, however,are not entirely adequate, since either they depend on a priori determination of relevance relationships which may not apply to the entire user

VI-3
population, or they impose a waiting period on the user, after which yet
another wait may be required for the processing of smother reformulationof his request.
The advent of time-sharing computer systems allows the user totake a more active part in achieving the satisfaction of his request.

Inaddition to providing a convenient means of man-machine communication,
time-sharing appreciably reduces the time necessary for interaction between
man and machine.

With such a computing system, it is practical to propose

an iterative method of information retrieval in which the user returns
relevance judgments for the documents retrieved.

This can be done after

the abstracts, tables of contents, or full texts of the documents are read.
The computing system, using this relevance feedback information, modifies
the query last used in searching the document set by adding multiples of
the relevant document vectors to the previous query.

The system then.performs a new search and retrieves a new set of documents on the basis of
this modified query.

The iterative process may be continued by the user

until he feels that his needs have been adequately satisfied. [2,3]This paper analyzes an information retrieval system which is based on
an iterative query modification process, using relevance feedback information.
Neither a time-shared computing system nor a user population were available
for the study; but a FORTRAN program, run on a CDC 160^ computer was used
to investigate various updating strategies as applied to a set of 82
documents and 3^ queries. A priori judgments of the relevant documents fora given query were available for the document set and were used to simulate
the user's relevance feedback information.

The development of the various

updating strategies is discussed in the next part; then the experimental

VT-^
results are presented; finally, conclusions are drawn from the study*

2.

Principal Methods
The relevance feedback information is used iteratively to perturb thequery vector in the following manner: [2]

— —

r* ""

r

=

Q

—

+ a

R

•

W
»»

mxl

mxl

mxn

—j

«j

nxl

where m = number of indexing terms
n = number of retrieved document
R = mxn matrix in which the ith column is theconcept vector of the document of rank i
W « nxl vector of relevance weighting factors
It is assumed that query Q has caused a set, R, of n documents to be retrieved*
Using the a priori judgments of the relevance of the retrieved documents,
the relevance weighting factors are determined, and are used to define
W, a vector of length n, so that the relevance weight of the document ofrank j is the jth element. The relevance weight of a document reflects
the relevance of that document to the query (the determination of themagnitude of these weights is discussed later). a is a multiplier controlling the strength of the perturbation to the query Q. The end result of
this modification is that same linear combination of the kth elements of
the retrieved documents is added to the kth element of the query vector
Q, for all k of the indexing terms, thus producing a new query $'•In order to determine the effects of varying the parameters a, n,
and the relevance weighting factors, sample runs are made using selectedvi-5
queries.

The results of these sample runs, presented in the following

paragraphs, are used to develop strategies which are then applied to theentire set of queries.

A)

Determination of the Number of Documents Retrieved

The number of documents, n, that are returned to the user is set at
fifteen.

The sample runs show that if this number is reduced to eight,

the effectiveness of the updating process is diminished.

In the case citedin Figure 1, returning fifteen documents leads to the retrieval of four
relevant documents after three modifications are made, while returning onlyeight documents leads to the final retrieval of only two relevant documents
after the same number of modifications.

This implies the need to return

initially as many relevant documents as possible so that more information
can be used in the updating procedure.

(The number of relevant documents

initially retrieved also depends on the correlation function, as is discussed
later.)

Further, in determining the number of documents to be retrieved, acompromise must be made between the desirability of retrieving a large numberof documents and the desirability of not imposing a large reading task on the
user.
B)

The Effect of the Correlation Function

The result of an iteration is a list of n documents ranked by theircorrelations with the query.

These correlations are determined by one of

the following correlation functions:
Cosine correlation function: [7]m
2 (q,d.)

c-

I n~~

a_

m

./ ( 2 a,*. ) x ( Z d.d. )VI-6Co-occurrence correlation function:[6]
m

da)
i*l
m
m
( 2 q.^ ) + ( S d dm) - ( J <J.d. )Simple vector matching correlation function:

1=1mwhere m = the number of indexing concepts
q.= the ith concept weight of the query vector
d.= the ith concept weight of the document vector
The effect of these different correlation functions on the relevance
feedback process is not known, so the correlation function is included asanother parameter in this investigation.
C) Determination of the Relevance Weighting Factors
In determining the relevance weighting factors the assumption is madethat no information concerning the relative ranking of the relevant documents is available*

That is, there is no way of knowing if one relevant

document is more relevant than another. This is consistent with theproposed information retrieval system, in which the user returns only .
"relevant" or "non-relevant" judgments, without indicating the degree of
relevance of each document retrieved. This implies that the numericalinterpretation of the relevance information should be "binary; therefore aweight of 1 is used as the relevance weighting factor of a relevant document
The simple vector matching correlation function, as stated, is strictly
suitable for use only with binary document and query vectors. Its use
with other than binary vectors, without the addition of a normalization
factor, does, however, preserve the relative rankings of the documents.

VI-7
and 0 is used as the relevance weighting factor of a document for which therelevance is not known.

However, the relevance weighting factors can beused to assign weights other than 1 to the relevant documents. In particular, the weights of the relevant documents can be defined as theircorrelations, thus giving the relevant documents with higher correlation
more weight in the modification process. This method is also used to assign
weights to the relevant documents in this investigation.
Negative relevance weighting factors can be employed in the perturbationof the query.

Essentially, these negative factors indicate that the documentsare "irrelevant" rather than the non-committal judgment that the relevanceis not known.

Since the relevance judgments available do not include any"irrelevant" indications, it is not feasible to simulate fully such
judgments by a user.

But some use of negative relevance weighting factors

can be helpful when, after modification of the query, documents of unknown
relevance are still retrieved with high correlations. Figures 3 and kshow the effect of applying negative relevance weighting factors to aquery for which no additional relevant documents are retrieved after the
first modification.

Documents 1102, 302, and 109 axe ranked 2, 3, andh for both the original and updated queries but are of unknown relevance.
Updating the modified query with negative relevance weighting factors
yields a query which retrieves three relevant documents, a significant
improvement.

Further updating, using the normal 0 and 1 relevanceweighting factors, eventually produces a query which retrieves all five
relevant documents.
The improvement resulting from the use of negative relevance weighting
factors suggests that the following heuristic method would prove useful.

vi-8
When no relevant documents are retrieved, the two most highly correlated
documents are given -1 relevance weighting factors. However, without"irrelevance" judgments the use of this heuristic method is at best anarbitrary procedure, and it should be noted that in some cases thenegative relevance weighting factors are not helpful.

In particular, theresults shown in Figure 5 indicate that the use of too many negative
relevance weighting factors overly perturbs the query vector, so that themodified query is unable to retrieve more relevant documents.
The use of negative relevance weighting factors implies that semeof the concept weights in the query vector will become negative.
concept

Negativeweights have no significance in a term-document matrix, since

an indexer cannot indicate the non-appearance of a certain concept.

It isplausible, however, to have negative weights in a query vector after it
has been perturbed because the relevance feedback information can properly
indicate that certain concepts are irrelevant.
D) Determination of the Value of a
The parameter a is necessary to control the manner and the strengthof the perturbation to the query vector caused by the relevance feedback
information,a may affect the modification in one of three ways. First,

if oc is increased with every iteration, the new relevance information will
have the same weight as the sum of all the previously-gathered relevance
informations.

For example, if one relevant document with concept vector(10101) is retrieved initially by a query with concept vector (10100),the modified query (when oc equals 1) will have a concept vector (20201).
If this modified query, in turn, retrieves relevant documents having

vi-9
concept vectors (10101) and (11100), then these vectors should have theirconcept weights multiplied by 2 to make their concept weights comparable
to those of the modified query.

Second, cc can be kept at a constant value

for all the iterations, which would cause all types of relevance information
to be treated in the same manner, regardless of when they are obtained in
the process. Finally, if cc is decreased with every iteration, more emphasis
will be placed on the relevance information obtained during the early
iterations.
Figure 6 shows the results obtained in sample rims using various
progressions and magnitudes of a.

in general, the use of small fractional

values of cc yields, after two or three iterations, the results which canbe obtained in one iteration by the use of a larger, integral value of a.
For this reason, only integer values of cc are used in further investigations.
The decreasing progression of a is also discarded because it is not
significantly better than the other progressions. Also, a decreasing
progression is illogical, since if the iteration process is converging onsome area of n-space, then more weight should be given to relevance information
obtained later in the process.
Thus two strategies involving cc are used in the final investigations.
In the first, cc is set initiallysucceeding iteration.

to 1, and is increased by 1 at eachIn the second, cc is held constantly equal to 2 (thevalue 2 is used to represent a typical value rather than an optimal one).
However, it should be noted that whenever the correlations of the relevant
documents are used as the relevance weighting factors, ex is held constantly
equal to 1; this is denoted in the figures by the label "ALPHA = CORRELATIONS".

VI-10£)Termination of the Modification Process

Updating is terminated when all relevant documents have been retrieved(since the user's needs are then satisfied as fully as possible), or whenat most three modifications have been made (since the results presented
in Figure 6 indicate that, with the progressions of OL used in the final
investigations, the iteration process can safely be terminated in general
after three modifications have been made).

3.

Experimental Results
In general, the modification of a query using relevance feedback

information leads to an improvement in both the number of relevant documents retrieved and in the ranks of all the relevant documents. The modification normally yields an increase in both precision and recall (asshown in Figure 2 ) , regardless of how a is applied, provided that theset of relevant documents lies in one basic cluster in n-space.

If therelevant documents cluster in two separate regions in n-space (as a resultof the indexing scheme used), the results are as shown in Figure 7*
When such a dual clustering of the relevant documents exists, Rocchio
suggests the use of multiple queries. [2] This is good theoretically, whena priori relevance judgments, which list all the documents relevant to agiven query, have been made. However, in a real system, the user is
uncertain of the existence of other relevant documents and the technique
is impossible to carry out. A possible solution is the use of a list thatguarantees, for example, that whenever document X, Y, and Z are deemed
relevant, then documents A, B, and C are also relevant and are returned* The statement that the iterative retrieval process does not significantly
depend on the particular strategy of applying oc (for the progressionsof a used in the final investigations) is supported by the data given in
Figure 6, for the progressions used and the correlation function.

VI
to the user.

But this procedure assumes that what is relevant to one user

is also relevant to another.
best.

Such an assumption is of doubtful value at

Therefore, there does not appear at present to be any feasiblemeans of returning documents to the user which are actually relevant but
which, because of the indexing scheme, lie in a region of space far removed
from the query and the other relevant documents.
Figures 9f 10, and 11 show the effect of using the negative relevanceweighting factor heuristic method.

Application of the heuristic method to

two of the queries show results in a modified query which eventually
retrieves the relevant documents.

In the third case shown, query Q B H ,no definite conclusion can be drawn after three modifications have beenmade; however, it appears that the query vector is being modified
correctly, and that it is moving toward the correct region of the document
space.
The magnitude of the initial value of a affects the speed at which
the modified query converges to a position in n-space where it is capableof retrieving all of the relevant documents. For the query shown in Figure
12, when a is initially equal to 1, the final ranks of the relevant documents are 1, 2, 3, 5> and 19#

If, on the other hand, ex is initially set

equal to 2, the final ranks are 1, 16, 18, 28, and 3^. This dependence
on the initial value of ex can be explained graphically (see Figure 13) •Assume that documents A, B, C, and D are relevant to query q, and thatq retrieves documents B, C, and D. Document D has camparitively large
concept weights so that when q is modified using an ex of 2, the resulting
query q" is strongly biased toward D.

Consequently, q" is unable toretrieve document B. With a milder modification using a smaller value ofVl-12ct, the query qf is obtained and all the relevant documents are retrieved.
The final values of recall and precision depend on the number of
relevant documents retrieved on the successive searches, since more
information will obviously perturb the query to a greater extent.

Inparticular, there is a dependence on the number of relevant documents
retrieved initially, which is, in turn, dependent on the correlation
function used.

(In this investigation, the dependence is actually on the

denominator of the correlation formula, since all of the functions tested
possess the same numerator.)

If only a few of the relevant documents are

retrieved initially, then convergence is slow.

In other words, given aquery having three relevant documents, the probability of retrieving all
three is higher if two of the documents are retrieved initially rather
than only one.

As shown in Figure l4, for query QA15 the cosine correla-tion function initially retrieves three relevant documents, while the cooccurrence and simple vector matching correlation functions retrieve two
and four respectively.

Since the simple vector matching case now includes

more information concerning the concepts in the relevant documents, thefinal values of recall and precision achieved by the modification process
are higher when simple vector matching is used as the correlation function,
than when either of the other two functions is used.

These results suggestthat it is unwise to restrict the proposed retrieval system to the use ofa single correlation function.

h.

Conclusions
The implicit assumption underlying this investigation is that relevance

feedback is a necessary part of the overall retrieval process. As theVI-13feedback process is studied in detail, the validity of this assumption isdemonstrated repeatedly; for practically every query tested, the recall is
eventually increased.
The explicit assumption basic to this investigation is that therelevance feedback and query modification proccesses can be optimized insome sense. Although the analysis demonstrates that no uniformly "bestprocess exists, the investigation of the relevance weighting factors, thesearch correlation function, and the iteration strategy shows that semecombinations of these parameters produce methods that work well in manycases.
Specifically, the following choices of the parameters constitute an
optimal system since each optimizes that part of the modification process
which it affects most directly.

First, the simple vector matching correlation

function is optimal in that it retrieves more relevant documents on theinitial search than do either the cosine or co-occurrence correlation
functions.

Second, the iteration strategy in which OL is increased withevery iteration is optimal in the sense that the scale of the query
updating information is effectively equal to the scale of the current query.
Finally, the use of zero and one as relevance weighting factors, including
the negative relevance weighting factor heuristic method, is optimal in
the sense that it adequately describes the feedback information provided
by the user.

Therefore, this combination of parameters yields a retrieval

system which is optimized for high precision and recall through the use of
relevance feedback information.

Of course, it can be argued that some ofthese choices for the parameters may lead to inefficiencies in the retrieval
process, but this objection is of no consequence in this investigation

VI-14
because the search time is not included in the investigation.
The investigation indicates two specific areas where more researchis possible. Of primary importance, since it is now known that negative
weights are useful in the relevance feedback process, is a more specificinvestigation of negative relevance weighting factors using relevancejudgments given by an actual user population.

Also, the nuriber of

iterations needed to obtain the best possible recall and precision should
be studied.
In addition to its use in document retrieval, the relevance feedback
process provides an efficient method for testing the efficiency of document indexing schemes, since external disturbances, such as incorrect
formulations of queries, are minimized.

Theoretically, one could possibly

develop an indexing scheme which would eliminate the need to use relevance
feedback information.

However, the present investigation has shown how arather simple use of relevance feedback information can greatly improve
the recall achieved by the document retrieval system. Thus, it would
seem more practical to use the best of the present indexing schemes, anddirect further investigation to the area of retrieval improvement by theefficient and optimal employment of user relevance feedback information.

VI-15

References[l]M. E. MSaron, and J. L. Kuhns, On Relevance, Probabilistic
Indexing and Information Retrieval, Journal of the Association
for Computing Machinery, Vol. 17, No. 3, July I960, pp. 216-244.

[2]

J. J. Rocchio, Harvard University Doctoral Thesis, Report ISR-10,
to the National Science Foundation, Chapters 3 and h.

[3]

J* J* Rocchio and G. Salton, Search Optimization and Iterative
Retrieval Techniques, Proceedings of the Fall Joint Computer
Conference, Las Vegas, Nov. 1965.

[4]

G. Salton, A Document Retrieval System for Man-Machine Interaction,
Proceedings of the 19th National Conference, ACM Philadelphia,
1964, pp. L2.3-1 - L2.3-20.

[5]G. Salton, The Evaluation of Automatic Retrieval Procedures Selected Test Results Using the SMART System, American Documentation,
Vol. 16, No. 3, July 1965, pp. 209-222.

[6]

G. Salton, Evaluation of Computer-Based Information Retrieval
Systems, FID Congress, October 1965.

[7]

G. Salton, Progress in Automatic Information Retrieval, IEEE
Spectrum, Vol. 2, No. 8, August 1965*

vi-16APPENDIX AFORTRAN Program Operating Instructions'for CDC loo**- run
The program appears in two forms. The first form reads information
concerning the document and query set from an input tape (tape number 138
as the program is presently written) . The second form reads the document
and query vectors from cards. The two programs operate identically, theonly difference being that the tape version requires the input tape to be
mounted on unit two, whereas the card input version must have a data deck
appended to it.

(The card input version qualifies as an IS job.)

Both programs are driven by input data which should be organized asfollows:
1.

Relevant document information.

One card per query, in the orderin which the

Eleven three-column fields (using

queries appear.

columns 1 through 33) specify the numbers of the relevant documents(these numbers are the sequential

numbers of the documents asthey appear in the document set, not the identification numbers).
A single three-column field (columns 3U through 36) gives thecount of the relevant documents for the query.
2.

Factorial information for use in the evaluation formulas.

3.

Concept number data deck if card input version is used.

Eachdocument and query have a set of cards which give the following:a)First card contains the alphameric identifier of the
document or query in columns 1-16 and the total number ofconcepts in columns 20-21.

b)

The second through last card contain the concept numbers
and total weights paired together (10 pairs to a card) inan Ik format, e.g.
column1t
10

7 • • • •12 78 36 198

12

VI-17indicates that concept

has weight

10
78
198
4.

Program driving information.

12
36
12
The program can only set up thequery-document correlations for a maximum of 17 query designations
at a time, but these query designations need not be unique*
Therefore the driving information should appear so that thedesired analysis is grouped into packages of 17*

Each package

has the following format:a)First card contains in columns
1-3
4-6

7-9
b)

number of documents retrievedcorrelation function indicator
1 -* Simple Vector Matching
2 •* Co-occurrence
3 •* Cosine
number of analyses to be made(maximum of 17)

Second card contains in columns
1-3
4-6
49-51

c)

number of the query used in 1st analysis
w
"
"
"
"
"
" 2nd
"

"

"

"

"

" 17th"Following cards, one for each analysis contain incolumns 1-3
4-6
7-9
10-12

initial value of a numerator
increment added to 0t numerator after
each iterationadenominator
total number of iteration modifications
allowede.g. if the progression of a desired is 3,5,J,*.*
with
only three modifications allowed, the card pertaining
to this analysis would be:
3 2
1 3
whereas the progression 3/15* 7/15, ll/l5#"* witha maximum of 6 modifications would require:3 fc 156

vi-18The program as it is presently written is for a document-query set
containing 82 documents and 3*4- queries. DO loop indices, tables offactorials, and matrices are all set for this size set, and would have to
be changed before the program could be used on a set of different size*
A data deck to perform a cosine correlation analysis on queries,
2,7, and 33, using the increasing a, and a equal to the correlations
strategies for both 15 and 30 documents retrieved would appear as follows:
15

3 6

2

2 7

set number retrieved at 15
7 33 33

query numbers concerned

1 1 1 30 0
1 3

a strategy cards for query 2

1 1 1 30 0
1 3

a strategy cards for query 7

1 1 1 30 0
1 3

a strategy cards for query 33

30 3 6change number retrieved to 30

2query numbers concerned

2 7 7 33 33

1 1 1 30 0
1 3

a strategy cards for query 2

1 1 1 30 0
1 3

a strategy cards for query 7

1 1 1 30 0
1 3

a strategy cards for query 33

vi-19AEHBHDEC B
Evaluation of Relevance Feedback Methods;
E. M. Keen
Some of the results are here presented in summary form for searches
employing relevance feedback, using averages over 22 search requests.
Results are computed for three different correlation functions: cosine,
co-occurrence and simple vector matching; and also for three different
feedback strategies: increasing alpha (1,2,3), constant alpha (2,2,2)
and alpha equal to the correlations of the relevant documents (c,c,c).
Only 22 of the original 3^ requests are averaged, since full results were
not available for 12 of. the 3U. Nine of these 12 were not processedby

all of the above procedures, some because the initial search result wasvery good and no iteration was needed, and the other three because each
had only one relevant document, and averages were therefore not believed
to be meaningful.
Tables Bl through B9 give average results using the measures ofnormalized recall, normalized precision and normalized overall. In Table
Bl for example the cosine correlation function is used with the increasingalpha strategy, and the normalized measures indicate the improvement in
performance that results from each update.

Tables B2 and B3 also illustrate

the use of the cosine correlation function, but the increasing alphastrategy is altered to constant alpha and alpha correlations respectively.
Tables B^ through B6, and Tables B7 through B9 cover these same threealpha strategies but use the co-occurrence correlation function and thesimple vector matching correlation function, respectively.
Comparing the three correlation functions alone, on the initial searchVI-20result the normalized overall measure is better for cosine than forco-occurrence, and better for co-occurrence than for simple vector
matching*

This can be seen in Table BIO where all searches are comparedfor each correlation function, with the correlation functions ranked inorder of merit according to the normalized overall score achieved by each.
The cosine correlation function works the best on the initial search andalso on the updated searches using the alpha correlation strategy.

The

co-occurrence correlation function is the best for the updated searches
using the increasing and constant alpha strategies.

With one exception,

the simple vector matching correlation function performs the worst on
all searches.
Evaluation of the relevance feedback methods requires examination of
the effectiveness of the updated searches, and the tables given show only
two cases where updates result in a drop in performance.

This occurs

when the simple vector matching correlation function and alpha correlation
strategy is used, and Table B9 shows that the second and third updates
had a performance progressively worse than the first update. A single
request contributed largely to the result, since in request QA9 thenormalized overall measure dropped from 1.^677 to O.6887 after the second
update, and dropped further to 0.5989 after the third update.

However

all the combinations of correlation functions and alpha strategies resulted
in a considerable improvement in performance with the updated searches
compared to the initial search.
An order of merit of the nine combinations tested is given for thethree updated searches in Table Bll, where merit is based on the increasein the normalized overall measure achieved by each update compared with

VI-21the initial result.

The co-occurrence correlation function with theconstant alpha strategy achieves the greatest increase in the first andsecond updates, and the same correlation function, but with increasingalpha strategy, gives the best result for the third update.

Combinations

using the co-occurrence and cosine correlation functions together withthe constant and increasing alpha strategies perform the best.

Combinations

using the simple vector matching correlation function and the alphacorrelations strategy always Jiave low merit.
Another method of displaying the increase in retrieval performanceachieved by relevance feedback is the use of a plot of precision versus
recall.

Table B12 gives such a plot of the same results as those given

in Table Bl, in which the cosine correlation function and increasing alphastrategy is used.

The results of the 22 requests are averaged by the useof a cut-off made after examination of m consecutive documents (m ranging
from 1 to 20), and the precision and recall values are averaged over all
requests at each cut-off point to obtain the nine average points for eachcurve.

The four performance curves show the initial search and the three

updated searches.

The first update gives the biggest improvement in

performance, and the updated searches achieve a very substantial
improvement in performance at the high precision end of the curves.

This

improvement decreases as recall increases, and is almost lost when 0.7
recall is reached.

If performance is compared at a cut-off of four

documents, for example, the third updated search achieves an improvement
over the initial search of more than 0.3 in both precision and recall.
To summarize, evaluation of the relevance feedback methods averaged
over 22 search requests shows a very considerable improvement in performanceVI-22
with all the updating methods used.

The co-occurrence and cosine

correlation functions with constant and increasing alpha strategies
are a little superior to the simple vector matching correlation
function and the alpha correlations strategy.

VI-23
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.76010.5566

I.3167First Update

O.8083

O.69671.5050

Second Update

0.8231!-0.7^81

1.5715

Third Update. 0.8267

0.755^

1.5821

Average Search Results for Cosine Function
and Increasing Alpha
Table Bl
normalized iMeasures
Search Type
RecallPrecision

Overall

Initial Search

0.76010.55661-3167First Update

0.8100O.69871.5087

Second Update

0.819.'

0.7381

1.557**

Third Update

0.81890.7^011.5590

Average Search Results for Cosine Function
and Constant Alpha
Table B2
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.76010.55661.3167First Update

O.80380.6873

1.1*911

Second Update

0.81190.71381.5257

Third Update

0.81500.71531.5303

Average Search Results for Cosine Function
and Alpha Correlations.
Table B3

VI-*
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.75520.55^51.3097First Update

0.81290.70271.5156

Second Update

O.8377

0.7^93

I.583O

Third Update

O.83850.75501.5935

Average Search Results for Co-Occurrence Function
and Increasing Alpha
Table Bk
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.75520.55^51.3097First Update

0.81770.70981.5275

Second Update

O.83900.7^901.5880Third Update

O.83860.7511!-1.5900

Average Search Results for Co-Occurrence Function
and Constant Alpha
Table B5
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.75520.55^51.3097First Update0-78990.6590I.UU89Second Update

0.801*+

O.6898

1.1*912

Third Update

0.8121+0.70^81.5172Average Search Results for Co-Occurrence Function
and Alpha Correlations
Table B6

VI-25
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.74870.53561.2*3

First Update

0.7958

O.6789I.47V7

Second Update

0.7991*0.68641.4858

Third Update

0.8001

O.68851.4886

Average Search Results for Simple Vector Matching
Function and Increasing Alpha
Table B7
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.74370.53561.2843First Update

0.7968

O.6782

1.4750

Second Update0.8029O.6966

1.4995

Third Update

0.80330.69841.5017

Average Search Results for Simple Vector Matching
Function and Constant Alpha
Table B8
Normalized Measures
Search Type
RecallPrecision

Overall

Initial Search

0.74870.53561.2843First Update

0.79440.6712

1.4656

Second Update

0.77530.6566

I.4319

Third Update

0.77440.65471.4291

Average Search Results for Simple Vector Matching
Function and Alpha Correlations
Table B9

VI-26

Merit

InitialSearchFirst
UpdateSecondUpdateThird
Update

INC.

CON.

CORE.

INC.

CON*

CORK.

INC.

CON.

CORR.

1

COS

CO-OC

CO-OC

COS

CO-OC

CO-OC

COS

CO-OC

CO-OC

COS

2

CO-OC

COS

COS

SVM

COS

COS

CO-OC

COS

COS

CO-OC

3

SVM

SVM

SVM

CO-OC

SVM

SVM

SVM

SVM

SVM

SVM

INC = alpha increasing
CON = alpha constant
CORR = alpha correlations

COS• cosine function
CO-OC « co-occurrence function
SVM- simple vector matching function

Order of Merit assigned by Normalized Overall for the Correlation Functions
Table BIO

SecondUpdateFirst
Update

Merit

Third
Update1| CO-OC

CON

CO-OC

CON

CO-OC

INC.

2

CO-OC

INC

CO-OC

INC

CO-OC

CON

3

COS

CON

COS

INC

COS

INC

V

SVM

CON

COS

CON

COS

CON

5 . SVM

INC

SVM

CON

SVM

CON

6

COS

INC

COS

CORR

COS

CORR

7

SVM

CORR

SVM

INC

CO-OC

CORR

8

COS

CORR

CO-OC

CORR

SVM

INC

CO-OC

CORR

SVM

CORR

SVM

CORR

L 9

INC * alpha increasing
CON • alpha constant
CORR * alpha correlations

COS• cosine function
CO-OC • co-occurrence function
SVM• simple vector matching
function

Order of Merit assigned by Normalized Overall for Different
Functions and Alphas when the Increase in Performance ofeach Update Compared with the Initial Search is Considered
Table Bll

VI-27

i

8
-p
CO fc
-P <u cft

<1 a; 02

:P C -P

•H

O C

a 5

as <H S a;a> oHo +> O fc

S3

CO

CVJ
H
pq

<0

OJ

I

Lin

o

*

o
• a>
rO tr

CVJ

r—

Q

<Q

m

ro

CVJ

-.

O

VI-28INDEX OP FIGURES
Page
Figure 1

Ranks of Relevant Documents After Each Update
as a Function of the Number of Documents. Returnedin Answer to a Query

VI- 30

Typical Results of Query Modification Using
Relevance Feedback Information in the Case when
the Relevant Documents are in One Region of N-space

VI-30

The Effect of Negative Weightings Applied to NonRelevant Documents which Consistently Appear on
the Output List with a Relatively High Correlation

VI-31

Figure h

Successful Use of Negative Weighting

VI-32

Figure 5

Unsuccessful Use of Negative Weighting

VI-37

Figure 6

Results of Various Progressions of Alpha

VI-^-2

Figure 7

Typical Results of Query Modification Using
Relevance Feedback Information in Case when the
Relevant Documents Split into Two Regions in
N-Space

VI-45

Shifting of Queries in N-Space Due to Query Modification Using Relevance Feedback Information

VT-U6

Selected Results Showing Various Effects of the
Negative Weighting Heuristic Method

VI-^8

Figure 2

Figure 3

Figure 8

Figure 9

VI-29INDEX OF FIGURES (continued)

Figure 10

Use of Negative Weights

Figure 11

Continued Use of Negative Weights when Use in

Page
VI-^9

the First Update Fails to Move Query into
Correct Region of N-Space
Figure 12

VI-58

Comparison of Results After Three Iterations
as a Function of the Value of Alpha used in
First Update Cosine Correlation Function

Figure 13

Effect of the Initial Value of a used in the
Iteration Process

Figure ik

VI-62

VI-70

The Result of the Query Modification by Use of
Relevance Information as a Function of the Number
of Relevant Documents Initially in the Fifteen
Highest Correlated Documents

VI-71

15 Documents
Returned

8 Documents
Returned

Initial Ranking of
Relevant Documents

3, 12, 17, 33, W-

3, 12/17, 33, **Ranking AfterFirst Update

1, 2, 3, 15, 30

1, 3, 10, 30, &Ranking After
Second Update

1, 2, 3, fc, 19

1, 2, 11, 12, 29

Ranking After
Third Update

1, 2, 3, *, 19

1, 2, 11, 13, 30

Query QA10GR0UP
Cosine Correlation Function
Increasing Alpha StrategyRanks of Relevant Documents After Each Update as a Functionof the Number of Documents Returned in Answer to a QueryFigure 1

Ranks of
Relevant
Documents

Normalized
RecallNormalized
Precision

Rank
RecallLog
Precision

Initial Results

1,2,19,27

.877

•739

•2<*

.U58

Results AfterFirst Update

1,2, 6,33

• 899

.805

.238

• 531

Results After
Second Update

1,2, 3,21

.9^6

.885

• 370

.657

Results After
Third Update

1,2, 3,18

• 956

.895

.kn

.679

Query QB10CHEMIST
Cosine Correlation Function
Increasing Alpha Strategy
Typical Results of Query Modification Using RelevanceFeedback Information in the Case when the Relevant
Documents are in One Region of N-Space
Figure 2

VI- 31

Document
Rank
Name

Relevant

Weighting
AppliedInitialResults

1
2
3
h
5
6
7

307ROLE OF /1102CENTRALI
302RECORDING
109STATUS RE
1212T0WARD A
1201ACCREDIT
1107THE AUTO

X

1
0
0
0
0
0
0

Results AfterFirst Update

1
2
3
h
5
6
7

307ROLE OF /1102CENTRALI
302REC0RDING
109STATUS RE
1212T0WARD A
1201ACCREDIT
H07THE AUTO

X

1
-1

1
2
3
h
5
6
7

307ROLE OF /902D0CUMENT
214ELECTRONI
2 0 M SYSTEM
209THE REDUC
903THE USE 0
203NEW PHOTO

X

1
2
3
h
5
6
7
•
•
10

307ROLE OF /209THE REDUC
203NEW PHOTO
214ELECTRONI
902DOCUMEJYT
201A SYATEM
211A. COMPUTE

Results After
Second Update

Results After
Third Update

206AMLYSIS

-1
-1
0
0
0

X
X
X
X
X

X

—1 1

Query QA12J0URNAL
Cosine Correlation Function
Constant Alpha Strategy
The Effect of Negative Weightings Applied to Nonrelevant
Documents which Consistently Appear on the Output List
With a Relatively High Correlation
Figure 3

1
0
0
0
1
0
1

VI-32

Results for Query No. 12

QA12JDURNAL

Document

No.

Name

Relevant

Correlation

k

307ROLE OF /1102CENTRAL1
302RECORDING
109STATUS RE
1212TOWARD A
1201ACCREDIT
1107THE AUTO
201A SYSTEM
1206THE EDUC
1209SCIENCElOO^DENSITY
1119RECENT A
603COMPILATI
1215GRADUATE
303C. M. HER

X.64956980.3825^603.30096^63.29774567.2449^897.20*1*15.20^12Ul5.2013^682.19658927.17234550' .16823165.16439899.163663^2•15861032.15713^

ko

19
k6
60
10
81

75
1*3
21

35
13
57
h5
38

Query(1) = Query (0) + Alpha

3 *2 WEIGHT(I) # D o c u m e n t 0 f Rank T
l
1*1
'

5 6

7 8

9 10 11 12 13 lk 15

Document Rank 1

2

Weight1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Alpha * 2(a) Initial Results (Cosine Correlation)
Successful Use of Negative Weighting(Five relevant documents in the collection )Figure k

VI-33Results for Query No. 12

QA12J0UBNAL

Document

No.

Name

k
ho

Relevant

Correlation

X•991^6269.34989921.30969005.25531^80.22165517.22003279.21004201.19837301.18670^01.17389250.163^9177.16114324.15^03081.15244319.15105098

307ROLE OF /1102CENTRALI
302REC0RDING
109STATUS RE
11017INF0RMA
301D0CUMENT
1212T0WARD A
1107THE AUTO
91^THE RAPID
108SEARCHERS
1004DENSITY
902D0CUMENT
306PROGRESS
906MICROFILM !
813AUTOMATIC

19

hS

33
17
60
81
22

71
35
32

37
73
69

Query(2) = Query (l) + AlphaZ
1=1

Document Rank

6

Weight

1

2

3 ^

5

TTp- * Document of Rank IW7

8

9 10 11 12 13 lk 15

1 - 1 - 1 - 1 0 0 0 0 0 0 0 0 0 0 0

Alpha = 2(b) After First Update (Cosine Correlation)
Figure 4 (continued)

VI-^

Results for Queiy No. 12
Document
No.
Name

k

hk
k2

7k

69
73
Ik
35
33
28
22

Query(3) - Q*«y(2) + Alpha

WeightRelevant

Correlation

X.535715^1.10271^70.05685CA7.02853186.0271135^.017531^5.01711912.01171571.00936073.00^72350

307ROLE OF /902DOOMENT214ELECTR0NI
201A SYSTEM
209THE REDUC
903THE USE 0203HEW PHOTO
305THE USE 0813AUT0MATIC
906MICROFILM
518EDUCATIONlOO^DENSITY
llOl/lllFORMA
506THE ROLE
91^THE RAPID

32
63
75
25

Document Rank

QA12J0URKAL

1

2
1

3
0

k
0

X
X

0
-.01430356-.01569842-.01592627-.0231^03^

2 WEIGHT ( I ) # D o c n a n e n t o f R a J l k
1=1
vl)
5

0

6
1

7
0

8
1

9 10 11 12 13 1^ 15
0

0

0

0

0

0

Alpha • 2(c) After Second Update (Cosine Correlation)
Figure k (continued)

0

0

x

VI

Results for Query No. 12

No.

4
25
42
63
32
75
7
73
17
9
69
44
11

71
33

Document
Name

Relevant

Correlation

X
X
X.70658278.44245670.39855516.21694460.15644221.14526777.13650823.1171631*7.11570746.10580301.IO3873IO.10070364.08802188.08792684.08539239

307ROLE OF /209THE REDUC
203NEW PHOTO
214ELECTR0NI
902D0CUMENT
201A SYSTEM
211A COMPUTE
906MICROFILM
301DOCUMENT
206ANALYSIS
813AUTOMTIC
903THE USE 01117PLANNING
108SEARCHERS
1101/INF0RMA

Query(4) * Query(3) + Alpha

QA12JDURKAL

X
XE^IGHT(l) * Document of Rank Iw

1=1
Document Rank
Weight

1

2

3 4

5

6

7

8

9 10 U 12 13 l 4 15

1 1 1 0 0 0 1 0 0 1 0 0 0 0 0

Alpha « 2(d) After Third Update (Cosine Correlation)
Figure 4 (continued)

Results for Query No. 12

QA12J0URNAL

Document

No.

Name

Relevant

Correlation

4

307ROLE OF /209THE REDUC
203NEW PHOTO
211A COMPUTE
206ANALYSIS
214ELECTR0NI
201A SYSTEM
902D0CUMENT
906MICROFILM
304PHOTOGRAP
301D0CUMENT
813AUTOMATIC
914THE RAPID
1115S0ME OBS
108SEARCHERS

X
X
X
X
X

.67270^72.58764284.5632^980.^269^230.37116251.2^151839.23915765.20116064.19796U93.19125152.17945400.17232484.16768022.16539875.16145026

25
42

7
9
63
75
32
73
20
17

69
22
51
71

No Update for these Results(e) After Fourth Update (Cosine Correlation)
Figure 4 (continued)

VI

Results for Query No. 9

QA9AKALYSIS

Document
| No.

Name

39

517RESEARCH
1116RETRIEVA
1117PLANNING
809A MECHANI
1113THE MATE
8I4A NEW EFF
307ROLE OF /301D0CUMENT
206ANALYSIS
815A FACET A
504IS RELEVA
1209SCIENCE
1104THE IBM
108SEARCHERS
601ENC0DING

k7

11

62
12
C\J

If
17
9
23
27
21
1
71

Mol

Query(1) = Query(O) + Alpha
Document Rank
WeightRelevant

Correlation.43301270.42135049.40209035.39900373•34815531• 3^299717. 32478490.32075015.29329423.29329423.28867513.28724249.28647316.28517591.28306926

N
WEIGHT(I) „ _. ._ . _Z/nx* Document of Rank I
1=1

7 8 9 10 11 12 13 14 15-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
1 2

3 4

5

6

Alpha • 2(a) Initial Results (Cosine Correlation)
Unsuccessful Use of Negative Weighting
There are two relevant documents in the collection.

Figure 5

VI-38Results for Query No. 9

QA9ANALYSIS

Document

No.

Name

1 31

207PRACTICAL
305THE USE 0603COMPILATI
610CHARACTER
902D0CUMEOTlOO^DENSITY
U18ENGLISH-L
1122STATE-0F
506THE ROLE
901A SELECTE
21UELECTR0NI
1119RECENT A
1001DESIGN 0
716A NAIIONA
505THE PLACE

7^
57
x
5
32
35
50
18
28
*

63
x
3
8o
•67
65

Relevant

Query (2) = Query(l) + Alpha

Weight

1 2
0

3
0

^
0

2
5

0

-.06676&+6-.08U985U5-.08688523-.09306588-.09^3151^-.09719073-.1071^3^7-.10990208-.IIU06536-.12266150-.13^92076-.135^572-.13721113-.138^392-.1386^335

X

WEIGHT (I) # D o c u m e n t o f R a n k jI«lDocument Rank

Correlation

W

6
0

7
0

8
1

9 10 11 12 13 1^15
0

0

0

0

0

0

Alpha * 2(b) After First Update (Cosine Correlation)
Figure 5 (continued)

0

0

VI

Results for Query No. 9

QA9ANALYSIS

Document

1 No.

Name

Relevant

Correlation

50
31

U18ENGLISH-L
207PRACTICAL
603COMPILATI
305THE USE 0
610CHARACTER
902D0CUMENTIOCADENSITY
506THE ROLE
1122STATE-0F
9O0A SELECTE
505THE PLACE
1001DESIGN 0
21^ELECTR0NI
1119RECENT A
716A NATIONA

X.2268128-.0587^55^-.0796570^-.085^5551-•09358063-.09^83680-.09508698-.09926725-.11050995-.11563119-.13300052-.13500295-.13566701-.13615^65-.13920965

57
7*
x
5

32
35
28
18
5^
65
80
63
1
3
^67

N
Query (3) = Query(2) + AlphaZ

1=1

WEIGHT(I)* Document of Rank I(«

Document Rank 1 2 3 fc 5 6 7 8 9 10 11 12 13 Ik 15
Weight1 - 1 - 1 - 1 0 0 0 0 0 0 0 0 0 0 - 1
Alpha * 2(c) After Second Update (Cosine Correlation)
Figure 5 (continued)

Results for Query No. 9

QA9ANALYSIS

Document

No.

Name

Relevant

Correlation

1 50

418ENGLISH-L
902D0CUMENT
1122STATE-0F
901A SELECTE
lOO^DENSITY
207PRACTICAL
518EDUCATION
214ELECTR0NI
212A STATIST610CHARACTER
506THE ROLE
903THE USE 0
1001DESIGN 0
209THE REDUC
30 3C. M. HER

X.12680691-.09522233-.10680565-.11023756-.1193^192-.13022519-.13^266-•13599739-.13606583-.13951793-.1429^77^-.1^71603^-.153020^7-.15309977-.15538235

32
18
54
35
31
14
63
29
15
28
44
80
25
38

I'JFTPHTiTi

Query(U) = Query(3) + Alpha
Document Rank

1

2

3 *

Weight1-1 -1-1

TTT^
W

S
1=1

5 6

7 8

* Document of Rank I

9 10 11 12 13 1^ 15

0 0 0 0 0 0 0 0 0 0 0

Alpha = 2(d) After Third Update (Cosine Correlation)
Figure 5 (continued)

vi-fciResults for Query No. 9

QA.9AMALYSIS

Document

J No.

Name

Relevant

Correlation

50

4l8ENGLISH-L
1004DENSITY
506THE ROLE
212A STATIST
518EDUCATION
207PRACTICAL
610CHARACTER
505THE PLACE
1001DESIGN 0
214ELECTR0NI
903THE USE 0
804C0MPUTERI
209THE REDUC
424 PRODUCING
1119RECENT A

X.22115640-.10861615-.123^3199-.12639036-.12844204-.13875402-.14017633-.14537614-.15220234-.15303001-.15481228-.15488673-.15707111-.16395750-.17453190

35
28
29
l4
31
x
5
65

8o

63
1*
48
25
58
13

No Update for these Results(e) After Fourth Update (Cosine Correlation)
Figure 5 (continued)VT-^2

Progressionof Alpha

Correlation
Function

1,1,1

Cosine1,1,1Co-occurrence

SimpleVectorMatching

1,1,1

Cosine2,2,2Cosine

3,3,3

k,k,k

Cosine1/15,1/15,1/15 Cosine

2/15,2/15,2/15 Cosine

3/15,3/15,3/15 Cosine

1

.

•

Ranks of theRelevant Documents
Normalized
Iteration Appearing in the
RecallTop Fifteen

0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3
0
l
2
3
0
1
2
3
0
1
2
3
0
l
2
3

Normalized
Precision

3,^,15
1,2,5 .
1,2,5
1,2,5

• 706
.675
.667
.667

.51*
.612
.608
.608

2,7

.662

1,5,10
1,3,7
1,3,10

.&7

.^57
.528
. .580.565

2,8,9,11
2,3,^,1^
1,3,^,1^
1,3,^,1^

.725
.760
.766
.768

.550
.6^3
.682

3,M*

.706
.667
.665
.660

.51*
.608
.607
.605

.706
.667
.661
.662

•51*
.608
.605
.606

.706
.665
.662
.662

.51*
.608
.606
.606

3,^,1^
2,5,8
2,5,7
2,3,7

.706
.701
.690

.68k

.51*
.532
.530
• 5*9

3,^,1*
2,3,5
1,3,5
1,2,5

.706
.708
.699
.686

.51*
.576
.605
.618

3,*,1*
1,3,5
1,2,5
1,2,5

.706
.708
.686
.682

.51*
.609
.618
.615

1,2,5
1,2,5
1,2,5

3A,1^
1,2,5
1,2,5
1,2,5

3>,1*
1,2,5
1,2,5
1,2,5

.673
.671

—

Query QA15C0ST

Results of Various Progressions of Alpha
Figure 6.683|

VI-43
Progressionof Alpha

Correlation
FunctionU / 1 5 , V 1 5 , V 1 5 Cosine

Iteration

0
1
2

3
1 Cosine

1,2,3

0
l
2

3
Cosine

2,3^

0
1
2

3
Co-occurrence

2,3^

0
1
2

3
2,3ft

SimpleVectorMatching

0
1
2

3>f5

Cosine0l2

1/15,2/15,3/15 Cosine

0
1
2

3

3

3
2/l5,3/l5,Vl5 Cosine

012

3/l5,Vl5,5/l5 Cosine

0I
2

Cosine

0
1
2

3

3
3f2,l

3
L—•

Ranks of theRelevant Documents
Appearing in the
Top Fifteen

3,4,l4
1,3,5
1,2,5
1,2,5
3,4,l4
1,2,5

1,2,5
1,2,5
3,4,l4
1,2,5
1,2,5
1,2,5

2,7
1,6,9
1,3,9
1,3,12
2,8,9,11
1,3,4,14
1,2,4,14
1,2,4,15
3,4,1*
1,2,5
1,2,5
1,2,5
3,4,l4
2,5,8
1,3,6
1,2,5
3,4,l4
2,3,5
1,3,5
1,2,5
3,4,l4
1,3,5
1,2,5
1,2,5
3,4,l4
1,2,5
1,2,5
1,2,5

'
Figure 6 (continued)

Normalized
RecallNormalized
Precision

.706
•697
.682
.677
.706
.675

.51^
.603
.616
.613

.666
.660
.706
.667
.662
.662
.662
.643
.662
.665
.725
.766
• 771
• 773
.706
.667
.662
.662

.514
.612
.608
.606
.51^

.608
.606
• 606

.457
.522
.564
.555
.550
.682
•705
.705
• 514
.608
.606
• 606

.706
.701
.703
.686

.514
.532
.598
.618

.706
.708
.686
.682.

.514
.576
.598
.616

.706
.708
.686
.677
.706
.667
.662
.660

.514
.609
.618
.613
.514
.608
.606
.606

Vl-kkProgressionof Alpha

Correlation
Function

1 4,3,2

Cosine5,4,3Cosine3/15,2/15,1/15 Cosine

4/15,3/15,2/15 Cosine

5/15,4/15,3/15 Cosine

Correlations

CosineRanks of theRelevant Documents Normalized Normalized
Iteration
Appearing in the
RecallPrecision
Top Fifteen
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3

3,4,14
1,2,5
1,2,5
1,2,5

• 706
.665
.662
.662

.514
.607
.606
.606

3,4,14
1,2,5
1,2,5
1,2,5

.706
.662
.662
.662

.514
.606
.606
.606

3,4,14
1,3,5
1,3,5
1,3,5
3,4,14
1,3,5
1,2,5
1,2,5
3,4,14
1,2,5
1,2,5
1,2,5*
3,4,14
1,3,5
1,2,5
1,2,5

.706
.708
.686
.686

.514
.609
.598
.618

.706
.697
.686
.682

.514
.603
.618
.616

.706
.693
.680
.677
.706
.677
.668
.662

.514
.620
.614
.613

Figure 6 (continued)

•514
•595
.609
.591

VX-45

Ranks of
Relevant
Documents

Normalized
RecallNormalized
Precision

Rank
RecallLog
Precision

InitialResults

3
k
Ik
19
k9
68

1117HiAMIIJG
306PROGRESS
1110C0MFUTER
1106A NEW CE
902D0CUMENT
716A NATIONA

.706

.512

•13^

.407

Results AfterFirst Update

1 1117ELAI^TNG
2 1110C0MPOTER
5 306PROGRESS
k2 1106A HEW CE
52 716A NATIONA
67 902D0CUMSNT

.670

.613

.12fc

.k&

Results After
Second Update

1 iai7HAMING
2 1110C0MFUTER
5 306PROGRESS
kj 1106k HEW CE
51 716A NATIONA
68 902DOCUMENT

.669

• 607

.121

.464

Results After
Third Update

1 H17PLAUT1ING
2 HlOCOIiFJTER
5 306PROGRESS
k$ IIO6A HEW CE
51 716A NATIONA
68 902DOCUI-EITT

.665

•605

.119

M9

Query QA15C0ST
Cosine Correlation Function
Increasing Alpha Strategy
Typical Results of Query Modification Using RelevanceFeedback Information in Case when the Relevant Documents
Split into Two Regions in N-Space
Figure 7

Initial
Results

Results AfterFirst Update

Rank

Name

Rank

Name

2
22
2^

306PROGRESS
504IS RELEVA
llO^THE IBM
518EDUCATION
505THE PLACElOOlDESIGN 0

1
15
39
^5
16
50

306PROGRESSllOUTHE IBM
505THE PLACElOOlDESIGN 0
518EDUCATION
50^IS RELEVA

h5
h9
51

Results After
Third Update

Results After
Second Update
Rank
1
2

he
63
6k
69

Name
IIOUTHE IBM
306PROGRESS
50*MIS RELEVA
lOOlDESIGN 0
518EDUCATION
505THE PLACE

Rank

Name

1
2
^5
63

1104THE IBM
306PR0GRESS
50UIS RELEVA
lOOlDESIGN 0
518EDUCATI0N
505THE PLACE

66
69

Query QA13EVALU
Cosine Correlation Function
Increasing Alpha Strategy) Typical Results of Query Modification Using Relevance Feedback
Information Shoving the Shifting of the Ranksof the Relevant Documents
Shifting of Queries in N-Space Due to Query Modification
Using Relevance Feedback Information
Figure 8

VI-^7

D1,Dp,D«, and D^ are relevant
documents, "but only D. is ranked
in the 15 highest correlated
documents.

a)

Initial Configuration

Query has been shifted and D- andT> appear ranked in the top 15.

Query shifted further so that D-ijD^
and D

now appear in top 15, but

D- now appears above D.»c)

After Second Update(b) Two Dimensional Representation of Query Modification Showing
the Shifting of the Ranks of the Relevant Documents
Figure 8 (continued)

VI-48
1

I

I

I

Relevant Document
Ranks

Normalized
RecallNormalized
Precision

InitialResults

33,50

.506

.172

Results AfterFirst Update

6,28

.506

.k$k

Results After
Second Update

1,2

1.000

1.000

InitialResults

21

.756

.309

Results AfterFirst Update

kk

.kj6

.141

Results After
Second Update

kl

• 512

. -157

Results After
Third Update

22

.Jhk

•299

InitialResults

16

.817

•371

Results AfterFirst Update

65

.220

.053

Results After
Second Update

37

.561

.181

Results After
Third Update

7

•927

.558

A. Query QA9ANALYSIS

B. Query QB11INDEX

C. Query QB8C0STRET

Cosine Correlation Function and Increasing
Alpha Strategy used for all Three Cases
Selected Results Showing Various Effects of the
Negative Weighting Heuristic Method
Figure 9

Vl-49Results for Query No. 9

QA9ANALYSIS

Document
Rank | No.

Relevant 1 Correlation

1
39
2 47
3 11
4
62
5 12
6
2
4
7
8 17
9
9
10
23
U
27
12
21
1
13
14
71
15 | 70

Name
517RESEARCH
1116RETRIEVA
1117PLANNING
809A MECHANI
1113THE MATE

.43301270.42135049.40209035.39900375•34815531•34299717.32478490.32075015.29329425.29329423.28867513.28724249.28647316.28517591
1 .28306926

8I4A NEW EFF

307ROLE OF /301D0CUMENT
206ANALYSIS
815A FACET A
504lS RELEVA
1209SCIENCE
1104THE IBM
IO8SEARCHERS
601ENC0DING

Ranking took .120000 seconds
Relevant Document Ranks
Document
Rank No.
[

33
50

82
50

Name

Relevant

Correlation

308MACHINE R
418ENGLISH-L

X
X.20701967.12038585

Normalized Recall * .5061728
Rank Recall • .0361

Normalized Precision * .1717605

Log Precision * .0936

QMery(l) * Queiy(O) + Alpha
Document Rank

1 2

Weight-1.

S >reiGHT(l) # j^^^t o f R a n k x
1=1(1)
3 * 5 6 7 8 9 10 11 12 13 1^ 15- 1 0 0 0 0 0 0 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha = 1(a) Initial Results
Use of Negative Weights
Cosine Correlation Function Increasing Alpha StrategyFigure 10

VI-50Results for Query No. 9

QA9AHALYSIS

Document
Rank 1 No.

Relevant

1

k

OJ

l-jk
i 31

3
k
5
6
7
8
9
10
11
12

13
Ik

15

48
57
82
15
25
13
68
k6
8o
75
18
28

Name

Correlation.Q5kQk'jk'l.01273911

307ROLE OP /305THE USE U
207ERACTICAL
80*KX)MFUTERI
603C0MPILATI
308MACHINE R
610CHARACTER
209THE REDUC
1119RECENT A
811THE RELAI
109STATUS RE
1001DESIGN 0
201A SYSTEM
1122STATE-0F
506THE ROLE

0
0
0
0

X

-.01836796-.029U8198-.03039738-.03125382-.03303192-.03^283-.03722904-.03827795-03896U33

Ranking took .166667 seconds
Relevant Document Ranks
Document
Rank No.

6
28

82
50

Relevant

Name

308MACHINE P.
^l8ENGLISH-L

X
X

Normalized Recall « .8086^20

Correlation
0-.06677806

Normalized Precision * .^535267

Rank Recall « .0882 Log Precision * .1353
Query(2) = Query(1) + AlphaN
I S ^ B i S i * Document of Rank I
1=1
W

Document Rank

1 2

5 6

Weight0 0 0 0 0

3 ^7 8 9 10 11 12 13 1^ 15

1 0 0 0 0 0 0 0 0 0

N • Number of retrieved documents
Alpha * 2(b) After First Update
Figure 10 (continued)

VI-51Results for Query No. 9

QA9AMLYSIS

Document
Rank No.

Name

Relevant

Correlation

308MACHINE R
4l8ENGLISH-L
307ROLE OF /213ADAPTIVE
603COMPHATI
516AN EXPERI
811THE RELAI
511AMLYSIS7
901A SELECTE
506THE ROLE
201A SYSTEM
1103FEASIBIL
211A COMPUTE
505THE PLACE
305THE USE 0

X
X.80137^97.159776^7.06465833.06020315.05430365.04789131.03738971.02824476.02715182.02330699.02226901.01279949.01236548.01128809.00762006

Name

Relevant

Correlation

308MACHINE R
418ENGLISH-L

X
X.80137497.15977647

1 ' 82
2 50

3
4
5
6
7
8
9
10
11
12
13
14
15

4
77
57
61
68
4i
54
28
75
5
7
65
74

Ranking took .166657 seconds
Relevant Document Ranks
Document
Rank No.
1
2

82
50

Normalized Recall = 1.0000000
Rank Recall = 1.0000

Normalized Precision = 1.0000000

Log Precision = 1.0000

No update for these results(c) After Second Update
Figure 10 (continued)

VI-52
Results for Query No. 9

QA9MALYSIS

Document
Rank 1 No.

Relevant

1
2
3
4
5

39

U7

11
62
12
2
4
17
9
23
27
21
1
71
70

6

7
8
9
10
11
12
13
14
15

Name
517RSSEARCH
1116RETRIEVA
1117FLANNING
809A MECHANI
1113THE MATE
8I4A NEW EFF
3L7ROLE OF /301D0CUMENT
206ANALYSIS
815A FACET A
504lS RELEVA
1209SCIENCE
1104THE

Correlation.43301270.42135049.40209035•39900373•34815531.34299717.32478490.320750.5.29329423.29329423.28867513.28724249.28647316.28517591.28306926

IBM

lOSSEARCHERS
601ENC0DING

Ranking took .166657 seconds
Relevant Document Ranks
Document
Rank No.
82
50

33
50

Name

Relevant

Correlation

308MACI-HNE R
418ENGLISH-L

X
X.20701967.12038585

Normalized Recall = .5061728

1

Normalized Precision =* .1717605

Rank Recall = .0361 Log Precision * .0936'

Query(l) = Query(0) + Alpha2

3 4

T.n?TpTTV( T\Z -—7T\
I»l
tl)

5 6

7 8

* Document of Rank I

Document Rank

1

9 10 11 12 13 l4 15

Weight- 1 - 1 0 0 0 0 0 0 0 0 0 0 0 0 0

N « number of retrieved documents
Alpha « 2(d) Initial Results
Use of Negative Weights
Cosine Correlation Function Constant Alpha Strategy
Figure 10 (continued)

VI-53Results for Query No. 9
Document
Rank No.

k
1
2 \Tk

3
k
5
6
7
8
9

31
k8
57
15
82
68

Li!

7

QA9ANALYSIS

Name

Relevant

307ROLE OF /305THE USE 0207PRACTICAL
ot^COMFUTERI
603COMPILATI
610CHARACTER
308MACHBIE R
811THE RELAI
1119RBCENT A
1122STATE-0F
506THE ROLE
518EDUCATION
1001DESIGN 0
209THE REDUC
211A COMPUTE

13
10 18
11 28
12 1^
13 1 80
Ik 25

X

Correlation.00855386.0060^851
000-.017^216-.02726146-.02967852-.03608157-.0363^862-.037000^0-.C&08216U-.0^1^7700-.0^199^00-.0^907625

Ranking took .166657 seconds
Relevant Document Ranks,
Document
Rank No.
7
28

82
50

Name

Relevant

Correlation

308MACHITCE R*H8ENGLISH-L

X
X-.027261^6•.07926529

Normalized Recall « .802^691
Rank Recall = .0857
Query(2) = Query(l) + Alpha
3 ^Normalized Precision *Log Precision = .1313
Z ^IGHTIJ) * Document of Rank I

Document Rank 1

2

5 6

7 8

9 10 11 12 13 i t 15

Weight0 0 0 0 0 0 1 0 0 0 0 0 0 0 0

N • Number of retrieved documents
Alpha = 2(e)After F i r s t Update

Figure 10 (continued).h&5lk6

Results for Query No. 9

QA9ANALYSIS

Document
Rank No.

Name

Relevant

Correlation

1 308MACHINE R
4l8ENGLISH-L
603COMPILATI
307ROLE OF /811THE RELAI
305THE USE 0
506THE ROLE
207PRACTTCAL
804C0MPUTERI213ADAPTIVE
901A SELECTS
610CHARACTER
1122STATE-0F
211A COMPUTE
201A SYSTEM

X
X.52017756.06772233.03682715.02923295.OI26783O.00516770

1
2
3
k
5
6
7
8
9
10
11
12
13
14
15

0
0
0
-.00510350-.00613786-.01490215-.01552769-.01677182-.02265329_

1

Ranking took .166557 seconds
Relevant Document Ranks

Document
Rank No.
1
2

82
50

Name

Relevant

Correlation

308MACHIIIE R
4l8SNGLISH-L

X
X.52017756.06772233

Normalized Recall = 1.0000000
Rank Recall = 1.0000

Normalized Precision « 1.0000000

Log Precision = 1.0000

No Update for these results(f)After Second Update

Figure 10 (continued)

VI-55Results for Query No. 9

QA9AIIALYSIS

Document
Rank |No.

Relevant

1 39
2 \k7
3 11

k 62
5 12
6 2
7 U
8 17
9 9
10 23
11 27
12 21
13 1

Ik 71

15 70

Name
517RESEARCH
1116RETRIEVA
1117PLANNING
889A MECHANI
1113THE MATE
81^-A NEW EFF
307ROLE OF/
301D0CUMENT
206ANALYSIS
815A FACET A
50U1S RELEVA
1209SCIENCE
llO^THE IBM
108SEARCHERS
601ENC0DING

Correlation
A3301270.^21350^9A0209035•39900373.3^815531.3U299717.32478^90.32075015.29329^23.29329^23.28867513.2872^2^9.286U7316.28517591.28306926

Ranking took .150000 seconds
Relevant Document Ranks
Document
Rank No.
82

^9 50

Name

Relevant

Correlation

308MACHINE R
U18ENGLISH-L

X
X.20701967.12038585

Normalized Recall * .5123^57

Normalized Precision = .17^2522

Rank Recall « .0366 Log Prscision • .0938

Query(1) = Query (0) + Alpha Z ^ I & m i )U ;ML* Document of Rank I

Document Rank 1 2 3 k 5 6 7 8 9 10 11 12 13 l4 15
Weight- 1 - 1 0 0 0 0 0 0 0 0 0 0 0 0 0
N = Number of retrieved documents
Alpha • Correlations(g)

Initial Results

Use of Negative Weights
Cosine Correlation Function Alpha = Correlations Strategy
Figure 10 (continued)

VI-56
Results for Query No. 9

QA9ANALYSIS

Document
Rank |No.

Relevant

1 1
OJ

k
82

307ROLE OF/
308MACHINE R
1 201A SYSTEM
305TP3E USE 0
109STATUS RE
81 ;A FACET A
209THE REDUC
1102CENTRALI
1210IN INFOR
1201ACCREDIT
207PRACTICAL 1
306PROGRESS
1215 GRADUATE80^C0MPUTERI
603COMPILATI

3 75
k

17*

5 k6
6 23
7 25
8 ko

9

Name

3

10 10
31
n
12 37
13 ^5
Ik U8
15 57

Correlation•18272327.0i%9256^•03933260•03230139•02442883.01289122.01A5912.007^729^

X

0
0
0
0
0
0
0

Ranking took .166557 seconds
Relevant Document Ranks

• 2
26

0 ro

Document
Rank No.

Name

Relevant

Correlation

303MACHINE R^lSENGLISH-L

X
X.Oc%925&-.02116537

Normalized Recall • .8^56790 Normalized Precision = .5981638
Rank Recall » .1071 Log Precision « .175^
Query(2) = Query(l) + Alpha
Document Rank
Weight1 2

E ^IGHT(l) * Document of Rank Ii»iID

3 *• 5 6 7 8

9 10 11 12 13 Ik 15

0 1 0 0 0 0 0 0 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha « Correlations(h) After First Update
Figure 10 (continued)

VI-57
Results for Query No- 9

QA9AHALYSIS

Document
Rank |No.

Name

Relevant

Correlation

308MACHINE R
307ROLE OP/
201A SYSTEM
305THE USE 0
109STATUS RE
209THE REDUC
4l8ENGLISH-L
815A FACET A
511ANALYSIS
603C0MPILATI
1102CENTRALI
213ADAPTIVE
1210IN INFOR
1201ACCREDIT
2L7FRACTIGAL

X.22188008.17902224.05009643.03164696.02393390.01831005.01382437.01263004•00977531.00939705.00732154.00260448

1
2
3
4
5
6
7
8
9
10
11

R?
4
75
74
46
25
50
23

4l
57

4o

12 , 11

13

3
10
15 31
lk

X

0
0

°_j

Ranking took .156667 seconds
Relevant Document Ranks
Document
Rank No.
1
7

82
58

Name

Relevant

Correlation

308MACHINE R
418ENGLISH-L

X
X.22188008.01382437

Normalized Recall = .9691358 Normalized Precision = .8454909
Rank Recall = .3750 Log Precision = .3562
No Update for these results(i) After Second Update
Figure 10 (continued)

VI-58Results for Query No. 28
Document
Rank No.

1 x5
2 B
3 ! 6
k 30
5 65
6 7
7 9
8 61
9 48
10 28
11 20
12 57
13 42
14 51
15 80

QB11IKDEX
1

Name

Relevant

610CHARACTER
1109A PROGRA
1106A NEW CE
U09RELATI0NA
50 5 THE PLACE
211A COMPUTE
206ANALYSIS
516AN EXPERI
804COMFUTERI
506THE ROLE
304PHOTOGRAP
603COMPILATI
203NEW PHOTO j
1115S0ME OBS
1001DESIGN 0

Correlation.561951^9.48112524.43643578.41478068.38490018.31622777.31108551.30618622.28867513.26490647.25819889.23145502.18983160.18983160.18490007

Ranking took .056667 seconds
Relevant Document Ranks
Document
Rank No.

Name

Relevant

Correlation

21 36

1110C0MPOTER

X.13801311

Normalized Recall = .7560976 Normalized Precision = .3091181
Rank Recall = .0476 Log Precision = 0
Query(l) = Query(O) + Alpha
2

3 4

N
S ™EGHT(I) * Document of Rank I
1=1(1)

Document Rank

1

5 6

7 8

9 10 11 12 13 l4 15

Weight- 1 - 1 0 0 0 0 0 0 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha = 1
Update plus matching took 7.416667 seconds(a) Initial Results (increasing Alpha Strategy)
Continued Use of Negative Weights when Use in the First Update
Fails to Move Query into Correct Region of N-Space
Cosine Correlation Function
Figure 11

Results for Query No. 28

QBllINDEX

Document
Rank [No.

Relevant

1

5

CM

!52

Name

3 -55

k 82

5 7^
6 35
7 50
8 77
9 ^3
ID 5^
11 53
12 k
13 39
lk
23
15

iJ

1103FEASIBIL
1121SC0PE AN
907SATIRE 01
308MACHINE R
305THE USE 0lOO^DENSITY
418ENGLISH-L
213ADAPTIVE
1206THE EDUC
901A SELECTE
1118D0CUMENT
307ROLE OF/
517RESEARCH
815APACET A
121US0ME HUM

VI-59

Correlation

0000-.007782^0-.009501^3-.01019877-.015371^5-.01665^53-.OI&8685-.02109709-.0220119^-.02U4.5580-.0248^712-.02638095
1

I

Ranking took .183333 seconds
Relevant Document Ranks
Document
Rank No.

kh

36

Name

Relevant

Correlation

1110C0MPUTER

X-.08267569

Normalized Recall = .U756098 Normalized Precision « .1U12683
Rank Recall • .0227 Log Precision • 0Queiy(2) = Query(l) + Alpha

Document Rank
Weight

1

2

3 ^

N
E W E I G ^ I ) # D o c u m e n t of Rank I
I»l
(1)

5 6

7 8 9 10 11 12 13 lb 15- 1 - 1 0 0 0 0 0 0 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha - 2Update plus matching took 8.716667 seconds(b) Results after First Update (increasing Alpha Strategy)
Figure 11 (continued)

vi-6oResults for Query No. 28

QEUINDEX

Document
Rank No.

Relevant

1 50
2 77

3 54
29
5 35
6 31
7 18
8 82
9 2
10 38
n 74
k

12 11
13 32
lfc Ik

64

15
1

Name
4l8ENGLISH-L
213ADAPTIVE
901A SELECTE
212A STATIST
1004DENSITY
207PRACTICAL
1122STATE-0F
308MACHINE R
8I4A NEW EFF
303C. M. HER
305THE USE 0 1
1117FLAMING
902DOCUMENT
518EDUCATION
507PARAGRAPH

Correlation-.00576985-.OO889623-.01045874-.02426927-.02687667!-.03415810-.03968813-.03968813-.04383773-.04518692-.04843090-.05139033-.05146744-.05200079-.05325330

1

Ranking took .150667 seconds
Relevant Document Ranks
Document
Rank No.

41

36

Name

Relevant

Correlation

111C0MFUTER

X-.17773708

Normalized Recall = .5121951 Normalized Precision = .1572932
Rank Recall = .0244 Log Precision = 0
Query(3) = Query(2) + Alpha
Document Rank

1 2

3 4

Weight- 1 - 1 0 0 0

Z M I ^ M l ) ^ Document of Rank I
1=1

5 6

7 8
0

0

9 10 11 12 13 l4 15
0

0

0

0

0

0

0

0

N = Number of retrieved documents
Alpha = 3Update plus matching took 10.650000 seconds(c) Results after Second Update (increasing Alpha Strategy)
Figure 11 (continued)

VI-61Results for Query No. 28

QBllIMDEX

Document
Rank |No.

Name

Relevant

1 35
2 32
3 Tk
k 18
5 31
6 k
7 5k
8 13
9 25
10 llf
11 63
12 38
13 8l
Ik 6k
15 71

18&-DENSITY
902D0CUMENT
305THE USE 01122STATE-0F
207PRACTICAL
307ROLE OF/
9 0 M SELECTE
1119RECENT A
209THE REDUC
518EDUCATION
21^ELECTR0NI
303C. M. HER
1107THE AUTO 1
507PARAGRAPH
108SEARCHERS

Correlation
i -.02580081-.03087952-.O36982U8-.cA-762^30-.05123559-.063608^8-.08127568-.0850939^-.0917015^-•09805563-.10501212-.IO8UJ-530-.10956911-.II821871-.12027386

Ranking took .183333 seconds
Relevant Document Ranks
Document
Rank No.

Name

Relevant

Correlation

22 36.

1110 COMPUTER

X-.13189557

Normalized Recall = .7^3902^

Normalized Precision = .2985615

Rank Recall = .0^55 Log Precision = 0
Mo Update for these results(d) Results after Third Update (increasing Alpha Strategy)
Figure 11 (continued)

VI-62
Results for Query No. 12

QA12J0URNAL

Document
Rank [No.

Name

Relevant

Correlation

307R0LE OF/
1102CENTRALI
302F.ECORDIITG
109STATUS RE
1212T0WARD A
1201ACCREDIT
1107TH3 AUTO
201A SYSTEM
1206THE EDUC
1209SCIEI7CE
lOCtDEHSITY
1119RECEKT A
6O3C0MPILATI
1215 GRADUATE

X.6^956980.3825^603.30096U63.2977^567. 2^9^897.20^+12^15.20^12^15.2013^682.19658927.1723^550.16823165

1 k
2 ko
3 19
h

^6

5 60
10
6
i
81

8 75
9 Jv3
10
ll
12
13

21
35
13

57

111- >i-5

1 J^J 38.l6h39899.163063^2.15861032.157131+8t

303C. M. HER J

Ranking took .056557 seconds
Relevant Document Ranks
Document
Rank Ho.
1
16
17
25
33

1k
7
9
25

Name

Relevant

Correlation

307ROLE OF/'
211A COMPUTE
206ANALYSIS
209TH3 RZDUC
203NEW PHDTO

X
X
X
X
X

.&956980.1^907120.1^66^712.3.0629880.06711561

Normalized Recall = .8C256U1

Normalized Precision = .5599956

Rank Recall = .1630 Log Precision • .3886"'vrVTC'\VY( "Z\S"^/-.v —1=1^}

Query(l) * Qucry(O) + Alpha
Document Rank
Weight

1

2

3 ^

5

6

7

8

* Document of Rank I

9 10 11 12 13 lk 15

1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha = 1(a) Initial Results (Increasing Alpha Strategy)
Comparison of Results After Three Iterations as a Functionof the Value of Alpha used in First Update
Cosine Correlation Function
Figure 12

VI
Results for Query No. 12

QA12J0UEHAL

Document
Rank [No.

Name

Relevant

Correlation

307HOLS OF/
1102CENTRALI
302REC0RDING
109STATUS RE
1212T0WARD A
301D0CUMENT
1107THE AUTO
11017INF0RMA
91^THE RAPID
lOO^DENSITY
108SEARCHERS
902D0CUMENT
206ANALYSIS
305THE USE 0813AUT0MATIC

X•97316392. 3666^-099•3l88lU8l.2703^708.222^087^.21357210•20593^02. 20^90176.1853^062.16972328.16782759.16250565. 1553^7^.15291796.1^99^79^

1 k
2 \ko

3 19

k 46

5 60
6 17
7 8l
8 33
9 22
10 35
11 71
12 32
x

3

9

Ik 74
15 69^

X•1

Ranking took .150000 seconds
Relevant Document Ranks
Document
Rank No.

Name

Relevant

Correlation

•1 k
9
13
18 k2
7
27
32 25

307ROLE OF/
206ANALYSIS
20 31®J PHOTO
211A COMPUTE
209THE REDUC

X
X
X
X
X

.97316392.1553^7^.1^2192^5.11279^71.09651717

Normalized Recall = .8051282 Normalized Precision = .5660867
Rank Recall = .16^8 Log Precision = .3919
Q,uery(2) = Query(l) + Alpha
Document Rank
Weight1

2

3 k^ J * Document of Rank I

Z

I»l

W

5 6

7 8

9 10 11 12 13 l4 15

1 0 0 0 0 0 0 0 0 0 0 0 1 0 0

N • Number of retrieved documents
Alpha = 2(b) Results after First Update (increasing Alpha Strategy)
Figure 12 (continued)

Results for ftjiery No. 12

QA12JbtffiitAL

Document
Rank [No.

Name

Relevant

Correlation

307ROLE OF/
206ANALYSIS
1102CENTRALI
302REC0KDING
109STATUS RE
11017INF0RMA
1212T0WARD A
211A COMPUTE
201A SYSTEM
301D0CUIvffirTT
914THE RAPID
1107THE AUTO
813AUTOMATIC
906MICROFILM
108SEARCHERS

X
X.88970072.5726^-7^2.40983981.276800^3.27678442.27389171.26646264.25798837.24558033.24107313.23551009.22878123.22320053.21701659.20838206

1
2

k

9

3 4o
4 19
5 ke
6 33
7 60
8 7
9 175
10 17
11 22
12 81

13 69
14 73
15 71

X

Ranking took .183333 seconds
Relevant Document Ranks
Do ciune nt
Rank Mo.

Name

Relevant

Correlation

4
1
2
9
8
7
16 42
24 25

307R0LE OF/
206AIIALYSIS
211A COMPUTE
203NEW PHOTO
209THE REDUC

X
X
X
X
X

.88970072.57284742.25798837.20575649.16819640*

Normalized Recall • .9076923 Normalized Precision • .7701336
Rank Recall * .2941 Log Precision = .5488
Query(3) = Query(2) + AlphaZ HSJ2SILI) * Document of Rank I

W

1*1
Document Rank

Weight

1

2

3 4

5 6

7

8

9 10 11 12 13 l4 15

1 1 0 0 0 0 0 1 0 0 0 0 0 0 0

N = Number of retrieved documents
Alpha = 3(c) Results after Second Update (increasing Alpha Strategy)
Figure 12 (continued)

VI
Results for Query No. 12

QA12J0URNAL

Document
Rank [No.

Relevant

Correlation

X
X
X.76756620.68759^01.532839M

X.31186705.28223262.2795152U.278IIU15•275^557^.27181938.2632^6^3.26323561.25698269.25179059 .2^952061

Relevant

Correlation

X
X
X
X
X

.76756620.68759^01.532839^.31186705.22573736

1
CVJ

3

• h

9
7

h ho

5
6
7
8
9
10
11
12

k2
75
33
60

he

22
56
hQ
13 J73
Ik 81
15 20

Name
307ROLE OF/
206ANALYSIS
211A COMPUTE
1102CENTRALI
203NEW PHOTO
201A SYSTEM
11017INF0RMA
1212T0WARD A
109STATUS RE
91^THE RAPID
1112THE USE
80^C0MFUTERI
906MICROFILM
1107THE AUTO j
30^PH0T0GRAP,khjjh693

Ranking took -183333 seconds
Relevant Document Ranks
Document
Rank No.

k
1
2
9
3
7
5 ^2
19 25

Name

307ROLE OF/
206ANALYSIS
211A COMPOTE
20 3JEW PHOTO *209THE REDUC

Normalized Recall * .9615385 Normalized Precision - .9089968
Rank Recall * .5000 Log Precision - .75^5
No Update for these results(d) Results after Third Update (increasing Alpha Strategy)
Figure 12 (continued)

vi-66
Results for Query 12
Document
Rank |Ho.

1 If.
2 4o

3 19

k k6

5 60
6 10
7 81
8 75
9 h3
10 21
11 35
12 13

13 57
h5

lk

15 38

QA12JOURNAL

Name

Relevant

Correlation

307ROLE OF/
1102CENTRALI
302REC0RDING
109STATUS RE
1212T0WARD A
1201ACCREDIT
1107THE AUTO
201A SYSTEM
1206THE EDUC
1209SCIENCEIOCADENSITY
1119RECENT A
603COMPILATI
1215 GRADUATE
3030. M. HER

X.6^956980.3825^603.3OO96U63.2977^567.2U49^897.20^12^15.20^12^15.2013^682.19658927.1723^550.I6823165.16^39899.163663^-2.15861032.15713^8^

Ranking took .083333 seconds
Relevant Document Ranks
Document
Rank No.

Name

1 Relevant

Correlation

1
k
16
7
17
9
25 25
33 k2

307ROLE OF/
211A COMPUTE
206ANALYSIS
209THE REDUC
203NEW PHOTO

X
X
X
X
X

.6U956980.1^907120.1468^712.IO629880.06711561

Normalized Recall = .80256^-1 Normalized Precision = .5599956
Rank Recall = .1630 Log Precision = .3886
Query(l) = Query(o) + Alpha

£

W E I

{1)

1=1
2

3

h

5

6

^ ( 1 ) ^ . DoC ument of Rank I

Document Rank

1

7

8

Weight1 0 0 0 0 0 0 0 09

10 11 12 13 1^ 15
0 0 0 0 0 0

N = Number of r e t r i e v e d documents
Alpha = 2(e) Initial Results (Constant Alpha Strategy)
Figure 12 (continued)

VI
Results for Query No. 12

QA12J0URNAL

Document
Rank No.

Name

Relevant

Correlation

307ROLE OF/
1102CENTRALI
302RECORDING
109STATUS RE
11017INF0RMA
301D0CUMENT
1212T0WARD A
1107THE AUTO
91^THE RAPID
108SEARCHERS
lOO^DENSIlY
902DOCUMENT
306PROGRESS
906MICROFILM
813AUTOMATIC

X.991^6269.3^989921.30969005.25531^80.22165517.22003279.2100^201.19837301.18670^01.173689250.1631*9177.1611^32^.15^03081.152^319.15105098

• 1

h

2 ko
3 19

k 46
5 33
6 17
7 60
8 8l
9 22

10 71

35
n
12 32
13 37
Ik 73
15 69

Ranking took .150000 seconds
Relevant Document Ranks
Document
Rank No.

Name

Relevant

Correlation

1 k
16
9
19 k2
28
7
31 25

307ROLE OF/
206AMLYSIS
203NEW PHOTO
211A COMPUTE
209THE REDUC

X
X
X
X
X

.991^6269.15089864.1I+963301.10226200.091150^7

Normalized Recall - .79^8718 Normalized Precision « .5505320
Rank Recall = .1579 Log Precision « .3835

Query(2) * Query(l) + Alpha L m i ^ { x h DoCument of Rank II»lW
Document Rank 1 2 3 k 5 6 7 8 9 10 11 12 13 Ik 15
Weight1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
N * Number of retrieved documents
Alpha = 2(f) Results after First Update (Constant Alpha Strategy)
Figure 12 (continued)

VI-68Results for Query No. 12

QA12J0URNAL

Document
Rank [No.

Name

Relevant

Correlation

307ROLE OF/
1102CENTRALI
302REC0RDING
109STATUS RE
11O17HSF0RMA
301D0CUMENT
1212T0WARD A
1107THE AUTO
914THE RAPID
I08SEARCHERS
214ELECTR0NI
902D0CUMENT
1004DENSITY

X•99758179. 33804448.30263667.24496349.23067222.22284037.20152621.19281829.1865983^.17669065.159862^3.15951821.15891377.15743547.15674261

Name

Relevant

Correlation

307ROLE OF/
203NEW PHOTO
206AMALYSIS
211A COMPUTE
209THE REDUC

X
X
X
X
X

.99758179.15338289.14746183.09539051.08745493

1 4
2 4o

3 19
4 46
5 33
6 17
7 60
8 81
9 22
10 71
11 63
12 32
13 35
It 73
15 37

9O6MICROFILM

306FROGRESS

Ranking took .150000 seconds
Relevant Document Ranks
Document
Rank ITo.
1
16
18
28
32

4
42
9
7
25

Normalized Recall = .7948718

Normalized Precision « .5518355

Rank Recall = .1579 Log Precision = .3842
Query(3) = Query(2) + Alphas ^ 1 ^ ( 1 ) * Document of Rank I
v1)

1-1
Document Rank
Weight

1 2
1

3 4
0

0

5 6
0

0

7
0

8
0

9 10 11 12 13 l 4 15
0

0

0

0

0

0

0

0

N = Number of r e t r i e v e d documents
Alpha * 2(g) Results after Second Update (Constant Alpha Strategy)
Figure 12 (continued)

VI-69QA12J0URNAL

Document
Rank No.

Name

Relevant

Correlation

307ROLE OF/
1102CENTRALI
302REC0RDING
109STATUS RE
11017INF0RMA
301IX)CUM5NT
1212TOV7ARD A
1107THE AUTO
914THE RAPID
I08SEARCHERS
214ELECTR0NI
906MICR0FILM
902D0CUMENT
306PR0GRESSIOO4DENSITY

X.99887910.33^2788.29979583.2Uo97886.23373192.22365318.198*818.19062321.1863871^.1775^277.16331062.15910194.15878053.15758186.15710467

1

4

OJ

Results for Query No. 12

4o

3 19
4 46
5 33
6 17
7 6o
8 8l
9 22
10 71
11 63
12 73

13 32
lk
37
15 35

Ranking took .133333 seconds
Relevant Document Ranks
Document
Rank No.

Narae

Relevant

Correlation

k
1
16 42
3.8 9
28
7
34 25

307R0LE OF/
203HEW PHOTO
206AJMALYSIS
211A COMPUTE
209THE REDUC

X
X
X
X
X

.99887910.15460210.14607761.09280767.08603237

Normalized Recall = .7897436

Normalized Precision = .5482948

Rank Recall = .1546 Log Precision = .3823

No Update for these results(h) Results af^er Third Update (Constant Alpha Strategy)
Figure 12 (continued)

Only relevant documents axe shown.
q is the initial query.

Initial configuration

qf obtained using smallvalue of aq" obtained using larger
value of a

Configuration after one modification

Effect of the Initial Value of a used in
the Iteration Process
Figure 13

VI-71
Correlation
Function
CosineNormalized
RecallNormalized
Precision

.706
•680
.669
.665

•512
.513
.607
.605

.706
.669
.669
.665

.512
.607
.607
.605

.706
.677
.671
.667

.512.
•593
.609
.592
.455
•533
.570
.557

l,l,i£, 35,51,7**

.662
.654
.675
.673
.662
.650
.675
.669

.455
.528
.57**
.562

2,7,25,30,48,65
1,2,26,39,57,60
1,2,16,43,57,70 I
1,2,2^3,50,75

.662
.643
.636
.649

.455
• 513
.546
.531

2,8,9,11,1*6,72
2,2,5,11,32,77

.725
.762
.768
• 773

Strategy

Iteration

Increasing
Alpha

0
1
2
3
0
1
2
3
0
1
2
3

1 i,it,i!bi9,l*9,68

0
l
2
3
0
l
2
3
0
1
2
3

2,£25,30,48,65
1,2,2*^^7,75
1,2, 9,34,50,74
1,2,12,3^,1*9,73
2,7,25,30,48,65
1,1,8,45,47,76

Constant
Alpha

Correlations

Co-occurrence Increasing
Alpha

Constant
Alpha

Correlations

SimpleVectorMatching

Relevant
Document
Ranks

Increasing
Alpha

Constant
Alpha

Correlations

0
1
.2
3
0
1
2
3
0
1
2
3

1,2,57*2,52,67
1,2,5,1*7,51,68
1,2,2,1*9,51,68
l,i*,l£,19,l*9,6&
l,2,p*7,51,68
1,2,5,47,51,6s
1,2,2,^9,51,68
3,4,14,19,49,68
1,^,2,39,55,67
1,2,2,47,48,70
1,2,1,45,49,71

l,i,I,36,49,7l|-

i,2,J,l2,30,77
1,3,5,15,28,78
1*2,5,^,27,78

.725
.768
.768
•773

.548
.646
.682
.706
.548
.684
.683
.706

2,8,9,11,46,72
1,^,1,15,29,79
1,2,7,24,65,80
1,2,8,25,65,79

.725
•751
.658
.656

.548
.647
.604
.600

l,-3_,S,2t>29'77

1,2,5,1*, 27,78
2,8,9,11,46,72

Query QA15C0ST
The Result of the Query Modification "by Use of Relevance
Information as a Function of the Number of Relevant Documents
Initially in the Fifteen Highest Correlated Documents
Figure Ik