XI-1

XI.

An Experiment in the Use of Bibliographic Data
as a Source of Relevance Feedbackin Information Retrieval

M. Amreich, G. Grissom, D. Michelson, E. Ide

Abstract
In order to determine whether bibliographic information provides auseful tool for relevance feedback, parallel retrieval and feedback searches
are made with and without such information within the framework of the
SMART retrieval system.

Bibliographic material operating as a sole sourceof feedback is shown to be comparable in efficiency to subject material
alone.

1.

Some recommendations are made for further investigation.

Introduction
If the processes of storing documents for future use and of re-trieving the appropriate stored documents in response to a given inquiry
are to be automated using todayfs type of computer, storage capacity andsearch time dictate that some short but meaningful representation of the
document be preferred to storage of the full-length natural language text.
Such representations generally consist of subject indicators, classifications of document content, comparable to the subject card in a librarycatalog.

These subject indices may be assigned on the basis of human judg-

ment of the relevance of each document to the subject, or on the basis ofsome mechanical procedure approximating such a classification.

One suchmechanical procedure is the assignment of indices based on key words

XI-2

mechanically judged indicative of content by their relative frequency, either
directly, each keyword having its own index, or through the intermediary ofa thesaurus, the same index being assigned to words sharing some common property, e.g. synonyms, words differing only by suffix, and so on.

In practice,these sets of "concepts11 often turn out to be either insufficiently refined,or sensitive to minute differences in the wording of the userfs request.
This motivates the present attempt to evaluate the utility of a second typeof information, bibliographic data, as an additional guide to document retrieval in a keyword classification system.

2.

The Bibliographic Assumptions [l]The underlying assumptions implicit in the use of bibliographic

information as a classificatory device are the following:1.

A document cites a work if and only if that work has contributed information (facts, methodology, etc.) to the document,
and, as a corollary, since both the work and the document deal
with that contributed information, they must share at leastthat much common subject matter.

2.

By extension, two documents citing the same work frequently
share some common subject matter or tend to deal with thesame subject.

Note that no claim is made for the converseof this: it is not assumed that all documents dealing with agiven subject need be linked bibliographically.
3. Documents by one author tend to deal with the same or related
subjects. Note that this is a weaker claim.

XX-3

3. The Problem and Method
It should be emphasized that the present experiment, consistent
with common practice, does not attempt to classify documents -uniquely bybibliographic information, but rather uses this only as a supplementary
indication of content. Thus, the method here employed is a form of feedback
starting from documents already known to be relevant. The basic test applied here to the assumptions is the comparison, within a controlled framework, of the retrieval capacities with and without the use of bibliographic
data. [2]The tripartite framework consists of a retrieval system, a document
collection, and a set of requests for information from that collection,
along with human judgments of the relevance or non-relevance of each document to each query, against which to check the machine's decisions. Chosen
by reason of availability, the framework used for the present experimentconsists of the following:a.

The retrieval system known as SMART, as implemented on
the Control Data l6ckb.

at Cornell University.

The ADI document collection (already coded on SMART
except for bibliographic information), consisting of
82 abstracts from short papers given at the 1963 meetingof the American Documentation Institute [3Lc.

The set of 35 queries associated with the ADI collection.

The specific structure of this framework results in certain restictions
on the method of investigation.
is therefore in order.

A detailed consideration of that structure

XI-4

Although initial queries in a keyword classification system could
contain bibliographic information (e.g. "What has X written on the subjectof Y?,f), the queries available with the ADI collection contain only subject
information.

To introduce bibliographic information into the system, aprocedure called relevance feedback is used. [2] After an initial retrieval
operation, the userfs request is modified to reflect the contents of those
documents which were retrieved and declared relevant by the user.

In thepresent experiment, this modification includes bibliographic informationfrom the documents declared relevant. Therefore, the value of bibliographic
information for retrieval is judged here by its usefulness in a relevance
feedback environment, rather than in initial retrieval. Relevance feedback
provides an equally valid test of the assumptions of part 2.
The ADI collection, the only document collection on SMART withreadily available bibliographic information, is atypical in two respects.
Firstly, since the collection includes only short conference papers, there
is a paucity of bibliographic citations. Slightly over 20 per cent of the
documents have no bibliography, and several others attach no authors to theworks they cite. Secondly, since the entire collection was published simultaneously, there can exist no cases of documents
both

i

and

j

i

citing works

j within the collection.

SMART'S representation of the ADI collection is based on the abstractsof the papers, with concept numbers assigned by a thesaurus. Each document
is stored as a vector so that the entire collection may be viewed as amatrix of terms (concepts) against documents. Thus, for concept
document

j , the matrix entry,

i and

C. , has an assigned value. While, intheory, the term-document matrix may be binary (i.e. (T » 1

impliesT.

XI-5

occurs in D ; C. = 0 implies Tjjdoes not occur in D.) in the ADI representJ-jation, C. takes on larger values to show the greater importance of a termjfor one document than for another. These values are called "weights"• To
save space in the actual machine representation, only those values of C.
J

greater than zero are stored. To indicate the position of the weight of
C. in the document vector

C. , the code number (i) for the concept calledjJthe "concept number" is paired with the non-zero value.
Extending the notion of "concept number" to allow the inclusion ofproper nouns, i.e. authors1 names and titles of cited works, permits the
direct addition of bibliographic data to the document vector in the following manner:
1. Dictionaries are manually constructed to assign concept
numbers to the authors (both of documents in the collection
and of works cited by documents in the collection) and to
the titles of the works cited.
2.

The completed dictionaries are used to generate triplets(D,N,W), where D is the number of the document in whichauthor or title concept N occurs with a weight of W.

3. These triplets are sorted by document number D. Then,for each document, the pairs (N,W) are sorted by ascending
value of N, since the particular implementation of SMART
used in the present experiment requires that the concept
numbers be stored in the vector in strict ascending order.
These ordered additions of bibliographic information arethen added directly to the original vector.
Each document is thus represented by a tripartite vector, in symbols,
OAC, where 0 is the original vector of subject information.

A represents

xi-6

the authors of the document or of the works cited by the document, and C
represents the citations.
Note that little information is lost by grouping the authors of the
documents in the collection together with the authors of the works which the
documents cite, since the following weighting scheme is used:

relativeweights of 3, 2, and 1 are assigned respectively to authors of documents,
citations made by documents, and authors of those citations.

(When acitation is listed in the document by title only, the author of the citation is not added to the vector.) The reasoning behind these relative values
is as follows: listing the author of a cited work allows his authoring of
two related works to link two documents each of which cites only one ofthose works; however, two documents citing the same work are more likely
to be related; finally, the author of the document has presumably contributed more to the document than the cited references. These basic weights
are combined by simple addition when necessary. Thus, the author of twoworks both cited by a given document has a relative weight of 2 in the vectorof that document.

Note that citing 3 works by an author renders his weightcomparable to that of the author of the document.

Note also that if anauthor cites his own works, his weight is strongly increased [l].
This representation further implies that working with authors of
documents alone or with authors of citations alone is not possible,

A dis-

tinct range of concept numbers for each group would not solve the problem,since SMART as it currently operates has no table look-up to identify thesame author in his two functions.
Rather than "retrieve" a given subset of the collection in response

XI-7

to an incoming query, SMART ranks all the documents in order of increasingcorrelation with the query, thus presenting the user first with the documents which the system deems most appropriate to his stated needs.
The correlation of query vector with document vector may be done byvarious different formulae. Of the several correlation coefficients programmed into SMART, two in particular invite consideration for present
purposes: the "overlap" and the "cosine".

in3°k C kCosine: R* = R?
0

i

m

m

E •*• E

<c*>2

k=l

k=l

m
V

min (cj, c£)k=lOverlap: R* = R3±
m
min

m

(E <•!•*)
1

k=l

k=l

'

where m is the highest concept number in the system. Each of these coefficients, has a serious drawback. As indicated by the formulae, the cosine's
denominator is sensitive to document vector length (or, more precisely, to thenumber of non-zero weights regardless of match with request), and the overlap's numerator is insufficiently sensitive to varying weights above thexi-8

minimum.

Thus, in view of the reliance on a relative weighting scheme, des*-cribed above, the overlap is distinctly undesirable.

Furthermore, the earlier

relevance feedback experiments on SMART (using only subject material) do notuse overlap, and therefore, cosine provides a better basis for comparison.
The choice of the cosine for this study, however, is not made without reservation. The ADI collection is typical in having a varying number of nonzero weights from one document to another; farther, the revised document
vectors (OAC) are all longer than the original document vectors (0), but notby any fixed length or fixed proportion of the length of 0.

Since the pre-sent experiment works only with the feedback capabilities of bibliographic
information, the set of initial queries associated with the old ADI collection
remains unaltered and still contains only subject (o) information.

The initialquery can match only on 0 information; nevertheless, because of the changed
length, the cosine correlation value of a given query matched against the
OAC representation of a document will be lower than that of the same querymatched against the 0 representation of the same document.
Because this lowering of the cosine coefficient could affect the
document ranks, two parallel classes of initial and feedback retrieval
searches are made.

In one class (denoted by the symbol K) all query sets,initial and feedback, are matched with a constant set of document vectors
containing all types of information (OAC), regardless of the type(s) of
information in the query set.

In the other class (denoted by V), the setof document sub-vectors searched is allowed to vary appropriately, eachtype of query set being matched only with those sections of the document
vector containing the same type(s) of information as the query set.

xi-9

k*

Query Alteration in Feedback
The standard procedure in feedback is to expose the user to alimited number n of documents which the system suggests after an initial
search (in SMART, the n top ranking documents) and to alter the query infavor of those documents the user indicates are relevant. How limited anumber the user examines is an arbitrary decision.

Any relevant documents

above this arbitrary cutoff are considered "retrieved" and become the sourceof feedback information.

It follows that if no relevant documents are re-trieved on the initial search, no feedback procedures may be employed. To
minimize such cases, a cutoff of 15 is employed in the present experiments.
This cutoff increases the risk of retrieving all relevant documents, inwhich case the application of feedback techniques can produce no additional
relevant documents. However, the cutoff at 15 leaves 3 out of 5 of thequeries associated with the collection in the ideal range for feedback,retrieving some, but not all, of the relevant documents.
The results of this study are presented as average Quasi-Cleverdon
curves [k] of precision plotted at each 0.05 of recall. Recall is the ratioof relevant documents retrieved to the total number of relevant documentsin the collection and precision is the ratio of relevant documents retrievedto the total number of documents retrieved.

For purposes of comparison withother studies, all 35 queries are averaged in the results, although not allof these queries can be improved by relevance feedback.

5. Evaluation. Three iterations of standard relevance feedback are applied to both

XI-10the original (0) vectors, which have only subject information, and to theaugmented vectors (OAC) with added bibliographic information. Both sets ofvectors are searched starting with the same set of 35 queries, which contains only subject information.
As shown in figure 1, the initial search with this query set yields
different results on the two vector representations of the same document
collection, differing only in length (see the discussion of the cosine coefficient in Part 3). The application of the three feedback iterations
produces for each initial search comparable improvement in performance.
From this result alone, it is unclear to what degree the bibliographic data
furnishes material for feedback.

In particular, it is uncertain if the im-provement in the case of the augmented vectors can be attributed to thesubject (0) material alone, or in part to the author (A) and citation (C)material.
To answer this question it becomes necessary to abandon the standard relevance feedback technique of treating the entire vector uniformly,
and to feed back only certain types of information. Two new feedback strategies are employed:a.

Only author and citation information are added to the original
query (symbolized as 0

A 1 C 1 for the first iteration, 0° A 2 C 2

for the second).
b.

The bibliographic feedback is further examined in complete isolation from the initial query. That is, the initial query 0°
1 1
11
is discarded and A C , and even A and C separately, areused as feedback query sets.

XI-11

I.Oi

I

o

O

o

.0-0.
Ok 4

o—o—o.

V

0.8 h

^

*cO

x

o.

\ ,
\ .

*o

O ^ i

C

o
* 0.61
o

o^
1

^o.

\„

^ 0

°\,

^*0 V

Nk*%,

O

0 ^

B 0.4

M

^o

if)

0.2 r-

l:(0AC) u

Initial search with

2:(0)°

the original queries

3:(0AC)83^8= •8

Third iteration

4:(0)3

0.0

oC°o

0.2

x
x
0.4
0.6
Standard Recall

Standard Feedback Process(initial search and third iteration)on Original and Augmented (OAC) Vectors
Fig. 10.8

1.0

XI-12

As a control for the sensitivity of the cosine coefficient to vector
length (Section 3), parallel K and V searches are made using each of the
feedback strategies mentioned above. K indicates that the full length (OAC)
document vector is used consistently, while V indicates that for each
initial or feedback search, the query is correlated only with those document vector segments with which a match is possible. For example, theinitial query (0 ) in a V search is correlated with only the 0 segment of
the document vectors, and AC

queries (using the b strategy above) arecorrelated only with the bibliographic segment AC. Results of the searches
described above are shown in Figs. 2 to 5*
Comparison of Fig. 1, where the original (0) subject information isused for feedback, with Figs. 2 to 5; where no 0 data is fed back, showsthat the same type of improvement over the initial searches occurs witheach of the various query sets. The runs using citation information (C )alone show the least improvement, since over twenty per cent of the documents
include no citations. This firmly establishes that bibliographic data alone,even in a citation-poor collection, can be used as a successful instrumentof relevance feedback, either in complete isolation from subject classi ications, or as modification of the original query.
Direct comparison of the feedback results in Fig. 1 with those in
Figs. 2 to 5 is misleading, because Fig. 1 shows the result of three iterafi 2

tions, while Figs. 2 to 5, except for one second iteration 0

A

2

C

, show

only one iteration. Fig. 6, however, which presents selected comparisonsat the same iteration level, shows no differences greater than four percent

XI-13

I.Or

o—o—o.

"AV0.8

I 0.6|
o

o—o—o

o
c
o

0.4

LO

1 : K - 0 ° — ( 0 A C ) ° Initiol Search
2 : K-C 1
0.2

3 : K-A 1
4 : K-A'C 1

0.0

0.2

_L
0.6

±

0.4

Standard Recall

First iteration feedback of citation (C)and author (A) information alone,with full length document vectors (K)Fig. 2

0.8

1.0

T

1

1

1

1

r

T

r

'

'

Search2 : V-C 1
3 : V-A 1
4 : V-A'c 1

J

I
0.2

I

l__l

0.4

I

0.6

I

0.8

Standard Recall

First iteration feedback of citation (C) and author (A)information alone, with variable document vector segments (V)Fig. 3

1.0

XI

I.Oi

o__o-o^

0.8

.2
0.6
in
o
a>

o—o—o—,

k.

\

V

O

2 ^

\

•o

g 0.4

-°^o

c/>

N

I : K-0°.1^1
2 : K-A'C'— Curve 4 from figure 2

0.2

<N>.

3 : K-O^'C 1
4 : K-0°A 2 C 2

0.0

0.2

_L
±
0.4
0.6
Standard Recall

0.8First and second iteration feedback of author (A)and citation (c) information retaining theoriginal (subject) query, with full-length document
Fig. k

1.0

-16
I.Oi

o—o—o—o-^ 0

0.8 h

c

o—o.

o

NX

</> 0 . 6
*o
<u

X.

NS2s

v

o

^

V

I 0.41CO\

I :V-0°2 : V - 0 ° A ' C1

0.2

3:V-0°A2C2
4: V - A1 C' -curve from figure 3

0.0

0.2

-L
X
x
0.4
0.6
Standard Recall

0.8First and second iteration feedback of author (A) and citation (C)information retaining the original query, with variable
document vector segments (V)Fig. 51.0

XI-17

0.06

T

i

r

c
o
•—

l:(0)' - ( V - A 1 C1)'§ 0.04

2 : ( 0 ) 2 - (V-0°A2C2)

v.

1

1

r

I

L

CL
O
C

/° °

?

2 0.02r\Jo§ o.oo I—o—o—o^-o/

b

?

\

-0.02,
0.0

I

0.2

L.

'

'

0.4
0.6
Standard Recall'

0.8First and second iteration comparisons of feedbackof subject (0) and bibliographic (AC) information
Fig. 6

1.0

XI-18

between the results of feeding back only subject data (0) and those of
feeding back only bibliographic data (AC). This implies that the usefulnessof bibliographic data for feedback is of the same order as that of subject
descriptors.
The difference between the (OAC)J and (0) results in Pig. 1 istherefore not due to any lack of merit of the bibliographic information*
There is evidence, in fact, that this difference is primarily attributable
to the difference in initial search results, which is caused by the sensitivity of the cosine coefficient to vector length. Since the initial
searches are different, different documents are available for first iteration feedback; the discrepancy in performance is similarly propagated tothe third iteration.
Pig. 7 shows difference curves which compare results from Pigs. 2
and k (K) with those from Pigs. 3 and 5 (V). It is obvious that the V runs
produce consistently better results than the K runs regardless of the typeof feedback information used.

The forms of these difference curves are sosimilar that a single cause is indicated. The only constant difference
between all V and K runs is the initial search, which for all K runs is(OAC)° and for all V runs is (0)°.
Pig. 8 shows the difference curve for (0)^ minus (OAC) from Pig. 1.
This curve has the same shape and magnitude as the difference curves in Fig. 7.
This similarity indicates that the difference between OAC and 0 feedback isalso attributable to the difference in initial search results.

xi-19

O.IOi

0.08
c
o
0)

0.06

a.
k.

O
"O
C

2

0.04

</)
c
•—
a
c

£ 0.020)Liiv-o^'c1
2:V-A-K-O0A'C'

3: V-oVc^-K-O^C
4:V-A'c'-K-A'C1
\

0.00

-0.02

j_

0.2

j_

s

l^tr0-0- •8
UOMMQUJ.

0.4
0.6Standard Recall

0.8

1.0

Differences between corresponding V and K curves (V - K)
Fig. 7

0.10 |

1

1

r-^T

1

r-

1

1

1

1

0.08

o

\

0.06
/

o—o-o
\

0.04
(0)3-(0AC)3
t

0.02

M

0.00

0.0 1
%0

1

t

1

0.2

1

-J

1 — 1 —.

0.4
0.6
Standard Recall

L

1

1

0.8

Difference between feedback based on original vector
and on augmented vectors (OAC)
Fig. 81
1.0

XI-21An investigation of the effect of added vector length on results ina keyword system using the cosine coefficient is obviously necessary before
the addition of a second type of information to the classification can berecommended.

If the added bibliographic vectors "were to vary randomly inlength from document to document, the changes in the cosine correlation
would be random, and the resulting changes in ranking would give the same
average retrieval performance.

In fact, the shorter (0) document vectors

give significantly better initial search results on the initial query set
than do the full (OAC) vectors, so that it is clear that the added vector
length lowers the ranks of some documents that are relevant to some queries.
One would guess from this observation alone that there is a direct relationship between the length of the bibliographic sub-vector and the relevanceof the document to the query set.
Fig. 9 shows that in fact, in the ADI collection, documents relevant to four or more queries have on the average longer bibliographic
vectors and more non-zero bibliographic concepts than do documents relevantto three or fewer queries. The twelve highly relevant documents thus havelower relative cosine coefficients when the full length document vectors
are used.

Twenty-three out of the thirty-five queries name at least oneof these twelve documents as relevant. Therefore, roughly two thirds ofthe requests are affected by the length-relevance relationship shown in
Fig. 9»The up to ten per cent differences in feedback performance ob-served in Figs. 7 and 8 are no longer surprising.
Two important questions remain concerning the combining of different
types of information in a keyword retrieval, system.

First, to what extent

XI-22In documents relevant to
0 - 3 queries

k - 7 queries

Average number of non-zero
bibliographic concepts

8.0

11-3

Average length of bibliographic vector

71.0

80.7

Average number of non-zero
subject concepts

16.2

18.1

Number of documents involved

70

12

Relevance of a document to the query set as
related to bibliographic and subject information
Pig- 9

XI-23is the observed length-relevance relationship true of other types of information than bibliographic.

Fig. 9 shows a less striking tendency in

the ADI collection for the more relevant documents to have more non-zero
subject concepts. This fact may explain an otherwise strange result in
Fig. 5, that the V curve using the initial query (V - 0
than the curve that discards the initial query (V - A
K and V difference curves of A C

- 0

A

C

A

C ) is lowerC ).
are shown in Fig. 10.

The K curves are essentially the same, with no differences above two per
cent.

The V curves, however, differ by as much as seven per cent. The

length-relevance relationship of Fig. 9 probably causes the difference
between the V and K curves. Both K runs and the V - 0

AC

run usethe full length document vectors. However, the V - A

Crun is correlatedusing only the bibliographic document subvectors (A C ). Again the twelve
highly relevant documents, identified by longer subject subvectors, haverelatively higher cosine coefficients in the V - A

C run which does not

use the subject subvectors. This effect of adding subject to bibliographic
information is similar to but weaker than that observed in Figs. 7 and 8,
where bibliographic information is added to subject information.
The second remaining question concerns the likelihood of observing
relevance relationships found for the ADI query-document set in other retrieval environments as well.

This question deserves more investigation.

It seems reasonable, however, that in a typical document collection, works
covering a wider subject range, which by the assumptions in Part 2 should
also contain larger bibliographies, are likely to be relevant to a wider
range of requests than are works dealing with narrower topics.

In the ADI

XI-*

0.08

o
a

o
c
o

CD

o
c

0.4
0.6
Standard Recall

Effect of adding subject Information
to a bibliographic vector
Fig, 10

XI-25collection, only 12 such documents out of 82 caused significant changes inretrieval performance.

In most retrieval environments, a similar or greater

proportion of such general documents might be expected.

Thus, length-relevance

relationships are likely to be important in any attempt to use more than one
type of information for retrieval in a keyword system with cosine correlation.

6.

Conclusions and Recommendations
This study indicates that bibliographic information can be used ef-

fectively in a mechanical keyword retrieval system as a source of feedback
information.

In fact, bibliographic information used alone seems as val-

uable as subject information alone in the retrieval environment studied.
Since the bibliographic information is useful for relevance feedback, itshould also prove valuable for initial retrieval searches.
A direct relationship between document vector length and relevanceof the document to the query set is observed in the ADI collection of 82
documents and 35 queries. Documents relevant to more queries tend to have
longer bibliographic description vectors and, less strongly, longer subject
descriptor vectors. This length-relevance relationship lowers retrieval
performance when a query containing only one type of information is correlated(using the cosine coefficient) with document vectors containing both types.
This situation can probably be observed in most retrieval environments.
These conclusions support the following recommendations for keyword
retrieval systems using the cosine correlation coefficient:a.

Bibliographic information (the author of the document, citation
titles, and citation authors, each treated in this study as
single concepts) should be used as well as subject descriptors
to classify the documents;

XI-26b.

the user should be permitted to use bibliographic informationin his initial request;c. whenever a request (initial or feedback) contains only subject
or only bibliographic information, the request should be correlated only with the appropriate subvector of each document
descriptor vector.

Several topics remain for further investigation:a.

Joint subject and bibliographic feedback should be attemptedin the present retrieval environment using the retrieval orderof the initial subject search (V - 0 ) for first iteration
feedback.

That is, recommendation c. above should be imple-

mented for joint feedback. This procedure will provide a more
valid comparison of subject and joint feedback than does Pig. 1;b.

other document-query collections should be investigated to see
if length-relevance relationships are observed;c. the document collection used for this study (ADI) is citationpoor.

The value of bibliographic information should be in-vestigated in more normal collections;d.

only one relative weighting of the four types of keywords(subject, document author, citation, citation author) is usedin this study. Some other relative weighting schemes might
improve retrieval;e.

initial queries using bibliographic information should be
utilized.

XI-27

References[l]G. Salton, Associative Document Retrieval Techniques Using
Bibliographic Information, Journal of the Association for
Computing Machinery, Vol. 10, No. 4, October 1963, pp.
kk^hkj.

[2]

W. Riddle, T. Horwitz, and R. Dietz, Relevance Feedback in
an Information Retrieval System Report No. ISR-11 to the
National Science Foundation, Section VI, Department ofComputer Science, Cornell University, June 1966.

[3]

H. P. Luhn, editor, Automation and Scientific Communication,
Part 2. Short papers contributed to the 26th Annual Meetingof the American Documentation Institute, October, 1963.

[h]G. Salton, The Evaluation of Automatic Retrieval Procedures —Selected Test Results Using the SMART System, American Documentation, Vol. 16, No. 3, July 1965, pp. 209-222.