X-X.

An Analysis of the Documentation Requests

E. M. Keen

1.

Introduction
Conclusions from the experiments performed with SMART have so far

been made on the basis of the average results achieved for a set of searchrequests.

Studies presented in sections III, V, VI, VII, VIII of this report

have presented such results, and given in addition some elementary data about
the individual requests, and examples of detailed system performance.

Thepurpose of this study is to uncover still more of the variables and characteristics of the requests themselves in the context of the test environmentand of the capabilities of the SMART system, in order to reveal basic problems and suggest improvements to the system.

The 35 requests associated withthe 82 document collection in documentation are used because the author's
knowledge of the subject field aids the investigation.

2.

Request Preparation
In section I, a brief description of the ADI documentation collec-

tion, requests, and relevance decisions is given.

The collection consistsof 82 documents, both in abstract and short text form.

Requests were preparedby two non-users of the system, and the relevance decisions were also madeby these persons by examining every document for potential relevance.
The request preparers were graduate students at Harvard University,
one an engineer, and one an applied mathematician.

Since neither person

X-2
was familiar with the subject field of documentation, it was suggested thatsome familiarity with the subject would be gained by looking at the documents in the collection.

The task of request preparation should then follow.

Specifically, requests thought likely to be asked by workers in the fieldshould be devised, but these requests should not build in any particular
documents in the collection.

No suggestions were made regarding request

length.
An examination of all documents in the collection should follow,
and every document should be judged not relevant or relevant.

Neither per-son was familiar with the SMART system.
The full text of the 82 documents was supplied in the form of computer print-out, and none of the KWIC indexes or subject categories printed
in the published volume were supplied.

Both preparers worked independently,

one producing 17 requests and the other 18.

The task appears to have been

carried out as instructed, except that it is suspected that some requests
may have been prompted by particular documents in the collection.

Thisfactor is not necessarily a weakness, since full relevance decisions wereobtained, and thus the testis not based on the "source document" techniqueoften criticized.

Some comments on the task will be made when discussing"Uhclear Requests"(part 2D) and "Relevance Decisions" (part 3 ) .

3.

Characteristics of the Requests

A)

LengthExcluding non-subject words contained in the standard "common"
word list used by SMART, the stem dictionary gives an average of 8.0 stems
per request, and the thesaurus dictionary reduces this to 5.1 concept numbers

x-3
per request.

The eight most frequently used thesaurus concepts are given

in Figure 1, together with the numbers of requests and frequency of use
in the collection.

More than half the 35 requests use one or more of these

8 concepts.
Comparison of request length between the two different preparers
shows that person A constructed longer requests than person B; this matteris considered in part 5C.

B)

Important Request Words

Since the requests are fairly short and many use words of high frequency in the collection, it would be expected that some requests would
contain one or two quite important words that are vital to the request
demand.

For example, request A15 reads MHow much do information retrieval

and dissemination systems cost?", and request Bh "Automated information in V
the medical field".

The words "cost" and "medical" are very important in therequest statements, and render otherwise general requests much more specific.
Since SMART gives weight to each request word in part on the basis of frequency of occurrence in the request and collection, these important words
are liable to fail to receive the desired weight; this problem is taken up
again in part 5D.

C)

Multiple Need Requests.

Nearly all requests express the need as a single topic, but two requests demand documents on two topics.

Request Bl asks for information

on both coding and matching in machine systems, and request Al requires
information on "titles", meaning journal titles, organization names andpresumably their abbreviations; and also "titles" meaning the subject state-

NUMBER OF REQUESTSTOTAL |*"A" |*"B"

FREQUENCY IN
COLLECTIONSTEM*) RANK*

1

2k

11

13

998

1st

Retrieval

5

Ik

12

2

255

lUth

Automat

19

Ik

7

7

86

86th

Comput

3

7

2

5

326

8th

Dissemination

108

6

j 3

3

k7

12kth

Article

21

6

5

1

86

86th

Index

k

5

1

k

k2k

3rd

Scienc

10

5

1

300

9th

TERM
(STEM)THESAURUS
CONCEPT NO.

Information

J* Requests prepared by Persons A and B
f Frequency is based on word stems, therefore "Information" is themost frequent stem (Rank 1st) and is used 998 times.

Eight Thesaurus Concepts Most Frequently Used in the
Requests Giving Frequency Data for Each Major Stem

Figure 1.

ment as applied to scientific papers.

There is nothing inherently wrong about

having multiple need requests, but for testing puproses such requests sometimes cause difficulties when binary relevance decisions are used.

For

example, of the three documents assessed as relevant to request Al, two clearlyanswer the first part of the request only, and one answers the second partof the request only; thus it is never ppssible, even for a perfect system,to establish a complete match between the request and the relevant documents.
It is believed that where multiple needs are expressed separate search requests
will give superior results.

D)

Unclear Requests

Two requests in particular are unclear.

In request Al does thephrase "approximate titles" mean abbreviated titles?

Does request A8 ask

for documents in information retrieval as practiced in countries other thanthose speaking English, or, is it asking about information retrieval (practiced anywhere) of documents written in languages other than English?

Thefull request statements may be examined in Appendix B.
Several other requests may be charged with perpetuating the unclear
terminology that abounds in the field of documentation.

Request B2, what doesan "automated" information system include and exclude?

Request B3 requires

documents either describing the shortages that exist of information personnel,
or some solutions to the problem such as the need to provide suitable training.
Request Bll uses the words "index system", and since only one document on
the cataloging

of books is judged relevant, "index system" has been taken

to b e synonymous with book cataloging only.
Such requests would, in an operating situation,

be clarified byinteraction with the questioner; this advantage is, however, denied to theX-6
SMART tests.

It is surprising to note that the five requests quoted do

perform quite well in the retrieval runs made, and of the total of 13
relevant documents involved only 3 receive consistently poof rank positions(below 15) on all search options.

E)

Difficult Requests

Of considerable interest in the analysis of a system such as SMART
is the identification of requests that may be quite reasonable in themselves,
but that nevertheless create problems due to some system weakness.

Six

examples are given.
Request A2 contains the following negative statement:
to references or entire articles themselves ...".

"...As opposed

SMART cannot recognize

the significance of the negation, and a search will be made for the ideas
as stated.

Unless rules to recognize negative statements can be added to thesystem, users or request preparers must be advised to avoid negatives.
In request A8, "other languages" is a very important part of therequest, but the idea of "languages other than English" is another negative
statement which cannot be handled.

Even if "other" were replaced by "foreign",

correct matches with relevant documents would be difficult to achieve sincea thesaurus concept that links "foreign" with all possible named languages
or countries might work well for this particular request, but would at thesame time provide an unhelpful grouping for other requests asking for one
language in particular.
Request A10 contains the homonym "abstract", here used in the senseof "abstract mathematics" rather than the frequent collections use in thesense of a summary of a document.

The use of phase recognition would copewith this problem, except that the phrase list in use does not contain therequired phrase.

A synonym problem also exists, because none of the relevant

X-7
documents use the phrase "abstract mathematics".

Only in one case was "abstract"used in a sense other than "document summary", namely, in "abstract trees".
Interaction with the requestor seems necessary here, or a demand to list atleast some example of "abstract mathematics".
Request All reveals the problem of ordinarily common words being
used in a technical sense.

Words such as "evaluation" and "need" axe re-legated to the common word list when the thesaurus is used, thus leaving arequest specification only in terms of high-frequency words in the collection*
The stem dictionary uses both words and gives a better performance result;however, since such words frequently occur in non-technical senses, two of thefour relevant documents receive poor (lower than 15) rank positions.

There

seems to be no way of coping with such problems except to get requestors to
supply alternative and less ambiguous words where possible.
Another example of this kind is in request A13, in which "criteria","objective" and "evaluation" appear.

In this case inclusion of these request

words in the stem dictionary results in good rank positions for 5 of the 6
relevant documents.

Where several such ambiguous words occur, the co-occurrenceof all of them in a document in the incorrect sense is less likely; an improved
type of phrase dictionary may overcome the problem.
A problem of synonym recognition is raised by request BI3.

The

phrase "physical sciences" is really ambiguous and not very well chosen, since
examination of the relevant documents reveals that it covers notions suchas "materials", "chemistry", "engineering", "technology", "missiles andspace technology" and "environmental engineering".

The use of such wide

ranging relations in a thesaurus concept would be reflected by a concept number
with very many corresponding words; this would not serve all types of requests
equally well, and would in any case require some recognition of phrases rather

X-8
than single words.
These examples of difficult requests point out two areas in which
future work is required.

The first is the problem of ambiguity caused by-natural language, which may partially be handled by sophisticated recognition
procedures (to include negative statements, for example), but may in othercases only be handled by introducing constraints to the free statement of
the request.

The second problem is that of making synonym connections incases where a generic term is used.

A possible solution is the provisionof more than one synonym dictionary, which would include one containingsome quite large groupings of many words into few concepts to handle thedifficult cases.

k.

Relevance Decisions
Since both request preparers were alone responsible for the requests

and relevance decisions produced, there arose no possibility for disagreement
during the setting up of the test.

Evidence suggests that a consistent andconscientious job was done, although it is very easy to argue that the judges
were not competent, or that real information needs did not arise, and soon.

Measurement of the accuracy with which this artificial procedure cansimiulate real user requests, needs, and relevance decision awaits a carefully controlled comparative test.

To submit the actual judgments made toa panel of judges for their opinion would undoubtedly reveal disagreementswith the request preparers, and probably among the judges themselves; sucha procedure would then serve no real purpose at the present time.

A cursory

look at the decisions has been taken, and some discrepancies are noted inorder to illustrate the probable types of deficiency that may exist.

x-9
The discussion of unclear requests in part U D is closely linked to
the relevance decisions, since as J. O'Connor shows [l], relevance disagreements are often due to unclear request forms; furthermore, since many requests
that are thought to be clear are not so in fact, one is led to different
request interpretations and hence to different relevance decisions.

Probably

more examples than the five given in part U D exist, but by the stringent
criteria for clarity suggested by O'Connor, many real user requests would
be regarded as unclear also.
Several of the requests deal with quite similar topics, and sometimes
do not have as many relevant documents in common as the requests suggest.
Examples are requests A15 and B8, Bl and Bl6, B9 and Bll, and A5, B3 and B6.
A clear error of judgment is seen for document 7, where the photo cam position
method that is described for producing NASA's "Scientific and TechnicalAerospace Reports" is thought relevant to request A7, which demands documentson systems for producing original papers by computer.

The request preparer

probably did not realize that NASA STAR is not a series of original reports.
No specific examples have been found of documents that should have beenrecognized as relevant, except in those cases where two or more requests
seem very similar, as noted.

5.

Request PerformanceA)

General Performance Analysis Methods.

It was intended to divide the individual requests into three groups,
namely:a)

Requests which perform badly on all processing options;b)

Requests which perform well on some options and badly on others;c)Requests which perform well on all options.

X-10
Definitions of good and bad performance are arbitrary, but it is thoughtthat good performance requires the rank position of a relevant document to
be at least 15, and anything positioned lower them this is a poor result.
Any requests which fall into groups a) and c) were thought to be particularly
useful for analysis; in practice, however, all 38 requests fall into group b ) .
Requests B6 and Bit perform well on nearly all options, but occasionally oneof the relevant documents falls below rank position 10.

There occurs a sur-prisingly large amount of change in the ranks of the relevant when options
axe tested; Figure 2 gives an example for one request and two relevant documents.

In this request, all the options that are found on average to be thepoorest, such as titles only, the use of cosine logical, and the "Hastie"
Thesaurus give the best results.
Since the division into groups by performance achieved does notassist in the analysis, another method of analysis is suggested:

this isto look for strong correlation between measurable request characteristics andthe use of particular performance options.

A summary of possible request

characteristics is given in Figure 3, some of which have been described previously; these can now be used to look for direct correlation between characteristics and performance, as attempted in sections 5B, 5C, and 5D.

B)

Variation in Generality, Length and Concept Frequency

Request generality refers to the number of documents in the collection that are relevant; using this principle, the request set may be divided
into specific and general requests.

With the 35 requests divided into sets of

17 and 18, request generality data is given in Figure k together with evaluation
results of normalized recall and precision, comparing the stem and thesaurus
dictionaries.

As has been observed previously [2], the specific requests give^s.
DICTIONARY
DOCT>v.
LENGTH^.
CORRELATION'^.
FUNCTION
^s.

TEXT
m™-m

TEXT

THESAURUS -2 ( HASTIE)

DOCT.50

DOCT.82

DOCT.50

DOCT.82

DOCT.50

60

1+3

59

36

ko

65

Cosine. .,Logical

9

kk

P

2U;

©

"

Cosine,Numeric

52

29

61

8

kl

38

kS

22

T

mnvm

THESAURUS-1

DOCT.82

Overlap. I
T
Logicaltmnwwr%

STEM

TEXT

AT

63

TITLE

i°Sin*
Numeric

67

o

D
O

Relevant Document having best rank
Relevant Document having second best rank
Request having best performance

r

1

i
i

i
i

Request having second best performance

Rank Positions of the Two Documents Relevant to Request A9,Comparing 3 Dictionaries, Two Document Lengths, and Three Matching Functions.

Figure 2.

X-12

REQUEST CHARACTERISTICS

1.

Number of documents judged relevant in testcollection, or request generality.

2.

Length of request, using all words, subject
words only, or dictionary concept numbers.

3.

OBJECTIVE SUBJECTIVEyyyClarity of request demand and quality of
relevance decisions.

k.

Request processing in the system used, suchas SMART.

Factors such as quality of dic-tionaries used and frequency of use in thecollection of the request words.

5.

yTest environment variables, such as the useof several personnel to prepare requests.

yySunmary of request characteristics showing which are objective
and which subjective from an evaluation viewpoint.

Figure 3.

X-13

ALL,
35 REQUESTS
1-33 RELEVANT,

MEAN k.9

SPECIFIC,
17 REQUESTS
1-3 RELEVANT
MEAN 2.1

GENERAL,
18 REQUESTS
k-33 RELEVANT,
MEAN 7.fc

Normed. Recall
Normed. Precision

.7601
.5326

.8018
• 5358

.7209
.5295

Thesaurus 1 Normed. Recall
Normed. Precision

.8016
.6069

.8U82
.6126

.7576
.6016

DICTIONARY

Stem

EVALUATIONMEASURE

Performance Results Comparing All the Requests, the Specific Requests
only and the General Requests only, with Two Dictionaries.

Figure k.

X-lU
a better performance than the general ones, although normalized precisionshows only a small difference.

There is no correlation at all between request

generality and dictionary in these results•
Request length results are given in Figure 5> with the requests
again divided into two sets.

The long requests perform better than the

short ones, but do so by a much greater amount with the stem dictionary than
the thesaurus dictionary, so that for the long requests normalized recallshows stem to be slightly superior to the thesaurus.

This correlation suggeststhat the generally inferior stem dictionary may be adequate for long requests.
Requests may also be characterized by the frequency of use in thecollection of the request concepts.

Two methods of obtaining averages for the

35 requests are given in Figure 6, each method supplying an arithmetic mean
and a median value.

The average frequency per average request concept has

been found to be the more satisfactory of the two, and requests are again
divided into two sets by this principle in Figure J.

Requests having low

frequencies per average concept axe seen to perform best, with no realdifferences between stem and thesaurus dictionaries.
It is to be expected that these three characteristics of generality,
length and concept frequency are strongly inter-connected, since specific requests are probably often long ones, and also probably have low average concept frequencies.

A visual representation of the correspondence between thethree characteristics is given in Figure 8; in Figure 9 it is shown that
19 of the 35 requests fall exactly into the two expected combinations of threecharacteristics each.

These characteristics seem to be the only available

objective means of stating whether requests are broad or narrow in a subject
field sense; although perfect correspondence is not obtained, there is

ALL,
35 REQUESTS
1+-18 CONCEPTS,
MEAN 8.0

LONG,
17 REQUESTS
8-18 CONCEPTS,
MEAN 10.8

Normed. Recall
Normed. Precision

.7601
.5326

.8363
.6191

.6883
A509

Thesaurus 1 Normed. Recall
Normed. Precision

.8016
.6069

.8352
.6353

.7699
.5801

DICTIONARY

Stem

EVALUATIONMEASURESHORT,
18 REQUESTS

h-J CONCEPTS,
MEAN 5.^

Performance Results Comparing All the Requests, the Long Requests
Only and the Short Requests Only, with Two Dictionaries.

Figure 5«

x-i6

•

Average Total Frequency(i.e. Sum of Concept
Frequencies)
Per Request

Average Concept Frequency

MEAN

MEDIAN

1,5*2

1,605

220

220

-i

Overall Concepts Per Request
1

Data Showing Two Methods of Computing the Average Frequency
of Use in the Collection of Request Concepts, using the Stem Dictionary.

Figure 6.

EVALUATIONMEASUREALL,
35 REQUESTS
FREQ. 26-483
MEAN 220

Normed. Recall
Normed. Precision

.7601
.5326

.8154
.5671

.7080
.5000

Thesaurus 1 Normed. Recall
Normed. Precision

.8016
.6069

.8527
.6717

•7533
.5457

DICTIONARY

Stem

LOW FREQUENCY, HIGH FREQUENCYJ
17 REQUESTS
18 REQUESTS
FREQ. 26-218
FREQ. 223-483
MEAN 130
MEAN 305

Performance Results Comparing All the Requests, the Requests Having
Low Average Frequency Concepts and the Requests Having HighAverage Frequency Concepts, with Two Dictionaries
Figure 7.

x-l8

GeneralityNumber of
Relevant
Documents

LengthNumber ofStemConcepts

J-JL

RequestNumbersI 1 I I 1 1 I 1 I 1 JL-L

i

i i

i

i

i

i i

AAAABAAABAAAAA8ABAABSABBBABBBBABBBB
7 1 2 5 14 15 1718 6 4 8 9 12 16 5 1315 6 II I 17 10 8 106 3 3 7 9 II 14 2 4 12 13

Data for the 35 Individual Requests Showing Visually theCorrespondence Between Length, Generality and Average Concept Frequency.

Fig. 8.

Data for the 35 Individual Requests Showing the Numbersof Requests that Fall into the Eight Possible Categories ofLength, Generality and Average Concept Frequency

Figure 9«•X-20
possibly a strong correlation between these characteristics and real breadthof requests.

The superiority of the specific, long and low frequency requests

is again seen in performance figures for the 19 requests that exactly fall
into the expected combinations as seen in Figure 10.

The thesaurus dictionary

is seen, as expected, to give the most improvement to the general short andhigh frequency requests.

Further analysis of this type awaits suitablecomputer programs, since the hand analysis methods used are too time consuming.

C)

Comparison of Requests of the Two Preparers

Since two persons were responsible for request preparation, anyvariation in the measurable characteristics of generality length and frequency already noted may be correlated with the different preparers.

Figures11, 12 and 13 show that a quite strong correlation does exist, since therequests from preparer "A" are on average more specific, longer and hence
have lowermean frequencies than requests from preparer "B"' (Figure 11).
Figure 12 repeats the data of Figure 9> adding the request preparer distinction, and Figure 13 shows that if the eight sets of results in Figure 12axe divided into two sets of four each by the diagonal line in Figure 12,
correspondence is quite marked and is probably statistically significant.
The previously examined subject request characteristics such as
studies of unclear requests, requests having a multiple need, and requests
containing identifiable important words (see part 5D) are almost equallydivided among requests of the two preparers; thus, although the requests
prepared by person "A" are expected to give the better performance, it is
not correct to assume that "A" did a better quality job than "B".

The

six requests judged difficult for the system (Part 3E) comprise five MA"
requests and one "B", but as has been noted only three of these requests

•
X-21

( $ f ^ ; '."•-EVALUATIONMEASURE

SPECIFIC,
LONG ANDLOW FREQUENCY,
9 REQUESTS

GENERAL,SHORT ANDHIGH FREQUENCY,
10 REQUESTS

Normed. Recall
Normed. Precision

.8309
.6085

.5812
.1+072

Thesaurus 1 Normed. Recall
Normed. Precision

.8617
.6702

.72U7
.5870

DICTIONARY

Stem

Performance Results Comparing Two Subsets of the Requests Having the
Expected Correlations Between Length, Generality and Average Concept Frequency

Figure 10.

X-22

GENERALITY, LENGTH
AND FREQUENCY

ALL 35REQUESTS

Generality, Mean Relevant

PREPARER "A",18 REQUESTSPREPARER "B",
17 REQUESTS

3-3

6.5

Length, Mean Concepts

8.0

9.6

6.3

Frequency, Mean Frequency

220

157

287

Comparison of Length, Generality and Average Concept Frequency for all
Requests, Requests Made by Preparer "A" and Requests Made by Preparer "B".
Figure 11.

LO NG
LOW
HIGH
FREQ
FREQ

SHORT
LOW
| HIGH
FREQ
FREQ
'
'

'
LOW
FREQ
SPECIFIC
HIGH
FREQ

LOW
FREQ

A8
Bl

X

A2
BO

^

>^J

\

Al
Bl

AO
BU

AO
B2

A3
Bl

GENERAL
HIGH
FREQ

^

Al
Bl

A3
B7

Data for the 35 Individual Requests as Given in Figure 9 but Here
Divided into the Requests of the Two Preparers.

Figure 12.

FREI>ARER"A""B"§2/3 BROAD

Ik

3

17

£2/3 NARROW

k

14

18

18

17

35

Correlation Between Request Preparers and Request Type, Where "Broad"is the Sum of the h Categories to the Left of the Diagonal in
Figure 12, and "Narrow" is the Sum of the Other h.

Figure 13.

X-24
actually performed poorly (two "A" and one " B " ) •The performance results in Figure lk

compare the "A" and "B" requests

for two dictionary runs each made on two different document lengths, text
and abstracts.

The expected superiority of the "A" requests is seen in thestem dictionary results, but with the thesaurus, the "B" requests perform
slightly better.

Since the "B" requests are quite inferior to "A" with thestem dictionary, this does leave a greater opportunity for improvement withthe thesaurus; the initial inferiority on stem requires an explanation,
however, as well as the fact that the thesaurus does not much improve the"A" requests.

It is difficult to isolate any fundamental reasons for thisresult, because individual problems with both the stem and thesaurus dictionaries seem primarily to be the cause, as the following example shows.
Request BIO, with a normalized recall of 0.8205 with thesaurus andO.3718 with stem, has four documents assessed as relevant, and the thesaurus
produces improvements in rank positions of 22, 26, 32, and 60 compared withstem.

Reasons for the superiority of thesaurus in this case are:a)

The thesaurus provides additional matching concepts between

the request and all four relevant documents, including the important concept "computer".

The stem dictionary fails to match thisconcept, because the suffix routine used does not conflate all wordforms, and. "computation" is separated from "compute" and "computer".
b)

The thesaurus does not contain "system" but regards it as a common

word to be ignored, and although the stem dictionary uses it andestablishes matches with all four relevant documents, this highfrequency word also establishes matches with many non-relevant
documents.
c)

The very important request concept "chemistry" is grouped withsynonyms in the thesaurus which successfully increase the weightEVALUATIONMEASURE"A" REQUESTS"B" REQUESTS

Text, Stem

Normed. Recall
Normed. Precision

.8028
.5917

• 7516.5209

Text, Thesauras-1

Normed. Recall
Normed. Precision

.8124

,60hl

.8294
.6519

Normed. Recall
Normed. Precision.7830• 5^03.52kk

Normed. Recall
Normed. Precision

.8000

.8033

.5826

.6327

INPUT AND DICTIONARY

Abstract, Stem

Abstract, Thesaurus-1

Normed.
RecallNormed.
Precision

.8250
.8000
.7750
.7500
•7250
^6500
.6250
.6000
< .5750
.5500
.5250

.7358

Text
Abstract

STEM
3TE

THESAURUS

Comparison of "A" and "B" Requests for Two Dictionaries
and Two Document Lengths.

Figure l^f.

X-26
given to this concept compared with stem, producing increases of 1
to 18 1/2, 2 to 11 1/2, 3 to 25 and k to 11 1/2.

These three reasons for superiority of the thesaurus process arethought to be typical for other requests also.

There is also a strong corre-

lation of reasons a) and b) between the "A" and "B" requests, since as Figure
15 shows the concepts "computer" and "system" appear a total 16 times in the"B" requests, and only 6 times in the "A" requests, thus giving the "B"requests greater opportunity to benefit from the superior handling of theseconcepts in the thesaurus.
This treatment of the different sets of requests only scratches thesurface of the problem and points mainly to some of the factors known to
be involved.

D)

The Recognition of Important Request Words

The presence of quite specific and important single request words
and the problem of giving them a weight in proportion to their importance
was noted in part 3B.

In order to discover whether increases in the weightof such important words does improve retrieval performance, the 35 requests
were examined (without knowledge of search results, relevant documents orconcept frequency) to see whether such important concepts could easily be^identified.

Seventeen requests were found to possess such important concepts,

and each of the concepts was tripled in weight.
in Appendix B.

These decisions axe recordedThis simulates a quite feasible requestor rule which would

ask for any important concepts to be underlined in the request statement,
and which the system could recognize and correspondingly increase in weightby some factor.

Six requests in addition to the seventeen were also slightly

modified, request Al was divided into two as suggested by part 3C; in request

X-27• > * * *TERM(STEM)

REQUESTS OF
PREPARER
ft A It"B"

COMPUTER

STEM

11

Occurrence of Two Particular Words in the Requests,

Figure 15«

X-28
A2 the negative statement was removed; in requests A8 and Bll the difficulties caused by common words used in a technical sense prompted selectionof one or two synonyms for the given words; and in two requests keypunchingerrors which preserved hyphenated words were corrected.

These six modifi-

cations are all thought to represent reasonable demands that would be madeto users of an operational system.
These 2k requests are now processed together with the 12 requests
for which no modification was made; they are described as "Hand Modified";a total of 36 results because request Al is split into two.

Comparison of

retrieval performance of the modified with the original unmodified requests
is made for six retrieval runs in Figures 16 and YJ.

All precision recall

curves for the hand modified requests show them to be superior over the whole
performance range, with increases in precision at most recall values of morethan %,and in the middle recall ranges of nearly 10$.
Using the Abstract thesaurus result for analysis, the six requests

that were quite severely modified did not perform very well, only Bll was
notably improved, and some of the others received a worse performance.

Of

the seventeen requests that had triply weighted important words, ten were
improved, five has a worse performance, and two remained unaffected.

Fourof the ten that were improved are shown in Figure 18, and the two that were
worsened by the greatest amounts are given in Figure 19 > with rank positions
for all the relevant and normalized measures.
It is of interest to note that at present these hand modifications
do produce a superior result to the relevance feedback process described
elsewhere [3]. Figure 20 includes a comparison, using an evaluation technique that differs from the plots in Figures 16 and 17 in order to achievea fully user-oriented evaluation [U,5].

Further work on relevance feedback

• la

x-29

a
09
CD

H
w
•\
-p

CD
CO

X
0)

cd
^

H

cd

21
£

cd

CO

-p
w
0)

I

&
CD

d

^

(0

o

d

OJ

CVJ

o

no

x—.
H
cd

C!

GO
CD
co

3

0)

U
o

bO

co
0

M

Q

§

s

00

d

^
\

\

/

°s

/

#

S

P

00

d

CO

01
EH

>
CQ

o
Q
OJ
OO

•H

01

o
cc;

CD

-p

0)

CD

=

o

CD
CD

-p
CD

3

-p

i

I

CM

CD

pq

d

8

§ §

BQ

U
cd

-P
cd
o

«

&
9

I

T3

ft

00

d

J

/ ,
\

\

m

«! o
O a

CVJ

d
.2 «_l_
d

to

d

O
Q

a
CD

-P

CO

>

d

00

CD
O

CM

d

d

-p
X
01
EH

O

Q

CM

vo
00

a
CL>

d
•P
a

6

-H

02

•H

to

t>o

a;
>

CO

d
6

H
-P

BO
cd

o

CVJ

j>

2 +±

1

* a:
d

/

o

&

O

•H

30

O

H

H

H

l

o

GO

6
^
(O

\

—
—

cd
co

O o g

\

0)
Q:

d

S.
Q.

1
00

d

CNJ

d

<0

.J

<t

L

d

d

d

X

w
-p
o
4
M
-P

3

CO

<

CO
CO

CD

I

o

CM

u^
nn

?

CD

SO
•H

o

M
Q

H

CO

£

o

1)

00

/

a;
>
<

b
to —

a
CO

\

T3

CD

|
d

fl

-p

O 7» +»

\

-p

a

T3
CD

>
•H
CD
O

0)

cr:
co

a
CO

CO

d

o
o

-p
o

Q

p
CO

CD

CMi

CSJ

d
^•^

J

I

J

L

CO

ao

d
o
*H
•P
O
CD

o
a
a
o
•H

13
|
CD

CD
CD

:*

-p
0)
pq
CO

§
CO
•H
JH

cd

§
o
CD

o

§

o
CM

U

REQUESTRELEVANTDOCUMENTNUMBERRANKS OF HELEVANT AND
NORMALIZED MEASURES"ORIGINAL""MODIFIED"

33

1

82

50

50

NR .5000

27
NR .8U38

NP .1718

NP .6780

5

1
2

65

17
k2

k2

27

NR .7553

NR .8354

NP . 1+392

NP .7683

2

09

1
2

1

he.

19

3

69

27

1+

70

NR .8750

NR 1.0000

NP .7387

NP 1.0000

211

NR .7531

NP 1.0000

NP .3091

NP 1.0000

Request A9(2 Relevant)Request B280(3 Relevant)Request BIO(h Relevant)Request Bll(l Relevant)•36

Results of Four Requests in Which Manual Modification
by Important Term Weighting is better than the Original
Unmodified Search.
Figure 18.

REQUESTRELEVANTDOCUMENTNUMBERRANKS OF RELEVANT AND
NORMALIZED MEASURES"ORIGINAL""MODIFIED"

Request Ak(2 Relevant)

1

1

29

30

k3

63

NR .8250

NR .7^37

NP .6660

NP .6216

1

2

19

6

Ik

07

8

16

09

25

19

ko

NR .9038

NR .8686

NP .7279

NP .5916

Request A7(k Relevant)Results of Two Requests in Which Manual Modification
by Important Term Weighting is worse than the Original
Unmodified Search;

Figure 1Q.

X-33
5
Relevance Feedback

Initial Search

10 First Iteration, Five New Documents Examined
11

15

Second Iteration, Five New Documents Examined

Precision
bU
\

.55

\
\

.50
.45
\

.40

\
\

.35

\

V
\5

.30

v6\i

\V

.25

\
<

^,\io\
.20

*i*

.15
.10 —

•
^

.05 —
-I

0

.10

J
L_ l
.20 .30 .40 .50 .60 .70
Recall
I

I

I

I

I

.80 .90 1.00

ADI Abstracts Thesaurus, "Pseudo-Cranfield" Cut-off, Macro
Evaluation Over 35 Requests.

Comparison of Performance Effectiveness of Relevance Feedback
witfe Manual Modification by Important Term Weight Increases

Fig. 20

I

X-31*
is expected to result in improvements, but for high precision requirements
hand modified requests may always be superior (at the cost of increased user
effort).
The treatment of important request words might be made more drastic,
for example by the use of an "essential" word rule, which would only present
to the searcher documents that contain the noted important words.

This

strategy could in fact be achieved by assigning very high weights to theimportant concepts (wedghts of several hundred would be needed in text runs);
alternatively, modifications could be made to the search algorithm.

It

is almost certain that this procedure would imply that some relevant documents would never be found, although large increases in precision might
be possible.
For example, for seven requests containing important concepts chosen
at random, only 9$ of the relevant items would be lost, and although actual
precision results cannot be calculated, of the 86 non-relevant documents thatwere given rank positions above 16 in the output, 32 would be excluded bythis rule.

Other requests subsequently examined occasionally produce a muchgreater recall ceiling, and also a greater precision improvement, so that thisprocedure is worth further experimentation.

6.

Performance Effectiveness and Search Procedures
A comparison of the retrieval results obtained in the documentationcollection with the performance of the aerodynamics and computer sciencecollection shows the documentation results to be quite inferior.

Dataconcerning this fact were given in Section I, where it was seen that with astem dictionary in use, at 0.80 recall, the amount of non-relevant examinedX-35
as a proportion of the total non-relevant in the collection is 0.^5 with
documentation, 0.23 with aerodynamics and 0.17 with computer science.

At

the other end of the scale, examining the output until 0.03 of the nonrelevant is encountered gives recall values of .16 with documentation, .U3
with aerodynamics and .51 with computer science.

Factors in the text en-

vironments differ between the collections, and matters such as the qualityof terminology in the subject language, as well as the testing of techniques
used for collection gathering, request preparation and relevance decisions
all contribute to the differences observed in unknown proportions.
It seems likely that the imprecise terminology encountered in documenation which appears in both the documents and requests is a major causeof the poor performance, and in order to overcome these problems extra
human intellect may be needed in the system.

It may not be possible tobuild synonym dictionaries that will entirely provide for this, but good dictionaries together with a good choice of search strategy is likely to improveM
performance considerably.

Some proof of the value of carefully chosen searchwords is given in Figure 21, where a hand search of the KWIC type concordance to the abstracts is compared with a SMART abstracts Thesaurus result.
The hand searches chose up to five keywords for each request and was allowed
to use any words that might be considered useful as suggested by the request
statement.

A comparison with SMART in Figure 21 a) is made after fitting

the SMART results to the hand searches by making cut-offs in the SMART ranked
output in such a way that the number of documents retrieved for each request
axe identical to the hand searched results, Figure 21 b) shows two fuller
SMART curves obtained by making a series of cut-offs after one document,
two documents, and so on, up to the last document in the collection.

It is

not surprising that the hand searches work better, since the free choice of

X-36

o
ro
T

O

p
0)

T

T

o
a>
T

b

<5

o

•

H

1
3

i

o

o

a
0)

>
tf

o
co
O

§

o

P
H

a>
P
o

1o
O
H

CL

CO

CO

H

P
o

3J
CD
o

O

Q

p
CO

*ft

tya
ro
P
g
gP £•1

o
c+
co

CO

3
a

-^ CO

>•

g oo

£

CO

c+

H*

1

3

^ o

•*

CO

0)

p
uo

o

»

p
3

CO

o

o

0)

rt

^.

o

o
o
o

N

p
!fr

T

p

o

a>

CO

T

en

(0

0)
CO

*

w

1

\

\

\

(0

O
PO

0)

3
a

anc

2

£

3

o o

^ jq

P
c+

P

o

c+

-p-

ro

>
Cf
CO

CO

ft

H*

p

0

P
>i
o

&
CO
0)
p
•1
o
S3T

o

O

CO

CO
H-

co

ct-

of
•i
P

s

c+
*i
P

HO

d

3
OQ

CO

0
c+

Manu

o

•
\
CO

O
A
O
<y>

o
b

g
>-3
.3
d
P
a H
o
»
*
C
O
ro
CO 2
p
O ft) H- CO e
H - CO c+ 3
COo A <+ a> 0)
<
1
o
CO
O ft
p
g £ft ^ o5P HCO OQ3 o5p »-b
0) CO
•1
3
c+ X
p
C
zr
o
&
o
Hi
C
O CO tr
cf
O
g
1

o

»-b H

HO

^ P
t>
CO c f

H-

30
CD
O

Q

p
cfp
H

o

o

o

3

>
a*

<g o
P^
ft

H
p

8
cf

"SO •1

C+

i

c+
•1
p
o
c+
co

^3
CO

arD

gCL OO

p
rt-

CO

cf

& o
p

CO

3

&
c
n
3
CO
0)

H-

<

>
&
C
O

>•

tr* 3
ft Ctt

0)
M
CO

s
0

H-

•~b
1
CD

ac 5 oO P M>
2c+ H* H

c*

o

fD

CO

3 p
B
2
CO

P
H
H

CO

>
a^
CO

et

1

P

P^ o
CO

co

P
£
H
P

CO

3

-P- o

o
•i

X-37
synonyms made on an individual request basis should work better than the
obligating use of the pre-constructed set of synonyms contained in the thesaurus.

The hand searches do not use any feedback to obtain a fair comparison,since the search keywords were chosen before any reference was made to the
KWIC concordance.

The result of SMART using the hand modified "important"concept with increased weights is included in Figure 21 b ) ; the curve now
lies much closer to the hand result.

Naturally a hand system permitting

coordinate keyword searches would extend the hand curve to higher precision
values, and choices of more than five keywords per request would enable higher
recall ratios to be reached.
This result does not condemn the automatic indexing procedures,
because hand searches are less easy to conduct in the sort of situationin which SMART would operate, such as a large file of individually long document surrogates.

It is clear also that in an operational use of SMART;

search strategies employing several dictionaries in a variety of possible
ways could be used, and for users willing to employ some intellect tostrongly interact with the system, quite large performance gains may be expected.

Ways in which a system might operate axe:

the use of several dic-tionaries successively, until the required performance is reached; the useof several dictionaries with "merged" output

results [6]; use of a manual v

or automatic method of making an accurate pre-search best dictionary choice,
yet to be developed; use of dictionary display methods to allow users willingto strongly interact to delete or add synonyms; and the use of relevance
feedback methods to iterate searches and improve performance.
Of these suggestions the idea of making a pre-search dictionary
choice has been explored but with no success so far.

If, for example, longrequests work better with the stem dictionary, and short requests need theX-38
thesaurus, then a simple automatic choice could be made by the system.
Using the Abstract Stem versus Abstract Thesaurus result, Figure 22 showsthat more of the factors of generality, length or concept frequency arecorrelated with either dictionary, and even the requests made up by thetwo preparers do not markedly prefer particular dictionaries.

Other

criteria may be discovered to aid such a pre-aearch cAoice, and If a perfectchoice were achieved the result would be as given in Figure 23, where thecurve based on choice of the best dictionary is seen to be better than useof either dictionary exclusively.
The possibility of achieving a satisfactory automatic subject recognition is considered in an extensive analysis performed by J. O'Connor[7].

It seems certain that some loss of performance due to inability tocorrectly recognize and match with ideas asked for in requests will be
experienced unless very sophisticated procedures can be developed.

However,

failure in matching occurs also in manual systems due both to errors andinability to cope with the tasks of manual indexing and vocabulary control;
it is thus by no means certain that automatic systems will in practice proveinferior.

GENERALITY
SPECIFIC I GENERALFREQUENCY
PREPARERLENGTHLONG I SHORT LOW FREQ. I HIGH FREQ. "A" I "B"

Stem,
10 Requests
Thesaurus,
25 Requests

13

12

U

Ik

13

12

12

Attempt to Use Factors of Generality, Length and Frequency
to Make a Pre-search Dictionary Choice Between Stem and Thesaurus,
Abstract Results, Showing Numbers of the 35 Requests that
Fall into Each Category.

Figure 22.

13

Word Stem Dictionary-Precision

Thesaurus Dictionary-

0.6 r

Result of Stem for 10
Requests, and Thesaurus\

for 25 Requests, Simulating Perfect Pre-search
Choice.

J Recall
0.2

0,4

0.6

0.8

l.O

ADI Abstracts, "Pseudo-Cranfield" Cut-off, Macro Evaluationof 35 Requests.

Comparison of Perfect Pre-search Choice of Two
Dictionaries with each Dictionary Used Exclusively.

Fig. 23.

References[l]J. O'Connor, Relevance Disagreements and Ifaclear Request Forms, Draft

Report, 1966.
[2]

G. Salton, The Evaluation of Automatic Retrieval Procedures —Selected Test Results using the SMART System, American Documentation,
Vol. 16, July 1965.

[3]

J. J. Rocchio and G. Salton, Information Search Optimization and Iterative
Retrieval Techniques. In Proc. of the 1965 Fall Joint Computer Conference, Washington D.C., Spartan Books, 1965*[h]E. M. Keen, User Controlled Search Strategies. To be published in Proc.
of the Fourth Annual National Colloquium on Information Retrieval,[5]

H. A. Hall and N. H. Weiderman, The Evaluation Problem in RelevanceFeedback Systems, Report ISR-12 to the National Science Foundation,
Section XII, Computer Science Department, Cornell Uhiversity, June 1967*[6]

J. Rocchio, Combinations of Analysis Methods — The Merged Output
Results, Report ISR-9 to the National Science Foundation, Section
XIX, Harvard Computation Laboratory, August 1965-[7]

J. O'Connor, Automatic Subject Recognition in Scientific Papers:
An Empirical Study. Journal of the ACM, Vol. 12, October 1965.