ISR-9
August 1965

XX. ANALYSIS OF STUDENT REQUESTSGo Harris

Introduction
A test of the retrieve,

e of the SMART system wasconducted In conjunction with a graduate course in Applied Mathematics
given at Harvard during the F
to obtain

1964•

The object of this test wasof retrieval requests from a group of persons in no way

connected! with the SMART reseai

i thus without previous

knowledge of the document collection, theionaries or the system.

Previous tests and retrieval results have been described in this series,
Information Storage and .Retrieval, Report Noo I8R-8, Sees. IV and X, andwill be referred to as Staff results *The processing options used in this experiment were thefollowing3,(1) thesaurus derived concept classes*, document vectors
derived from document titles onlyg(2) thesaurus derived concept classes with request alteredby addition of terms nupn

in the concept hierarchy;(3) thesaurus derived concept classes, concept vectors
weighted logically (that ±s$> with weights of 0 or 1 ) ;(4)

concept classes derived from word stems on a one-to-one
basisi(5) thesaurus derived concept classes with phrase detection(simulated syntactic search)5XX-2(6) thesaurus derived concept classes with request
altered by addition of terms "down" in theconcept hierarchy^(7) thesaurus derived concept classes (no further
modification)j(8) thesaurus derived concept classes with phrase
detection (statistical)*All preceding methods employ a cosine correlation scheme*
All concept vectors are weighted numerically, with the exception of
method 3 which is weighted logically©

Statistical phrase detection isthe detection of co«occurrences of phrase components within a sentence*
Simulated, syntactic search eliminates phrases whose components bear no
syntactic relation* The analysis which follows deals with the phrasingof the queries, the comparison of the evaluation parameters and theeffectiveness of the dictionaries*

1*

Description of the Student Bequests
To keep the requests as unbiased as possible, a minimum of

information about the SMHT system and the document collection wasgiven to the students*

Each student was requested to submit two

queries on a subject likely to be found in the computer literature.
No instruction was given as to the language to be used in phrasing arequest^ furthermore, the document collection and SMART dictionaries
were not made accessible at that time. The total number of requests
submitted by the students was 42 > ranging from one word to 78 words inXX-3
length. Although most requests were written in complete sentences,seven consisted of descriptive terms onlyo
After the requests had been prepared* the requestors wereasked to study a listing of the document abstracts, and to supply alist of relevant documents to be used in obtaining recall and precision
measures as described in Report No. ISR-8, Sees. IV and X. The 11 requests which could be satisfied by more than six documents were considered "general," in the terminology of the previous tests, (seeISR-8, Sec. IV-7) and the remainder were termed "specific."

Of thespecific requests, 7 were found by their originators to have no relevant
documents in the test collection, leaving 24 specific requests which
could be used in the test. It should be noted that the lists ofrelevant documents were provided by the requestors without consulting
the computer retrieval lists. In several instances, students included
with their relevance judgements the qualifying statement that they had
found no truly relevant documents, but that the ones named were somewhat
related. This well-intentioned stretching of relevance would tend tolead to poor retrieval performance9 and should be kept in mindo

2.

Retrieval Results
The Student requests were submitted to SMART and the results

analyzed automatically. Average values of the normalized recall,precision and over-all measures for the general and specific requests
are compared in Fig. 1 with the results of the Staff requests previously
reported on. A number of results are at once apparent from Fig. Is

Normalized
Precision

Normalized
Recall

1.00

1.00

Staff
.90

.90

.80

.80
Student

Requests

Requests.70.70

Student

Requests

.60

.60

.50

7

8

Processing.50
MethodPROCESSING METHODS^

Normalized
Over-All
Staff

2.0h

Requests

1.0

1.

thesaurus-

2
3.
4.
5.
6.
7
8.

thesaurus- hierarchy up
thesaurus- logical vectors
word stems - f u l l

_i_

1

J_

_L

_i_

_L

Requests

text

thesaurus- syntactic phrases
thesaurus- hierarchy down
thesaurus- numeric vectors
thesaurus- statistical
GENERAL

Student

titles only

phrases

REQUESTS

SPECIFIC REQUESTS

_L

2
3
4
5
6
Processing Method

Comparison of Averager! Measures f o r S t a f f an^f S t u d e n t Requests
Figure 1

(a) the general trend of the results Is similar for both
Staff and Student requests;(b) all measures obtained for Student requests are lower
than for Staff requests;(c)specific requests are consistently better than generalrequests, as had previously been reported;(d) the use of hierarchy expansion in addition to theregular thesaurus Is more effective "down" than "up;"(e) the grouping of concepts by phrases, both syntactic
and statistical, was less effective with the Student
requests than with Staff requests•
The discrepancies between Staff and Student results have been
examined in detail and certain valuable conclusions were reached• The
generally lower normalized measures obtained in the later test arelargely attributable to the test conditions, since relevance judgements
and query statements were made without the originators being coachedinto the proper approach to SMART. However this provided in itself aworthwhile opportunity to judge the value of the system dictionaries*
Whereas Staff relevance judgements were reached as a consensus, andwere carefully pondered, the Student test relevance judgements received
no such careful analysis. This in itself is enough to affect adversely
the evaluation measures throughout the test.
The language of many student requests was also such that the
SMART dictionaries would unfortunately, but unavoidably, assign muchweight in the query concept vectors to rather insignificant ideas. As
an example, consider the following request;"Tell me about numericalXX-6

integration on computers oM

The three concepts of this query are 13CALC(numerical)j, 384TEG (integration) 110AUT (computer). Figure 2 gives theConcept
Identification

Terms

Frequency(No. of Documents)

13CALC

arithmetic,calculate,compute,evaluate,figure,interpolate,numerical,plot,reckon,recompute,value

U5384TEG

integral,integrate

13110AUT

automaton,calculator,computer,data-processor,processor

219

Excerpt from the Thesaurus and Corresponding Frequenciesof Occurrence from the Document Concordance
Figure 2
thesaurus lists of these concepts and their frequencies of occurrencein the document collection as determined by a concordance of concepts
numbers in the whole collection.

It is immediately apparent that inthis instance the broad classifications of the thesaurus are ahindrance to retrieval rather than a help, and this explains why thisparticular request was best answered by use of the null dictionary(see ISR-8, See* VII-3) which is made up of single word stems obtained
from the whole text of all documents• Figure 3 shows the normalized
over-all measure for this request for all processing methods.
Processing by word stems is method 4 in Fig. 1 and Figo 3. Again,from the data of Fig. 2, it is apparent that no processing option will
be able to improve much on the performance of the regular thesaurus

XX-7

Normalized
Ovei>all

1.5

1.0

S

J_
1

2

3

U

5

6

7

8

Processing Method
Normalized Over-all Measure for the Request "Tell me about
Numerical Integration on Computers11
Figure 3
run (method 7)ssince there are no phrases in the request, phrase

processing is of no help; since two of the concepts are broad,hierarchy expansion will only further confuse the retrieval of
relevant documents; only processing by logical vectors is an
improvement, since multiple occurrences of high-frequency terms
within a document will no longer carry extra weight*

In conclusion,

the performance of this request might be improved by restricting thethesaurus classes, by recognition of the phrase "numerical integration"(which was not in the phrase library at the time the request was submitted) and by avoiding the word "computer/'1 which is assigned to 219
documents as might be expected in a collection of computer abstracts.

XX-8

By contrast, let us examine a request which performed much moresatisfactorily*

Figure 4. gives the normalized over-all measure for all

Normalized
Over-all
2o0 _

1.0

I

3

k

$

6

Processing Method

Normalized Over-all Measure for a Student Request
on Pattern Recognition
Figure 4
processing methods for the following querys"What is there written

about pattern recognition in configurations made up of dotso

Thatis, tell me about recognizing patterns where the figure is recognizedXX~9by difference In the density of the dots."
request is shown in Fig. 5-

The concept vector of this

The vector represented in Fig. 5 includes

statistical phrases weighted 3•1 as specified in the SMART phrase option
for this test; 5 phrases were found•

2INPUT

4

13CALC U

58PLAN 28

107DGN 12

129NUM

130MEA

4

I84.DEC 6

2100UT

4

246HRD332SEE 216

340LET 12

393UIF 6U2FLD

6478CEN 12

NOTEs

6

4

The number to the right of each concept
identification is the weight assigned
to it •

Numeric Vector for a Student Request on Pattern Recognition(Phrases Included)
Figure 5
From Fig* 5> it is seen that the regular thesaurus run withnumeric vectors (method 7) is far superior to the run with word stems(method U) • This is an indication that the thesaurus classes of concepts in the query vector are properly constructed.

In addition, theweights attached to the concept numbers, as shown in Fig. 5, tell usthat the request is properly worded so that significant concepts carrya high weight, while ideas that are less useful, do not. When theweight difference is removed, as in the logical vectors option(method 3)> a sudden dip in the over-all measure takes place as
expected*,
It is also expected that a request that contains 5 statisticalphrases should show a marked improvement when phrases are processed.

XX-10A look at Figo U shows that this did not happen*

To explain it, wemust go to the actual rank order lists of documents for methods 7
and 8 which are reproduced in Fig. 6.
REGULAR THESAURUS (method 7)
Top 10Documents

351
350
353
348
82
163
335
92
48
113

Relevant
Rank Document

351
2

350

5

82

6

163

18

1

32

205

From these lists we are able
THESAURUS WITH PHRASES (method 8)
Top 10Documents

353
82
351
350
225
3U
163
162
1
68

Relevant
Rank Document
2

82

3

351

4

350

7

163

9

1

34

205

Rank Order Lists with and without Phrases
Figure 6
to see how the rank of a particular document was affected by phrase
processing.

Document 205 remains approximately at the same rank

because it contains no phrases; document 1 is brought up fromr = 18 to r = 9| documents 82, 351, 350 and 163 maintain their highranks except that first place has been lost, so that the precision
value is much reduced. Upon examination, the irrelevant documentsin the phrase run with ranks 1, 5> 6 and 8 prove to be also on thesubject of pattern recognition, but not on the recognition of wdot

XX-11
configurations*11

The thesaurus category for pattern recognition includes

the recognition of speech patterns, and the phrases correctly identified
and weighted in these four documents were misunderstood in answering thequery.

In conclusion, the retrieval success was not better because thethesaurus was not capable of distinguishing between specific types ofpattern recognition; the phrase dictionary did not include phrases
describing different kinds of patterns and equated all phrases that included the word "recognition."

3.

Combined Processing Methods
In Report No. ISR-8, Sec. IV-27, the use of combinations ofprocessing methods was described*

It was conjectured that performance

characteristics might be improved by processing a search request by
several methods and combining the respective outputs.
In the present test, twelve pairs and nine triple combinationsof methods were calculated by SMART. However, in order to obtainmeaningful analyses, it seemed necessary to re-examine each individual
request in relation to the validity of its relevance judgements and toits performance on SMART with single processing methods, since there
is a potential error in averaging results over requests that are notcomparable. As an empirical means of selecting comparable requests,
that is requests with reliable relevance judgements and reasonably
clear wording, those requests were selected for study which had
exhibited a relatively adequate performance with single methods.

XX-12This point of adequacy was arbitrarily set as the retrieval of morethan one half the relevant documents in the first 15 for the specificrequests, and more than one third the relevant documents in the first
15 for the general requests. The selection reduced the list of queries
to 11 specific and 5 general, or approximately half of the original
group.
The normalized over-all measure averaged over the selected requests is shown in Figs. 7 and 8 for combinations of the methods ofword stems, thesaurus and thesaurus with phrases. In all cases, theAveraged
Normalized Over-all

11 Specific Requests

2o0

1.5

1.0

_L

1

_L

2

(1+2)

1

3

(1+3)

_L

_L

(20)

Processing Method
1 m word stems

2 = regular thesaurus

3 = phrases

Normalized Over-all Measure for Various Combinations of Processing
Methods Averaged over 11 Specific Requests for whichMore than Half the Relevant Documents were Retrieved
Figure 7(1+2+3)

XX-13

Averaged
Normalized Over-all

5 General Requests

1.5 h

1

2

(1+2'

1 = word stems

]

3

(1+3)

• oe< ssxng Methoa
2 = re mi l a r t h e s a u r u s

_.!
2

i

3

(2+3)

3 ••= phrases

Normalized Qver-all Measure for various Combinations of Processing
Methods Averaged over 5 General Requests for which More than
1/3 the Relevant Documents were Retrieved
Figure 8paired methods give a better performance than the component methods
taken singly, but the triple method is not an improvement over thebest pair. When we remember that the evaluation measures for the
combined methods are calculated from a combined rank list obtainedby merging of the component rank lists, we see that it is importantwhich nonrelevant documents occur among the relevant documents on
the single rank lists. The results of the merging of rank lists is(1+2+3)

xx-Htherefore not predictable. The best results are found when the twoprocessing methods to be combined each retrieve the relevant documents with different ranks, so that the last ones on one list are thefirst on the other, but with the same intervening irrelevant documents*
Retrieval of relevant documents with different ranks is most likely tohappen when the processing methods are basically different.
As an example of an unsuccessful merging of methods, Fig. 9
compares combinations of the regular thesaurus, thesaurus with phrases

Averaged
Normalized Over-all

1.5 I-

5 General Requests

1.0

_L

1

(1+2)

1

3

(1+3)

2

3

(2+3)

P r o c e s s i n g Method
1 = regular thesaurus

2 = h i e r a r c h y down

3 = phrases

Normalized Over-all Measure for Various Combinations of Processing
Methods Averaged over 5 General Requests for which More than
1/3 the Relevant Documents were Retrieved
Figure 9(1+2+3)

XXand thesaurus with hierarchy processing.

In this case, expansion down

the hierarchy introduces so many extraneous documents that the rankorder lists are diluted and any combination that includes the "hierarchy
down" method is at once worse than the best single component.

4.

Revision of Requests and Dictionaries
It is apparent from the preceding analyses that revision isnecessary both in the dictionaries and in the form of some of the requests*

On the whole, the results of this test were similar to thoseobtained for the set of Staff requests, but the generally lower performance of Student requests suggests several possible improvements;(a) the broadest thesaurus classes need to be studied
and refined to prevent ambiguity;(b) the phrase dictionary needs to be greatly expanded
if full advantage is to be taken of the statisticalphrase option;(c) the hierarchy structure must be limited to groupingsof concepts which are unambiguous and should preferably
not contain concepts with a high frequency of occurrencein the document collection}(d) requests should be written in an unambiguous way andnot contain high frequency concepts unless they are
specific to the query](e)familiarity with the SMART dictionaries is desirable.

An extensive revision of the SMART dictionaries has been madealong the lines indicated above. All concept classes in Thesaurus 2

XX-16have been reviewed with the help of a document concordance and a frequency listing of word stems. The broader, high-frequency concepts
have been split up, and Thesaurus 3 now consists of 736 concept
classes, as against 511 concepts in Thesaurus 2. However, the total
number of word stems has not been changed.
The phrase dictionary has been increased from 98 to 373
phrases covering a wider variety of combinations of word stems.
In particular, numerous phrases have been included which specify
the usage of common terms such as "function," "time," "system,""control}11 (typical phrases are "manual control," "traffic control,""accuracy control," "control system," "feedback control," "flightcontrol")«
Because of the extensive changes in the thesaurus andphrase dictionary, the hierarchy has been completely re-written.
Figure 10 shows the format which, the hierarchy attempts to follow.
(le ss p r e c i s e ,[more frequent

precise,frequent[more p r e c i s e ,\_less frequent
Hierarchy Format
Figure 10

XX-17

In establishing a three-level hierarchy, the aim is to restrict the
distance between related nodes, and thereby increase the likelihoodthat hierarchy processing will produce matching concepts a Highfrequency concepts are, in general, untied to the hierarchy except in
those instances where a filial connection is unambiguous.

In addition,

an extensive system of cross-references links various sections of thehierarchy*

These cross-references can be called by the hierarchicalexpansion option REFER.

Hierarchy expansions "up" (STEM) and "down"(LEAF) within concept groupings are the ones which have been tested in
the current tests.
Number ofConcept Classes
300 ,

200

h

100 f-

1

I

I

i

I

I

I

0

20

U0

60

80

100

120

'

I
1U0

.

I

I

160

180

• I
200

Number of Documents
Frequency Distribution of Concept Classes in the Document Collection
Figure 11

XX-18

Concept Noo

Frequency(No. of Documents)

601

230

computer,DAC0WP (dataprocessor) , data-processor,
electronic-computer

496

135
108

system

615
608

Terms

operateprogram

603

104
101

32

98

dictate,impose,necessary,ne ce s sit,need,quali,require,requisite,restrict

614

88

apply,applic

605

87

calculate,compute

202

85

complete ,entire,full,general,
total,universal,whole

92
600

84
82

DIGCMP (digital computer)

215

80

acquire,detect,find,get,note,obtain,recover,refer

321

79
79
76

memory,store

74
71

problem

623
609
602
68

digitalmachinecircuit,network
designapparatus,device,equip,
hardware,instrument

604.

70

analog

119
618

70
69

artifi cial,automat,me chan

14.7

67
67

227

function
handle#process
charge,electric,electron

Most Frequent Concept Classes According to Thesaurus 3
Figure 12

XX-19INSTRUCTIONS TO REQUESTORS

1. Requests to SMART may consist of full grammaticalsentences or of series of terms•
2. Most functional words (such as prepositions, adverbs,
conjunctions, auxiliary verbs) and many common
descriptive terms are ignored by SMART and can safely
be included•

However, idioms which contain ambiguous

words may be misunderstood*

For example, "in order to,11"in addition," *a number of11 have the meanings of"order," "addition" and "number."3»

It is important to select words which are unambiguous
and which specifically define the desired subject* Itis especially desirable to use terms which are specificto the subject, rather than the broad terms which apply
to a wide range of computer technology.

4.. Additional, weight is given by SMART to repeated terms.
Therefore it is useful to write more than one sentence
and to repeat the significant phrases. Cautions

therepetition of broad terms, such as "computer, system,
operate, structure, design, calculate. o»" which arefrequent in the document collection, should be avoided•$• Affirmative sentences should be used since SMART
ignores negative words.
6.

Synonyms of the more esoteric technical terms could be
used, or the SMART Thesaurus should be consulted, if
one is available, to insure that all significant
meanings will be found•

Sample Instructions for Submitting Requests to SMART
Figure 13

XX-20In phrasing requests to SMART, it is desirable to avoid
high-frequency terms unless they are thought to be distinctly
useful in retrieval of related documents©

The document concordance

for Thesaurus 3 is shown in Fig* 11, and the concept classes which
occur in more than 65 documents are listed in Figo 12 o Sample
instructions for submitting requests to SMART are given in
Fig* 13.