VIIIE V A L U A T I O N

VIII o 1

E X P E R I M E N T S

Review of experiments

As noted earlier,, the major evaluation tests of manual indexing
systems which have been carried out in the last decade provide thebackground for the evaluation of automatic indexing0 Thus theComparative Systems Laboratory (Saracevic 1968^ 1971), Cranfield(Cleverdon 1966), Inspec (Aitchison 1970) and ISILT (Keen 1972, 1973)
tests all found simple manual keyword indexing very competitive withother languages*(Collection details for all the tests referred to inthis section are summarised in Table 1.) In particular, reallyelaborately controlled or structured languages typically performed
less well, though there is some evidence that relatively simple
controlled languages have a slight performance edge. There is also
some evidence that manual thesauri can be improved by taking accountof vocabulary distribution properties: see, for example, Salton 1968a.
Unfortunately these evaluation tests were done on different
collections, and there was no comparison between collections byindividual projects. The experiments were also confined to smallcollections,, Tests of operational systems like that conducted by
Lancaster 1968a may well involve large collections, but do not provide
the comparative information we need here. Evaluation tests with
automatic indexing techniques tend to suffer from the same defects
as these manual ones. Many of the Smart experiments, for example,
have been carried out on the 35x82 ADI collection*, However tests
with several different collections have been conducted by the Smart
project, and by Sparck Jones, and the rather variable results obtained
serve as useful reminders that the relative performance of different
devices may vary over collections * Fortunately, the common use ofparticular collections is increasing, which makes it more rational toattempt to relate different project results to one another0 The 42x200
Cranfield collection has been widely used, and there are several others
which have been exploited by more than one project. It should, however,
be noted that different versions of these collections may be involved„
which inhibits detailed comparisons. Special difficulties also arise in
making comparisons when different matching functions are usedo Strict
Boolean searches, simple coordination level matching, and normalising
matching coefficients all produce rather different forms of output. Again,
averaging over sets of requests may be done differentlyG
In Section II, I categorised indexing techniques as simple or complex.
An example of simple indexing is the use of keywords taken from titles,
with word truncation allowed in searching. Complex indexing is illustrated
by the use of a controlled language like MeSH* The evaluation of automatic indexing must refer to the comparative performance of simple andcomplex manual indexing, and itself requires comparisons between simple
and complex automatic techniques, and between these and manual ones.

8.1

It is a pity that the evidence available for making all thesecomparisons is so fragmentary. The situation is complicated by
the fact that different indexing sources (say title or abstract)
may be used^ and by the fact that automatic indexing systems may
have manual componentss an example is Sparck Jones' use of manual
keyword lists as input to statistical classification0 The discussion
which follows is necessarily rather schematic: thus points like theexhaustivity implications of different sources are disregarded, andapproaches are defined as manual or automatic according to theirpredominant characteristics. After all, at this stage what we areinterested in is the overall performance of indexing methods, rather
than the reasons for it (this is particularly important in relation
to differences between projects associated with the use of different
matching techniques)«
The following attempts to summarise the useful results which havebeen obtained. I shall use the symbol
A= B

to mean that the performance of methods A and B is not noticeably
different,

A> Bto mean that the performance of method A is noticeably better thanthat of B,

A^>B

to mean that the performance of method A is materially better thanthat of B, andto mean that the performance of method A ranges from the same asthat of B, to noticeably better than that of B.
Manual indexing

The experiments referred to above suggest that where complex
indexing methods refer to fairly straightforward subject headings orthesaurus terms, and simple ones to extracted keywords, or phrases, withtruncations
complex^simple.

Automatic indexing
Relevant comparisons here are those concerned with the use ofstatistical association techniques versus keywords* (Actual experiments
have been based on both automatically and manually extracted keyword
lists0) Relevant tests are those by Lesk 1969, Minker 1972, 1973, Sparck
Jones 1971a, 1973c and Vaswani 1970. The result, as appeared in Section IV,
iscomplex ZZ

simple0

Allowance should be made for the fact that there is some variation

8.2

in the results with simple techniques. Subsidiary comparisons are
therefore of interest0 These concern a) titles versus abstracts andb) weightingca)
Comparisons between simple automatic indexing from titles
and abstracts have been made by Aitchison 1970, Barker 1972a, Cleverdon
1966 and Tell 1970, as well as the Smart project (Salton 1968a,c).
There is some variation in the results, probably due to different
matching techniques, Aitchison, Barker and Tell found that titles
gave better precision than abstracts, but worse recall, on Boolean
matching. With coordination levels Aitchison and Cleverdon foundtitles superior to abstracts (except for the recall ceiling)$ whilewith cosine correlation Salton found abstracts better than titles athigher recall, or overall. Assuming some interest in recall, the final
result must beabstract^title.

b)
Experiments with keyword weighting schemes of various types
by Salton 1972c, 1973b,c and Sparck Jones 1972, 1973e show that someforms of weighting may be useful, and specifically that collection
frequency based weighting can be helpful. These tests involved bothautomatically and manually obtained keywords, so the results areapplicable to simple manual indexing as well as simple automaticindexing. Allowance should therefore be made in the comparisons which
follow for the possibility that simple indexing performance can be
improved«

Automatic versus manual indexing
The number of experiments directly comparing automatic and manualindexing is small,
10

simple automatic v. simple manuala)titles vQ keywords*

These have been compared by Aitchison 1970, Barker 1972a and Hansen
1973 for Boolean search, Aitchison and Cleverdon 1966 for coordinationlevels, and Salton 1968a,c for correlation matching0 Aitchison foundtitles better on precision and worse on recall, Barker found titles gaveslightly better precision and worse recall, and Hansen found the two thesame. Aitchison found keywords superior with coordination levels while
Cleverdon found them much the same. Salton*s one test with the Cranfieldcollection also shows them much the same.
So we concludekeywordsb)^titleabstracts v« keywords.

These have also been compared by Aitchison 1970 for Boolean matching,
Aitchison, Cleverdon 1966 and Sparck Jones 1973d for coordination levels,

8.3

and Salton 1968a,c for correlation matching. Aitchison*s Boolean
matching showed the same performance. With coordination levels all
three projects found keywords superior, while Salton (see also Lesk1968) also shows a slightly better performance with keywords in his
comparison for the Cranfield collection. We therefore have
keywords ^>2oabstract.

simple automatic v. complex manual

The comparisons here mostly concern titles versus fairly straightforward subject headings or thesaurus terms. Comparisons have been madefor Boolean searching by Aitchison 1970, Miller 1971a, Olive 1973,
Saracevic 1968, 1971 and Tell 1971? for coordination levels by Aitchison
and Cleverdon 1966, and for correlation by Salton 1968a,c. The results
tend to show keywords performing better, though there is variation in
the test findingsa In coordination level matching Aitchison found titles
inferior, while Cleverdon found their performance much the same. In theSmart experiments they were inferior. Cleverdon and Salton also compared
abstracts and manual indexing, in both cases finding the former inferior.
However the recent Smart experiments with weighting (Salton 1972c, 1973b),
directly comparing improved simple automatic indexing for abstracts withsubject headings, showed the same performance. Nevertheless the overall
conclusion must becomplex manual ^3osimple automatic.

complex automatic v. simple manual

Most of the projects investigating statistical association techniques
compared performance with simple keywords, but in some cases the keywords
were extracted automatically. Comparisons with manual keywords were made forcoordination levels by Sparck Jones 1971a, 1973c and for cosine correlation
by Lesk 1969. The results were very variable, so it must be concluded
thatcomplex automatic ZZ simple manual.
40

complex automatic v. complex manual

Apparently the only comparison between these two forms of indexingis that made by the Smart project. Lesk 1969 illustrates the results
for three collections. The balance of the evidence is in favour ofthesauri as opposed to statistical associations, so we havecomplex manualj ^ , complex automatic.

These remarks are based on the explicit results given for particular
tests in the publications concerned. The fact that different tests andcollections tend to be involved under the various headings may mean thatthe conclusions I have drawn are not necessarily consistent. I shallattempt to pull the threads together in Section IX.

8,4

VIII . 2The SMART Project

The Smart project is the largest and longest term research projectin automatic information retrieval. There are few guestions connectedwith automatic indexing on which it has not done some work over the lastten years* Individual publications have been mentioned as appropriate?
but it is useful to look briefly at the Smart research as a whole, to see
what overall conclusions about automatic indexing can be drawn from it,
Details of the work are given in the projects Information Storage^
and Retrieval Reports (Salton 1966-)0> Early research is summarised in
Salton 1968a,cc Selected papers are reprinted in Salton 1971a<, Theinvestigations can be grouped under four heads. The relevant papers
are listed in Table 2D The test collections exploited are listed inTable 3olo

Experiments on basic automatic indexing,

20

Experiments on automatic index language generation.

3oExperiments on document clusteringc

4oExperiments on relevance feedback techniques„

5<.

Experiments concerned primarily with methodological questions•1<. Early experiments investigated the simple use of non-trivial keyword
stems from titles and abstracts, and compared these with manual indexing
on the one hand, and manual and automatic methods of organising theextracted vocabulary on the other. The results showed that simple keyword
methods, though performing less well than manual thesauri, did not
perform conspicuously less well* These tests were also concerned withthe effects of withm-document frequency weighting, and with different
matching coefficients0 They showed that alternatives here could affectperformance, and that on the whole the best results were obtained withweights, and a normalising matching coefficient like cosine correlation,,
2Q
Initial experiments with index language generation involved statistical association lists for expanded documents and requests* These were
not particularly successful, no real improvement over simple keywords
being obtainedc Recent experiments in controlling an indexing vocabulary
either by deleting non-discriminating or common words, or by differential
weighting using collection frequencies, show that noticeable performancegains can be obtained, the results being very competitive with independent
manual indexing using a controlled vocabulary*3*
Clustering experiments have been based on centroid techniquese The
ob3ect of clustering is seen as economic? the results show performancelossesc This line of work has not been very productive, perhaps because
the clustering methods were not really adequatee
4cTests with relevance feedback have examined the use of information
for relevant and non-relevant documents retrieved in an initial searchin modifying requests for new searches? relevant terms in the request
may be upgraded, and non-relevant ones downgraded. The tests generally
show that noticeable performance improvements can be obtained by8.5

techniques which do not involve the user in very much effort, and which
represent automatic revision of indexing. The correct characterisationof performance in this type of experiment is not readily determined,and care is needed in interpreting the ordinary recall/precision graphs
given
Tests have also been carried out showing that permanent changes
to document descriptions may be useful.
5o
The range of experiments done has raised a number of methodologicaland related questions, and some experiments have been carried out toexamine points involved in feedback evaluation (1970c), relevance judgement
variation (1968d), generality (1972b) and the use of mixed language databases (1970b). It cannot be claimed that the problems involved are allresolved, but more obvious criticisms may be overcome.
It should be emphasised that direct comparisons with manual indexing
have been few: for the 42x200 Cranfield collection, and, recently, forthe 29x450 Medlars collection. The latest results are promising forautomatic indexing, but it would be nice to have more comparative evidenceof this sort.
The main weakness of the Smart experiments has been the small scaleof the collections mainly used, apparent in Table 3. A further difficultyis that different tests may have been carried out with different collections,making detailed cross checking rather difficult. Further complication
appears where different versions (stem, thesaurus) of particular collections
are involved. It is to be hoped that some larger scale experiments will be
carried out with the more successful techniques, and that fairly rigorous
cross comparisons will be made. A point which should perhaps also be madeis that where statistical significance tests may justify the assertion thatmethod A is better than method B, in many cases the real difference inperformance is not large.

8.6

Table 1

Evaluation test collections

Aitchison 1970
Barker 1972a
Cleverdon 1966
Hansen 1973
Keen 1972,1973
Lancaster 1968aLesk 1968,1969

Miller 1971a
Minker 1972
1973

Olive 1973
Salton 1968a,c

1972c,1973b
1973c

Saracevic 1968,1971
Sparck Jones 1971a1972,1973c,e

1973dTell 1971
Vaswanil970

Table 297 x 542
nK
193
48
nK
221 1400
42200
20nK
63
800
302
nK
42200
3582
34780
25nK
34780
18273
34780
18273
3582
60 12765
42200
3582
34780
29450
24450
2442424425
24600
42200
42200
63797
97541
42200
47407
7nK
53nK
93 11571

InspecCAC
CBAC/POST
Cranfield
Cranfield

CAC
I SILT
Medlars
Cranfield

ADI
IRE
Medlars

IRE
Medlars

IRE
Medlars

ADI
NSA
Cranfield

ADI
IRE
Medlars
Medlars
Cranfield
Time
Cranfield
Cranfield
Keen (ISILT)
InspecCranfield
Keen
InspecPOST

Electrical engineering
Chemistry
Chemistry
Aeronautics
Aeronautics
Chemistry
Documentation
Medicine
Aeronautics
Documentation
Computing
Medicine
Computing
Medicine
Computing
Medicine
Documenta tion
Nuclear science
Aeronautics
Documentation
Computing
Medicine
MedicineAeronautics
World affairs
Tropical diseases
Aeronautics
Aeronautics
Documentation
Electrical engineering
Aeronautics
Documentation
Electrical engineering
Polymer science
mixed

Smart publications; Salton unless otherwise indicated1« basic

1968a,c, 1970a, Keen 1967, Lesk 1968

2. inde^ language

1968a,c, 1969a, 1972a,c, 1973b,c, Lesk 1968,19693, document clustering

1968b, 1972d, Rocchio 1966, Dattola 1969, Kerchner 19"M,
Murray 1972

4o relevance feedback

1968b, 1969b,c,d, 1972d, Brauen 1969, Ide 1969,
Kerchner 1971

5, methodology

1968d, 1970b,c, 1972b8.7(8.8 blank)

CMT1HO00

c^in
CMin
CM

CNinCO0)

T?

Oin

Si en
<t*CM

XXoX

inOD
CN
000Juft
GOX

co m«*• oX

COCNin
CN
CN

X

OXoO
Oinin

C

CN-d 4oU

T5CN

CNorH0)
;H

CNcCN
•P
W

rdU

CNCN<L)•P

co-P

CNoo(NCNoo

X X X

X X X

X X X

X X X

X

X

CO(U•HCN•9o
CO

CNinminmin
COCN
COX X X

X X X

Iusi

C O O O C O O O C ^ C ^ C h O O O O
VOlOkO^vOVDvDVOfHiHCSlCNCNCNCnrOrnr^COOC^CriC^r-ICN^ ^ ^ r H f H f H f H r H r H i - l r - l r - l r H r H r H i H r ^ r H r - l H r H i H i H r - l r H r H i H i HUCOc (0^0)

03

S38.9

C -10) 0
3 4J
fd -P <u( Q Q HsI