14

An experiment: search strategy variationsin SDI profiles
Lynn Evans14.1 Introduction
This research project was one of a number concerned with mechanized
information retrieval carried out at INSPEC and supported by grants from,originally, the Department of Education and Science's Office for Scientificand Technical Information (OSTI) and, later, the British Library Researchand Development Department (BLRDD).
Thefirstproject, in the period 1967-69, had investigated the performance,
economics and acceptability to users of a computerized SDI service inelectronics1. As a result of this study, in 1970 INSPEC offered publicly an
SDI service in electronics.
Another research project2, overlapping the SDI investigation, hadestablished the comparative effectiveness of natural language as a medium
for mechanized searching so that, from 1971, all documents input to the
INSPEC database were assigned free index terms. These are significant
words and phrases from the title, abstract and text of the document, selectedby the information scientists as representing the meaningful concepts treatedin the document. In April 1971 the cost-recovery service in electronics was
replaced by a commercial SDI service covering the whole of the INSPEC
subject areas with the important addition that free indexing was now
available as a search medium.
In the original INSPEC SDI investigation an EJC-type thesaurus wasused for document indexing and profile generation. The complete thesaurus
was never produced in printed form and so was not available to users forcompiling their own profiles. The introduction of free indexing removed thisbarrier and prompted another study3 into the optimum degree of user
participation in SDI profile generation. This concluded that users preparedto familiarize themselves with the 'mechanics' of the system compiled theirown best profiles as measured by precision (although indirect evidence
indicated lower recall values than in the profiles compiled by INSPEC staff).
However subscribers were reluctant to become too involved in that 45 percent of them chose to delegate profile compilation to INSPEC staff and only
14 per cent opted to completely manage their own profiles. The main reason
offered in explanation was that users found the whole procedure of getting a285

286 An experiment: search strategy variations in SDI profiles
profile started too complicated and having to read a detailed User Manual,including grappling with the intricacies of nested boolean logic, was apositive discouragement.
At that time INSPEC profiles were almost exclusively of the boolean type.
The general assumption was that full advantage of the machine facility
should be taken and profiles compiled with complex logic to match the user's
statement of information requirements. However if the initial user-systeminteraction was to be eased then simpler search strategies was one area where
this might be achieved. There was some evidence of a non-quantitative
nature which suggested that, even in profiles incorporating sophisticatedlogic, many of the relevant matches obtained were derived from comparatively simple parts of the logic statement.
In addition to doubts about the need for sophisticated logic, work at
Cranfield in the field of precision engineering4 showed that the case for any
boolean logic was not proved and that a straightforward co-ordination of all
the profile terms might be equally or even more efficient.
Quite separately it was also felt that the issue of simpler search strategies
was to become important in the development of online retrospective searchsystems. For these to be operated directly by the people requiring theinformation rather than by intermediary information scientists and librarians
then the man-machine interaction needed to be much simpler than was
currently the case. In the event, some 5 years later, online searching of
bibliographic databases is still very much the prerogative of intermediaries
and is likely to remain so until the development of truly interactive retrieval
software.
This then was the rationale leading to the particular experiment describedand analysed in this paper, viz. search strategy variations in SDI profiles5.
The overall aim was to develop the most cost-effective search strategy bystudying the cost, retrieval performance, and ease of use of a number ofdifferent search strategy types. In the general framework of information
retrieval experiment this work was most like a laboratory test but perhaps
lacking the rigid control necessary for a 'true' laboratory experiment.
Although carried out in the environment of an operational system it was inno way an investigation of the operational system. It is probably mostaccurately categorized as a developmental project, i.e. one pursued with asmuch experimental rigour as possible but not at the expense of losing touchwith 'real-world' conditions.

14.2 Experiment
Very broadly a conventional evaluation in information retrieval requires: (1)a collection of documents with various attributes (titles, abstracts, assigned
index headings, etc.); (2) a set of queries whose subject area is covered by the
document collection; and (3) knowledge of which documents in the collection
are relevant/non-relevant to particular queries. In addition, in this experiment, the important considerations were the search strategies, profile
compilation procedures, and search software.

Experiment

287

Documents
The documents used were not special in any way and were obtained on a
weekly basis from the INSPEC current file. The most important consideration
was that the document collection should be typical of the INSPEC database.
Eight weeks' documents were taken in two groups of 4 weeks each separated
by a time gap. Four consecutive weeks was considered a reasonably adequate
span in that the cycle of most journals would thereby be covered. The reason
for having two separate 4-weekly groups was to allow for profile performance
analysis and modifications in between the first 4 and last 4 weekly SDI runs.
In all over the 8 runs more than 20 000 documents were matched against the
profile file, the individual weekly totals being 3095, 2640, 2582, 2194, 2542,
2781, 2123, and 2860.
As with many aspects of this experiment deciding the size of the document
collection was empirical. Later, when the pattern of retrieval performance
was found to be similar from week to week it was not considered worthwhile
completing the performance calculations for all 8 runs so that 20 000
documents would seem to have been more than an adequate number andsomething like 5000 would have been quite acceptable. Since then of course
the question of document collection size has been thoroughly considered aspart of the design of an 'ideal' information retrieval test collection 6-7 .
Queries
It was considered important that the profiles should be based on real rather
than artificial questions and that they should operate in a 'near-real'environment. This was because one of the main objectives was to detect
differences between the retrieval performances of the search strategies whichshowed through despite the hazards associated with real user assessments.
Experience in the INSPEC SDI investigation had highlighted the fact that
real users' relevance assessments were often not entirely based on subject
content but might be influenced by such factors as the language of theoriginal document, its source, the author's reputation, etc. It was also not
unknown for slight changes of interest not to be notified and only become
apparent on investigation of poor profile performance.
Despite these known irritations which only tend to cause confusion the
feeling was that they are an inevitable feature of operational systems andshould not be ignored by the use of artificial queries. On the other hand it was
not thought politic to use paying customers of INSPEC's commercial SDI
service because of the danger of their alienation when asked to assess
documents over and above those retrieved by their optimum profiles.
The compromise decided was to recruit a user group from UK university
research workers, the arrangement being that for their agreement to provide
subject interest statements and to assess the relevance of document outputs
they would gain, at no cost, experience of a mechanized current awarenessservice. In selecting the university research workers the latest issue of the
Department of Education and Science's annual publication Scientific Researchin British Universities and Colleges was used as a random source of names,
addresses and subject interests.
Of the 100 people originally approached (40 Physics, 40

288

An experiment: search strategy variations in SDI profiles

Electrical/Electronics, 20 Computers/Control) the 55 who actually suppliedstatements of their subject interests comprised
23 from Physics Departments,
27 from Electrical/Electronics Departments, and5 from Computer/Control Departments.
Given the inevitable delay between their original undertaking toparticipate in the experiment and their receipt of the first notifications forassessment, the response from users was quite gratifying. Over the 8 runs
relevance assessments were received from, on average, 82 per cent ofparticipants, the returns for the individual runs being 84, 89, 86, 87, 76, 78,
78 and 76 per cent.
As with the number of documents the decision to use about 50 users was'gut-feeling' rather than statistically reasoned. The question of what is anacceptable number of queries cannot of course be considered in isolation andis intrinsically bound up with the number of documents against which thequeries are matched. If Q represents the number of queries and D thenumber of documents it would be tempting to speculate that there is an
optimum value for the product QD with minimum acceptable values for Q
and D. However it is not only a matter of the numbers of queries anddocuments but also how many documents are relevant to particular queries.
The fewer relevant documents there are the less confidence there can be in
the particular recall or precision ratio so there is great doubt associated with,say, a recall figure of 1/1 which could so easily have been 0/1. These arematters which can only be completely controlled in an artificial situation andthe statistical bases of relevance assessment have been considered in detailmore recently8.
Search strategies
The strategies to be compared should ideally represent different degrees ofintellectual effort when compiling profiles and, also, should require different
degrees of sophistication of computer facility. The strategies selected forcomparison were all variations of two basic types, viz. those consisting of asingle list of terms, and those containing groups of terms, where the groups
represent subject concepts in the original query. The list of search strategy
types was:
(1) Co-ordinate matching of terms without weights (CT)The output is ranked in order of term co-ordination level, i.e. in order ofthe number of matching profile and document terms.
(2) Term-weight cumulation (TWC)
The profile terms are assigned weights in accordance with their relative
importance to the query. The weights of all matching terms are 'summed'
to produce a 'document score'. The output is ranked in order of documentscores.
(3) Co-ordinate matching of terms with weights (CTW)
The profile terms are weighted as in (2). The output is ranked, first inorder of term co-ordination level (i.e. number of matching terms), andthen by sum of all matching-term weights.

Experiment

289(4) Co-ordinate matching of groups of terms without weights (CG)The profile search terms are divided into groups representing the various
concepts in the query. The output is ranked, first in order of group coordination level (i.e. number of matching terms where each is from adifferent profile group), and then in order of total number of matchingterms.
(5) Group-weight cumulation (GWC)
The profile term-groups of (4) are weighted according to the relative
importance of the groups (concepts) to the query. The weights of allmatching groups are summed to produce a document score. The output
is ranked in order of document scores.
(6) Group- /term-weight cumulation (GTWC)
The term-groups of (4) are weighted according to their importance to thequery and the individual terms are weighted according to theirimportance within their group. The output is ranked, first by sum ofmatching-group weights, second by sum of highest-weighted matchingterms from each group, and third by sum of all matching-term weights.
(7) Co-ordinate matching of groups of terms with weights (CGW)The profile term-groups of (4) and the individual terms are weighted. The
output is ranked, first in order of group co-ordination level, second inorder of sum of matching-group weights, and third in order of sum ofmatching-term weights.
(8) Boolean logic (B)The profile term-groups of (4) are governed by boolean statements whichmust be satisfied before any output is obtained. The output is in document
number order, i.e. unranked.
(9) Boolean logic with weights (BW)The profile term-groups of (4) are governed by boolean statements
which must be satisfied before any output is obtained. After theboolean equations are satisfied the ranking of output may be based ongroup- and/or term-weight cumulation procedures. In our experiment
only term weights were used.
Basically procedures (1), (2) and (3) involve search profiles comprising asingle list of terms (weighted or unweighted) while procedures (4)-(9)inclusive involve profiles comprising groups of terms (in which groups and/or
terms may be weighted or unweighted).
In the weighted profile versions two types of weights were used. In
procedures (2), (3), (5) and (9) above, the weights were subjectively assignedby the compiler, while in procedures (6) and (7) 'powers of T weighting wasused9. In 'powers of 2' weighting the weights are assigned routinely once theorder of importance of individual terms and/or term groups in the search
profile has been intellectually decided. This ordering was again decided by
the compiler.
Profiles incorporating automatically-assigned weights were not includedin the study mainly because the necessary statistics (term frequencies, etc.)
were not immediately available. They were to become available subsequently
from another INSPEC research project.
In addition to the 9 search strategies listed above, as the project proceeded
it was decided to include a further two types:

290

An experiment: search strategy variations in SDI profiles(10) Co-ordinate matching of restricted list of terms with weights (CRTW)
The original list of search terms chosen to represent a particular user's
interests in strategies (l)-(9) above was not restricted in any way andcomprised, on average, more than 40 terms. It had been felt, mainly asa result of experience with INSPEC's commercial SDI service where
similar-sized profiles pertained, that there may be a significant numberof unnecessary search terms in profiles, viz. those search terms which'hit' too infrequently to be useful and those search terms which 'hit' too
often to be selective.
A further argument was that although the search profiles used in online
retrospective searching contain, in general, far fewer terms than thosein current-awareness batch SDI systems, their retrieval performancedoes not seem to be noticeably inferior.
For these reasons it was thought worthwhile to include for comparisona profile version in which the number of terms was restricted to 20
irrespective of the number in the original list used for search strategies(l)-(9) above. The 20 terms were selected subjectively in order ofimportance from those used in strategies (l)-(9).
(11) Controlled-language boolean strategy (CLB)
In the main experiment the medium used for matching profiles anddocuments was natural language, viz. the free-index terms assigned to
all items added to the INSPEC database. In general the free-index
terms are words or phrases occurring in the original document which
represent the meaningful concepts treated in the document. In addition
to the free indexing, the subject content of all items in the INSPEC
database is indicated by two other elements: (i) classification codes,
which govern the location of the item in the published abstracts journals,
and (ii) controlled subject headings, which appear in the six-monthly
indexes to the abstracts and are used mainly for manual retrospective
searching. The classification codes and controlled subject headings canof course also be used in machine searching and to this end boolean-type
profiles using only classification codes and/or subject headings wereprepared.
Originally the main purpose of these controlled-language boolean
strategies had been to act as 'back-up' profiles to retrieve relevant
documents which the other versions (based on the free-index terms)
might miss because of inadequate free indexing or profiling. Knowledgeof these additional relevant items would of course mean that the recall
performance figures would be that much nearer to being measures of thetrue rather than the relative recall. However the data available also
allowed a direct comparison of the retrieval performance of controlledlanguage against free-language boolean profiles. Brief details of thissecondary experiment are given in section 14.3 (p. 309).

Profile compilation
All the tasks associated with translating the original user statements into
profiles incorporating the various search strategies were carried out by one
person under controlled conditions. These are described now.

Experiment291

Standard tasks
To measure the intellectual effort involved in compiling profiles incorporating
the different search strategies, the profile compilation procedure was divided
into a number of standard tasks and times recorded for completion of eachtask.
It was considered important that, for meaningful comparison of the searchstrategies, all the profile versions compiled from a particular user statement
should use the same basic set of search terms. The first standard task wastherefore to produce a list of search terms. The complete list of seven standard
tasks, T1-T7 (with corresponding task completion times ^-^ 7 ) wasestablished as follows:
Tl—from user statement prepare list of search terms using various aids suchas INSPEC thesaurus, dictionaries, known relevant documents, etc.
T2—arrange the terms of Tl into groups representing the concepts in theoriginal query.
T3—assign boolean equations to govern the groups of T2.
T4—assign weights subjectively to the individual terms of T l .
T5—arrange groups of T2 in order of their importance to the original query.
T6—assign weights subjectively to the groups of T2.
T7—arrange the terms in the groups of T2 in order of their importancewithin the group.
Figure 14.1 shows the relationship between the various standard tasks andhow they lead to the profiles incorporating the different search strategies.
The compilation times are then calculated by adding the appropriate taskcompletion times, e.g. tx for strategy CT, tx + f4 for TWC, and so on.
Strictly there is no compilation time for strategy CRTW since the profile
terms were obtained in rather an indirect way. However it seems reasonable
to assume that, starting from a user statement of interests, the time taken to
produce a list of the 20 most important search terms (effectively strategy
CRTW) would not differ very much from that taken to prepare a complete
list of all the search terms likely to be useful (strategy CT). On balance theformer task could well involve less time. As an approximation we can take
both compilation times as being equal to tx.
The compilation of profiles incorporating strategy CLB was a completely
separate and self-contained exercise.
Clearly, with the standard tasks, there is a question as to howinterdependent they might be, e.g. in doing task Tl (preparing list of searchterms) does one immediately in one's mind start grouping them into concepts,
i.e. task T2, and even consider possible boolean equations, i.e. task T3. Also
it might reasonably be argued that the 'natural' thing to do starting with theuser statement is to isolate the concepts first and then expand them toproduce the search terms.
With these considerations in mind the user statements were first divided
at random into two groups A and B. In group A all the standard tasks for aparticular user statement were completed consecutively 'at one sitting'whereas for group B each task was completed in isolation, i.e. separated intime from the other tasks, so that the memory of doing one task had largely
disappeared before the next one was started.

292

1

c

<
>
/
c:

assi
lean
atio

U)

3
i ° O"<*> J2h- OJ <L>

C

o
cfl

a
i

^
H
u

c o
O)
if)

-*-<

CO
-*•>

H-

wei

en F
1 d
1
-H

u

£

h-

i

O

en
o
a
c
j *
on

T3
ret

»-

*o
C

Experiment 293
If the tasks are interdependent then, on average, the sum of the taskcompletion times for group A user statements should be less than for group
B statements; and, the greater the interdependence the greater the differencein the total task times. It is assumed here that with a total of more than 50
user statements involved there is unlikely to be any bias between groups A
and B caused by difference in subject matter or complexity.
The average task completion times for groups A and B were as given in
Table 14.1.
TABLE 14.1
Users

Group AGroup B

Average standard task completion times (min)Average

h

h

r3

u

t5

h

h

34
39

13
15

11
15

11
11

7
10

9
9

3
3

totals
88
102

Not surprisingly the total of the average task completion times is lower forgroup A (88 min) than for group B (102 min), thus indicating some degree ofinterdependence of the tasks. No doubt also part of the difference is due tothe effect of having to 're-understand' the user statement each time in group
B.
Alternatively it might be argued that groups A and B are not exactly
similar or are not quite large enough to discount the effects of individualstatement characteristics on the overall average task completion times. Thereis some support for this in that the average number of search terms is 44 forgroup A profiles and 50 for group B profiles.
However the average standard task completion times for groups A and B
are similar enough to encourage the conclusion that the degree ofinterdependence of the standard tasks is not so great as to invalidate theoriginal division of the profile compilation process into the 7 standard tasks.
Compilation times
Combining the data for groups A and B the overall average standard taskcompletion times (min) were: *i=36, t2 = \4, f3 = 13, t4—ll, t5 = 9, f6 = 9,
and t7 = 3 min, which gave the average compilation times for the different
search strategies as in Table 14.2.
TABLE 14.2. Search strategy compilation times (average)
Search strategy

Average compilation times

(min)

CT
TWC
CTW
CG
GTWC
CGW
B
GWC
BW

fj
tx+u
f,+r 4tx+t2/,+f 2 + f5 + f7
f,+f 2 + f5 + /7
f,4-f2 + t 3
r, +f2 + f5 + f6
tx+t2 + t2 + U

36
47
47
50
62
62
63
68
74

294

An experiment: search strategy variations in SDI profiles

Table 14.2 shows that, in terms of information scientist effort, the simplest
strategy, CT, takes almost exactly half as long to compile as the most complex
strategy, BW.
Modification times
As mentioned on p. 287 above the profiles were analysed and (perhaps)
modified just once, viz. after completion of the first group of 4 SDI runs but
before starting the second group.
The initial analysis procedure adopted was standard for all profiles
irrespective of whether any relevance assessments had been received fromthe users. Ten minutes were taken for an examination of the profile
performance after which time a decision was taken as to whether or not any
basic modifications were necessary. Twenty-two users' profiles were in fact
amended. It is emphasized that the time taken for any particular modification
is assigned in full to all the search strategy variations incorporating thatmodification, e.g. if 20 min are spent on amending the boolean equations
then this time is allocated to both strategies B and BW.
Averaging the modification time data (including all profiles whether
modified or not) and adding the 10 min initial analysis time the average
strategy modification times obtained were: CT=13, CG = 13, CTW=14,
CGW=14,TWC = 15,GWC = 15,GTWC = 15,B = 21,andBW = 22min.
Discussion
Before leaving the profile compilation procedure it may be useful to discuss
the standard tasks in more detail—in particular to consider some of theconflicts that occurred in trying to achieve a balance between experimental
rigour and what common sense indicated should be done in a real situation.
It has already been stated that for a valid comparison of search strategies
it seemed essential that, for a particular user statement, the same basic set ofsearch terms should be used. In fact occasions arose when this was contrary
to the needs of particular strategies, e.g. in the use of negative weights, NOT
logic, and WITHIN logic, which facilities do not feature sensibly in the coordination strategies CT, CG, CTW, CGW and CRTW. Examples of theuse of these facilities are detailed in the original report and the extent towhich they were used is indicated by the fact that, of the 55 statements
received, negative weights were included for 10 users, NOT logic wasincluded for 8 users, and WITHIN logic was included for 2 users.
Another general problem occurs when the original user statement reallycovers more than one basic subject interest or question. With boolean
strategies, if nesting or sublogic facilities were available, there would be no
problem but with co-ordination strategies it seems nonsensical to mix searchterms from what are essentially different questions. In those cases where
obviously more than one subject interest was involved the user statement was
divided and treated as 2 (and once 3) completely separate questions. It is now
felt that this should have been done for more of the user statements than was
in fact the case, viz. 6 users.
Some specific problems encountered when executing the individual
standard tasks were:Experiment295(1) 77—List of search terms
The initial intention was that, excepting perhaps chemical compounds(e.g. gallium arsenide), the search terms should be strictly singlets. It was
felt that any degree of pre-coordination would positively bias against
some strategies particularly CT (plain co-ordination of terms). Howevereven with simple CT there are difficulties, e.g. the free-index phrase 'field
effect transistor' may also appear as the abbreviation 4 FET' and of course
both versions have to be catered for in searching. If the singlets-only rule
is applied in the search profile then matching on the former produces aco-ordination level of 3 but matching on 'FET' is at a level of 1 only. Thisanomaly could be avoided by including the term 'FET' in the profile
three times but this would be merely simulating weighting techniques; todo so would invalidate the comparison with strategy TWC (weighted listof terms). Other similar examples encountered were STEM/scanningtransmission electron microscope, IMPATT/impact avalanche transit
time, and LEED/low energy electron diffraction.
Some 65 per cent of the profiles included one or more non-singlet terms
although, as a percentage of total search terms, non-singlets amounted toless than 5 per cent.
On the other hand some pre-coordination of terms might positively
favour some search strategies. For example, for the CT strategy, theconcept 'digital circuit*' would naturally be treated as two terms 'digital'and 'circuit*'. However for a boolean strategy it might be consideredsafer to search on the term 'digital circuit*' rather than 'digital' A N D'circuit*' since experience has shown that the latter usually throws up alarge number of false drops. Of course searching on 'digital circuit*' will
fail to match phrases like 'digital logic circuit*'. In an operational systemusing a boolean strategy the final decision would probably rest entirely on
which performance measure the user was more interested—recall or
precision.
Another difficulty encountered in preparing the basic list of search terms
for a particular user statement was the problem of what to do withnebulous terms like 'measur*', 'propert*', 'design*', 'observ*', etc. It was
fairly certain that they could do no harm when used in a weighted list ofterms (and might even improve the uniqueness of the ranked output) but
their usefulness in a boolean search is not clear and as likely as not to bedamaging.
In the interests of retaining the same set of terms in all the search strategy
variations for a particular query, some compromise had to be acceptedoccasionally in the final choice of terms used.
(2) T2—Arrange terms into groups (concepts)
In practice it was found that a large degree of latitude was possible individing the terms into concepts. At one extreme, for a user interested in'high power gas and liquid lasers', it might be argued that there are only
two basic concepts involved, viz. a device (laser) and a characteristic(power). Alternatively it could be said that there are five separable
concepts, viz. high, power, laser, gas, and liquid.
The general policy pursued was to divide into as many concepts as
possible. In fact the average number of concepts per user statement was
15, ranging from a minimum of 4 to a maximum of 25.

296

An experiment: search strategy variations in SDI profiles(3) T4—Assign weights to terms
In assigning weights subjectively to individual terms, values in the range
1-10 were used but where a single concept seemed paramount to thequery the scale was extended to cover weights of 1-20. The facility for
utilizing a wider range of weights was available ( — 999 to +999) and in
an operational system much higher positive and negative weights could
be used for ranking highly or excluding altogether items in certain
journals or languages, etc.
An interesting characteristic of assigning weights subjectively toindividual terms is that almost invariably one finds oneself separating theterms into groups (task T2), mentally ordering the groups into theirrelative importance (task T5), and setting imaginary threshold weights ina manner very similar to constructing a logical search. It is not suggested
that one necessarily has to formally perform tasks T2 and T5 in order toassign individual weights to a single list of terms; only that it is difficult
to proceed directly from task Tl to task T4 without thinking in terms oftasks T2 and T5, and even simulating T3 (assigning boolean equations).
A reservation concerning subjective weighting of terms is its possiblelack of appeal to individual users interested in compiling their own
profiles. No doubt some individuals would delight in the facility; others
might be quickly frustrated by the problems of unanticipated homonyms,etc.
(4) T5—Arrange groups in order of importance
Normally it is a fairly quick task to arrange subjectively the groups ofterms in order of their importance to the query. When 'powers of 2'
weighting is being used this task becomes quite crucial to strategy
GTWC. The feeling persisted that 'powers of 2' weighting would bebetter suited to controlled- rather than free-language searching, i.e. it
would operate better with greater pre-coordination of terms. If this is sothen strategy GTWC may have suffered somewhat from the approachadvocated for task T2 which was to divide into as many concepts as
possible.
(5) 715—Assign weights to groups
The same general procedure was followed in assigning weights subjectively to the groups of terms as was described above in (3) for theindividual terms.
Relevance assessments
An important factor when considering the mechanics of the experiment was
the role of the user group. It seemed desirable that they should operate in a'near-real' situation but at the same time it was necessary to have as many
documents assessed as possible to ensure the validity of the recall figures.
The users were not, and did not need to be, aware that a number of profile
versions (representing the different search strategies) had been compiled
from each statement of interests. On a particular search run the separate
outputs from all the profile versions prepared from the user's originalstatement were merged to produce a single set of notifications without
duplicates in random (document number) order. This was not only the mostconvenient procedure for the user but was also methodologically necessary inExperiment

297

that it ensured that the relevance judgements were completely independentof the search strategy and the position of the document in any ranked output.
The only question concerned the number of items from the different searchstrategy outputs that should be merged in the first place.
The profiles in INSPEC's commercial SDI service, operating on subject
interests similar to those of our experimental user group and with the same
document collections, were at that time producing an average of 12-15
notifications per profile per week. With this figure as a guide it was decided
that, for merging, the full output from the (optimum) boolean strategies
should be taken with at least the top 25 items from each of the ranked-output
strategies. Allowing for duplicates it was anticipated that the merged output
would comprise at least 50 notifications per user per run. In those subject
areas known to be more productive the full boolean output and the top 30, oreven 40, items from the ranked-output strategies were merged. In fact over
the total 8 runs the average weekly number of notifications sent to eachmember of the user group for assessment was 59.
Figure 14.2 illustrates in broad outline the operation to the point where the'single set of notifications without duplicates' has been produced for
despatching to the user for relevance assessments. The actual format of thenotifications (6 in x 4 in cards) followed that used in the commercial INSPECSDI service. They included the main bibliographic information (title, author,
affiliation, source reference) plus all the free indexing terms and the mainentry classification codes. The user also received a summary card of the hit
document numbers on which he indicated the relevance of each document
notified.
In making his relevance assessments the user was asked to apply a threecategory relevance code 1 as follows:
1—highly relevant documents;
2—partly relevant documents;
X—non-relevant documents.
To avoid misleading value judgements, the user was also requested to basehis assessment purely on the subject matter and to ignore such things as thelanguage of the original document, the quality of the journal in which itappeared, etc.
This three-category code was deliberately chosen for its relative ease of use
by the user. Highly relevant and completely non-relevant items are in generalquite quickly assessed, with relevance category 2 providing a useful 'dump'
for the difficult or doubtful documents, e.g. those which the user is quite
pleased to see but would not be concerned if they had not been retrieved.
Other relevance categories have of course been proposed and used in
document retrieval experiments. For example in evaluating operational
systems it is useful to distinguish between relevant documents which the user
has already seen before being notified via the system from those which are
new to him. As a generalization it might be said that too many re!
*categories are not advisable with 3 or 4 probably being the optimum numbc i
A more fundamental issue than relevance categories is the whole conceptof relevance. Its nebulous nature has been emphasized increasingly overrecent years even to the extent of raising it to the realms of philosophicaldiscourse. Nearly ten years ago Cooper 10 emphasized th disTinction b^tveen

60

o

-3

o
si

a
ffl

3
.00

Results 299
logical relevance and utility in an information retrieval context before
arguing in favour of the latter as the better basis for a measure of retrieval
effectiveness11-12. The point was succinctly made thus—'the purpose of
retrieval systems is (or at least should be) to retrieve documents that are
useful, not merely relevant'. Usefulness is defined as the user's subjective
evaluation of the personal utility of a retrieval system's output to him.
Recognizing the difficulties of such a subjective evaluation it is suggested
that more efficient compromise measures may be feasible although no readymade solutions have been presented.
The subject still features in the literature13 and on a practical level a real
distinction certainly exists between relevance meaning 'aboutness' andrelevance meaning 'pertinence'. In the former usage a relevant document is
simply one which deals (to a greater or lesser extent) with the same subject
matter as that of the query whereas for a document to be pertinent it has tocontain information which is new and useful to the originator of the query inthe subject area of the query. Obviously knowledge of pertinent documents
is more important than knowledge of those which are only about the same
subject as the query. However to establish the pertinence of documents
requires real users, with real queries, who have the inclination to peruse
entire documents. The availability of such a committed user group is rare.
In the experiment reported here relevance was used with the meaning of'aboutness'. Although the distinction was never spelled out to the users thefact that their assessments were based on less than the whole document
ensured this. Also no attempt was made to establish the extent to which users
followed up the documents notified to them.
Search software
Software for the project was specially written by the INSPEC Systems
Development Department. A generalized search package was developed
rather than separate optimum programs specially tailored to the requirementsof each search strategy.
The various search facilities available in the package are detailed in theoriginal report5 and are not of major concern here. Suffice it to say that thefollowing were included: boolean AND, OR, NOT, with (practically)
unlimited nesting, quorum logic, contextual logic, positive and negative
integer weights (decimal or 'powers of 2'), matching in upper and lower caseand in normal, inferior and superior alignment, left and right handtruncation, universal character, etc.
The most important point concerning the software was that with ageneralized search package rather than separate optimal programs, theamount of information obtainable on computer costs for the different searchstrategies was limited. This is discussed further on pp. 306 et seq.

14.3 Results
Retrieval performance
The doubts raised in the literature over the years concerning the rather
intangible nature of the 'relevance' concept have naturally been extended to300

An experiment: search strategy variations in SDI profiles

the performance measures based on relevance. In particular the measures
first quantified by Cleverdon as recall ratio (proportion retrieved of the total
number of relevant documents in the collection), precision or relevance ratio(proportion relevant of the number of retrieved documents), and fallout ratio(proportion retrieved of the total number of non-relevant documents in thecollection) have been increasingly questioned.
Despite the reservations that have been expressed about them and despite
the fact that theoretically more rigorous alternative measures may have beensuggested, it was considered that recall and precision were still the mostuseful and usable measures of retrieval performance. They are easily
understood and do provide answers to two of the most important questions
asked of bibliographic retrieval services, viz. 'What proportion of therelevant documents have been retrieved?' and 'What proportion of the
documents retrieved are relevant?'.
The recall figures established in this experiment were strictly measures of
the relative (or matched) recall rather than the true recall. Matched recall is
the percentage retrieved (by a particular strategy) of the total relevant
documents found by all the searches for a query. With document collections
averaging about 2500 per run and with a 'non-captive' user group it was notconsidered sensible to try and obtain relevance assessments for all the
documents actually searched. However, given the very loosefilteringprocess
utilized in the experiment to control the number of notifications sent to theuser for assessment (see p. 297 above), it is probable that the recall values
obtained were quite close to the true recall.
In the main experiment 10 search strategies were being evaluated. Of
these, 8 produced a ranked output of (effectively) unlimited size, thus
allowing free choice of cutoff points at which the relative retrieval
performances could be compared. The other 2 strategies B and BW, being
boolean type, produced a strictly limited output which of course varied fromuser to user depending on the subject interest covered. The problem was that
there did not seem to be any basis for comparing the boolean-type strategies
with the rest other than at one point, viz. the number of items retrieved by
the boolean.
It was decided to make two types of comparison:(1) a comparison involving all the non-boolean strategies, based on Salton's
rank-order cutoff-point procedure14, and(2) a profile-by-profile comparison, involving all the strategies, in which thebasis for comparison was the boolean output.
Ranked-output comparison
The raw retrieval data included in the original report need not be reproduced
here. Three of the eight search runs were evaluated, viz. runs 1, 5 and 6. The
consistency in the relative retrieval performances of the search strategies
over these three runs indicated that analysis of the remaining runs was
unlikely to yield any different information.
For runs 1 (46 queries), 5 (45 queries) and 6 (46 queries) the cumulative
totals of relevant documents retrieved by the different search strategies were
aggregated at the following 9 ranked-output positions: 5, 10, 15, 20, 25, 30,
35, 45 and 55 notifications. The corresponding recall and precision figures

Results

301(averages of numbers) were also calculated. Perhaps it should be pointed outthat the precision values were rather superfluous since, with the rankedoutput cutoff procedure, all the available information is contained in therecall figure—at a particular cutoff point if the recall figure for one strategy
is better than that for another then the precision figure is automaticallysimilarly so.
The single performance figure usually associated with the ranked-output
cutoff procedure is normalized recall15. Strictly this requires knowledge ofthe positions of all relevant items in the ranked output. The data used here
were the recallfiguresat 9 cutoff points down to the 55th ranked document.
Taking the average of the recall figures at the nine cutoff points gave an'average' recall value which approximated to the normalized recall in its
effect of ranking the search strategies in an order of merit. This order together
with the approximated normalized recall values for the eight non-boolean
search strategies for runs 1, 5 and 6 are shown in Table 14.3. No great weight
can be given to the actual values of the approximated normalized recall since
they depend on the number and positions of the cutoff points used. Howeverwith different cutoff points the relative positions of the search strategies
would not be expected to change.
TABLE 14.3. Ranking of search strategies by normalized recall (based on 9 cutoff points, averagesof numbers)
Order Relevance Rl documents

Relevance Rl/2 documentsojmerit Search strategy {normalized recall)

1
2
3
4
5
6
7
8

Search strategy (normalized recall)

Run I

Run 5

Run 6

Run I

TWC
(51.5)GWC (51.4)
GTWC (49.0)
CRTW (47.1)
CGW (47.0)
CTW(46.1)
CG(43.1)
CT(41.8)

TWC(58.5)GWC (57.3)
GTWC (55.7)
CRTW (53.0)
CTW(52.8)CGW (52.6)
CG(48.6)
CT(47.3)GWC (56.4)(54.7)TWC
GTWC (53.4)
CGW (52.0)(50.3)CG
CTW(50.0)CRTW (49.4)
CT(48.5)

TWC
GWC
CGW
CTW
CRTW
CG
CT
GTWC

Run 5
(44.5)(44.2)(42.5)(42.4)(41.9)(40.1)(39.4)(39.4)

GWC
TWC
CGW
CTW
CRTW
GTWC
CG
CT

Run 6
(49.7)(49.2)(46.9)(46.3)(45.6)(45.2)(43.8)(42.6)

TWC(47.8)GWC (47.5)
CTW(45.9)CGW (45.8)
CG(44.3)CRTW (43.9)
CT(43.7)GTWC (43.2)

Some points to emerge from Table 14.3 are:(1) The two strategies TWC (term-weight cumulation) and GWC (groupweight cumulation) are always the best, occupying first and second
positions on all three runs for both relevance Rl and Rl/2 documents.
(2) Strategy CT (simple co-ordination of terms) performs the worst, always
being in one of the last two positions.
(3) Perhaps surprisingly, strategy CRTW (co-ordinate matching of restricted
list of terms) holds its own with the others, mostly taking the middle
positions.
(4) Strategy GTWC (group/term 'powers of T weight cumulation) is unusual
in being always third best for relevance Rl documents but well down thelists when considering Rl/2 documents. This suggests that the harsher
consequences of 'powers of 2' weighting are not particularly suited toretrieval of 'in-between' partly relevant documents.

302

An experiment: search strategy variations in SDI profiles

In an attempt to obtain more detailed information the raw retrieval datafor run 5 were reworked using an additional two cutoff points (positions 1 and
2 in the ranked outputs) and calculating both by averages of numbers andaverages of ratios. In the former, the (overall) mean recall ratio is calculatedby dividing the total of relevant items retrieved for all the queries by the totalof known relevant items for all the queries. In the latter, recall ratios areindividually calculated for each query and the mean of these ratios represents
the overall system (or, in our case, strategy) recall ratio. It is not obvious
which method of calculation is the better; only that if the number of relevant
items varies widely from query to query then the two methods may not givethe same results. Also, if the two methods do not point to the sameconclusions, there must be some dubiety about drawing any conclusions at
all.
Table 14.4 shows the results of these additional calculations on the run 5
data. There is agreement that TWC and GWC are the best two strategies but
below that the relative positions of the search strategies are not constant. The
most extreme difference concerns strategy GTWC for relevance Rl
documents—by the 'averages of numbers' calculation GTWC is third best
but by the 'averages of ratios' it is in sixth position. Other less drastic
differences are also apparent. Incidentally comparison of the 'averages ofnumbers' results in Table 14.4 with the run 5 data in Table 14.3 confirms thatthe use of the two additional cutoff points in the calculation of normalized
recall has not changed the relative positions of any of the search strategies.
TABLE 14.4. Run 5 data—Ranking of search strategies by normalized recall (based on 11 cutoff
points, averages of numbers and ratios)
Order of

1
2
3
4
5
6
7
8

Relevance Rl documents

Relevance Rl/2 documents

Search strategy (normalized recall)

Search strategy (normalized recall)

Average ofnos.

Average of ratios

Average of nos.

Average of ratios

TWC (49.3)
GWC (48.3)
GTWC (46.8)
CRTW (44.7)
CTW (44.4)
CGW (44.4)
CG(40.9)
CT(39.6)TWC (54.3)
GWC (50.9)
CRTW (50.7)
CTW (49.7)
CGW (47.7)
GTWC(47.1)
CG(44.4)
CT(43.6)GWC (41.5)
TWC (41.1)
CGW (39.1)
CTW (38.7)
CRTW (38.0)
GTWC(37.8)
CG(36.5)
CT(35.5)GWC (47.7)
TWC (47.1)
CGW (46.7)
CTW (45.6)
CG(44.6)CRTW (44.1)
CT(42.7)GTWC (40.5)

The values given in Table 14.4 still lack statistical significance so the run
5 data were further analysed by pairing all the search strategies in turn and,using the normalized recall figures for the individual queries, the results
tested for significant difference using the sign test. Table 14.5 shows which
search strategies are significantly different at the 0.1, 1, 2, and 5 per centlevels. Wherep>0.05 the differences are treated as not significant. Perhaps
with more confidence than before it can be concluded that strategies TWC
and GWC are the best two (but not distinguishable from each other) and that
CT and CG are inferior (particularly when relevance Rl documents only are
considered).

Results

303

TABLE 14.5. Run 5 data—sign test for significant difference
Searchstrategy

Significantly better
than

Relevance Rl documents
TWC
CG(*2 = 16.9,/?<0.001)
CT(x 2 = 14.4,/?<0.001)
CTW0( 2 = 10.0,/?<0.01)
CGW (x2 = 6.4, p< 0.02)
GWC

CRTW
CTW
CGW
GTWC
CG
CT

CG
CT
CGW
CT
CG
CT
CG
CG
CT
CT
CG

(X2 = 12.1,/?<0.001)(X2 = 11.0,/?<0.001)(*2 = 9.0,/?<0.01)
0( 2 = 8.1,/><0.01)(*2 = 4.9,/?< 0.05)(*2 = 18.2,/?<0.001)(X2 = 11.0,/?<0.001)(X2 = 1.2,p<0M)(*2 = 6.4,/?< 0.02)(X2 = 5.6,p<0.02)(X2 = 5.6,p<0.02

—
—

CGW

CT
CG
CT
CG

GTWC

—

CG
CT

CT

—

GTWC (*2 = 3.03), CRTW (x 2 = 3.03)
GWC (*2 = 0.03)

CTW (x 2 = 3.6), GTWC (*2 = 2.03)
CRTW (x2 = 2.03), TWC
GTWC (x2 = 0.03), CTW (x2 = 0)
CGW (x 2 =0), TWC, GWC
CGW (x2 =0.03), GTWC (*2 = 0.03)
GWC, CRTWGTWC (x 2 = 0.23), CRTW, CTW
TWC, GWC, CRTW, CTW, CGW
C T ( * 2 = 1.6)
CG

Relevance Rl/2 documents
TWC
CT(x 2 = 12.8,/><0.001)
GTWC (x 2 = 9.8,/><0.01)
CG(x2 = 7.2,/?<0.01)
CRTW (*2 = 9.8,/><0.01)
GWC
GTWC (X2 = 8.9,/?<0.01)
CT(X2 = 8.0,p<0.01)
CG(X2 =6.4, p< 0.02)
CRTW

CTW

Not significantly
different from(*2 = 21.4,/?<0.001)(X2 = 6.4, p< 0.02)(X 2 =H.8,p<0.001)
iX2 = 5J,p<0.02)(X2 =4A,p<0.05)

CRTW (*2 = 3.76), CGW (x 2 = 2.69)
CTW (x2 = 1.8),
GWC (x 2 = 0.02)
CTW (x 2 = 3.76), CGW (x 2 = 3.76)
TWCCT(x2 = 1.09), CGW (x2 = 1.09)
GTWC (x2 =0.56), CG (*2 = 0.36)
CTW {x2 = 0.36), TWC
CGW (x2 = 0.09), GTWC (x2 = 0.02)
TWC, GWC, CRTWGTWC (x2 = 1.8), TWC, GWC
CRTW, CTW
C T ( * 2 = 0.8),CG(x 2 = 0)
CRTW, CTW, CGW
CRTW, GTWC
CRTW, GTWC

It might be argued that a more powerful test such as the Wilcoxon matchedpairs signed ranks test could have been used since the magnitude of thedifference in the normalized recall between pairs of search strategies was
known for all the queries. This was not pursued because of unease concerning
the validity and overall effect of those queries with very few relevant
documents; as mentioned earlier, in the extreme case of a query with only 1
relevant document a recall ratio of 0/1 could so easily be 1/1, and vice versa.
Boolean comparison
The method used for this individual profile-by-profile comparison of all thestrategies was as follows:304 An experiment: search strategy variations in SDI profiles(1) The output for the boolean strategy B (the only strategy producing an
unranked output) was counted (say x items), as were the number of
relevance Rl (say y) and relevance Rl/2 (say z) documents it contained.
(2) For the remaining strategies in turn, starting from the top of the ranked
output, the numbers of items taken to retrieve y relevance Rl and z
relevance Rl/2 documents were noted—say xx and x2 items respectively.
If both xx and x2 for a particular strategy are less than the boolean output
x, then that strategy is performing better than the boolean strategy B for both
Rl and Rl/2 documents. If x1<x<x2, then that strategy is performing
better than strategy B for Rl documents but not for Rl/2 documents. And soon.
This procedure gave a best strategy (or joint best strategies) for each queryon each run. The main disadvantage of the method is that it is entirely
dependent on the boolean output; if there is no boolean output or if theboolean output contains no relevance Rl or Rl/2 documents, there is nobasis for comparison. Also, under the conditions of this experiment, acomparison is not valid if the boolean output is very large.
Aggregating, from runs 1, 5 and 6, the results for the top 3 search strategies
only for each query a ranking of search strategies was obtained (Table 14.6).
This shows the percentage of times each search strategy occupied one of the
top 3 positions. Incidentally, in the calculations, where, for example, two (ormore) strategies were equal best they were both (all) ranked first but the next
best was ranked third (or fourth, etc., as appropriate). That is, a jointly held
first position was considered of equal merit to a uniquely held first position.
It is seen from Table 14.6 that, by this boolean comparison method, clearly
the two best strategies are respectively BW and GWC, with strategy TWC,
not so decisively, third best. The least promising strategy according to thismethod is CG. It is interesting that in the ranked-output comparison basedon normalized recall (p. 300), strategies GWC and TWC are practicallyindistinguishable whereas in the boolean comparison strategy GWC comesout better. A possible explanation for this difference was thought to lie in themethods of comparison—the ranked-output method is essentially 'neutral'
but the boolean comparison, being based on the boolean output, may be
more oriented towards those strategies which comprise term groups (e.g.
GWC) rather than a single list of terms (e.g. TWC). However this is notconfirmed by the relative positions of strategies CT (single list of terms) andCG (term groups) in the two comparisons.
In addition to the disadvantages of the boolean comparison method
mentioned above it is now thought that there may be more fundamental
objections. Not only are strategies B and BW considered to be misleadingly
rated relative to the other strategies (B too low, BW too high) but the wholeconcept of evaluating an optimum boolean (yes/no) output against a ranked
output may be questionable—it is not comparing like with like. On the other
hand one of the principal criticisms of the experiment might be said to havebeen the failure to develop a valid method for comparing optimum boolean
outputs with ranked outputs. To use a less-than-optimum boolean statement
as a weak filter in afirstpass of the document collection and then ranking theresulting output by some weighting scheme is a useful experimental
convenience but it does not correspond to the strategy BW in this experiment.

o r*- NO NO NO r-( <N<y>m§ o NO men

N© -<t o

«N

I/-,

NO

••-•

CQOUUhOuPPUUo^NO CN ON ON OO r- r- TI- Tf —<»o <N oo oo r-

M fN fS (N M

OQOHUuUfflUOu^ ^ m f n m ^ h h v o i '

,'°

<3

UU^ H ££
fflOHUUuOfflUU

O ^ rj- (N M ON —' O N V"> CN
TT vO OO Tf Tf CN O

r~» ^ m mR ,£ £ £ Hmr-H O ONCN ^H

hpiaOhCQOhUfflOUUUUIoo NO IT) «fr ,-r^ m m Tf —,mwn
NO lO CN
en CN m m m
m

3D

-4i
R
53

CN

CN

£ > £ H H I* H O C
fflOhOUUfflUUU
r-HmmooooooNONOfNcN

Q

I
K

• ^

o §

<<

fflOOhUUuUUffl-^(NcnTtiONor-ooON©

306

An experiment: search strategy variations in SDI profiles

Our boolean statements were meant to be optimum ones but to have optimum
boolean profiles producing ranked outputs suggests a conceptual contradiction. To have used less-than-optimum boolean statements would be no realtest of boolean logic.
Costs
In a computerized search system the costs of the system may be allocated toa number of factors—original database production or purchase; computer
operations such as searching, output preparation and printing, profile
maintenance, etc.; and non-computer operations such as the time of
information scientists, keyboarding, general clerical, etc. Two of thesefactors were considered in this investigation, viz. information scientist timeand computer search costs. Now, nearly five years later, the values used forsalaries and computer processing costs are dated. Rather than simply updatethem this section considers in more general terms what was learned in theexperiment about the various cost factors and how they related to conditionsin an operational system.
Intellectual (information scientist) effort
Although an accurate record of events in the experiment, the times obtainedfor profile compilation and modification (pp. 293 and 294 above) aresomewhat artificial in the context of a real system in that: (1) only one
analysis/modification stage was undertaken; (2) the times for trivial (butnecessary) profile modifications were not recorded; and (3) only the actual
compilation/modification times were included, i.e. there is no allowance for'dead-time' (tea breaks, etc.) which in real life might involve an additional 20
per cent or so.
To translate the figures on pp. 293 and 294 to a more real situation we
could: (1) assume an average of 3 significant profile modifications per year;
and (2) add 20 per cent dead-time to the above compilation/modification
times. If that is done comparative estimates are obtained for the information
scientist effort required per query per annum for the different search strategies{Table 14.7).
TABLE 14.7. Estimated annual information
scientist effort per query
Search strategy

Information scientist
time (min)

CT
CTW
CG
TWC
CGW
GTWC
GWC
B
BW

90
107
107
110
125
128
136
151
168

Assuming a 35-hour week and 47 weeks worked per year, the number of
profiles handled per year by an information scientist working full time would

Results 307
range from about 1000 for CT-type profiles to about 550 for BW-type profiles.
These figures still tend to the optimistic in that they account only for thepurely 'intellectual' aspects of profile handling. Normally time will also be
spent liaising with users (existing and potential), checking subscriptions, etc.
This variable, however, related more to job responsibility than any basic
profile compilation or modification procedure, would not be expected tochange by much the relative intellectual effort required by the different
search strategies.
Computer search costs
With a generalized search package rather than separate optimal programs,
the amount of information obtainable on the actual computer costs for thedifferent search strategies was limited. Tests with the software showed that
it is the matching of profile and document terms rather than profile logic
evaluation which controls the search rate; in other words, the number of
profile search terms is paramount and the profile logic evaluation represents
a minor part of the total computer processing cost.
In the experiment, for each user statement, all search strategies except
CRTW used the same set of search terms, and it may be assumed that thecomputer search costs for the different search strategies except CRTW would
not differ by more than 20 per cent.
Financial considerations dictated that the different search strategy types
were not run separately against the document collections and that thecomputer matching runs were conducted as a background job timesharedwith a variety of other tasks which differed from run to run.
In these less-than-ideal conditions the computer search costs per query(search strategy) per year were some 3-5 times the cost of information
scientist time. The other costs mentioned earlier but not investigated would
diminish further the contribution of information scientist time to perhaps
10-15 per cent of the total overall costs. The changing balance in the manmachine equation probably means that the information scientist time would
now contribute a greater share to the total costs than was the case 5 years ago.
Clearly there are two main approaches to effecting savings in the computersearch costs, viz. reduction in the number of search terms and simplificationof the term-matching procedure. The latter might be achieved by forgoing some of the more sophisticated matching facilities, e.g. simultaneousleft- and right-hand truncation, ability to distinguish upper and lower casecharacters, and the universal character. Individual profiles can be foundwhere one or more of these facilities is very useful and convenient but in most
instances their absence can be surmounted by the use of additional searchterms at a lesser overall cost. The value of these facilities varies with subject
area and in the chemical field the truncation facility might be consideredvital. A valid evaluation of their importance would probably require a large
number of search profiles for any significant effect to be apparent.
Cost-effectivenessIt became apparent early in the project that insufficient data would beobtained to enable any absolute conclusions to be drawn concerning the308

An experiment: search strategy variations in SDI profiles

overall most cost-effective search strategy. The discussion on p. 307 suggeststhat the computer search costs for all search strategies except CRTW would
not differ by more than 20 per cent because they all use the same basic set ofsearch terms. However strategy CRTW, which by definition always contains
20 search terms compared with an average of 46 terms for the other strategies,
is thereby much more economic in terms of computer search costs. If its
retrieval performance is on a par with the other strategies then strategy
CRTW must be the most cost-effective of all the 10 strategies evaluated in
the main experiment. The results presented on pp. 299 et seq. show that thisis in fact the case.
A more modest possibility with the available information is a comparisonof the cost-effectiveness of all the search strategies in terms of information
scientist cost only. A suitable measure of effectiveness is 'relevant documents
retrieved' and, certainly in the SDI situation, the number of relevant
documents retrieved at cutoff points of, say, 15 or 25 documents would seemto be the most appropriate.
The basic 'effectiveness' data included in the original report show, forexample, that at a cutoff point of 15 items, strategy CT retrieved 164,127 and120 relevance Rl documents, respectively, on runs 1 (46 queries), 5 (45queries) and 6 (46 queries). This gives an average figure of(164+127+120)/(46 + 45 + 46), i.e. 3.0 relevance Rl documents retrieved
per query per run by strategy CT at a cutoff point of 15 items. Assuming 50
runs per year (weekly SDI service) this figure becomes 150 relevance Rl
documents retrieved per query per year. Dividing this number by the one
given for information scientist effort on strategy CT in Table 14.7 gives afigure for the cost-effectiveness of strategy CT at a cutoff point of 15 items,
i.e. 150/1.5 or 100 relevance Rl documents retrieved per year per hour of
information scientist effort. Repeating the calculation for all the strategies at
cutoff points of 15 and 25 items gives the comparative cost-effectivenessfigures shown in Table 14.8.

TABLE 14.8. Cost-effectiveness of search strategies (information scientist effort only)
Order

of

Strategy cost-effectiveness {relevant documents retrieved!yearj hour of information
scientist time)

merit

1
2
3
4
5
6
7
8
9

Relevance Rl documents

Relevance R1J2 documents

Cutoff 15

Cutoff25

Cutoff 15

Cutoff25

CRTW (108.8)
CT(100.0)TWC(98.2)
CTW(91.9)
CG(89.0)GTWC (83.3)
CGW(81.8)
GWC(81.3)
BW(62.8)CRTW (150.9)
CT(139.4)TWC (136.0)
CTW (124.7)
CG(121.4)GTWC (114.8)
GWC (109.5)
CGW (108.3)
BW(77.8)CRTW (251.1)
CT(233.6)
TWC (211.6)
CTW (204.5)
CG(202.8)CGW (180.8)
GWC (175.5)
GTWC (169.9)
BW(128.0)CRTW (355.0)
CT(347.0)TWC (311.4)
CTW (303.3)
CG(294.1)
CGW (264.9)
GWC (251.9)
GTWC (245.5)
BW(167.9)Notes: (i) The information scientist effort for strategy CRTW was assumed equal to that tor strategy CT (see p. 291J(ii) Thefiguresfor boolean strategy BW are not strictly comparable with the others since some of the boolean outputs
were less than 15/25 items. Strategy B was omitted.

Results

309

Some points of interest to emerge from Table 14.8 are:(1) Strategy CRTW is confirmed, without qualification, as the most costeffective search strategy.
(2) Compared with the ranking of strategies based on retrieval performance
only (p. 301), strategies which have strikingly changed their relative
positions are CT (upwards), GWC (downwards), and BW (downwards).
(3) The 4 strategies comprising basically a single list of search terms, viz.
CRTW, CT, TWC and CTW, occupy the top four positions.
Free- and controlled-language comparison
Although not envisaged as part of the original project the data that became
available during the experiment was considered suitable for a directcomparison of free-language and controlled-language boolean profiles in the
INSPEC environment. The data covered profile compilation times, numberof search terms, and recall/precision performance figures. These are alldetailed in the original report and are only summarized here.
The average compilation time for the controlled-language boolean profiles(31 min) was just less than half that for the free-language boolean profiles (65min). The times recorded were for a compiler who was already familiar withthe controlled language concerned—INSPEC's thesaurus and unifiedclassification scheme. These compilation times may be slightly biassed infavour of the controlled-language profiles because invariably the freelanguage versions were compiled first. When the controlled-language version
came to be compiled there would probably be some memory of the originaluser statement even though the free-language version may have been
compiled some weeks earlier.
As well as having shorter compilation times, the controlled-language
boolean profiles were smaller than the free-language boolean profiles by afactor of 2|, averaging 19 terms and 47 terms respectively. It should bepointed out that one reason for the smaller number of search terms in thecontrolled-language profiles is that they were used in searching in a 'free-text'
way, i.e. extensive use was made of truncation in the search terms. Assuming
an approximately linear relationship between the number of profile searchterms and computer search time, this factor of 2\ would be largely reflectedin the computer search costs in favour of the controlled-language profiles.
There is a further saving of search time for controlled-language profiles
because the controlled-language searchable fields are smaller than the freelanguage searchable field in the INSPEC database. Statistics then current
indicated that the relative sizes of the threefieldswere in the ratios:Free indexing
10
Subject headings (thesaurus terms)
5
Unified classification codes
1
Because compilation of the controlled-language boolean profiles wasstarted after the main experiment had got under way the quantity ofexperimental data for the first few SDI runs was limited. Only therecall/precision figures for the last three of the eight runs were analysed, i.e.
those for runs 6, 7 and 8. Overall values for recall and precision calculated by
the usual two averaging methods (average of numbers and average of ratios)310 An experiment: search strategy variations in SDI profiles
are given in Table 14.9. Although they seem to show a consistently betteroverall retrieval performance for the controlled-language boolean profiles
further analysis using the sign test for significant difference did not support
this. Table 14.10 records the number of times controlled-language (CL) or
free-language (FL) profiles showed superior retrieval performance on runs 6,
7 and 8. The highest x2 value for this data is 2.0 so no significant difference
is indicated even at the 10 per cent level.
TABLE 14.9. Retrieval performance of controlled-language and free-language boolean profiles
Runno.

No. ofqueries

Profile type

Averaging method Retrieval performance
Recall (%)

Precision (%)

Rl

Rl/2

Rl

Rl/2

Av. of nos.
Av. of ratios
Av. of nos.
Av. of ratios

75.2
64.6
56.1
60.3

60.3
50.6
43.8
44.8

29.1
25.6
23.1
27.4

61.9
59.6
48.0
53.0

Av. of nos.
Av. of ratios
Av. of nos
Av. of ratios

58.0
57.6
51.4
57.2

49.4
46.5
39.7
42.8

20.8
25.6
17.6
21.4

57.6
58.6
44.4
49.8

Controlled-language

Av. of nos.
Av. of ratios

63.7
53.7

53.0
45.7

19.5
18.1

50.4
43.9

Free-language

Av. of nos.
Av. of ratios

57.9
53.6

45.9
42.8

15.8
14.3

39.0
40.7

Controlled-language
Free-language
Controlled-language
Free-language

TABLE 14.10. Retrieval performance of controlled-language (CL) and free-language (FL)boolean profiles
Run
no.

6
7
8

RecallPrecision

Rl documents

Rl/2 documents

Rl documents~~ ""Rl/2 documents

CL
FL
Same
better better

CL
FL
Same
better better

CL
FL
Same
better better

CL
FL
Same
better better

8
10
8

12
15
15

8
13
15

14
20
19

4
8
9

9
11
12

9
12
12

3
7
5

10
15
9

6
6
8

10
12
10

0
2
3

14.4 Conclusions
As is often the case with experiments in information retrieval whereconditions are peculiar to one situation or organization, the results obtainedin the major project may be valid only for the INSPEC database. In particulara factor that might be expected to influence the experiment would be themedium used for matching profiles and documents; in this case the freeindex terms assigned to all items in the database. INSPEC's operational
statistics at the time indicated that the free-index field contained on averageRetrospect

311

some 7 phrases/item, or, about 15 singlet terms/item. It might be argued that
this number may be small enough to be prejudicial against certain of thesearch strategies. No particular view is offered on this point other than that
it was not felt to be the case during the experiment and does not seem to be
obviously so. With the above qualification the following conclusions may be
drawn from the experimental results:(1) As measured by information scientist effort expended on the purely
intellectual aspects of profile compilation and modification, the simplest
search strategy, CT (co-ordinate matching of terms without weights),
occupied almost exactly half as much time as the most complex strategy,
BW (boolean logic with weights).
(2) The search strategies exhibiting the best retrieval performance were
GWC (group-weight cumulation) and TWC (term-weight cumulation).
In the boolean comparison of retrieval performance, strategy BW
appeared to do very well but it is now considered that the method of
evaluation was faulty and no conclusions are drawn concerning either ofthe boolean search strategies. The worst performer was strategy CT,
always being in one of the last two positions.
(3) Although the best retrieval performances were produced by strategies
using weighting techniques, experience gained during the project insubjectively assigning weights to terms suggested that the majority of
SDI users would not be particularly attracted to doing this task forthemselves.
(4) The most cost-effective strategy overall was CRTW (co-ordinate
matching of restricted list of terms with weights). In terms of information
scientist effort only, the most cost-effective strategies were CRTW, CT
and TWC, and, although not strictly comparable, the least cost-effective
was BW.
In the secondary experiment comparing controlled-language and freelanguage boolean profiles, the former: (1) were compiled more quickly (given
pre-knowledge of the controlled language); (2) comprised fewer search terms;and (3) showed comparable overall retrieval performance. Their maindrawback is that the use of controlled language is not likely to appeal to those
non-information workers who wish to prepare their own profiles. Although
not evident in this study another factor which can work against controlledlanguage profiles is that in subject areas where new terminology is being
introduced rapidly the controlled language may lag behind and be inadequate
until updated.

14.5 Retrospect
Looking back after somefiveyears the experiment is seen to have been in themainstream of information retrieval research at the time. On the whole its
methodology was based on established procedures and it also reflected thechanging emphasis in retrieval experiments, viz. whereas in the 1960s themain interest had been in indexing languages, by the early 1970s theconcentration was on search techniques. With the growing interest in312 An experiment: search strategy variations in SDI profiles
automatic indexing it is now being seen more clearly how interdependent areindexing and searching methods.
At various points in the above description some shortcomings of theoriginal experiment have been mentioned. It may be useful to conclude bygathering together and discussing these defects and also those questions
which were raised but remained unresolved. It is hoped that some activities
were performed adequately but inevitably they are of less interest and will
only be mentioned briefly.
Those parts of the investigation which are considered to have been soundinclude: a very adequate document collection; a meaningful range of searchstrategies; a realistic profile compilation method involving standard tasks
which allowed an accurate measure of the effort required from theinformation scientist on the different search strategies; a valid procedure forcollecting relevance assessments; and the recruitment of the user group andthe mechanics of the experiment in general.
Less satisfactory areas include: the rather low number of queries; retrieval
performance evaluation by the boolean comparison method; the absence ofautomatic term-weighting; the lightweight nature of the cost data; and thesignificance of the experimental results.
Concerning the number of queries it is now considered (although nowhere
proved) that perhaps twice the number of queries would have been moreconvincing; or, at least a number sufficient enough that the results of a few
individual queries do not obtrude on the overall results. In our experiment
this effect was exemplified by the differences observed when calculating by
the two averaging methods, numbers and ratios. With a greater number ofqueries it would also have been possible to ignore those queries for which
there were too few or, less importantly, too many relevant items in thecollection. It is not clear what the implications of such a practice are but
certainly the results would thereby be more reproducible. As has alreadybeen mentioned too many recall/precision ratios of the order 0/1, 1/1, etc.,
are not really acceptable. The problem could have been eased indirectly if amore drastic approach had been taken originally with some of the userinterest statements. Those that clearly comprised more than one question
could have been treated separately. This would have resulted in 'cleaner'
profiles of which fewer were overlong, some profile performances would
probably have been subject to less extraneous influences, and the number ofqueries would have been- larger. Although a token number of the userstatements were in fact split up more could have been and the experiment
would have been better for it. At the time the view taken was that as little aspossible should be done to change the conditions from that of 'real life' and,since these were statements very like those received from users of an
operational system, the less tampering the better. This is now deemed to havebeen misguided and to have done what is now suggested would not haveaffected the validity of the test in any way.
The most disappointing outcome of the whole experiment was the failure
to develop an acceptable method for comparing an optimum boolean strategy
with any strategy producing a ranked output. A few simple examples quickly
show the inappropriateness of using the boolean output itself as the basis forcomparison. Very little can be offered in the way of a solution even now andRetrospect

313

maybe it is not sensible to attempt such a comparison in that like is not being
compared with like.
An often-voiced criticism of the original report was that too little attention
had been paid to the statistical significance of the experimental results. An
attempt has been made to rectify that in this paper and undoubtedly it hashelped to clarify the picture. However establishing statistical significancewhen differences are not obvious is still a limited achievement and there ismuch appeal in the philosophy of what might be termed a Cleverdonian
maxim—if one needs to resort to statistical techniques to establish
performance differences in information retrieval experiments then the
differences are not worth knowing about.
It was mentioned almost in passing that assigning weights subjectively tosearch terms and term-groups was unlikely to appeal greatly to users
attempting their own profiling. For this reason alone it is a pity that automaticweighting of terms was not possible in the evaluation. Also since all profiling
was carried out by one experienced compiler there was no impression gained
as to the likely ease-of-use, and acceptance, by end-users (as opposed toprofessional information staff) of the different search strategies. This failing
was realized from the start but it was thought to be too difficult to surmounteasily. The decision to use only one compiler at least ensured the control ofthis variable.
It has been thought that perhaps not enough was attempted at the time toestablish the reasons why the strategies performed as they did. The questionof how much failure analysis should be done was considered at some length.
Where strategies performed as might have been anticipated (e.g. it was not
surprising to find that the best retrieval performance was produced bystrategies using weighting techniques) there seemed little purpose in detailed
analysis. In the evaluation of an operational system it is clearly important toobtain a measure of which activities are responsible for retrieval failures—inparticular what proportion can be allocated to poor indexing or to profile
compilation errors. In this experiment since, for each query, all strategies
shared the same basic list of search terms, such failures would be common to
all strategies. Thus the main interest was in distinguishing any differences
due to the characteristics of the strategies themselves. The most promisingprocedure seemed to be to examine those queries for which the strategies,
which performed best overall, did unusually badly. Pursuing this method
showed up one clear link between search strategy performance and type ofuser statement. In those strategies comprising a single list of terms (CT,
TWC, CTW) as opposed to those including term groups, there is the possible
deleterious effect of having one concept 'swamping' all others in the list of
profile search terms. The damage occurs when the document free-indexing issimilarly 'unbalanced'; this of course can quite legitimately be inevitable.
For example, the concept 'metals' comprises more than 60 individual metalelements and the literature is such that often a paper on some aspect of metal
behaviour deals with a number of different metals all of which are properly
included among the index terms. The result is that the outputs from singlelist strategies are top-heavy with the individual 'metal' terms which, in theterm-group strategies, are controlled by virtue of being in a term group which
contributes only once to the total weight irrespective of the number of314

An experiment: search strategy variations in SDI profiles

matching terms in that group. Although the effect was quite considerablewith the 'metals' concept it did not occur so obviously with other concepts,
because most documents are such that the same concept is not required to be
represented a large number of times in the free-index terms of a particular
document.
Perhaps the one surprising experimental result which did merit consideration was the comparatively good retrieval performance of strategy CRTW.
Of the other strategies CRTW was most similar in type to CTW. The factthat a reduction in the number of search terms by more than half (CRTW
always contained 20 terms and CTW contained, on average, 46 terms) had
produced no significant difference in retrieval performance was unexpected
and rather deflating. It suggests that the number of search terms in profiles
should be optimized rather than compiled exhaustively. The decision thatstrategy CRTW should contain a maximum of 20 terms for each query wasarbitrary and not much more than a reasonable assumption, and the optimum
number of terms would be expected to vary from query to query dependingon their subject matter. One explanation for the result, offered originally byCleverdon, is that the arithmetic product—'Number of profile search terms x
Number of document index terms'—has a critical value which, if exceeded,results in a deteriorating retrieval performance. This view is supported by
the results from another INSPEC project 16 in which profile search terms
were expanded automatically by reference to a thesaurus as a source of,successively, synonyms, narrower terms and 'see also' terms. It was found,against expectation, that although the retrieval performances 'were notusefully different. . ., the general trend was for poorer recall with eachexpansion'. The important factor was that the base profile version contained
41 terms on average and the final expanded version contained 122 terms on
average, figures which were probably well above the optimum. This effect of
profile length can of course be counteracted by the use of weights andgrouping of terms but it remains an interesting point when searching by thesimplest strategy, straight co-ordination of unweighted terms.

References
1. CLAGUE, P. SDI Investigation 1967-1969, 5 Vols, Report R71/6, INSPEC, Institution ofElectrical Engineers, London (1971)
2. AITCHISON, T. M. et al. Comparative Evaluation of Index Languages: Part 1-Design; Part 2Results, Reports R70/1 and R70/2, INSPEC, Institution of Electrical Engineers, London(1969, 1970)
3. EVANS, L. Optimum Degree of User Participation in SDI Profile Generation, Report R73/12,
INSPEC, Institution of Electrical Engineers, London (1973)
4. CLEVERDON, C. w. and HARDING, P. Report on an Investigation into a Mechanised Information
Retrieval Service in a Specialised Subject Area (CRISPE Project), Cranfield Institute ofTechnology (1970)
5. EVANS, L. Search Strategy Variations in SDI Profiles, Report R75/21, INSPEC, Institution ofElectrical Engineers, London (1975)
6. SPARCK JONES, K. and VAN RUSBERGEN, c. J. Report on the Need for and Provision of an 'Ideal'
Information Retrieval Test Collection, British Library Research and Development Report
5266, Computer Laboratory, University of Cambridge (1975)
7. SPARCK JONES, K. and BATES, R. G. Report on a Design Study for the 'Ideal' Information
Retrieval Test Collection, British Library Research and Development Report 5428, Computer
Laboratory, University of Cambridge (1977)References315

8. GILBERT, H. and SPARCK JONES, K. Statistical Bases of Relevance Assessment for the 'Ideal'
Information Retrieval Test Collection, British Library Research and Development Report
5481, Computer Laboratory, University of Cambridge (1979)
9. MATTHEWS, F. w. and THOMSON, L. Weighted term search: a computer program for an
inverted coordinate index on magnetic tape, Journal of Chemical Documentation 7, 49-56(1967)
10. COOPER, w. s. A definition of relevance for information retrieval, Information Storage andRetrieval!, 19-37(1971)
11. COOPER, w. s. On selecting a measure of retrieval effectiveness, Journal of the American
Society for Information Science 24, 87-100 (1973)
12. COOPER, W. S. On selecting a measure of retrieval effectiveness: Part II—Implementation of
the philosophy, Journal of the American Society for Information Science 24, 413-424 (1973)
13. swANSON, D. R. Information retrieval as a trial-and-error process, Library Quarterly 47, 128—
148(1977)
14. SALTON, G. Automatic Information Organisation and Retrieval, McGraw-Hill, New York(1968)
15. CLEVERDON, c. and KEEN, M. Factors Determining the Performance of Indexing Systems:
Volume 2—Test Results, ASLIB Cranfield Research Project, College of Aeronautics,
Cranfield(1966)
16. EVANS, L. and GOULD, A. M. Automatic Aids to Profile Construction: Expansion of Search Terms
by Thesaurus, Report R76/25, INSPEC, Institution of Electrical Engineers, London (1976)