•tS^U^v

U

^

^

tL<t

:J

i (_^-y~^^ V-

REPORTON
THE NEED FORANDPROVISION OFAN'IDEAL'INFORMATION RETRIEVALTEST COLLECTION

K. Sparck-JonesC J Van Rijshergen

University Computer Laboratory
Cambridge

December 1975

i>X.^fc

Preface

The preparation of this Report was supported by a grant fromthe British Library Research and Development Department. The
Report was used as a discussion document for a Workshop held
on December 11-12, 1975. The authors are very grateful to theWorkshop participants for their stimulating reactions to theReport, and for their favourable response to the suggestionthat an "ideal0 test collection of the kind indicated should
be constructed as a material aid to retrieval research overa wide area.
V7e have not attempted to incorporate the many comments madeinto the final version of the Report, as this would haveeffectively required a wholly new document. We are instead
issuing the Report as prepared for the Workshop with only
minor clerical corrections, in the hope of receiving further
comments from other potential users of the 'ideal" collection,
which could be input to a more detailed design study for thecollection construction.

K.S.J.
C.v R.
December 1975

1

O.

SUMMARY OF REPORT, .AND RECOMMENDATIONS
Summa^
This studya) investigates the need for an ideal test collection(s) for information retrieval research;h) discusses the requirements it should meet;c) outlines the characteristics it should have; andd) considers the administrative implications of setting it up andmaintaining i t .

T h e study is in three parts:

1. deals with the need for, and properties of, the ideal collection(s);2. deals with organisational aspects;
3. deals with (roughly) estimated costs.
The Appendices provide details of existing collections.

Our conclusion is that there is a genuine need for a well-designed multipurpose test collection. The least collection satisfying these needs
would consist of a large document set with core propertiesr having several
small, enriched, subsets, and a number of associated collections comparablewith the subsets in size and having other properties. Higher-grade ideal
collections would provide more alternatives, large and small. At least
some of the basic material could probably be obtained from existing services
or projects, but this would certainly have to be supplemented.
The ideal collection(s) could be set up by a one-off project, but it must
be maintained and made available to research workers, and some person ororganisation is required to do this. The collection itself should hopefully allow a large range of uses, and while the primary intention is thatit should be made available to different projects, it could also benefit
research through in-housc exploitation by the holding organisation.
Very rough cost estimates suggest that the collection could be set up for
between £25K and £<?0R,- depending on how much use could be made of existingdata and the grade of collection chosen; and it could be maintained for
between C25K and £10K p.a., giving a total of between E25K and £50k for aminimum useful maintenance period of five years.

2

Recommendations
This document is primarily for discussion, and so presents alternatives atmany points. We nevertheless feel that it is useful to provide specificrecommendations based on our survey. These are1.

that an ideal test collection be set up to facilitate and promoteresearch;2.

that the collection be of sufficient size to constitute an adequate
test bed for experiments relevant to modern information retrieval
systems; and that it be characterised in a large number of ways suited
to different needs. Specifically/, that it consist of one or more large
document sets with core properties, each with several subsets enriched
in different waysf and each accompanied by other collections comparable
in size and richness with the subsets;

3.

that the collection(s) be set up by a special purpose project carriedout be an experienced worker, called the Builder;

4.

that the collection(s) be maintained in a well-designed and documented
machine form and distributed to users, by a Curator;

5.

that the curating project be encouraged to promote research via theideal collection(s), and also via the common use of other collectionsacquired from independent projects.

3

THE NEED FOR, AND CHARACTERISTICS OF, AN IDEAL TEST COLLECTION' •Introduction
At the recent Workshop on Automatic Indexing (Sparck Jones 1974, Wilson
1974) , it became apparent that there was a need for a well-founded
information retrieval test collection. This is not to say that this need
was not recognised before; it certainly was, but retrieval research has
fortunately not been significantly hampered by the lack of such acollection. It has nevertheless now become clear that future research ininformation retrieval will require better test data than is currently
available. The purpose of this report is to say why an 'ideal1 testcollection or set of collections* is needed, to characterise it f and to
consider how it might be provided.
We believe that research in information retrieval is entering a new phase.
Perhaps the best way to describe this hew.phase is to characterise thephase now ending as one of pilot studies. The last fifteen years haveseen the publication of many preliminary and isolated results. These
results were frequently obtained with data-bases which were not quite
right for the kinds of experiment concerned. 3ut since these were theonly available data-bases, the tests were the best that could be done;
and the experiments were therefore performed and reported, sometimes,
though not often enough; with suitable caveats. Other research workers
might want to use the same data, to compare their own results with theearlier oncs« For this alternative descriptions of the data night berequired, which were not readily available. The later workers would thus
be driven to other collections to satisfy their particular requirements,
making proper comparisons with the previous work impossible.
It is now time that many of the 'mini0 results reported so far areincontrovertibly established (or rejected)„ The major stumbling block
seems to be the lack of suitable test data. There is a widespread feelingamong research workers that existing test collections are inadequate because
they are small and/or careless and/or inappropriate. They nay also not be
fully ma chine-readable, or may be in an esoteric machine format.
At present, too many projects are working with different collections * This
leads to unnecessary data preparation effc^rt by each project. It also
accounts for much of the project disconnection ivhich exists. This makes
it difficult to correlate the results obtained by different projects.
Further, since the use of a collection is limited by the time span of asingle project, data tends to get Inst, particularly where it has been
temporarily extracted from an operational system.
Further, the recent growth of on-line retrieval services based on large
data-bases has changed the conditions and character of information
retrieval in many ways, and there is a new requirement for test collections
suited to experiments relevant to these services.

* abbreviated to "ideal test collection(s)".

4This state of affairs suggests a need for an 'ideal1 test collection(s),namely one satisfying requirements forcommonality between projects,
hospitality to projects,
adequacy for projects, andconvenience in projects.
The rest of this Section of the report considers the detailed implications ofthese requirements.

Levels of collection description
An information retrieval collection is a set of documents, requests, andrelevance judgements. Sometimes it may be more convenient to talk of test
data than a test collection, but these are to be taken as more or lessequivalent.
Past test collections referred to in this report are ones which have beenused mainlv for experiments in computer-based information retrieval. The
ideal collection (s) is also designed primarily for research in mechanised
indexing and searching, and some low-level requirements and pre, .ration
activities are associated with the provision of an easily-used computerbased collectionc The collection itself could,, however, in principle beused for manual experiments. ™e have not pursued this possibility here,
though it should be noted that listings for manual use are a simple byproduct of machine processing. The distinction between manual and automaticcollection of data is quite different, and the extent to which the collection
will be- characterised automatically appears in our detailed specification.

Tt7e discuss collections at four levels:-^ rGa^-;this refers to the entire documents, requests and relevancejudgements,in their full detail; however in many cases
the entire document texts have never been explicitly
considered, all processing like indexing, the making of
relevance judgements etc. being based on e.g. abstracts.

2) material: this refers to the form of the documents, requests andrelevance judgements actually input for considering orprocessing (i.e. directly used for tests).

^

keyed

:

^

f° rm ^tted: this refers to the keyed data after standardisation andclerical manipulation for easy use. It does not ordinarilyrefer to collections .specially formatted e.g. for operational
IP. systems, but is represented by KS3' s "standard
collections'1 where standardisation is represented bystemming and clerical manipulation by the use of numbers
for words and the systematic provision of a set of transformed data files in a consistent format.(For details see
Appendix D.)

this refers to the machine readable form of the material
data.

5

Past test collections
Since both research experience and the justification of demands for theideal collection (s) have been materially influenced by past and existing
test collections and their properties/ it is useful to summarise salient
facts about the more important of the collections used for evaluation tests,
primarily of mechanised indexing and searching. The British collections arc:
1.

Cranfield'22.

Inspec .

3.

ISILT

4.

UXCIS

5.

Medusa

6.

NPL

7.

Olive, Terry and Datta's

Details of the collections and the projects exploiting them are given in
Appendix A, under headings includingcollection size and subject
project objectivemode and source of indexing
form of relevance judgement, etc*
The striking feature of this set of collections is their incomnarability••
there is no one form of index description common to all; there was great
variation in the environmental conditions; the projects had quite distinct
objectives: and they presented their results in very different ways.
We have confined our detailed analysis to British collections because thesecollections are sufficiently representative, and because information about
them is more readily available. For reference, non-British collections ofany status include those used by the Smart Project, by Lancaster, and by
Jahoda. Summary information about these collections is given in Appendix 33.
Several of the British collections have been used by more than one project,and some have been processed for easv machine handling. Some American
data has also been shipped about, and the SMART Project collections inparticular are presumably available to interested parties. However mostcollections set up to date have not been easy to use or widely exploited.
The reports describing major experiments conducted with the British collections (for references see Append!xA) say very little about the detailed
design of the collections. In general it seems as if they were designed bydefault. It is true that the composition and nature of the test data, forexample how relevance was assessed, may be described in some details But
there is very little discussion of possible choices, say of index description
source, or indication of why one particular choice, for example of abstract,
was made. In some cases, of course, the choice was dictated by circumstances;
but even then its implications may not be explicitly considered. But perhaps
this lack of design is not surprising, since the prime concern of the6experiments has been the testing cf some major variable affecting
efficiency or effectiveness, in a given environment. The same seems tohold, for non-British collections.
Following up our stu^y of the literature, we have discussed specifications
for an ideal collection(s) which would meet both needs arising from present
lines of research and those likely to arise in the future, particularly inconnection with on-line searching, with research workers in the UK asfollows: Mr. Aitchison, Miss Barraclough, Dr. Brittain,
Miss Horsnell, Mr. Keen, Dr. Leggate, Professor Lynch, Mr. Robertson,
Professor Vickery and Dr. Wyatt.- We are very grateful to them and to
Dr. Holmes of BLP.DD for their help. We also sought suggestions from
Professor W.S. Cooper, Professor B. Griffith and from Professor Salton.
The discussion of collection requirements which follows is based on theexperience and predictions of ourselves and our fellow workers.
He emphasise that we have attempted to be forward looking. It is apparent
in particular that it is most important that the ideal collection (s) should
be a means of relating valid abstract studies of information retrieval andthose of operational systems and user behaviour. These both imply a large
test collection, with some properties not manifest in existing collections„
The specification of the ideal collection(s) are neverthless necessarily
derived mainly from experience with past collections, and we therefore makeno apologies for references to these. In general prediction can only bebased on previous findings. In information retrieval research in particular,
past results have been so fragmentary that some future research must beconcerned with validating them. At the same time it is clear that new
research topics are arising, for example in connection with on-line
searching, the availability of really o^werful computing and communications
facilities, the development of retrieval networks, and so on, for which
suitable test data must hopefully be provided. We have therefore sought tospecify a fairly 'open8 ideal collection(s).
Areas of interest
General areas of interest likely to be of study interest to users of theideal collection(s) are:
A. relative to collections and their users:
text ponulations
document populations
source (i.e. journal) populations
origin (i.e. author, organisation, country) populations
citation populations (differentiated under the preceding 3 heads)
request copulations
user populations
need populations
exnert populations (e.g. indexer, searcher)vocabulary ponulations, natural and index
language populations, natural and index
description populations
catalogue populations
input oooulations
subject nonulations* We were unfortunately unable to contact Mr. Cleverdon at the relevant time.

7

B. relative to computer manipulation:
file populations
network node populations
C. relative to economic management:
It is not easy to say exactly how economic questions could be studied
with a test collection which is necessarily abstracted from ordinary
use* However we think a large test collection could be used toprovide comparative information for soecific costing studies, andmore importantly, since many facts about it will be known, as ameans of validating simulations of some library management operations.

Requirements
Me can broadly distinguish two kinds of requirement to be met by an ideal
collection(s). The first is to ensure the validity of experimental results.
The second allows for the control of variables affecting retrieval
performance. The control of one variable may not be compatible with thatof another. F^r the moment we will ignore the implications of thisincorapatibilitv for the design of test collections, and sirmly list all therequirements that an ideal test collection(s) should independently meet.

General requirements, concerning the sets of documents, reauests andrelevance judgementsi) substantive requirements re these sets
The ideal collection(s) should be:large
re documents•:..•.,< 500 documents are of no real value
1-2000 documents are minimally acceptable for some purposes> 10000 documents are needed for some purposes

re requests:< 75 requests are of no real value
250 reauests are minimally acceptable> 1000 reauests are needed for some purposes.

reasons: real collections are large
statistically significant results are desirable
scaling up must be studied(Note that request and document set sizes are not necessarily correlated).

8

1

various in content
documents and requests should cover a range of subjects of varying content
and 'hardness9 e.g. science, social science, news
reasons:2

real collections are heterogeneousconsistency of devices must be tested by comparison

homogeneous in content
documents and requests should cover one subject intensively
reasons:3

real collections are homogeneous
discrimination of devices must be tested by exhaustion

various in type
documents should be of different types e.g. popular, specialised, survey,
review, patent; requests e.g. broad, narrow
reasons:4

similar to 2.1

similar in type
documents and requests should be of the same typereasons:5

similar to 2.2

various in source
documents should cover a range of journals and journal types
reasons:6

similar to 2.1

homogeneous in source
documents should cover one or a fev; similar journal types in depth
reasons:7

similar to 2.2

various in origin
documents should represent different author origins and status; requests
should represent different users and needs(link relevance)
reasons:

similar to 2.1

16Collection specifications^•

To satisfy general requirements for set of documents, requests and relevancej udgementsa)substantive

There should be at least 2 large collections for an

science
These should ezch

subject area respectively,

cover variations in

conL;er?.b, type, Source, origin, time and language.
They should profex.~L\Vy be i-};c;n from an operational system, i.e. both
documents and ire^ce'.-itf-j 3c;onJ..d be thos taken, and accompanied by genuineuser reJ evance ;j Lv^-ir^'its,

It should be possible to cxt.ra/jc from each of the large collections one
or more small subcollectioi;-; which are homogeneous with respect toconte.it, type, source, origin, time and language.

The subcollecflous should be operational as far as possible, but it ishighly probable that some requirements e.g. for alternative indexing etc.
can only be met by
dvsycjn.
b) formal
The large collection:-; .shou?o. he various in formal properties, and it
should be possible to extract homogeneous subcollections.

B•

To satisfy requi.r:-j:n^r>ts csty:err:.ing individual documents, requests andrelevance j udgerj.ji.rL-,::;The detailed specification of core and enriched properties for the ideal
collections primarily refers to individual documents, requests, andrelevance judgement':-:, rather than sets of documents, etc. Our choice ofproperties from the full lints of pp./(to which the numbers refer) is as
fo1lows t
11—13
C = core;

E = enriched17

Documents
Documents in large collections should be represented by
2 abstracts
3 titles4 free keywords, from abstracts
6 citations
)
,,
. . .
,
. -, ..
_,,.,,,,,,. abbreveiations to be avoided
7 author and bibliographic elements)
8 thesaurus or subject indexing, if available
Indexing should be by one simple indexer, and one expert, for 4
by one expert, for 8.

Documents in small collections should be represented core plus

4 free keywords, from text, title, to different exhaustivity
5 free sentence
8 thesaurus if not in core, and other controlled languages as available.
Indexing should be by various simple indexers, and various experts, for 4
by one simple indexer, and one expert, for 5
by various experts, for 9

Requests
Requests in large collections should be represented byla) verbal textb) coordinated termsc) Boolean formulation (which could consist of a cumulative log ofan on-line search)
Indexing should be by one user, and one expert, for (b) and (c).

Requests in small collections should be represented by core plusId) terms with weightse) edited forms of b, c, df) modified forms of b, c, d2source documents
3
verbal text from, source documents.
The total record of any request eliciting procedure should be preserved.
For example if a user is asked to mention appropriate known documents,
these should be indicated.
Note that while some experiments relevant to on-line searching could
exploit requests formulated during previous on-line searches, in othercases new searches would be required; for the latter the ideal collection(s)would provide an adeauate set of documents.

18
Relevance judgements
C

Relevance judgements in large collections should allow
2 grades (highly, partially)
one user, and one expert, as judges.
It is unlikely that exhaustive relevance judgements could be made forlarge collections; however some attempt must be made, e.g. by additional
searches, to estimate recall.

E

Relevance judgements in small collections should allow core plus
more grades, types
various users and various experts.

Collection reco^se• \dxi\'-ton;;
The general problem is that these ideal specifications may have to betempered by realism. The exact way in which the requirements lis tedabove can be met must be determined to some extent by what is available
in operational systems. A specific problem is that while some data may
be available in an operational system, it may not be in machine readableform. In general one night hope to extract material with most coreproperties from an operational system, but keying of items like abstracts
must be allowed for. .M.u^h of the data for enriched collections would
have to be specially supplied. Clearly, any project to set up theideal collection (3) would have to have an initial phase for a detailed
study of data sources.
Our specifications suggest the following as useful but realistic collection sizes:
large
1O-30O0O documents; 500-2000 requests
medium
2™. :.)OOo"
small 500- iOOO"
200- 500
The main problems are clearly those of satisfying the core requirements
for the large document sets which are needed for some purposes; and of
ensuring that small collections are experimentally valid while notmaking them too large for the capacities of independent projects whichmight contribute them.
Similar recommendations are needed for numbers of collections. In
particular, since choices of property requirement can be combined indifferent ways, it is convenient to distinguish three grades of ideal
collection(s) which might be built. They are
1
2
3

best,
accept.able, andleast.

Which grade is achieved is determined largely by the ease with which
enriched property descriptions can be supplied by operational systems.
It is likely that many will have to be supplied by design. In detail,the grades are as follows:19-*- k e s t

2 large collections,each with 3-5 subsets having substanial property
overlaps, of which 3 are designed and 2 are selected
subsets.
3-5 other small collections to complement these.

^ acceptable2 large collections,each with 3 subsets having some overlap,of which 1 is a designed and 2 are selected subsets.
2 other small collections.

3 least1 large collection,with 5 subsets having some overlapof which 3 are designed and 2 are selected subsets.
2 other small collections.

These subset specifications do not include ones which could be selectedby purely clerical operations, e.g. ones representing all the articles
from a specified journal or requests with the same number of terms.
Some such selections could easily be made initially, others to order.
Even the least collection would be of great value to research, particularly if it was supplemented by collections from other projects, especially
if these were of the good quality which might be achieved by 'bulge1
funding. In addition, if the collection was primarily extract.d froman operational system, it might be encouraged to grow through theoperational system. This would clearly be a very satisfactory way of
meeting many ideal collection needs.

Collection form
As noted earlier, it is intended that the ideal collection be machine
held. This means that the main collection data is machine held, and in
a convenient form. Referring to the categorisation of collection levels
on p.4, it is clear that some real information , like fulldocument texts, could hardly be keyed; but it should preferably be heldin microform. The material collection must, however, be keyed. It mustbe supplemented by adequate backup information and documentationa) characterising the content of the collection and how it was set up;
and b) detailing any processing applied to bring it from level 3 tolevel 4, and its format at level 4.
It is perhaps not reasonable to require that other projects, even whenfunded by 3LRDD, should provide total information about depositedcollections. But these collections should meet some minimum standardsof content and format, to make them sufficiently comparable to the corecharacterised ideal collection (s); and they should be suitably documented.
In principle collections set up by projects not funded by BLRDD might beof value to supplement the ideal collection(s); it might of course not be
possible to obtain such material in the desired form, but even rav;
magnetic tapes and primary documentation should be sought.

20

2.

ORGANISATIONAL ASPECTS
There are two questions here:
1) the ideal collection (s) must be set up, by someone whom
the Builder;we will call2) the collection (s) and possibly other generally useful ones must be
kept so that they can be made available to research workers, by someone
we will call the Curator.
These are distinct activities, so Builder and Curator need not be thesame person. The division between their concerns comes with the provisionof the ideal collection (s) at formatted level 4; this could be either the
final stage of the builclJ7>g project, or the first stage of the curatingone. We emphasise that there is little point in setting up the ideal
collection (s) unless proper management and maintenance is provided for.
Some organisation is required even to provide tape copies of the level 4
formatted collection. But v/e believe that the Curator could have the morepositive function of stimulating research through the use of the collection (s).
It will be clear that both setting up and maintaining the collection (s)are non-negligible enterprises. The implications of the specifications
outlined in the previous section, and possible ways of setting up andmaintaining the collection (s) are discussed below. We necessarily
assume that funding sufficient for the least ideal collection(s), isavailable. The higher grade collection(s) of p!9are clearly more
attractive to the research community, but we should not claim ".hat they
are necessary for the well-being of the community. In particular, since
the cost of setting up collections involving different primary document
sets must be largely additive, the grade to be chosen depends primarily
on BLRDD's willingness to provide funds. We think that a very good casecan be made for BLRDD's supplying the least collection, both to reducethe cost of individual projects and to promote the research that
information retrieval needs; and since managing this collection andsupplying it to users is a not wholly trivial task, some committment
to the future maintenance of the collection from BLRDD is also reauired.

The Builder

We do not think that this is the place for nit-grit recommendations as
to exactly how the ideal collection(s) are to be set up. This will
depend in part on the level of funding, and in part on how far suitable
input material exists in current operational systems or has beenassembled by research or development projects. However we feel that thegeneral approach to setting up the ideal collection (s) is independent ofsuch specific considerations. The important points are as follows.
Even if ideal collection building is funded only to achieve the least
output, the degree of control required, and effort involved, areconsiderable.

9

2.8

homogeneous in origin
documents and requests should represent one kind of author and user
reasons:2.9

similar to 2.2

range over time
documents should be of different dates; requests should be of different
dates both for different users and the same user
reasons:

similar to 2.1

2.10 coincide in time
documents and requests should be contemporaneous
reasons:

similar to 2.2

2.11 various in natural language
documents should be in different languages (or at least their titles
should, in which case translations should be provided)
reasons:

similar to 2.1

2.12 homogeneous in natural language
documents should be in one language
reasons:

similar to 2.2

Globally, it should be possible to use the ideal collection(s) toinvestigate or simulate
retrospective searching i.e. one request against all documents;
SDI searchingi.e. a repeated request against successive
document sets;

iterative searchingi.e. a modified request against some or all
documents;

multifarious searchingi.e. a request, modified request or set ofrequests against multiple document sets»in
It should also be possible to use the collection (s)/studying the. interfaces
between components in a mixed system incorporating, for example, data
retrieval, fact retrieval, document retrieval and computer-aided
instruction. This mav be called hybrid searching.

10ii) formal requirements of document, request and relevance judgement
sets.

1

documents and requests should be variable inreal length
material length (i.e. index source length)
index length
reason:2.2

to test consistency

documents and requests should be homogeneous inreal length
material length
index length
reason:to test discrimination

It is assumed that appropriate parallel substantive and formal propertiesof relevance judgements will follow naturally if the above specifications
for document and reauest sets are met.

11

General requirementsi)

re individual documents, requests and relevancej udgonents.

substantive, re documents

Document representation
It should be possible to use or study
1.

full text (this should be preserved even if not keyed to allow for
future new indexing, linguistic studies and questionr
answering experiments)

2oabstract

a)
b)

as is
all non-stop keywords, stemmed

3.

title

a)
b)

as is
all non-stop keywords, stemmed

4.

free extracted keyword or keyword string indexing
1)from full text )2)
3)from abstractfrom title))a) as wordsb) stemmed

where in general if 1 has exhaustivity x

2 has exhaustivity x and also y>x
3 has exhaustivity x and also y and also z>y5.

free quasi-extracted sentence, i.e. a single unit sentence incorporatin*
extracted keywords

6.

citations7.

author and other standard bibliographic details

3.

controlled indexing, including broad subject codesa)b)

in full detailin short code

1) using any standard existing thesaurus for the field(and or classification, as many as readily to hand)
1 from abstract
2 from title
9.

probabilistic indexing (using keywords)

10. usage statistics

12

Document indexing
Indexing should be carried out
re 4a) by a simple indexer; by one indexer at different times;by different indexersb) by an expert(also perhaps re 5)

by expert consensusr

re 8 by an expert

"

VB

:

»

by expert consensus

Request representation
It should be possible to use or study
1 verbal(given the same source text request)a) running textb) simple coordination formulationc) full Boolean formulationd) terms with user weightse) edited after consultation forms of the above
i.e. pre search, with librarianf) modified forms of the above at end search, withrecorded history of subsearches, changes etc.
i) off-lineii) on-line

2 source document as request
3 verbal (as above) from source documenta) where source document is relevantb) where source document indicates area of interest but is notnecessarily specificallv relevant
Request indexing
Indexing should be carried out
1

by user; by user at different times

2by expert; by expert at different times? by different experts,
by expert consensus

This indexing may be done with a specific relevance need in mind; if so,
this should be indicated with the query. Any other germane backgroundinformation should be recorded.

13
Index language
Ensure available, i.e. preserved, even if not used, if relevant language
exists at time collection(s) is set up.
1• 2
3

thesaurus
classification
switching language

Relevance judgements
Ideally these should be exhaustive. But if not some attempt should bemade to carry out independent searches using any available informationand device, to obtain a pooled output for more broadly based relevancejudgements than may be obtained only with simple user evaluation ofstandard search output. In this case some estimate of the recall sample
should be attempted.

It should be possible to separate
1) gradese.g. highly, fairly2) typese.g. novel, stimulatingof relevance judgement.

Judging should be done by
1 one user; one user at different times; one user specificallysequentially
2 one expert; one expert at different times; several experts;
expert consensus

Exclusions
The following 6o not seem
1
2

3
4
5

to be called for:books as documents'non-literary 1 items for documents e.g. technical record specifications,simple data records (e.g. stock,
personnel)
verification-type requests e.g. for publication dates
material in esoteric character sets
legal data

14

Other collections
The provision of a new test collection (s) , even if ideal, will not makeexisting collections redundant. This is in part because a good deal is
known about some existing collections, so they may be useful test beds
for new ideas. Some may also be of value for making comparisons with theideal collection. It must also be recognised that the ideal collection(s)is unlikely to meet every research need, and that future collections
associated with specific projects may be created. Thus in the future weshould allow foranda) some further comparison between existing collections;b) some comparisons between existing and new collections;c) some comparisons between existing collections and the ideal
collection;d) comparisons between new collections and the ideal collection.

These projections imply that steps should be taken to relate newcollections, in particular, to the ideal collection (s). They should be
regarded as a means of extending the ideal collection(s).

The ideal collection(s)It is obvious that the listed requirements for the ideal collection (s) areconsiderable.
In some sense they cannot be provided within a single collection, unless
this is no more than a mere aggregate.
The following pairs of collection requirements are particularly important:
1. The need for sub and super collections;2. The need for one and several collections;3. The need for operational and designed collections.

Thus experimental needs are in fact for different collections which canbe related to one another, and which have specific properties. Realism
suggests that it may be impractical to seek to ensure that each suchcollection has the maximum set of (compatible) properties (e.g. allvariations on the relevance judgement theme), and further that it isunlikely that such collections with all the requisite properties canjust be pulled out of operational retrieval systems. It appears morepractical to think in terms of large, not necessarily completely
characterised collections, with richer small subsets, selected as far as
possible from operational systems, but supplemented where necessary bydeliberately designed information (e.g. further sets of relevancejudgements, judex descriptions etc.). The former have 'core1 properties
while the latter are 'enriched*.

15

This suggests something like the following will turn out to be needed:

all large
mediumsmall

collections respectively to be of comparable size innumbers of documents and requests

The following sections work this scheme out in detailCore and enriched forms of collections'Core1 refers to essential properties possessed by all ideal collections
and subcollections; "enriched' refers to additional properties. Somecore property requirements are readily satisfied even for large collections: the problem is to specify a set of core requirements which areboth useful for retrieval experimentation and realistic for large
collections. Some enriched property requirements are very exigent: itis perhaps unrealistic to suppose that all compatible ones can besatis fied for every subcollection; on the other hand it would be nice if
different subcollections of a large collection had more in common thantheir all being subsets of the same set, with core properties. If
possible, some overlap in enriched properties should be provided, toallow for valid comparisons and extrapolations.

21This suggests
an experienced heada project of Ih - 2 years
a cost of £25-30k(ball-park figure).

We see the project as having three phases:
1) Design study, to by carried out by the Builder as an initial short
investigation.
This would survey existing operational or experimental services/ andalso test collections, to see how they might be exploited to provide
input; and it would discuss the mechanisms for collecting thedetailed data, with specific cost estimates.
2) Data assemblyThis would involve the extraction and bringing together of datafrom services, and the provision of new data, e.g. alternativeindexing, relevance judgements, etc.
3) Machine inputThis would include keying the raw input material and applying any
appropriate basic transformations to material already in machinereadable form. The boring but non-trivial job of raising this level3 material to formatted level 4 would be done either by E-.dlder orCurator, according to the resources available.
4)

Documentation
This w-'?uld cover a full account of the source material and the way
it was collected, with notes on the keying conventions• Level 4
processing if done would require documentation to match.

The most important requirement of the Builder is that he should be
experienced in setting up and using test collections. It would clearlybe ideal if Builder and Curator were one, but this is perhaps too muchto hope for. If they are not, it is most important that there should
be adequate liaison between Builder and Curator, perhaps in phase 2,
and certainly in phases 3 and 4. A suitable mechanism might be to have
the Curator as a consultant on the building project. Since bringing
the collection up to level 4 could be done under the maintenance project,
it is not necessary that the Builder have direct access to powerfulcomputing facilities. Keying of raw data could be done by a bureau, andbasic transformations of material selected from machine-based systems
could be done either by the supplier under contract, or a bureau, or by
the Curator.
Since willingness to do the job, and the necessary experience, are more
important than anything else, we do not feel obliged to specify the
Builder's locale. He could be
an independent research worker;
a member of an existing retrieval service organisation;a member of a consulting estafilishment like Aslib;
on the staff of, or associated with, BL.

22
The Curator
As mentioned above, a minimal view of the Curator's activities would imply
that he did no more than hold the established ideal collection(s) anddistribute magnetic tapes and descriptive documentation. However otheractivities for the Curator are implied by the suggestion in Section 1that the ideal collection(s) might be supplemented by other project
collections. The Curator's brief could therefore include the following:
1. Maintaining and distributing the ideal collection (s). .
2a.Obtaining existing reasonably solid test collections,
if necessary vamping up at level 3
andprocessing at level 4.
b.Acquiring new collections from individial projects, particularly if
BLRDD requires or encourages deposition; if necessary vamping andprocessing.
3. Carrying out (documented) benchmark retrieval runs; gathering basic,e.g. statistical information about collections.
In terms of day to day operations these activities would imply:.. holding, over a long period;
2. obtaining, and vamping/formatting;
3. clerical processing e.g. of magnetic tapes;
4. providing and distributing documentation and advice;
5. carrying out simple experiments and counting.

Effective curating over this range of activities would require a Curator
experienced in both retrieval work and computing, and fairly powerfulmachine facilities, and would depend on relatively long-term support at
the appropriate level. But it must be emphasised that the ideal collection
would probably have a long life, so a long-term commitment of funds, even
if maintenance is only on a low level, is needed.
Again, it is rot for us to recommend a specific organisational setup for
the Curator. The following are alternative possibilities for BL:
1. entering into a non-personal contract with a computing service(commercial or university), for the provision of tape copies, etc;
or a similar contract with a retrieval service;
2. establishing a personal Curatorship attached to a Library School,
Computing Department or Retrieval Service;
3. establishing a curating project with specified Curator, attached
as in 3, with the intention that this should act as a focus ofresearch;
4. setting up

an institution with Curator, with intention as 3.

The first of these would almost certainly not promote the fullest use of
the ideal collection(s), aid the assembly of other collection material,or ensure benchmark testing. The fourth is objectionable as very
expensive and liable to be a white elephant. The second and third

23

alternatives seem the best bets. In particular they would promote theuse of the ideal collection (s) as a focus of research, and hopefully
prevent the mere accumulation of dead material. Assuming a more than minimal tape-copying service, these alternatives would imply something likea half-time Curator and half- or probably full-time programmer of somecalibre, with suitable support, i.e. an annual cost of between £5K and£10K = £50K over five years ( not including machine time). A deliberateattempt to encourage extended research using the collection(s) in
association with curating would imply higher costs.

Advisory panel
Since building and curating the ideal collection (s) are significant
projects, we advocate a panel or steering committee with the following
functions:a) advising the Builder and Curator on project operations; .
b) maintaining technically acceptable standards of data managementand distribution;c) encouraging collection use by advertisement;d) vetting proposed uses;e) ensuring general continuity.

24

3.

COST ESTIMATES
The cost estimates given in the following pages should only be taken as
ball-park figures. The difficulties in giving accurate estimates now, are
1. Insufficient data on which to base estimates/
2. Data out of date (1971),
3. For some parts commercial rates will not apply.
The figures used are mainly based on a report by Peter Vickers (1974).
There is an additional difficulty in allowing for inflation. Althoughin general a 30% increase may be applicable to the costs quoted we havenot adjusted them for the simple reason that in some cases the cost(e.g. computer processing) has gone down. Rather than try and estimate
the trend of the cost associated with each item we have stuck with 1971
prices. We also give the raw data on which our estimates are based(taken from Vickers, 1974) in Appendix C. We only give detailed costsof the Building phase of the operation since the costing of the Curatingphase depends heavily on what is actually implemented. We do however
list some of the major factors determining the cost of the Curatingphase.
We ignore the cost of housing the projects and the fact that some of thecosts may be borne by separate small projects.
Building(all figures in US dollars; halve for pounds)

The reason for giving most of the costs in US dollars is that we wish to
maintain comparability with the figures in Appendix C.
a. Documents
cost of buying a data-base of some 50000 items from an operational
system.
Tape with citations+ descriptors+
abstract
Low-level reformatting

750
1500
2500
1000

25

Data prepration at .05 cents/char.
50000 docs.

30000 docs.

10000 docs.

12500
25000
50000

7500
15000
30000

2500
5000
10000

6250
12500
25000

3750
7500
15000

1250
2500
5000

500 chars
1000 chars
2000 chars

312.5
6250
12500

1875
3750
7500

625
1250
2500

500 charslOOO chars
2000 chars

21875
43750
87500

13125
26250
52500

4375
8750
17500

16666

10000

3333

If we have to keyboard per item
500 chars (e.g. index terras)
1000 chars (e.g. abstracts)
2000 chars (e.g. everything)
Proof-reading is about half the
keyboarding cost
500 chars
1000 chars
2000 chars
Equipment is about half theproof-reading cost

Total

Computer processing of input at about
0.33 dollars oer item

We estimate some of the costs associated with generating small enrichedcollections
2QOO docs.
Cost of indexing 2.5 - 5.00 per item
Cost of abstracting 1.5 - 6.5 " "Cost of acquisition of full text

5000 - 10000
3noo - 130CO?

b. Requests
We assume that the requests will be collected during a bona fide use ofan operational system. Therefore the cost per query will be mainly thatcharged by the system. One could estimate 5 - 1 0 dollars for this. Notehowever that corresponding to every information need we may have to run
5 - 10 formulations to estimate the relevance set.

c. Relevance judgements
In general one will have to assume that by providing a service free ofcharge to a user he will in return provide relevance assessments. The
exhaustive assessments of small subsets will have to be costed separately

26
e.g.
Acquisition of full text
Mailing
Clerical?
?
?

d. Cited references
One of the core requirements is that each document should have as part ofits representation the references it cites. Unless this comes with therepresentation extracted from the operational system the cost of obtaining
this further information will have to be estimated separately. The most
likely source of the cited references are the ISI tapes for theappropriate period.

e. Generating a collection at level 4
At this stage it is not possible to say whether level 4 should be createdby the Builder or the Curator. However this decision will mainly affectthe apportioning of costs between the two phases. If for the moment weassume that creating at level 4 is done by the Builder then we will haveto allow for
extra machine time,
file storage,
documentation costs.
f. Personnel
Builder)
Programmer)

up to £8000 p.a.

Library and clerical support £2000 p.a.
Travel (particularly in the early stages of the design study, see p. 21)
Adminstration (e.g. mailing, xeroxing)
The reason the cost of the Builder and programmer have been lumped together
is that to some extent there exists a trade-off between them. If the
Builder is experienced computationally he would not need a very experienced
programmer. On the other hand if the Builder is not acquainted with thecomputer technology his programmer will have to be of a higher standard.
Also, it may be that if creating at level 4 is left to the Curator thebuilding project would only require a half-time programmer.

27

Maintenance and Distribution
To some extent the costs of this operation will depend on the demand forthe data. The operation should be costed over 5 years following thebuilding phase. The main cost factors are
Curator (half-time?)
Programming support (h programmer?)
Equipment (e.g. tapes, terminals, etc)
File storage
Computing time
TravelAdvertising
ClericalIf the ideal collection(s) is to be added to over a period of time thenthe costs of the building operation will be applicable here pro rata.

28

AppendicesA.

Details of British test collections

B.

Summary of non-British test collections

C.

Costs table

Do

Standard c o l l e c t i o n

References

format29

Al

CRANFIELD 2

Project nameFactors determining the performance of indexing systems.
Objectives'to deal with index language devices..(with)., precise measurement ofrecall and precision ratios'. To carry out a laboratory test, following
up and improving on Cranfield 1.
Chief person/Reference
Cleverdon et al, 1966
Size
221 queries. n_TA ^ j* several subsets, especially1400 documents
Subject
Aeronautics.

42 queries.
n^ ' 200 documents.

Indexing source
Full texts; also abstracts and titles/titles.
Index languages
3 types, in 30 forms, all applied manually:
single terms; with synonym grouping; with hierarchical reduction
simple concepts; with hierarchical reduction
controlled terms; with related terms.
Abstracts and titles/ titles indexed automatically.
Requests
Authors of selected recent papers asked to state reason (in form of aquestion) for undertaking research leading to paper, and to provide otherquestions related to this research.
Relevanceby authors, for own cited papers: exhaustively by experts to obtain
additional papers for author vetting. There were four relevance grades.*Relevance judgements were based on full text.
Document collection
Document set consisted of some recent papers, and their cited references,with some others.
Present state of test collection
Queries and single term index descriptions, abstracts and titles, with
relevance judgements, available at level 4.
Other users of test collection
Sparck Jones, van Rijsbergen, Salton and SMART Project workers; also
Minker, Svenonius.
(Some SMART tests with 24 or 155 queries and 424 documents) .

*n relevance grades does not include non-relevance as a grade.

30

A2

INSPEC

Project nameComparative evaluation of index languages.
Objectives
A comparative assessment of the retrieval performance/ in the INSPEC
system, of a number of index languages which might be used as the sole ormain means of subject manipulation.
Chief person / Reference
Aitchison et al, 1970
Size
97 queries
542 documents.
Subject
Physics, electrotechnology, and control.
Indexing Source
Abstracts and titles/titles.
Index Languages
1. Titles" „,, . ..,) not normally regarded as an index languageJ
2. Abstracts and titles
3. Printed subject index to Science Abstracts + free language modifierline
4. Controlled language using a thesaurus
5. Free language terms (applied by the SDI investigation staff to indicate'subject content of document1 before translation into 4 ) .
3-5 applied manually.
Requests
Questioners were asked to ensure that the questions were within the scopeof their SDI profiles ('it will need to be answerable by some of the
documents already notified to you by the SDI service'). Only questions
with at least one document at the higher level of relevance were usedin the evaluation. Queries screened by research team 'if detailed studyof the profile showed that its scope had been changed in the course of
the four SDI services' query would be discarded).
Relevance
Each questioner to be sent for assessment only those documents which hehad previously assessed as relevant to his profile. Relevance assessments made by the user on the basis of document texts. There were two
relevance grades.
Document collection
2/3 of documents relevant to some query.
Preconditions
1. SDI investigation was in progress
2. Queries were solicited from users who received all four services andhad assessed at least 12 documents as relevant to their profile.
Present state of test collection
Queries and free language index descriptions, with relevance judgements,
available at level 4.

31
A3

ISILT

Project Name
Information science index languages test.
Objectives'to compare the effectiveness and efficiency of different index languages
as used in subject retrieval systems'.
Chief person / Reference
Keen et al, 1972
Size
63 queries
800 documents
Subject
Documentation
Indexing source
Abstracts and titles/full texts
Index languages
5 kinds all applied manually
1. Comnressed term language - 300 terras from ASLIB + related terms added.
2. Uncontrolled - natural language text words underlying hierarchicalindex terms of 3. Specific indexing was followed by redundant indexing.
3. Hierarchically structured language - post-coordinate.
4. Same as 3 but pre-coordinate
5. Relational indexing
Exhaustivity and specificity of indexing were controlled.
Requests
These were miscellaneous real requests, formulated considerably later
than the dates of the documents.
Relevance
Exhaustive relevance judgements were made based on abstract and title for408 of the documents; for the rest full text was used. There was a scaleof relevance. The assessment was 'non-user relevance by simple subject
experts who were not requesters, indexers, or searchers in the test'.
Document collection
set I...408 documents from the Smart project. Abstracts from the period
1961-6 3 were available in machine readable form. These abstracts
claimed to be bad*set II..392 good abstracts dated up to 1968, some of these were quoted asknown relevant ones by requesters.
2/3 of the collection in fact relevant to some request.
Present state of test collection
Queries and uncontrolled index descriptions for whole collection,
abstracts and titles for Subset- I, with relevance judgements, available
at level 4.
Other users of test collection
Sparck Jones, Van Rijsbergen, Horsnell(Some tests with Subsets I or II .with automatic indexing from abstracts
and titles/titles for Subset I ')

32

A4

UKCIS

Project name
Retrieval experiments based on Chemical Abstracts Condensates.
Objectives
1. To gain experience using CA-Condensates tapes,
2. To compare the relative effectiveness of searching titles only, titlesplus -keywords, and titles-plus-digests.
3. To measure the variation in performance between profiles covering
different subject areas.
4. Investigate automatic profile construction.
Chief person / Reference
Veal / Barker, et al, 1974
Size and Subject
193 requests (subset 48)
documents
size
subject
CAC-1
11518
Biochemistry, organic chemistry
CAC-2
15629
Macromolecular, applied and physical chemistry
CBAC
1568
Biochemistry
POST-J
1412
Polymer science
POST-P
1442
Polymer scienceIndexing source
Full texts/titles
Index languages
Keywords applied manually; titles and digests effectively indexed
automatically.
Requests
Formulations were from SDI service users and were current for the documents
searched. Different versions were written for different data bases.
Relevance
Assessment of output (pooled if appropriate) by users, usually from titlesand digests, sometimes titles only. There were 2 relevance grades, andsometimes 2 non-relevance grades.
Document collection
Documents were taken from Chemical Abstracts Service files.
Present state of test collection
CAC-1 and"CAC-2 available at level 3 or 4.
Other users of test collection
Sparck Jones, van Rijsbergen

33

A5

MEDUSA

Project name
Medlars on-line search formulation and indexing.
Objectives
To compare the standard method of search formulation by a trained search
editor with a physician's using an on-line terminal.
Chief person / Reference
Barraclough/Barber et al., 1972
Size
58 queries
51000 documents
Subject
MedicineIndexing source
Full texts
Index language
MeSII, i.e. controlled language, applied manually.
Requests
Requests from real on-line systems users, with two formulations;, one by
the user and one by a trained search editor.
Relevance
2 grades of relevance, also 2 grades of non-relevance.
Assessment of output based on citation and indexing, by user for both
search formulations.
Document collection.:
Documents ta)~en from monthly files of regular Medlars service.
Present state of test collection
Available at level 3
Other users of test collection

34
A6

NPL

Project name
The National Physical Laboratory experiments in statistical word
associations and their use in document indexing and retrieval.
Objectives
1. To develop methods of clustering words on the basis of especiallycomputed measures of association between word pairs.
2. To explore and evaluate ways of employing these clusters and associations
to improve performance especially in the ability to recall relevantmaterial.
Chief person / Reference
Vaswani, 1970
Size
93 queries
11571 documents
Subject
Electronics/ computers, physics, and geophysics.
Indexing source
Abstracts and titles
Index languages
A dictionary of 1000 index terms (stems) was constructed based on a sample
of 1648 abstracts by semi-automatic means. These were used to index thedocuments.
1. Weighted
i u, J terms
Unweighted
2. Clusters derived from associations
3. Expansion through a connection network
Requests
20 people formulated requests based on source abstracts; but these only
specified subject and abstract not necessarily relevant.
Relevance
17000 relevance decisions made by the people who formulated the requests.
Report claims that 80% relevant documents uncovered by various strategies.
Document collection
Set from published abstract journal.
Present state of test collection

Other users of test collection

35

A7

UKAEA/NSA

Project name
SDI from Nuclear Science Abstracts
Objectives
A study of the relative performance of two computer matching techniques:(a) of Euratorn indexing terras aid (b) words in titles.
Chief person / Reference
Olive et al, 1973
Size
60 queries
12765 documents
Subject
Nuclear scienceIndexing source
Abstracts and titles?
Index languages
1. Natural language
2. Euratom index terms
3. NSA subject categories
2 and 3 applied manually.
Requests
Formulations were based on SDI service users' interests;
Relevance
User assessment of search output based on title, bibliographic elements
and assigned index terms. There were two relevance grades and an option
to state that abstract was required to make relevance decision.
Document collection
Documents taken from successive issues of NSA used
service
Present state of test collection
Available at level 3
Other users of test collection

in a regular SDI

36

Bl
Test collections used in SMART Project tests published by Salton andothers from 1968 are:Requests

Documents

ADI

35

82

Documentation

IRE

17
24
34

375
375
780

Computing

Cranfield

42
36
22
22
24
30
155
36
50
225

200
200
200
424
424
424
424
1400
1400
1400

Aeronautics

Ispra

48
48

1268
468, 1095

Documentation

Medlars

18
24
29
35
29

273
450
450
1033
852

Medicine

03
21

425
425

World Affairs('Pphth^ilcKDdicrcjy)

Time

These collections are automatically indexed from abstract and title (but
ADI from short full texts) ; some have indexing derived from a manual
thesaurus; the Medlars collections MeSK indexing is not held. The
collections are presumably available at something like level 4.
Recents tests have mainly exploited the Cranfield 24x424, Medlars 24x450
and Time 24x425 collections. For relevant information see Salton 1975aand b.

^

tr
G

CJ

CJ

G^

r-H
4J
-H

r-H

v->

r-H
4J
-rH

-rH

W

tt

H

En

H

EH

-P

ing

6

CO
(1)

rH
CJ

2:

EH

»o

tion and ition

ind x wit

aph theory

Med cus a

G

o

rH

Cn

CJ

>.

T3
G
H

rd
r-H

4->
CD

c

P

P

B

a
0)
g
3
u
o
Q

a
<y

,V
0

CJ

>§
a

-H
GH
-rH
CO
CO

0)

rd

l*H

r-H
O

U
CD

B
2

o
0
Q

>

-H
-P
rj
JH
(D
p
H

P
CX

0)

E

3

u
0
Q

p
-H

4J

X>

-H

CO

GH

M
CD

G

G
G

4->
CO

G
O

CO
-H
rH
fd

3

CO
-H
rH
fd
Oi

m
0

rx
a
c
u

rH
O

6

rH
CJ
EH

TJ

H
IH

0

£
CI)

«

-

CP

0
rd

G
•H

CO

X

E
M

CJ

P

TJ

CO

CJ
rH

G

Xi

H

s

•

M

<X

d abstrac

^
c
rH
u

G

P

-H

X
CD

CO
p

4-J

x.x
CJ

•44

03
-P

r

<V
G
,-H

P

^ XOJ

O

4J
rH
rH
r H Z. f d
3 • B
f H ^ C O

rH
=

-H

X

4J

CO
G)

CO nrd
XJ
r-C

Hi

a
0
u

a
c

CD
rX

:3

O
G)

co

-n

^J

T3

G

a
fd

•§
CO

fd

CO
CD
rH

CO
OJ
rH

P

P

•H

-H

H

EH

0

•H

GH
CO

G

H
CJ

CJ

G

4J

0

4J

P

rd
0,
CJ
0

G

O

rH

3
O
O
0

O'

G)
rH
144

s

CJ
p

a
fd
G

c

-H
-P
rd
•H
O

0

CO
CO
fd

X
CJ

1

n3

0

G

u

CO
-H

S

H

rH

B

Cn
G

fd

O

tr
G

-H
H
G)
4J
CO

z

z.

P

-H
GH
-H
CO
10
fd
rH

u

rH
O

p

TJ

H
O
z

z

CO
(J

'H
Cn

MH

3

•H
4-3
fd
^5
rH

a

4J
fd

rH

rd

u

U

CO

4J

jG

- G
l

CD

p

O

3
0

u
rd

a «' Oto -- Q

I

G)

H

0

«

a

s

P

a
0

u
CJ

c

£
3

U

=

0

JH

(^

*H

a)

CO
CO
G)

P

G
G)

CO
H
r:>
r-\

T3
Qj

GH
G4

Si

w

-H
4J

CD
CO

•f~\

>

0
rH
-p
-H
EH

CO

Cn
G

•H

CO
CO
CJ

CJ

CO

^

u

G

4->

X)

4J
U
G)

rd

G

-H
JH
CJ
4->
10

0

"0

rd
3

D>

^TJ
rd

^O

rd G

p ,

ings

eadin
W

tn

O
CO

c citatio

-

bib

strac

11

11 te
X3

ograp

ulary
d voc
trol
S

TD
G
rd

rC
CO

CD

X

o

c

=

n

rH

3

w

W

CJ

o

2

CO

Q)

fd

r-H
O

-H
rH
(D
Cq
X

X
13
G
H

rd
-rH
HH
•rH
CO
CO

i

X!

0

CJ
CO
rj

rH

CO

G

'TJ

-H
rH

ject

Q»

S £

G

CO
CJ

hing

CO

=

rd

lang

CO

0

G

Cn

ties and subj
hea

P
CO

X
o

4->
to

CJ
•d

•a

CO

XI

yword

u
CJ
CO
0)
rH
p
•r-\

Cn
G
-H
H
(D

-H
H

p

•H

fd

tura

M
0

>

B

CH

3
G

titl

co

W
•d

o
H

from abstra

assi
B

4J

tion

fd
*—?

u

P
X
CJ

T3
CJ

X

alys

CJ

P

fd

ghtin

G

3

rd

a)

rH

4->

edlars

()
en

X

>i
rH

ubject
index

M

>i
rH
cH
fd

3
rd
M
<D
,C
-P

144

nual

ned
rus
G

o

-H
P
0,
•H

C7>
•rH
CO
CO
fd

rom t

c

-H

cted

ce
•a

•a

r ter

les

rms from
X

CO
4J
O

CJ
C
Cn

-P
•H

CJ
4J

CO

4J

a

al iidexi

r>

m

>

O

G

CO

-P
0
rd
H
4->
to
.Q
rd

u
G)

CD

CO

-H
4J

a,
CO

-H
4J

>

rH

O
03

0

O

a<

u
P

CD
GH
GH

>

CD

cn

2

CD

B

W

rd
CO

0
CD
Oi
CO

ro

0

G

CD

•H
n
CO

u
'3
0
0}

1
0

0
Q

CO
4J

G
G)

LI

S

•
&
B

0

0
rj

rC
CO

O

^

0

u

c
<c a

rH
CM
OD

CM

a^
r^

0

CN

-:r

r>

0

<
CO
s

^
0*1

vo

UJ

^

<':

J

1

<c

<-K

0

0
0
r>

LO
rH

CN
ro

03

r>
ex?
OJ

CM

U

-H
GH

B

-H

G

»rj
QJ

fd
y 1

'
G
0
r ^

^

u

w

O
in

8

0
0

G
cd
r ^

4J
r<
'n.

O

UJ

rCvJ
rH

-H

CM

CM

- '-CO
rH

rd

0)

s

a.

m

rH
CO
CM

H
Ci
f"

w
rt
H

n^

CM
CD

CO

O

r>

8

CO

O
-H
CO
>1
rH

rH

f>
CM

eld

CO

-H
L;

ics

CO

p

CO

eld

try

a^

0)
0

-H
U

0

tics

&
rH

•H
ro
. 3
M

§
CM

W
K
H

O

m
G
rd

U

CO
rH
CJ
rH
IDCD

u

S

-r;f

\4
G

CO

ft

n:
w

<:
u

0

«

CO

CM

r*

v-r

co^ ' ^

CD

CM

a>

c

r>

co

<x>

CJ

8

G

EH

O

^H

rx

G

y.
G

^

O

r^

ro

G

CO

^

I

4J
CO

G)

I

i

f

LO
LTj

i

m
CM

rH

c
CM
fTl
0

^
(J
rM
CD
MH

CD
CX !

KD

X>\
CO
G4

0
P
•H

^

^

n

fd

r^

e
td

rd
T3

CO
4J
CO

r*

>i
-H

CJ^

CJ

^4

Xc

^

<c

rd
r-D

^
<

G
rd

x:
u

CJ

0

1

CO

p-

v

^
cn

CO
rH

1

rH

CM

ro

G)
P

r>

CO
rd

G

UD

CM

r-

r^
0
G

CM

n

4

CX

r72

CO
QJ

CO

O
r0
O^
rH
-H

>

0

r>

2
•H

ro

G
G
CO
CO

G

CJ

->.
CJ
Co'
>
CO
U

0U

G
fd

Cn

O

3
rj

^

CM

r>

rH

G
r^

rH

CJ
r ^

G

-H

:

z

B
G

-H
CJ

b

0
G
G

r>

x:
0
0 . v
u 0rH
0
u

2

rH
CJ
4J
CO
rd
O

cd
rd
r-H-

a)
CO

rH
f>

G

rj

fd

rH
rH
CD

Xi

&

LH

0
G

E

60
8
Q
o

9
^

2.

0 0 0 08 R

8 8 i* S*

X 5

I

I

I

I

3^

S^^S

6
r-i

CO

0

g S 3

9.

o

JN

5

o

5 8"*

0
0

•

°.

£5

jq
«->

i P.
O

O

O

O

Q

O

O

NO

ON

o

o

o

O
o

O
o

ON

ON

«r>

O

co

ON

%r\

VO

CO
ON

T*>C

CM

o

O
ON

CO
O

co

J

O

O

^

O
ON

<^

O

O

O

O

r~

r

H

O

O

O

O

N

O

8

vO

ft

O

O

O

o

d

d

O

ON

ON

CM
*C

$ £ ^

rH

o

«r>

0 0
o

0

£

£

8

£ &

NO

O- H-5 3o.J 22,

S

•»
L,

q
rH

o
O

»>
-K

^

M

3 2
W

8

5

.
•

8
fi
>s

V
U
P.

3

8

JS

o

vo

rj
8

t»

O

g 5 ^. ^ 3

-5

O
o

I

CO

,0

M

;*

^

ON

•

O

o

O

*»

O

*>
•

o

"d

*>

CD
CO

-4-

O

O

r-t

3 s s :

JS

I

I

O
O

O
O

O
O

O
O

O

O

O

O

KN

2

S

I

I

O

•-4
**

rH

CO

5

S

S °.

t

»H

-3
»
«

T)
«•
«

,-j •
*> U
tn m

U
>.

O
O

M
U
o

•
jq

u.

^J
t,
ai,

O
(-<

g

8 8 8 8

5

S

& 3

0
£

3

SN
O

O

O

0
o

•
<•

o

>*.

3

^

H I

^

O

g-2

ft S S

6

£

§

1

8

r-*

5

S

O

^

O

O

O

O

«r»

ft

C 5

5 I 8 V.
s ^
•(3

O

•

««

"i

* I?

g
O

«"\

J-

a .

O

O

O

O

f H H p- M

X
Q
LU
Q.

<

j l
i 5>

0.

g
- s. •*> « «
•

0

•J

P.

-J

0

«

X

"H

U.

0

0
2:

fr^

0

j

"<
• -

0

5

Z
B

c

c
.3
q

a)

15

&
O
r-i

c
.H
M

0
CI
£

4

*'

«0

rj

co

0

n

•<

«j
,-H
a

O

-^
«j
O

HI

C
=>
a

a)

->!

t>

x

H

f-|

H

* 2S325 2

•

?

So

•

'-; * I

3 % * s3 ^j
0
x

c

.3

0

l c*
c

-n

L,

U

S0
ft.

S

«

H

x: ft. W M
^

a

c

0

M

->
:

i

nl

0
^

3
(

o

*

^

col

~>

to

&H
u.
2
0

6
u

J2

ui

81

>

r

-

5

o

^

M

'

-

to
q

*>

o

U
3

i

i

-

>

w

a

«

V.

S.

?

ao

s

:

p

t
i

,

O

^

"

3

»

*

i

>

k

.

t^

>

o

u

n

«>

<

«

3

B

1

2

.. Z

4

a

08

E. a8
-

s

a .

c

«.-)

30

Dl
Sparck Jones 1 standard formatted level 4 collections are obtaineda) by processing the document descriptions and indexing vocabulary
automatically to delete stop words and generate stems with associated term
numbers; andb) by regularising the document, request and relevance judgement sets and •deriving basic files from them.
These files all conform to regular layout principles.
collection consists of files, or streams,as follows

Thus a standard

•

1/ 0

documents, with original document identifying numbers, and sorted
term numbers

1

documents serially numbered, with sorted term numbers

2

requests serially numbered, with sorted term numbers

3

relevance judgements, serially numbered, with sorted original
document numbers

4original document number - serial number equivalence list

5term dictionary, giving term numbers in serial = alphabeticalorder and alphabetically first variants in each word group witha common stem

6

term dictionary with words in alphabetical order, if cc responding
terms not serial

7

documents, with original numbers, and sorted term names

0

requests, serially numbered, with sorted term names

9

inverted documents, i.e. inverted stream 0

10

inverted requests, i.e. inverted stream 2

11

inverted relevance judgements, i.e. inverted stream 3

12

document frequencies, i.e. a list with the number of terms ineach document in stream 0

13

request frequencies, i.e. a list with the number of terms in eachrequest in stream 2

14

relevance judgement frequencies, i.e. a list with the number of
relevant documents for each reauest in stream 3

15

distribution data, giving the numberof items, maximum, minimum andaverage length, and length distribution of streams 1/0,2 and 3
and 2/0,1 and 2

2/ O

document term frequencies, i.e. a list with the number of documents
for each term in stream 91

request term frequencies, i.e. a list with the number- of requests
for each term in stream 10

2

document relevance frequencies, i.e. a list with the number ofrequests for each relevant document in stream 11

3

original request number - serial number equivalence list

* conventional numbering with historical rationale

40
D2

Collection innut data processing normally generates a variety ofother streams which, since they all conform to the common layout
conventions, constitute a natural extension of the basic standard collection.
These streams may include an alternative to
stream O, with within document frequencies of terms indicated1""
request3 with serially numbered documents
plus different sets of relevance judgements etc, listings of the full
dictionary, indicating truncation and grouping, and so on.
Note that a standard collection refers to a particular set of documents(and reauests) indexed in a particular way, i.e. to what may be called acollection version. Thus the Cranfield 1400 documents and requests indexedby manually assigned terms, and by terms automatically extracted from titles,
lead to two standard distinct collections. Also, when subsets of documents
and requests are selected, all the frequency information is different, sothese also generate distinct standard collections. Thus the Cranfield 200
manually indexed document collection is different from the 1400 one.
The particular form of standard collection just given is merely
illustrative. It is evident that more complex collections like the ideal
one(s), or collections with radically different characteristics, might
require more elaborate, or alternative, standard forms. But we feel theprinciple of standardisation is very important. Data formats, particularly
for operational systems, are not necessarily suited to research, so somemodification may be needed; but full-blooded standardisation is usuallymore convenient in the long run. We have certainly found standard
collections set up on the lines indicated very helpful. It should also
be pointed ou': that standardisation is a non-trivial operation, so there
are clear gains if it is done only once, in a competent way.

41
References

Aitchison, T.M., Hall, A.M., Lavelle, K.H. and Tracy, J.M. Comparativeevaluation of index languages, Part I, Design; Part II, Results,
Project INSPEC, Institute of Electrical Engineers, London, 1970
Akiyama, S. Automatic document classification systems, M.Sc. Thesis,
Department of Computer Science, University of Alberta, Canada, 1972
Augustson, J.G., and Minker J. Deriving term relations for a corpus bygraph theoretical clusters, Journal of the American Society for
Information Science, 21, 101-111, 1970
MarVor F.H., Veal, D.C. and Wyatt, B.K. Retrieval experiments based on
Chemical Abstracts Condensates, Research Report No 2, UKCIS,
University of Nottingham, 1974
llarber A.S., Barraclough, E.D. and Gray W.A. MEDLARS on-line search
formulation and indexing, Technical Report Series, No. 34, Computing
Laboratory, University of Newcastle upon Tyne, 1972
Cagan, C. A highly associative document retrieval system. Journal of theAmerican Society for Information Science, 21, 330-337, 1970
Chan, F.K. Document classification through use of fuzzy relations anddetermination of significant features, M.Sc. Thesis,Department ofComputer Science, University of Alberta, Canada, 1973
Cleverdon, C.W. , Mills, J., Keen, M. Factors .determining the performanceof indexing .systems, Vols 1 and 2, College of Aeronautics, Cranfield
1966
Feinman, R.D. and Kwok, K.L. Classification of scientific documents bymeans of self-generated groups employing free language, Journal of
the American Society for Information Science, 24, 382-396, 1973
Hansen, I.B. CA condensates as a retrospective search tool. A commentary,
Information Storage and Retrieval,9, 201-205, 1972/5
Horsnell, V. , Intermediate lexicon In .information science
Librarianship, Polytechnic of North London, 1974

School of

Jacquesson, A. and Schieber, W„D. Term association analysis, Information
Storage and Retrieval, 9_, 85-94, 1973
Jahoda, G. and Stursa, M.L.A comparison of a keyword from title index witha single access point per document alphabetic subject index,
American Documentation 20, 377-380, 1969
Keen, E.M. and Digger, J.A. Report of an information science index
languages test, Aberystwyth College of Librarianship, Wales., 1972
Lancaster, F.W. Evaluation of the MEDLARS demand search service, National
Library of Medicine, Bethesda, Md., 1968
Lancaster, F.W. Evaluating the effectiveness of an on-line natural
language retrieval system, Information Storage and Retrieval. _8
223-245. 1972

42
References (contd.)
Litofsky, B. Utility of automatic classification systems for information
storage and retrieval, Ph.D. Thesis, University of Pennsylvania, 1969
Lo, A.K. An automatic optimum iterative feedback document retrieval system,
M. Sc. Thesis, Department of Computer Science, University of Alberta,
Canada, 1972
Minker, J., Peltola, E., and Wilson, G.A. Document retrieval experiments
using cluster analysis, Journal of the American Society for Information
Science, 24, 246-260, 1973
O'Connor, J. Text searching retrieval of answer-sentences and other answerpassages, Journal of the American Society for Information Science,
24_, 445-460, 1973
Olive, G., Terry, J.E. and Datta, S. Studies to compare retrieval using
titles with that using index terms. SDI from 'Nuclear Science
Abstracts8, Journal of Documentation, 29, 169-191, 1973
van Rijsbergen, C. Further experiments with hierarchic clustering in
document retrieval. Information Storage and Retrieval, 10, 1974, 1-14
Salton, G. A 'theory of Indexing Regional Conference Series in Applied
Mathematics No. 18, Society for Industrial and Applied Mathematics,
1975
Salton, G. Dynamic .'information and llibrary processing, Englewood Cliffs,
N.J.: Prentice-Hall, 1975
Sparck Jones, K. Automatic indexing 1974,, Computer Laboratory, Universityof Cambridge, 1974 (OSTI Report 5193)
Svenonius, E. An experiment in index term frequency, Journal of the American
Society for Information Science, 23, 109-121, 1972
Tell, B.V. Retrieval efficiency from titles and the cost of indexing,
Information Storage and Retrieval, Jj 241-243, 1971
Vaswani, P.K.T. and Cameron, J.B. The National Physical Laboratory
experiments in statistical word associations and their use in
document indexing and retrieval, Publication 42, Division ofComputer Science, National Physical Laboratory, Teddington, 1970
Vickers, P. The costs of mechanized information systems, Directorate for
Scientific Affairs, OECD, 1974
Virgo, J.A. An evaluation of Index Medicus and Medlars, Journal of theAmerican Society for Information Science, 21, 254-263, 1970
Wilson, E. Report on Automatic Indexing Workshop, April 29-30, 1974 ,Computing Laboratory, University of Kent, 1974 (OSTI Report 5194)
Crouch, D.B. A clustering algorithm for large and dynamic document
collections, Southern Methodist University, Dallas, 1972.