R E P O R TONAD E S I G NFORTHE'IDEAL1T E S TK.

STUDY

I N F O R M A T I O N

C O L L E C T I O N

S p a r c k

R. G.

R E T R I E V A L

J o n e s

B a t e s

Computer L a b o r a t o r y
U n i v e r s i t y of Cambridge
Corn Exchange S t r e e t
Cambridge

fc\\^S
October 1977

SummaryThis Report presents the findings of a Design Study for the 'Ideal1
Information Retrieval Test Collection.
Part A, Design, covers a detailed collection specification, an
investigation of sources of document material and methods of obtaining
requests and relevance assessments, and estimates of the costs ofbuilding various versions of the collection. The conclusion is that acollection consisting of a main set of 30,000 scientific documents and
750 requests with adequate relevance assessments, plus a supporting setof 3000 social science documents with 250 requests and assessments,
providing a range of characterisations for the documents and requests
but not citation data, could be provided in a two year buildincr programme
for about £85K. A similar collection with citation data for the documents
could cost £94K. One with more supporting document sets and their own
requests and assessments, but without citations, could cost E109K, anda collection with this range of sets and citations could cost £123K.
In Part B, Uses, information on possible uses of the 'ideal1
Collection collected as part of the Design Study work is summarised, toallow some evaluation of the proposed collection as a tool for worthwhile
research and teaching.
The collection design and costings were considered by the Study
Project's Advisory Panel, and in Part C, Discussion, the main comments
made by the Panel are noted. Consequent chanqes to the initial designand costings are indicated, and a possible cut-price collection costing
perhaps £55k is presented. Professor Cleverdon has questioned the needfor a specially-built test collection, and more specifically argues thatsufficiently useful test material mav be obtained as a byproduct ofoperational system investigations. We therefore examine the relationshipbetween a purpose-built 'ideal' collection and an 'incidental' one, inthe context of the types of information retrieval research which may bydesirable; and conclude that unless very severe restrictions are placedon the kind of research done, an 'ideal' collection, if only a cut-priceone, is needed for effective and efficient research.

Acknowledgement
The Desiqn Study was carried out under British Library Research andDevelopment Department Grant SI/G/231. We are very grateful to the many
people who provided information for the Study.

Contents
PART A : DESIGN
1.

Objectives, scope and conduct of the Design Study2.

Summary of the collection specification input to the Design Study

3.

Conduct of the Study

4.

Document sets

5.

Request sets6.

Relevance judgement sets

7.

Document and request characterisations

8.

Citations9.

Summary of data base selections

10. Possible uses of the 'ideal'collection
11. Costs

PART B ; USES
1.

Research projects

2.

Teaching activities

PART C : DISCUSSION
1.

Comments on the Study2.

Retrieval experiments

References

Appendices1.

List of reference works consulted

2.

Sample entry from Williams and Rouse' Data Base Directory

3.

Example of data base questionnaire as sent out

4.

List of CA and CAB subbases

5.

Tabulated data base questionnaire replies

6.

CAB subbase sizes

7.

Analysis of relevance judgement requirements

8.

Research

9.

Teaching and on-line education questionnaires

roject questionnaire