1Introduction
Karen Sparck Jones

This book is about information retrieval experiment. Documentary information systems have changed in many ways in the last 20 years. The rapid
growth of specialized literature has encouraged an intellectual development,
post-coordination, and a technological development, the use of computers.
Many questions about effective methods of document identification, andabout efficient methods of document management, have naturally followed.
These questions about the substantive and the economic aspects of retrieval
systems have provoked a whole range of studies. Some of these may bedescribed as investigations; others can properly be described as experiments.
They are sometimes associated with generalizations or sometimes, morestrongly, with theories or models.
These studies taken together have produced some moderately solid results,
and have to some extent enlarged our understanding of the way retrieval
systems work. For example it appears that good retrieval performance lies in
the 40-60 per cent recall and precision area; and it seems that someprobabilistic models offer valuable insights into system behaviour. But
research progress has perhaps been less than might have been expected, andinformation system practice has in essentials been extremely conservative.
The 1958 International Conference on Scientific Information, held in
Washington, was widely felt to mark the beginning of a new era in
information processing. The novel ideas and techniques to be developed
were symbolized by the 'auto-abstracts' of conference papers produced byLuhn. By 1978 computers were firmly established in information work, but
primarily, and almost entirely, for clerical operations: the crucial information
processes of document and request characterization and matching are doneby human beings along conventional lines.
There is no very good reason to suppose that the conventional methods arebest, even in principle, let alone practice. There is in particular no goodreason to believe that they are based on any thorough understanding of thenature of information systems. Information systems are manifestly complex.
It is by no means certain that information science exists: but it is clear that
information processes need scientific study. The requirement for, and role of,experiments in such a study is clear. A good deal of experimental andinvestigative work has been done in the last two decades; but while results
12Introduction

have been obtained and experience gained, a review of this research serves
mainly to show how much more needs to be done. Individual experiments arestill far too often methodologically inadequate, in some cases in obvious
matters of design, in others in less obvious but nevertheless very importantmatters of scale.
Over the years, the character of experimental work in information retrieval
has changed. Many early tests focused on the new information processing
methods associated with automation: they were concerned either with theapplication of post-coordinate searching using manual indexing, or more
ambitiously, with wholly automatic indexing techniques. However the main
outcome of this work was the discovery that nobody really knew, in detail,how document retrieval systems behave or more importantly, why they
behave in the way they do. Many test results were unexpected or ambiguous,and it became evident that this was due to weak experimental design, which
was in turn a consequence of unexamined assumptions about retrieval
systems. The need for a better standard of experiment, preferably informedby an explicit characterization of system properties, has been slowly
recognized, and the quality of experiments has improved. Some research
workers have in particular been able to capitalize on previous work throughthe use of common data, and also through the application of common
techniques for system performance measurement. Thus although specifictests may have been done within the framework of particular operational
systems and their assumptions, much of the research that has been done hasbeen devoted to a painstaking analysis of typical system behaviour, over awide range of factors.
These developments are in many ways illustrated by the objectives of theCranfield 2 test, its results, and its subsequent influence on information
retrieval research. The test was designed to compare the performance ofdifferent indexing languages, including simple natural language, in alaboratory environment. It was intended to improve on Cranfield 1 inexperimental design, and was systematically conducted. The most striking
result, the competitive performance of post-coordinate natural language
terms, was not expected, but has been largely supported by subsequentexperiments. The test also indicated that only medium levels of performance
could be expected of retrieval systems. The inverse relationship between
recall and precision was clearly displayed, and subsequently adopted as ageneralization about retrieval systems, if not as a law.
The test was criticized for methodological inadequacies, for example in
the way the test data was generated, and for being too small in scale. Further,while the test's concern with index languages and their application wasclearly focused on the centre of retrieval systems, important factors, especiallythose involving users, were not studied. It has also been argued that recall
and precision have been overvalued as measures and, more generally, thatthe bottom-up approach to the understanding of retrieval systems represented
by Cranfield-type experiments may be unproductive or misleading, and thata top-down approach, guided by a theory or model, is preferable.
The impact of the Cranfield 2 test on later research has nevertheless been
considerable. Specific projects have applied Cranfield procedures and used
the Cranfield test data for comparative purposes. More broadly, much of theexperimental research done since Cranfield 2 has been in the same tradition,

Introduction

3and addressed to similar topics, though with more emphasis on automatic
natural language indexing techniques, and more interest in a wider range ofsystem factors. Moreover, in other work motivated by the alternative
approach, seeking to base experiments on theory and so explain rather thansimply exhibit system behaviour, the influence of Cranfield is often visible.
Thus while much experimental work is explicitly indebted to Cranfield 2,
much more has been coloured by it, This book is rightly dedicated to Cyril
Cleverdon.
The Cranfield test showed what hard work experimental information
retrieval is: collecting data, conducting searches, computing performance
figures all take substantial time and effort. There is work in the essentials of
experiment: maintaining systematic procedures, imposing controls onvariables; and there is work in the realities of research: varying approaches
in the search for understanding and explanation, and rehashing past studies
for consistency and solidity. Salton (Chapter 15) has noted that the Smart
Project has involved thousands of tests, implying a large use of man andmachine power; yet there are important aspects of retrieval system use andbehaviour which have not been studied in these tests.
Since information is claimed to be of increasing importance, at least in
twentieth century high technology cultures, while information systems arelittle understood, there is every justification for information system
investigation and experiment. Existing information systems reflect long andextensive experience, but it does not follow that such systems cannot beimproved, and that new technologies may not be exploited for wholly new
types of system. It is arguable that our current understanding of information
processing is like that of sixteenth century herbalists: it embodies someobservation and insight, but lacks detailed analysis and supporting theory.
Unfortunately, though many retrieval experiments and investigations
have been carried out in the last 20 years, much of the experience in theconduct of tests which has been gained is not very accessible. Publishedpapers tend to be cleaned up accounts of objectives, general methods, andresults. Project reports may be much fuller, but even here it is often extremely
difficult to find out exactly what was done, or why it was done. Though an
improvement in the quality of experiments is detectable, far too many of
those reported are defective, and in many cases defective in recognized ways,
for which remedies are available. As all but the most limited test is a major
enterprise, it is a pity that so much effort should be wasted. The best that canbe said about many reported studies is that even if they are individually
dubious they may collectively point in the same direction. This is perhapssomething, but it is not much. The object of this volume is to make available
the experience in information retrieval testing of its contributors, in the hopethat this will lead to more fruitful and useful testing in the future.
The book is designed to treat information retrieval experiment andinvestigation in a comprehensive way, relevant to both pure research and tooperational practice. Pure research naturally leads to experiment, but system
operators may also want to know how their system is working. Both research
workers and practitioners may know what they want to find out, but it may
not be at all obvious how it is to be found out, and in particular what the best
specific testing methods are. This is clearly seen in evaluation tests. Most
tests are intended to evaluate the performance of existing or proposed

4Introduction

retrieval systems or their individual components. A good deal has been
written about evaluation: what aspects of system performance, under thegeneral headings of effectiveness and efficiency, should be evaluated andwhat form the evaluation should take. Evaluation implies experiment, or atleast observation. But what to do and how to do it are distinct. For example,
there is a difference between an interest in studying a method of indexing,
evaluating it via recall, and actually carrying out the study by having certain
documents indexed by certain people and searched in a certain way to satisfy
certain needs. The general assumption tends to be that if you know what you
want to evaluate, with given evaluation criteria, the appropriate experimentis obvious. Experience shows that this is not the case, because thecharacteristics of retrieval systems are so difficult to determine and theirimplication for experiment so difficult to identify.
Much has nevertheless been learnt in the last 20 years about the conductof retrieval system tests; and this book attempts to bring the varied
experience of its contributors to bear on all aspects of retrieval experiment.
Part 1 deals with general topics applicable to all experiments andinvestigation: methodological issues, the relation between testing andevaluation, the problems presented by inaccessible system factors likemeaning, and the proper provision of data. Part 2 covers different types oftest: real-life tests and laboratory tests, with a separate treatment of manualand automatic systems in each case; simulation tests, and gedanken
experiments. Though the different types of test have much in common, theypresent distinct problems and demand different approaches in testing. InPart 3, specific retrieval tests are used to illustrate and amplify the points
made in the previous sections. Chapter 12 considers the test history of the last
20 years through major and representative tests; Chapter 13 examinesCranfield 1 and 2; Chapter 14 analyses a specific experiment in detail, andChapter 15 the long-term Smart Project. In the book as a whole, the emphasis
is on a detailed treatment of the issues involved in information retrieval
testing, and especially on the less obvious problems occurring in the designand conduct of tests. Throughout, examples are used to illustrate the points
made.
It will be evident that the theme of the book is a large one, with many
facets. The chapters moreover embody individual views of their specifictopics. For this reason, it is appropriate to conclude this Introduction with
the interpretations of the key terms used which underlie all the chapters.
These terms have been used in these senses so far, but to lead into Part 1 theyneed an explicit rather than implicit characterization. However, since thecharacterization is filled out by the book as a whole, all that is required here
is a summary indication of the use of the key terms informing the different
chapters.
Thus as this book is about information retrieval experiment, we need tosay what is meant by 'experiment', and how it is related to 'investigation'. An
experiment is designed to answer the question 'What happens if you do X?',
or 'What happens if you do X rather than Y?\ An investigation is designed
to answer the question 'What happens in System S?', or 'What happens, ingeneral?', or, more tentatively, 'What might be happening in System S or ingeneral?'; most modestly, an investigation is designed to answer the question'What data can we get which may tell us what might be happening?'

Introduction

5We can think of experiments as suggesting methods of investigation, andinvestigations as influencing the design of experiments. Experiment istypically hypothesis-guided, investigation hypothesis-generating. However
we must allow that the hypothesis underlying an experiment may be implicit
rather than explicit, and, further, may be of the weakest kind, namely 'Thisvariable is important and therefore worth study'.
But this difference between experiment and investigation is not thesignificant one: the essential difference between experiment and investigation
is in the application of control in experiment. One cannot test without
control. The problem in experiments is then to control variables and not tosuppress them.
Experiment implies, and investigation typically involves, measurement
and specifically, since an information retrieval system has a purpose,measurement related to system merit in some sense. However investigations
may be summed up in measurements of a descriptive rather than evaluativekind. Moreover both experiments and investigations may be concerned withindividual system elements only, and with direct measurements of these, soperformance measurement proper is indirect rather than direct.
Finally, although experiment and investigation may in principle referequally to laboratory or operational system studies, in practice there havebeen few operational system experiments, and experiment and investigation
tend to refer to laboratory and operational studies respectively.
In practice, this distinction between experiment and investigation may bedifficult to apply to work that has been done in information retrieval,particularly in relation to operational systems. Thus while focusing on
experiment, the book does include discussions of better-conducted and more
systematic investigations. Equally, many of the points applicable toexperiment are applicable to investigation. The word 'test' is therefore used
as a global, neutral term to cover both experiment and investigation, and also
as a stylistic variant to refer to either when the context makes the particularinterpretation clear.