Karen Sparck Jones

Butterworths

Information Retrieval Experiment

Information Retrieval
Experiment
Edited by

Karen Sparck Jones

ForCYRIL CLEVERDON
from whom we have all learnt a lot

Butterworths
London Boston Sydney Wellington Durban Toronto

All rights reserved. No part of this publication may be reproduced or transmitted in any formor by any means, including photocopying and recording, without the written permission of thecopyright holder, application for which should be addressed to the Publishers. Such written
permission must also be obtained before any part of this publication is stored in a retrieval
system of any nature.
This book is sold subject to the Standard Conditions of Sale of Net Books and may not be resold in the UK below the net price given by the Publishers in their current price list.
First published 1981
© Butterworth & Co. (Publishers) Ltd, 1981

British Library Cataloguing in Publication Data
Information retrieval experiment.
1. Information storage and retrieval systems - Evaluation
I. Jones, Karen Sparck025'.04
Z699
80-42222
ISBN 0-408-10648-4

Typeset by CCC in Great Britain by William Clowes (Beccles) Limited,
Beccles and London
Printed and Bound by Robert Hartnoll Ltd, Bodmin, Cornwall

Contributors

Elizabeth D. Barraclough
Computing Laboratory
University of Newcastle upon Tyne

Nicholas J. Belkin
Centre for Information ScienceThe City University

William S. Cooper
School of Library and Information Studies
University of California at Berkeley

Lynn Evansformerly INSPEC,
Institution of Electrical Engineers

Michael D. Heine
Department of Librarianship
Newcastle upon Tyne Polytechnic

E. Michael Keen
College of Librarianship Wales

F. Wilfrid Lancaster
Graduate School of Library Science
University of Illinois at Urbana-Champaign

Robert N. Oddy
Computer Centre
University of Aston in BirminghamCornells J. van Rijsbergen
Department of Computer Science
University College Dublin

Stephen E. Robertson
Centre for Information Science
The City University

Gerard Salton
Department of Computer ScienceCornell Universityv

vi

Contributors

Karen Sparck Jones
Computer Laboratory
University of Cambridge
Jean M. Tague
School of Library and Information Science
University of Western Ontario

Contents

Chapter 1 Introduction
Karen Sparck Jones
Part 11Testing in general7

The methodology of information retrieval experiment
Stephen E. Robertson

93Retrieval effectivenessCornells J. van Rijsbergen324

Ineffable concepts in information retrieval
Nicholas J. Belkin

445The pragmatics of information retrieval
experimentation
Jean M. Tague

59

Types of test

103

Evaluation within the environment of an operating
information service
F. Wilfrid Lancaster

105

Chapter 2Part 2
Chapter 6

7

Opportunities for testing with online systems
Elizabeth D. Barraclough

128

8

Laboratory tests of manual systems
E. Michael Keen

136

9Laboratory tests: automatic systems
Robert N. Oddy

156

10 Simulation, and simulation experiments
Michael D. Heine

179

11

Gedanken experimentation: An alternative to traditional system testing ?
William S. Cooper

199vii

viii

ContentsPart 3

Chapter 12

Actual tests211

Retrieval system tests 1958-1978
Karen Sparck Jones

213

13 The Cranfield tests
Karen Sparck Jones

256

14 An experiment: search strategy variations in SDI
profilesLynn Evans

285

15 The Smart environment for retrieval system evaluation—advantages and problem areas
Gerard Salton

316

Bibliography

330

Index343