Evaluation of Online Catalogues: an assessmentof methods
Micheline Hancock-Beaulieu
Stephen Robertson
Colin Neilson

British Library Research Paper 78
1990

British Library Research and Development Department

British Library Cataloguing in Publication Data
Hancock-Beaulieu, Micheline
Evaluation of online catalogues: an assessment of methods. (British Library research paper, ISSN 0269-9257; 78).
1. Libraries. On-line catalogues
I. Title II. Robertson, Stephen III. Neilson, Colin
IV. British Library, Research and Development Department
V. Series
025.3132
ISBN 0-7123-3226-X© T h e British Library Board 1990
The opinions expressed in this research paper are those of the authors and notnecessarily those of The British Library.
RDD/G/808

British Library Research Papers are published by The British Library Board anddistributed by The British Library Publications Sales Unit, Boston Spa, Wetherby,
West Yorkshire LS23 7BQ, UK. In the USA they are distributed by the American
Library Association, 50 East Huron Street, Chicago, Illinois 60611; in Canada by theCanadian Library Association, 200 Elgin Street, Suite 602, Ottawa, Ontario K2P 1L5;
in J a p a n they are distributed by Kinokuniya Co Ltd, P O Box 55, Chitose, Tokyo 156;and in India, Burma, Pakistan, Nepal, Bangladesh and Sri Lanka they are distributed
by Arnold Publishers (India) Private Ltd, A B / 9 First Floor, Safdarjung Enclave, New
Delhi 110029.

Printed in Great Britain by The British Library.

ii

Abstract
A project investigating tools, techniques and methods for the evaluation of interactive
library catalogues is described.
Methods used in other projects are reviewed. The emphasis of the investigation
was on diagnostic methods, and on the use of the catalogue in the wider context ofuser information seeking behaviour. Studies on users of the City University Librarycatalogue indicated that (a) traditional transaction logging is quite inadequate as amethod of evaluation; (b) various enhancements can be introduced to greatly increase
the usefulness of transaction logging; (c) a fuller understanding of the processes
involved requires study of the user's behaviour away from the catalogue.
Useful enhancements to transaction logging include:a) full-screen loggingb) a playback facilityc) pre- or post-search, online or offline questionnairesd) interactive, in-search questionnairese) talk-aloud.
A front-end system (Olive) has been developed to allow the first four enhancements.
The experiments also emphasise the extent and importance of subject-searching orhybrid searching activity, as opposed to specific item searches.

iii

Dr. Micheline Hancock-Beaulieu is Research Co-Ordinator at the Centre for Interactive
Systems Research, Department of Information Science, City University. Her researchhas been concerned with information seeking behaviour, evaluation methodology andinformation systems for the Humanities.
Professor Stephen E. Robertson is Head of the Department of Information Science inCity University and Director of the Centre for Interactive Systems Research in theDepartment. His research interests are in probabilistic models, relevance feedback,and the evaluation of IR systems.
Colin Neilson has previously worked on projects on Bibliographic Control of Computer
Files and on Classification for Community Information. His research interests include
computerisation and subject indication in working with the BLISS C L A S S I F I C A T I O N
ASSOCIATION.

iv

Contents
1 Introduction

12 Approaches to Evaluation for Library Catalogues

53Methodology

154

Olive—an enhanced logging facility

235 Results

35

6Assessment of Evaluative Methods

59

7

Future directions

63

References

65

Appendix A

69

Appendix B

73

Appendix C

79

Appendix D

83

Appendix E

87v