X m p ro v ± n gi n2.

^3 L_J tz> j j e? cc " to n l i n eRelevancer^epizr^iL^v^^Lcc a a ^ aa L CD eg L_J €E? ifa

feedbackand5 tephenqueryexpansion

WaLkerandRachelBritish

LibraryDe

VereResearch

Paper

72

1383

British Library Research and Development

Department

R r ep f* aa cz eeThis report is about improving subject access in computerized
bibliographic information retrieval systems which are intended for end
users rather than for information specialists. It contains a descriptionof some retrieval systems which were used to provide subject access to a
Library catalogue database, and of some experiments which were doneusing these systems. It reports the most recent of a series of projects
the Okapi projects - which began in 1982 and have been funded by theBritish Library Research and Development Department and hosted by thePolytechnic of Central London.
Gill Venner
Gill Venner, one of the team of three which designed, developed andreported the original Okapi system during 19B3-1S85, died while thisreport was being written. The report is dedicated to her memory.
Rcknowledgment sThis work would not na^e been possible without the help and support ofthe British Library Research and Development Department. Particularthanks are due to Karen Merry and Derek Greenwood, our present andformer project officers. We are also mast grateful to Neil McLean
CProject Head and formerly Director of Information Resources Services,
Polytechnic of Central London], and to the staff of the Polytechnic's
Computing Services.
People who gave valuable advice on the design of the retrieval systems
and/or the experiments include Win.' ec Rbbott, Maura Coghlan, Ellen
Gredley. Charles Hildreth, Mark Lansdale, Martin Porter, Stephen
Robertson and Peter WiLlett. Efthimis Efthimiadis, Michel.me
Hancock-Beaulieu, Colin Neilscn and Stephen Robertson read portions of
the manuscript and made valuable comments and suggestions.

Credits
It would not have been possible to design and program the experimental
retrieval systems used in this project without the foundation provided
by previous Okapi systems. Major contributors include Nicky Johns,
Richard Jones, Nathalie Mitev, Julie Porteous, Gill Venner and Stephen
Walker. The evaluation experiments for this project were designed andcarried out by Rachel De Vere as part of the work for her MSc inInformation Technology at Loughborough University. Parts of Chapters 3,4 and 5 of this report are adapted from De Vere's MSc dissertation.
Stephen Walker
City UniversityRachel De Vere
University of Sheffield-in-CCDNTENTS

1INTRODUCTION

11.1
1.2
1.3
1.4
1.5
1.6
1.7

The Okapi projects
Notes on terminology
Subject access in online catalogues
Subject access in Okapi '86
Query expansion
Feasibility studies
Development of the experimental system Okapi '88

11255672

QUERY MODIFICRTION THROUGH RELEVANCE FEEDBRCK

92.12.2
2.3Retrieval techniques
Term weighting and relevance feedback
Query modification
2.3.1 Sources of terms for query modification
2.3.2 Weighting and term selection
Query modification in end user retrieval systems
2.4.1 Term selection by the user
2.4.2 The CITE catalogue at the NLM
2.4.3 The online catalogue at the Scott Polar ResearchInstitute
2.4.4 The INSTRUCT system at the University of Sheffield

911131314141415

SYSTEM DESIGN a DESCRIPTION

132.4

3
3.1
3.2
3.3
3.4
3.5

Introduction
The bibliographic database
Hardware and software
Dialogue styleObtaining and processing user input
3.5.1 Obtaining input
3.5.2 Input preprocessing
3.5.3 Parsing and lookup
3.6 Search processing
3.5.1 Term weighting and the merge
3.6.2 Initial search results and options
3.7 Record displays
3.7.1 Brief record display
3.7.2 Full record display
3.8 Obtaining relevance judgments
3.3 Query expansion
3.9.1 Term extraction and weighting
3.9.2 Selection of terms for query expansion
3.9.3 Query expansion search - screen display and options
3.10 Shelf-order browsing
3.11 Other options
Screen illustrations

1617131920202020212122232424252627272829293032

EVRLURTION
1 Introduction
2 Formal, experiments
4.2.1 Rn example
4.2.2 Formal experiments with interactive systems
3 Experiments with real users
4.3.1 The NLM online catalogue comparison
4 Evaluation techniques for highly interactive systems
5 Rims of the evaluation
B Planning the evaluation
7 Performance measures
4.7.1 Recall, precision and efficiency
4.7.2 Relevance, recall and precision in the presentexperiment4.7.3 Transaction Log analysis
4.7.4 Users' opinions
B Method
4.8.1
4.8.2
4.8.3

3

Subjects
Tasks
Experimental

procedure

Objective relevance assessmentsRNRLYSIS & RE5ULT512345679Introduction
5.1.1 Terminology
5.1.2 Source and processing of data
General comparison of the systems
5.2.1 Statistics on efficiency and effectiveness5.2.2 User opinions on system ease and helpfulness
5.2.3 User opinions on system usefulness
Use and performance of the query expansion and classificat
browsing facilities
Query expansion in detail5.4.1 Statistics
5.4.2 QuaLity of Lists of records from query expansion
5.4.3 Users' comments on query expansion
Classification browsing in detail5.5.1 Statistics
5.5.2 QuaLity of Lists of records from classification
browsing
5.5.3 Users' comments on classification browsing"Objective" precision of chosen records
User comments about the systems
5.7.1 Choice of search terms and retrieval of non-relevan
records
5.7.2 Record content and display; the recognition ofrelevance
5.7.3 Problems connected with the list of chosen records
5.7.4 General interaction and presentation
How people searched the systems-vi-6

CONCLUSIONS & RECOMMENDATIONS

73

6.1
6.2

Performance of the experimental search systems
Query expansion
6.2.1 Towards heuristics for offering query expansion
6.2.2 The collection of relevance information
6.2.3 Computational implications of query expansion
Classification browsing
5.3.1 Why classification browsing was relatively unsuccessful
6.3.2 Classification browsing in a live system
6.3.3 R note on synthetic Dewey numbers
Further work
6.4.1 When and why does automatic query expansion work?
5.4.2 Rutomatic query expansion with non-catalogue databases
6.4.3 Catalogue systems with enhanced records
6.4.4 Towards highly interactive systems: user monitoring asa tool for guiding interaction
Concluding remarks

7373
74
75
76
71
77
78
73
80
80
80
81

6.3

6.4

6.5

81
82

RPPENDIXES
1
234
5
678

File structures
Input processing
Questionnaires
Task sheets
Annotated extracts from a log file
Assessors' instructions
Class browsing results
Example searches
1 Extracts from a live search en the query expansion system
2 Notes en a search of the "full" system

REFERENCES

83
85
36
31
35
35
101
104
104
107
'10-vii-