Okapi at City
An evaluation facility for interactive IR

Stephen Walker and Micheline Hancock-Beaulieu
with contributions byAy§e Goker
Lee McCluskey
Scarlett Palmer

Centre for Interactive Systems ResearchCity UniversityLondon
August 1991

British Library Research Report 6056

© British Library Board 1992

Contents
1Introduction
1.1 Introduction
1.2 Background
1.3 Development of Okapi at City University
1.3.1 The PCL Okapi system
1.3.2 Parameterization
1.3.3 Updating procedures
1.3.4 Databases
1.3.5 Logging
1.4 Installation and use of Okapi at City
1.4.1 Installation
1.4.2 Use66677899101010112Q u e r y expansion in a library catalogue
2.1 Introduction
2.1.1 Automatic query expansion
2.1.2 Installation of catalogue system
2.2 Background2.2.1 Library database
2.2.2 Library setting
2.2.3 Network access
2.3 Evaluation methodology
2.3.1 Transaction logs
2.3.2 Search replays
2.3.3 Pre-search and post-search questionnaires
2.3.4 Post-search interviews
2.3.5 Structured interviews of network catalogue users . . .
2.4 Findings
2.4.1 Overall usage of query expansion facility
2.4.2 Effectiveness of query expansion
2.4.3 Perceived usefulness of query expansion facility . . . .
2.4.4 Search intentions
2.4.5 Replay of searches with "MORE" option not used . .
2.4.6 Previous experience with online catalogues
2.4.7 Ease of use
2.4.8 Help facility

1313
1314
14
14
15
15
15
15
16
16
17
1718
18
18
1920
20
21
21
22I

2CONTENTS
2.4.9 User satisfaction with search outcome
2.4.10 Browsing references2223

3Transaction log analysis
3.1 Introduction
3.1.1 Source of the data
3.1.2 Sessions and searches
3.1.3 Transaction logs
3.2 Sessions and searches
3.2.1 Aborted searches
3.2.2 Failed searches
3.3 Records retrieved, displayed and chosen
3.3.1 Numbers of records retrieved
3.3.2 Effect of number of records reported on record display
3.3.3 Full records
3.3.4 Relevance judgements
3.4 Query expansion
3.4.1 Takeup of query expansion option
3.4.2 Results of query expansion searches
3.4.3 Source of records displayed and chosen
3.4.4 How useful was query expansion?
3.5 System use patterns
3.6 Differences between classes of users
3.7 Best match searching242424252527272828282930313232333434363637

4F r e q u e n t users
4.1 Introduction
4.2 Obtaining the data
4.2.1 Background information about users
4.2.2 Individuals' use of language
4.2.3 Search language in general
4.3 Analysing the data
4.4 Results
4.4.1 General points about frequent users
4.4.2 Suggestions for system modifications

39393940414141424242

5Towards an adaptive I R system
5.1 Introduction
5.2 The probabilistic model for an IRS
5.3 The learning component
5.3.1 Inputs to the learner
5.3.2 Formulating and evolving the context
5.4 Testing context formulation
5.4.1 The Okapi system
5.4.2 An example
5.5 Conclusions and future work

4444
4546
46
47
49
49
49
51

CONTENTS

3652
52
53
5455
55
57

Conclusions
6.1 Automatic query expansion
6.1.1 Query expansion in the library system
6.1.2 Query expansion in searching INSPEC
6.1.3 Heuristics for query expansion
6.1.4 Query expansion: conclusions
6.2 Towards adaptive IR systems

A Okapi system description

59

B Okapi transaction logs

71

C Questionnaires
C.l Questionnaires for library users
C.l.l Exhaustiveness of search
C.l.2 Previous experience
C.l.3 Post-search questionnaire
C.l.4 Structured interview
C.2 Questionnaire for network users

7777
77
77
78
78
79

D Notes on a frequent user80

List of Figures
3.1

Transaction log summary

26

A.l Welcome screen
A.2 Information window for welcome screen
A.3 Information offered after user identification
A.4 Search input screen
A.5 Information offered from search input screen
A.6 Search results screen
A.7 Replacement of misspelt word
A.8 Brief record display
A.9 Options from brief display
A. 10 Full record display
A . l l Brief record display with "MORE" option
A.12 Results of query expansion
A. 13 Brief display from query expansion search
A.14 Record from query expansion
A.15
A. 16 Records from second iteration of query expansion
A.17
A. 18 Options after query expansion
A.19 Choosing the PRINT option
A.20 Editing a search
A.21 Full record display: INSPEC
A.22 Brief record display: INSPEC

459
60
60
61
61
61
62
62
63
63
64
65
65
66
67
67
68
68
69
69
7070

List of Tables
2.1 Availability and usage of query expansion
2.2 Usage of the u MORE" option and search sessions
2.3 Items selected with and without query expansion
2.4 Items displayed and selected before and after query expansion
2.5 Items selected per search after query expansion
2.6 Search intentions
2.7 Query expansion in replayed searches
2.8 Users'experience of online catalogues
2.9 Ease of use of Okapi
2.10 Ease of use of Okapi compared to CLSI
2.11 User satisfaction
2.12 Perceived number of references looked at181819191920212121222223

3.1
3.2
3.33.4
3.527282930

Sessions and searches
Distribution of number of records retrieved
Percent of non-failed searches in which no records were displayed
Action following search with no records displayed (CAT1 set)
Full records displayed and chosen: searches in which somebrief records were displayed
3.6 Full records displayed and chosen: searches in which some fullrecords were displayed
3.7 Takeup of query expansion by search result
3.8 Takeup of query expansion by number of records chosen . . .
3.9 Records displayed and chosen from query expansion
3.10 Sources of displayed and chosen records: CAT1
3.11 Sources of displayed and chosen records: INSPEC
3.12 System use Jan-Apr 1991: all identified users
3.13 System use Jan-Apr 1991: people who had used system before Jan 1 1991
5.1

31313333343536373750o