An operational evaluationof weighting, ranking and relevance feedback
via a front-end system
S.E. Robertson
C.L. Thompson
Department of Information ScienceThe City University
Northampton SquareLondon EC1V OHB, U.K.

Final report to theBritish Library Research and Development Department
on Project Number SI/G/703(January 1985 - June 1987)
July 1987

Suggested keywords:
information retrieval, weighted searching, ranked output, relevance
feedback, front-end systems, Boolean searching, evaluation

Abstract

Cirt is a front-end system which allows Weighted searching (i.e. searchterm weighting, ranking of output documents by matching value, andmodification of the weights by relevance feedback) on a Boolean host.
In this project, Cirt was first modified to improve its usability, andthen used for a comparative evaluation of Weighted versus Boolean
searching in an operational environment. Searches were conducted byexperienced intermediaries in the presence of the end-users, in three
University of London institutions as well as at City University, mainlyon the Medline or Inspec databases. Each user was randomly allocated toeither Weighted or Boolean searching. Evaluation parameters includedsubjective reactions of the user and the intermediary and cost-related
factors as well as the more traditional relevance-related parameters.
Because of various delays in the early stages of the project, because
the independent-sample (as opposed to matched-pair) design of experimentrequires large samples, and because the differences between the systems
are generally small, few of the results obtained were statisticallysignificant. The implementation of Weighted searching in a front-end isa limiting factor, as it does not allow many terms to be used or any
form of query expansion. Nevertheless, it appears that Weighted
searching is a feasible way to do real searches, and that it gives
results which are comparable to those obtained from Boolean searching.

Preface

This report describes the second of two projects supported by theBritish Library, concerning the front-end system Cirt. The firstproject was concerned with the development of Cirt; this second project
has been aimed at evaluating weighted searching via Cirt underoperational conditions.
The project would not have been possible without the help of anumber of people. John Bovey, Mike Macaskill and Helen Mickleburgh wereconcerned at various stages with developing and maintaining the system.
Alina Vickery and the staff at the University of London Central LibraryServices were very helpful in setting up the contacts with the otherLondon University institutions. A number of such institutions,particularly the School of Hygiene and Tropical Medicine and Middlesex(and also the City University Skinner's Library), gave us the publicity
which we needed for the search service offered at City as part of theproject. A number of other institutions expressed interest in theproject, and had to withdraw from taking part for a variety of practicalreasons. Inspec provided some royalty-free searches on their databases.
Particular thanks are due to the intermediaries who undertook all
the searching and liaison with users, and contributed in this way a very
substantial amount of their valuable time to the project. Apart fromCatherine Thompson, these were:
Alain Besson
Sheila Dibley
Elizabeth Lyon

St. Bartholomew's
Imperial College
St. George's

Stephen Robertson
July 1987

Contents

Abstract
Preface
Contents
1. Introduction

2.

1.1.
1.2.
1.3.

The theoryAims
Functional description of Cirt

1.4.

Technical description of Cirt

1.5.

Structure of the report

Refinements and modifications to Cirt
2.1.

Initial work

2.2.
2.3.

Two-process system
Refining useability
2.3.1.
2.3.2.
2.3.3.

Search tree
LimitsDeleting

2.3.4.

Adding terms offline

2.3.5. Saving searches
2.3.6. Look mode
2.3.7. Printing offline
2.4. Discussion: Cirt and the relevance feedback model
2.4.1. Limits
2.4.2.

3.

Ignore2.4.3. Print off-line
2.4.4. Delete and save
Methodology
3.1. Sample size
3.2. Variables
3.2.1. Retrieval effectiveness3.2.2. User effort
3.2.3. Cost
3.2.4. Subjective user reactions
3.2.5. User characteristics
3.2.6. Request characteristics

3.3.

3.2.7,

Intermediary's contribution

3.2.8.

Search process characteristics

Data collection instruments
3.3.1.
3.3.2.

Questionaires
Evaluation of offline prints

3.3.3.

The Logs

3.4.

The participants

3.5.

Procedure for data collection

3.6.

Discussion

Results
4.1. Results from the presearch form
4.2. Results from the post-search user questionnaire
4.2.2.

Satisfaction
Assessment of the search

4.2.3.
4.2.4.

Search results
Searcher's contribution

4.2.5.

Closeness to original enquiry
Number of references expected

4.2.1.

4.2.6.
4.2.7.

References marked as relevant
4.2.8. References viewed online
4.3. Results from the post-search intermediary's questionnaire

4.4.

4.3.1.
4.3.2.

Overall satisfaction
Assessment of the search process

4.3.3.
4.3.4.

Results of the search
Reason for finishing

Results from logs
4.4.1. PSS packets sent and received
4.4.2.
4.4.3.
4.4.4.

Online time
Online citations
Offline citations

4.4.5. Terms used in the search, terms added or ammended
Results from the relevance assessments
4.5.1. Total number of documents assessed
4.5.2. Total relevant retrieved
4.5.3. Precision
4.6. Summary of results by category of variable
4.6.1. Retrieval effectiveness4.6.2. User effort
4.6.3. Cost
4.5.

4.6.4.
4.6.5.
4.6.6.
4.6.7.
Problems

Subjective user reactions
User characteristics, request characteristics
Intermediary's contribution
Perceived online time

6.

5.1.

The York Box and the size problem

5.2.

System crash

5.3.

Staffing

5.4.

Capturing searches

5.5.

Proposal for extension

Conclusions and recommendations
6.1.
6.2.

Experimental methodology
Proposals for future research

6.3.

Final remarks

References
Appendices
AlMacaskill, M.J.

Splitting Cirt into two processes

A2

Robertson, S.E.

On sample sizes for non-matched-pair 1R

A3

Sample logs

A4

Publicity for free searches

A5

Thompson, C.L.

A6

Random allocator cards

experiments

The Cirt manual

A7

The questionnaires

A8

Tables of results