British Library R & DReport 5553

R E S E A R C H
0 N
R E L E V A N C E
W E I G H T I N G
19

7 6-19K ^ S p a r c kC. A.

7 9J o n e sW e b s t e r

Computer Laboratory
U n i v e r s i t y of Cambridge
Corn Exchange S t r e e tCambridge CB2 3QGEngland

1980

i

ACKNOWLEDGEMENT

This work was supported by the British Library Research andDevelopment Department under grant SI/G/210 to Professor M.V. Wilkes ofthe Computer Laboratory, University of Cambridge. We are grateful to theUniversity Computing Service for permitting use of the University's
370/165 computer.
WethankDr S.E. Robertsoninparticular,
andalso
Professor C.J. van Rijsbergen, Dr D.J. Harper, and Dr R.M. Needham formany useful discussions.
Data
usedbytheprojectwassuppliedbyProfessor C.W. Cleverdon, by UKCIS via Dr D.C. Veal, by Inspec throughMr L. Evans,
andby
theNational
PhysicalLaboratory
through
Dr P.K.T. Vaswani. We are very much obliged to them all for making theirmaterial available.

Karen Sparck Jones
July 1980

iiABSTRACTThis report describes experiments on relevance weighting ofsearch terms.
The tests were designed to study the behaviour ofrelevance weights in relation to other forms of request modification, andin the context of different collection environments, specifically thosedefined by variations in the amount of relevance information available
for weight calculation.
The results show that relevance weighting can be very effective
in a wide variety of situations, and effective even in very unfavourable
circumstances.

iii

CONTENTSPARTJ[Section:TEXT.A:

Background

1

Object of the work

2

Test data

3

General testing strategy .

4

Performance representation

5

Tabulation of search outputSection

B:Experiments

1

Overview of the experiments

2

The framework for the experiments

3

Relevance weighting

4

The tests

4.1
4.1.1
4.1.2
4.1.3
4.2
4.2.1
4.2.2
4.2.3
4.2.4
4.3
4.3.1
4.3.2
4.3.3
4.3.4
5

Request membership
Regular environments
Variant environments
Conclusions on request membership
Request weighting
Weighting formulae
Prediction
Specific formula tests
Alternative performance representations
Variant collections
Qualitative conditions
Quantitative conditions
Specific condition tests
Cross checks and alternative representations
Iterative searchingiv
Section

CDiscussion

1

Analysis of the test results

1.1

Request membership

1.2

Request weighting

2

Boolean searching

3

Comparable research

4Conclusion
FIGURES, GRAPHS AND TABLES

PART
FiguresFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigure

A1
A2
A3
A4
A5
A6
A7
A8
A9
B1
B2

Test data summary
Test raw material
Test collection derivation
Test collections
The profile collections
Collection comparison
Subsidiary collections
Term retrieval facts
Performance representation methodsSpecial cases for weighting functions
Numbers of retrieved documents

GraphsGraphsGraph
GraphsGraphsGraphsGraphsGraphsGraphsGraphs

1- 234- 9
10-12
13-14
15-20
21-26
27-29
30-35

dv
redv
dv
dv
dv
reredv

Request membership
tift

Relevance weighting, F4
F1
H1
U1
F4
U1
variant collections

Tables
Note on the organisation of the Run Tables
Key Table:

Specification of Tables, Specification of Runs

V

Summary Table
Summary Tables, variant collections v1, v2
Main Tables MT, MW, MR, MS, MSW, MSR, MC, MCW, MCR, ME, MEW, MER
Main Tables,
variant collections v1/MR, v2/MR, v2/MSR, v2/MCR, v2/MER
Secondary Tables

SrcT etc., SrcR;
variant collections v2/SrcR, v2/SrcCR
SdrT etc., SdrR;
variant collections v2/SdrR
SprT etc., SprR;
variant collections v2/SprR
Str
variant collections v1/Str, v2/Str
SrrT etc., SrrR;
variant collections, v2/Srr
ScrT etc., ScrR
variant collections, v2/ScrR, v2/ScrCR

Other Table
REFERENCESGLOSSARY