'"J1,™«L ON UBWW RESOUROB. *5J0UNCIL ON«AHMII«»Nli»EVALUATIONof theMEDLARSDEMAND SEARCHSERVICEJanuary 1968

F.W. Lancaster
Deputy Chief, Bibliographic Services Division
National Library of Medicine
U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE
Public Health Service7^

"IN ORDER TO SURVIVE, A SYSTEM MUST MDNITOR ITSELF, EVALUATE ITSPERFORMANCE, AND UPGRADE IT WHEREVER POSSIBLE."

z69?
.^391 3I

* From report GER 12760 Measures of Effectiveness and Criteria forEvaluation of ja Document Processing System. Rome Air Development
Center, 15 July 1966.

P R E F A C EThis report presents the results of a detailed analysis by
the National Library of Medicine of the performance ofMEDLARS (its Medical Literature Analysis and RetrievalSystem) in relation to 300 actual requests made to thesystem in 1966 and 1967. Thanks to this study, we now know
considerably more about the requirements of MEDLARS users,
how well the system is meeting these requirements, and what
we must do to improve the overall performance level. The
investigation is timely: the Library is now planning asecond-generation MEDLARS, and the design of MEDLARS II
should benefit greatly from our knowledge of factors affecting the performance of the present system.
Since this is the first large-scale evaluation of a major
operating information system, and because of the diversityof subject areas covered by the study, it should be ofconsiderable interest to the scientific community at large.
Some readers, of course, may wish to take exception to partsof the methodology of the study or even view some of theanalyses with reservation. In an effort to make the study
as objective as possible, the design and results werereviewed by a distinguished outside advisory committee towhom we are most grateful.
To remain responsive to the demands of its users, a large
scientific or technical information system must examine
itself critically. We hope that a major benefit of thisinvestigation will be the establishment of a program forthe continuous quality control of MEDLARS products andservices.

Martin M. Cummings, M.D.
Director
National Library of Medicine

CONTENTSPART 1DESIGN AND EXECUTION OF THE EVALUATION PROGRAM
Introduction
MEDLARS:

general background

Objectives of the test program
Test design:general considerations

Selection of user groups
Establishing the performance figures
The pretest
Procedures used in the conduct of the test
Derivation of performance figures
Analysis of reasons for search failures

PART 2

THE TEST RESULTS
Overall performance figures
The individual ratios
Average MEDLARS performance for the test
requests
Analysis of causes of recall and precision
failuresAnalyses of failures:

explanatory notes

Recall and precision failures attributable tothe indexing subsystem
Failures due to exhaustive indexing or tolack of exhaustivity
Effect of exhaustivity levels
Failures due to lack of specificity inindexing
General observations on exhaustivity andspecificity of indexing within MEDLARS
Evaluation of indexing as part of the inputsubsystemv

Recall and precision failures attributable
to the searching subsystem

65

Recall losses resulting from failure to cover
all reasonable approaches to retrieval

65

Recall and precision failures due to variations
in exhaustivity of the formulation

67

Recall and precision failures due to variationsin specificity of the formulation

70

Use of "weighted1' index terms

76

Other causes of searching failures

79

Joint causes of system failures

81

Effect of the 6-5-4 levels on recall andprecision figures

81

Recall and precision failures attributable
to the index language

82

Failures due to false coordinations and incorrect term relationships

91

General observations on the MEDLARS index
language98

The relationship between indexing, index
language, and searching100

Recall and precision failures attributable tothe area of user-system interaction

101

Improving request statements

H?

Experimentation with modes of interaction

125

Recall and precision failures attributable tocomputer processing

127

The novelty ratio

128

Factors affecting performance of a MEDLARS
search130Variations in performance between five MEDLARS
centers

145

MEDLARS indexing coverage

157

Foreign literature usage factors

159vi

PART 3

PART 4

Journal usage factors in MEDLARS

165

Effect of MEDLARS response time

174

The serendipity value of MEDLARS searches

175

Output screening

176

Indexer consistency

180

Requests rejected by MEDLARS183

CONCLUSIONS AND RECOMMENDATIONS
Overall MEDLARS performance

185

Upgrading the performance of MEDLARS

191

User-system interaction

193

The MEDLARS index language

193

The MEDLARS searching strategies

195

The MEDLARS indexing

197

Computer processing

199

The relationship between indexing, searchingand MeSH

200

Use of foreign language material in MEDLARS200

The search printout as a content indicator

201

Continuous quality control of the MEDLARS
operation

201

Future use of the MEDLARS test corpus

202

APPENDICES
Appendix 1. Samples of MEDLARS vocabulary,sample request and search formulation, sample printout

206

Appendix 2. Specimens of forms used in evaluation

213

Appendix 3.

218

Specimen analysis worksheetvii

Appendix 4.

Complete set of recall and precision ratios

Appendix 5.

Analysis of the specificity ofsearch formulationsAppendix 6.

Analysis of the effect of subheadings on false coordinations
and incorrect term relationshipsviii

230