Final Report

Use of the Critical Incident Technique to Evaluate the Impact of
MEDLINE

September 30, 1989

Sandra R. Wilson, Ph.D., Project Director
Norma Starr-Schneidkraut, Ph.D., Project Coordinator
Michael D. Cooper, Ph.D., Consultant

Submitted to

Elliot R. Siegel, Ph.D., Project Officer
Office of Planning and Evaluation
National Library of Medicine
8600 Rockville Pike
Bethesda, Maryland   20894

Contract No. N01-LM-8-3529

-----------------------------------------------------------------

ACKNOWLEDGMENTS

     The authors wish to acknowledge the important collaborative
role of Elliot R. Siegel, Ph.D., Assistant Director for Planning
and Evaluation/NLM, and Barbara A. Rapp, Ph.D., throughout this
study.  Dr. Siegel's official capacity was that of Project
Officer, and Dr. Rapp was Alternate Project Officer.  However,
AIR's role as support contractor to NLM meant that, functionally,
both were an integral part of the study team, with Dr. Rapp
providing day-to-day project coordination on NLM's behalf.  Their
insight, guidance, and technical input at all points were key
ingredients in the successful outcome of the study.  Other
persons at  NLM who contributed to the study effort at various
stages of planning and implementation were Dr. Michael Ackerman,
Mr. Sean Donohue, Dr. Richard Hsieh, Mr. David Kenton, Dr. Henry
Kissman, Mr. Charles Kalina, Ms. Becky Lyon-Hartmann, Mr. Robert
Mehnert, Ms. Susan Slater, and NLM consultants Dr. Jay Goldman of
the University of Alabama and Dr. Edithe Levit, past President of
the National Board of Medical Examiners.  Special acknowledgment
is due NLM Director Dr. Donald A. B. Lindberg, who originally
conceived the notion of using the CIT methodology to evaluate
MEDLINE.  Any residual deficiencies in the study execution or
reporting, however, remain the ultimate responsibility of AIR and
the authors.

     The authors also wish to express their appreciation to a
number of current and former staff of the American Institutes for
Research for their help in this project.  Jeffrey McHenry, Ph.D.,
worked with Dr. Wilson in the development of AIR's approach to
the study and proposal.  Prior to the assumption of the role of
Project Coordinator by Norma Starr-Schneidkraut, Ph.D., Ms. Susan
Vintilla-Friedman served as Coordinator and conducted the
interviewer training and initial respondent interviewing.  Ms.
Nancy Hull, AIR Palo Alto Office librarian, participated in
respondent interviewing, taxonomy development, and preparation of
the case study abstracts.  Ms. Elise Mosse also wrote such
abstracts.  Special thanks are due to Mss. Emily Campbell, Carrie
Davis, Linda Erickson, and Irene Yurash for their expert word
processing of the many hundreds of incident reports and case
study abstracts, as well as drafts of the technical reports. 
Throughout most of the study, administrative support was provided
by Ms. Alice Cherveny. Appreciation is also due to Ms. Parisa
Ashrafi for her careful monitoring of the progress of the
recruitment, the interviewing, and the processing of the data, as
well as for her coordination of report preparation and for her
data processing support.  

     Credit for the preparation of the dBase IV file goes to
Bradley Hesse, Ph.D., and Frances Stancavage, M.A.; for
development of the SAS data file, to Mr. Mitchell Yee and
KatherineTroyer, Ph.D.; and for help in the mailing of letters of
invitation, to Winnie Young, MLS,  and Mss. Elaine Balek, Alison
McGrady, and Roberta Caldwell of the AIR Washington office. 
Thanks also go to two students, Mr. Kwame Burroughs and Mr. Peter
Hu, NIH Biomedical Research Support Grant summer interns, for
their help in a variety of project tasks.  

     Great appreciation is due the staff of the University of
Texas Regional Medical Library who identified and conducted
interviews with individuals requesting MEDLINE searches at that
institution.  These staff members include Ms. Jean Miller,
Director, Health Science Center Library; Mr. Spencer Marsh, who
coordinated the interviews; and Ms. Laura Wilder, Pennie
Billings, Alan Carr, and Helen Mayo, who carried out the
interviews.  Thanks are also offered to Elaine Graham, Associate
Director, Pacific Southwest Medical Library Service, and to the
staff of the University of California at Los Angeles Regional
Medical Library and the libraries of community hospitals in
Region 7, who identified a sample of individuals requesting
searches of these hospitals' librarians.  

     The assistance of Muriel H. Bagshaw, M.D., is also
gratefully acknowledged.  Her careful review of the patient care
incident reports and the case studies helped insure the accuracy
of their medical content and their general readability.  The
participation of Laurence Way, M.D., and Gary M. Arsham, M.D., in
the review of some of the initial incident reports is also
acknowledged.  

     Cristina Campbell, M.L.S., of the University of California
School of Public Health Library, working with Michael D. Cooper,
Ph.D., Professor, University of California School of Library and
Information Studies, carried out the verification of the search
parameters reported in the incidents and provided an analysis of
the transaction logs of ineffective searches.  

     Finally, great appreciation is due to the hundreds of
physicians, researchers, and other health professionals who
contributed their time to share their experiences in searching
MEDLINE and their perspectives on the value of and potential
improvements in MEDLINE and Grateful Med.  

TABLE OF CONTENTS

EXECUTIVE SUMMARY
INTRODUCTION
METHODS
OVERVIEW OF METHODOLOGY
     SAMPLE
          Sample selection
          Recruitment
          Response rates
     DATA COLLECTION PROCEDURES
          Protocol development
          Interviewer training
          Interviewing procedures
          Data processing
     DATA ANALYSIS
          Critical incident data
          Respondent and search characteristics
          Verification
          Analysis of reports and logs of ineffective searches
RESULTS
     CHARACTERISTICS OF RESPONDENTS AND SEARCHES
          Overview of the data
          Respondent characteristics
          Searches
     RESPONDENT COMMENTS CONCERNING NLM SERVICES
     THE INCIDENT REPORTS
          Examples of incident reports
          Verification of parameters of the end user searches
     TAXONOMY:  WHY INFORMATION WAS NEEDED FROM MEDLINE
          I.   Patient care
          II.  Research
          III. Teaching
          IV.  Learning
          V.   Administration
          VI.  Other
     REASONS WHY THE INDIVIDUAL CHOSE TO DO A MEDLINE SEARCH
          I.   Why the individual wanted to access the journal
          literature
          II.  Why the individual chose to do an online search
          III. Why the searcher used Grateful Med
     TAXONOMY:  IMPACT ON MEDICAL DECISION-MAKING
          I.   Patient care
          II.  Research
          III. Teaching
          IV.  Learning
          V.   Administration
          VI.  Consultation
     TAXONOMY:  IMPACT ON THE OUTCOMES OF PROFESSIONAL ACTIVITIES

          I.   Patient end results
          II.  Research end results
          III. Teaching end results
          IV.  Administrative end results
          V.   Legal end results
     RESULTS OF ANALYSIS OF INEFFECTIVE SEARCHES
          Analysis of user interviews
          Detailed analysis of searches
          Conclusions about searching and indexing
CONCLUSIONS
     METHODOLOGY
     RESULTS
          Reasons for using MEDLINE to obtain information
          Reasons for needing information
          Impact on medical decision-making
          Impact on outcomes
          User problems

IMPLICATIONS
     IMPROVING MEDLINE SERVICES
     INCREASING USE OF MEDLINE
     INCREASING SUPPORT FOR LIBRARY SERVICES
     FUTURE RESEARCH
          Improvements to Grateful Med
          Outreach efforts
          Research on continuing education and clinical reasoning
          Use of the research data base
BIBLIOGRAPHY
TABLES 
Table 1.  Location of Interviewer and Type of MEDLINE User, by 
Source of Sample
Table 2.  Recency and Results of Searches Reported by End Users 
and Mediated Users
FIGURES
Figure 1. Examples of Reports of Effective MEDLINE Searches
Figure 2. Example of Reports of Ineffective MEDLINE Searches
Figure 3. Reasons Why the Individual Needed Information from
MEDLINE
Figure 4. Reasons Why Individual Chose to Do a MEDLINE Search
(Rather than Use Other Means of Obtaining the Information)
Figure 5. Impact of the Information Obtained from MEDLINE on
Medical Decision-Making
Figure 6. Impact of the Information Obtained from MEDLINE on the
Outcomes of Professional Activities
APPENDICES
APPENDIX A. LETTERS OF INVITATION
APPENDIX B. CRITICAL INCIDENT REPORT FORMS
APPENDIX C. INCIDENTS AND TRANSCRIPTIONS
APPENDIX D. CHARACTERISTICS OF RESPONDENTS AND SEARCHES
APPENDIX E. EXAMPLES OF CASE STUDY ABSTRACTS
APPENDIX F. REASONS WHY THE INDIVIDUAL NEEDED INFORMATION
APPENDIX G. IMPACT OF THE INFORMATION OBTAINED FROM MEDLINE
ON MEDICAL DECISION-MAKING
APPENDIX H. IMPACT OF THE INFORMATION OBTAINED FROM MEDLINE
ON THE OUTCOMES OF PROFESSIONAL ACTIVITIES
APPENDIX I. SUMMARY OF ANALYSIS OF INEFFECTIVE SEARCHES