Computer Technology in Medical Education

and Assessment

September 1979

NTIS order #PB80-102528
Library of Congress Catalog Card Number 79-600146

For sale by the Superintendent of Documents, U.S. Government Printing Office

Washington, D.C. 20402 Stock No. 052-003-00704-5
PREFACE

This report reviews the state-of-the-art of the use of computer technology in medical

education and assessment. It both describes the technologies and strategies for computerbased education and assessment, the current state of medical education and assessment

activities, and computer applications in medicine. It also provides examples of systems

for the development and dissemination of computer-based educational materials. A summary and analysis of such computer uses are also presented. Although the focus of the report is on physician education and assessment, the implications of the activities described

are applicable to other health professions.

The Federal Government’s role in computer-based education and assessment has

been primarily in the research and development phases. However, the medical areas in

which such computer-based methods have been applied have been of long-standing interest to the U.S. Congress. Such areas include health professions education, monitoring

and evaluating physician services through Professional Standards Review Organizations

(PSROs), and efforts to contain the costs of such federally financed programs as Medicare and Medicaid. In all of these areas the use of computer technology is becoming the

rule rather than the exception.

This report examines these computer applications in various stages of the life-long

process of medical education from entry into medical school through active practice. It

therefore provides an overview that may be of value to two audiences: policymakers at

the Federal, State, and local levels, and those in the public and private sectors who are

directly involved in the use of computer technology in medical education and assessment.

The study was conducted by the OTA Health Group staff with assistance from an

advisory panel and contractors. The report was also reviewed by the OTA Health Advisory Committee. This report is a synthesis and does not necessarily represent the views

of any of the individuals involved in its preparation.

JOHN H. GIBBONS

Director

. . .

111
OTA HEALTH GROUP STAFF

Joyce C. Lashof, Assistant Director

Health and Life Sciences Division

H. David Banta, Health Group Manager

Ruann Ernst Pengov, Study Director

Lawrence Miike, Study Director

Page S. Gardner, Editor

Lynne Pepi Pietz, Congressional Fellow

Administrative Staff

William S. Burnett Shirley A. Gayheart

Contractors

Association of American Medical Colleges

Washington, D.C.

Health Care Management Systems, Inc.

Encinitas, Calif.

Human Resources Research Organization

Alexandria, Va.

OTA PUBLISHING STAFF

John C. Holmes, Publishing Officer

Kathie S. Boss Joanne Heming
OTA HEALTH GROUP ADVISORY COMMITTEE

Frederick C. Robbins, Chairman

Dean, School of Medicine, Case Western Reserve University

Stuart H. Altman Sidney S. Lee

Dean Associate Dean

Florence Heller School Community Medicine

Brandeis University McGill University

Robert M. Ball

Senior Scholar

Institute of Medicine

National Academy of Sciences

Bernard Barber

Professor

Department of Sociology

Barnard College

Columbia University

Lewis H. Buttler

Health Policy Program

University of California

Kurt Deuschle

Professor of Community Medicine

Mount Sinai School of Medicine

Rashi Fein

Professor of the Economics of Medicine

Center for Community Health and

Medical Care

Harvard Medical School

Melvin A. Glasser

Director

Social Security Department

United Auto Workers

Mark Lepper

Vice President for Evaluation

Rush-Presbyterian Medical School

C. Frederick Mosteller

Professor and Chairman

Department of Biostatistics

School of Public Health

Harvard University

Beverlee Myers

Director

Department of Health Services

State of California

Helen Ewing Nelson

President

Consumer Research Foundation

Mill Valley, Calif.

Charles A. Sanders

General Director

Massachusetts General Hospital

Kerr L. White

Deputy Director

Health Sciences

Rockefeller Foundation

vi
ADVISORY PANEL ON COMPUTER TECHNOLOGY IN MEDICAL EDUCATION

AND ASSESSMENT

Richard B. Friedman, Chairman

Associate Professor of Medicine and Human Oncology

University of Wisconsin—Madison

Edward N. Brandt, Jr. Edithe J. Levit

Vice Chancellor for Health President and Director

University of Texas System National Board of Medical Examiners

Laurel A. Cappa

Resident, Program in Social Medicine

Montefiore Hospital

Bronx, N. Y.

Lloyd R. Evans

Chairman, Federation Licensing

Examination (FLEX) Committee

Michael Goran*

Director, Health Standards and

Quality Bureau

Office of Professional Standards

Review Organizations

Department of Health, Education,

and Welfare

Frederick T. Merchant

Permanent Secretary of the Federation

Licensing Examination (FLEX) Board

Lawrence Miike * *

Consultant on Health Policy

Berkeley Springs, W. Va.

Harold F. O’Neill Jr.

Research Psychologist

U.S. Army Research Institute for the

Behavioral & Social Sciences

Richard Spohn

Director

California Department of Consumer Affairs

William G. Harless August Swanson

Union Graduate School/ West Director

San Francisco, Calif. Department of Academic Affairs

Association of American Medical Colleges

● Resigned position in October 1978; currently serves as Medical Director, Rockridge Health Care Plan, Oakland, Calif.

* *Assumed co-directorship of the study in November 1978.

vii
CONTENTS

Chapte r Page

1. SUMMARY AND ANALYSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Organization of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Implications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . , . . .

2. COMPUTER TECHNOLOGY IN EDUCATION AND ASSESSMENT. . . . . .

The Technology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Trends. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Computer-Based Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Limitations and Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. MEDICAL EDUCATION AND ASSESSMENT . . . . . . . . . . . . . . . . . . . . . . . .

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Undergraduate Medical Education. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Admissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Curriculum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Assessment of Student Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Graduate Medical Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Licensure and Specialty Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I.icensure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Specialty Certification . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . .

Emerging Developments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Clinical Practice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. COMPUTER APPLICATIONS IN MEDICAL EDUCATION

AND ASSESSMENT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Computer-Supptortecl Independent Study Program in the Basic Medical Sciences

The Use of Computer-Based Simulations in Education and Assessment . . . . . . . .

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Pediatric Fellowship Examination of the Royal College of Physicians and

Surgeons of Canada . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Computer-Based Examination (CBX) Project of the American

Board of Internal Medicine and the National Board of Medical Examiners . .

A Model for Evaluation andl Recertification Through Individualized Testing

(MERIT) of the American Board of Internal Medicine. . . . . . . . . . . . . . .

Computer-Based Medical Consultation Systems . . . . . . . . . . . . . . . . . . . . . . . . .

Computer Uses in Quality Assurance Data Management Systems. . . . . . . . . . .

Medicaid Management Information System. . . . . . . . . . . . . . . . . . . . . . . . . .

Professional Standards Review Organization . . . . . . . . . . . . . . . . . . . . . . . . . .

Computer-Stored Ambulatory Record System . . . . . . . . . . . . . . . . . . . . . . . . .

Problem Oriented Medical Information System . . . . . . . . . . . . . . . . . . . . . . . .

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

3

4

4

6

13

13

13

18

20

20

21

23

24

29

29

30

30

32

33

34

35

35

37

38

39

47

47

48

55

55

57

6 0

60

63

65

65

65

66

66

67

ix
— .———.

5. DEVELOPMENT AND DISSEMINATION OF COMPUTER-BASED

EDUCATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institutional Development and Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Ohio State University College of Medicine . . . . . . . . . . . . . . . . . . . . . . . .

The University of Illinois School of Basic Medical Sciences. . . . . . . . . . . . . . . .

Massachusetts General Hospital. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Networks, Consortia, and Other Dissemination Mechanisms . . . . . . . . . . . . . . .

The Health Education Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Computer-Assisted Teaching Systems Consortium . . . . . . . . . . . . . . . . , .

Milliken Communications Corporation , . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APPENDIX I–EXAMPLES OF SIMULATIONS OF THE PATIENT-PHYSICIAN

ENCOUNTER. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LIST OF TABLES

Table

Number

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

Computer-Based Education Course Development . . . . . . . . . . . . . . . . . . . . . . . .

A Sample Spread of Computer-Based Education Funding and Development

Sources as of 1974 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Thirty-two Studies of Computerized Simulations and Testing . . . . . . . . . . . . . . .

Number of Medical Schools, Graduates, and Physicians in Practice,

Selected Years . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Jurisdictions With Authority to Require Continuing Medical Education for

Reregistration of the License to Practice Medicine . . . . . . . . . . . . . . . . . . . . . . . .

Implementation of Recertification by the Specialty Boards, . . . . . . . . . . . . . . . . .

The Independent Study Program Curriculum Outline . . . . . . . . . . . . . . . . . . . . .

Sample Practice Profile to Select Appropriate CASEs. . . . . . . . . . . . . . . . . . . . . .

Institutions Producing Computer-Based Materials . . . . . . . . . . . . . . . . . . . . . . . .

Computer-Assisted Instruction Regional Education Network (CAIREN)

Membership Profile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Accreditation of Computer-Based Continuing Education Courses by Professional

Societies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Ohio State University College of Medicine Computer-Based Education

Programs Indexed by Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

College of Medicine Computer-Based Education User Group Summary. . . . . . . .

Computer-Based Education Utilization Summary for the College of Medicine . . .

Massachusetts General Hospital: Computer-Assisted Continuing Medical

Education Program (1979) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Health Education Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Continuing Education Seminars Currently Marketed bv Milliken

Communications Corporation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LIST OF FIGURES

Figure

Number

1. Sample Interactive Computer Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. Sample Interactive Computerized System in Which the Computer Simulates

the Patient . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . .

3. Sample Interactive Computerized System in Which the Computer Acts as

the Teacher/Tutor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

71

71

73

76

79

80

81

84

85

91

125

Page

19

24

25

32

38

39

494$)

62

73

74

75

75

76

77

80

83

86

Page

14

15

l b

x
4.

5.

6.

7.

8.

9. .

10.

11.

12.

13.

14.

15.

Schematic of a Tutorial Computer-Based Education Question . . . . . . . . . . . . . . . 22

The Medical Education and Assessment Continuum . . . . . . . . . . . . . . . . . . . . . . 31

Years of Graduate Medical Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Student Flow Through a Typical Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Part of a Student-Computer Dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Sample Computer-Based Management System Reports . . . . . . . . . . . . . . . . . . . . 52

Item Statistics Report. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Example of an INDEX Case Interaction Printout . . . . . . . . . . . . . . . . . . . . . . . . . 57

Cumulative Number of Courses Involving CBE Versus the Year in Which They

Were First Taught . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Data Base and CBE System User Group Overview . . . . . . . . . . . . . . . . . . . . . . . . 76

CBE Usage Profile for the College of Medicine (1977). . . . . . . . . . . . . . . . . . . . . . 77

Average User Dollar Expenditures: Computer-Assisted Instruction

Regional Education Network (CAIREN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

xi
ACRONYMS

AAFP –

ABIM –

ABMS –

ACEP –

AMA –

CAI

CAIREN -

CASE –

CAT –

CATS –

CBE –

CMI

COSTAR -

CPMP –

CPHA –

CQE –

FLEX –

GME –

HEN –

HEW –

ISP –

JCAH –

MCAT –

MCC –

American Academy of Family

Physicians

American Board of Internal

Medicine

American Board of Medical

Specialties

American College of Emergency

Physicians

American Medical Association

computer-assisted instruction

Computer Assisted Instruction

Regional Education Network

Computer-Assisted Simulation

of the Clinical Patient Encounter

computer-assisted testing

Computer-Assisted Teaching

Systems Consortium

computer-based education

computer-managed instruction

Computer-Stored Ambulatory

Record System

Computerized Patient

Management Problems

Commission on Professional and

Hospital Activities

Comprehensive Qualifying Exam

Federal Licensing Examination

graduate medical education

Health Education Network

Department of Health, Education,

and Welfare

independent study program

Joint Committee on

Accreditation of Hospitals

Medical College Admissions

Test (MCAT)

Milliken Communications

Corporation

MCE –

MERIT –

MGH –

MMIS –

NBME –

NRMP –

OSU –

OTA –

PACE –

PAS

PLATO -

PMIS –

PMP –

PSRO –

QAM –

RAMS –

SOPHIE –

S/UR –

TES –

TICCIT –

UNDDS –

UPRO –

Medical Care Evaluation (MCE)

Model for Evaluation and

Recertification Through

Individualized Testing

Massachusetts General Hospital

Medicaid Management

Information System

National Board of Medical

Examiners

National Residency Matching

Program

Ohio State University

Office of Technology Assessment

Physician Ambulatory Care

Evaluation

Professional Activities Study

Programed Logic Automated

Teaching Operation

PSRO Management Information

System

patient management problem

Professional Standards Review

Organization

quality assurance monitor

Retrospective Analysis of

Medical Services (RAMS)

Sophisticated Instructional

Environment

Surveillance and Utilization

Review

tutorial evaluation studies

Time-Shared Interactive

Computer-Controlled

Information Television

Uniform Hospital Discharge Data

Set

Utah Professional Review

Organization

xii
Chapter 1

SUMMARY AND ANALYSIS
1 ●

SUMMARY AND ANALYSIS

INTRODUCTION

In the past quarter century, dramatic reductions in the size and costs of computers

have increased their availability and led to their application in most aspects of our lives.

Projections regarding their prevalence and use in the 1980’s indicate that most stores, offices, and hospitals, and many homes will have them. During this same period, advances

in medicine have led to a virtual information explosion, making the contemporary medical care system more complex, more information-dependent, and more technology-oriented. Computers, as perhaps one of the most pervasive of the present technologies, can

contribute to the increasing complexity of medicine, as well as assist efforts to more effectively understand, employ, and manage the information and array of technologies used

in health care. Already, they have become integral elements in public health and biomedical research (e. g., statistical analysis, experimental modeling); medical education (e. g.,

electronic access to learning resources); patient care (e. g., on-line physiological monitoring); and enhanced information management (e. g., medical information systems).

This report focuses on the use of computers in medical education and assessment. *

More precisely, the report examines the methods by which students or practitioners interact with teaching and testing materials contained in a computer. Such methods are

generally called computer-based education (CBE).

It is now recognized that medical education includes not only the formal education

of physicians in undergraduate and graduate (residency) medical training, but also the

more informal education accompanying active patient care. This new recognition that

medical education is a life-long process is evidenced by two developments:

First, continuing medical education for physicians, initially implemented on a voluntary basis, is now a requirement for reregistration by many State medical licensing

boards and for recertification by some specialty boards.

Second, improvements in the quality of medical care are clearly related to continuing medical education. Criteria used in evaluating the quality of care are usually grouped

according to structure, process, and outcome measures. Structural measures include both

the availability of resources (e.g., facilities, equipment, and health care personnel) and

the qualitative aspects of medical care personnel (e.g., extent of educational background

and board certification). Process measures assess the appropriateness of the medical care

that has been provided. Outcome measures reflect the effect of medical interventions on

patient health status. These three ways of measuring the quality of medical care reinforce

the relationships between the formal education process and the more informal life-long

learning process that must accompany patient care.

*Assessment is defined to include both formal and informal methods of evaluation and testing.

3
4 ● Computer Technology in Medical Education and Assessment

ORGANIZATION OF THE REPORT

This report examines the application of computer technology in medical education

and assessment. Computer technology and its uses and capabilities in general education

and assessment strategies are described in chapter 2, followed by a description of the

process of medical education and assessment in chapter 3. Case examples of the application of computer technologies in specific areas of the medical education continuum are

then presented in chapter 4. The institutionalization of computers in medicine is documented in chapter 5 through summary descriptions of the development, dissemination,

and use of computer-based education and assessment materials.

In the following sections of this first chapter, the implications of computer technology developments and uses in medical education and assessment are summarized. Further discussion of the conclusions drawn in this chapter are provided in the body of each

chapter.

CONCLUSIONS

Computers can be compared to other medical aids upon which the physician has

come to depend. In many cases, computers have replaced other, less reliable technologies. Computer technology can: 1) assist in, and help manage, educational activities; 2)

assist in, and help manage, assessment activities; 3) assist in diagnostic, prognostic, and

therapeutic processes in patient care; and 4) help manage large amounts of data for a

variety of purposes. Computers save time, improve efficiency and skills, and provide

unique learning opportunities.

Due to their ability to store, process, and retrieve information almost instantaneously, computers can be used in ways that make comparisons with the capabilities of other

methods difficult or impossible. Such unique uses include individualization, simulation,

inquiry, interpretation, and data management.

Computers permit a wide range of learning responses in tutorial sessions and can

respond to individual learner backgrounds and needs. Computers enable students to pace

their progress through a curriculum. The independent study programs in the basic medical sciences are examples of computer uses in medical education that take advantage of

these capabilities to create a flexible curriculum.

Computers are capable of storing and rapidly searching and processing huge volumes of data. This capability permits simulations of some aspects of the patient-physician encounter. Computers, however, cannot assess the quality of the physical examination conducted by the physician; they can only present the findings of such examinations.

“Body language” cues that may be apparent in a live physician-patient interaction also

cannot be duplicated by a computer because it uses electronically printed words as the

predominant communications device. However, simulations can provide student exposure to specific diseases or clinical situations that are rarely encountered, as well as

familiarize the user with the many variations by which even common diseases can be

manifested. Simulations also can be used in examinations and tests to overcome some of

the subjectivity inherent in other techniques (e.g., essay questions, oral exams, clinical

rounds, and student “treatment” of patient actors). Finally, as in teaching, simulations

permit uniform nationwide testing of aspects of the patient-physician encounter that previously were not effectively or economically measured.
Ch. l—Summary and Analysis ž 5

Methods for inquiry and interpretation of data have been influenced dramatically by

the computer. The huge amount of data stored in computers obviates much of the physician’s need to retain detailed facts; and, much of the deductive reasoning process that

takes place in diagnosing and treating patients can be done with more consistent, comprehensive, and instantaneous recall and application.

The computer’s information-handling capabilities allow it to serve as a reliable extension

of the physician’s memory and expander of the physician’s information and synthesized

knowledge resources. Computers have been used to facilitate decisions through organization of patient data, improved classification of patients, decision analysis in clinical settings, and simulation of expert clinical reasoning. Computer programs are more successful in narrow, constrained, single arenas of medicine with much underlying pathophysiologic understanding and where decisions are based largely on hard laboratory

data. New models of synthetic reasoning that simulate expert clinical behavior show

promise of supporting complicated decisions concerning problems of multiple disease

(Schoolman and Bernstein, 1978).

The use of computers as “consultants” for specific diseases or problems, or for a particular specialty, illustrates these capacities.

Finally, computers have a unique role in the management of large volumes of data

for reimbursing health services; reviewing utilization; and planning, monitoring, and

evaluating medical care services. Computer systems are, or can become, integral components of both intra-institutional medical information systems and interinstitutional

health data systems.

Medical information system are defined as computer-based systems that receive

data describing patients; create and maintain computerized medical records based on

these data; and make the data available for the following uses: 1 ) patient care, 2) administrative and business management, 3) medical care service monitoring and evaluation, 4)

epidemiologic and clinical research, and 5) medical care resource planning. No existing

medical information system yet provides data for all the purposes enumerated.

Health data systems collect data from a variety of medical care institutions. Although health data and medical information systems are not now coordinated, these uses

of data cannot realistically be separated. Both systems should be highly integrated, interactive subsystems of the health care system. A well-designed computer system should facilitate and enhance data and other information flow among the various users and between types of information systems.

The use of computers in education and assessment inevitably will be linked to their

uses in medical information systems. Such linkage will allow, if not force, the formation

of new relationships between segments of the medical education and assessment continuum through the accumulation of large data bases on student characteristics and performance, on physician and institutional performance in patient care, and on patient outcomes following treatment. These data bases could serve as the thread of continuity between portions of the continuum. They could provide more objective and quantitative

feedback mechanisms from active practice to education and assessment.

Currently, the best measures of competence in learning do not necessarily predict

good performance in practice. Patient care assessments depend largely on comparison

with peers using standards (processes that should be followed) or empirically determined

norms (the average care provided). Computer technology could be used to improve the

linkage between medical education and patient care through the provision and maintenance of more specific and objective data bases for diseases and treatments. In addition

51-541 0 - 79 - 2
6 ● Computer Technmology in Medical Education and Assessment

to providing better data for generation of standards, computer data bases could allow

better comparison of present standards and norms of care with actual patient outcomes.

These data bases also could permit the development of computer consultant systems.

Feedback from medical information and health data systems could provide continuous

updating of the data bases.

Such data bases could be linked to education and assessment systems for improvements in both areas. Early in the medical education and assessment continuum, the emphasis is on the accumulation, retention, and application of knowledge on which decisionmaking in future practice will be based. This general medical education is quite different from the specialized training that follows undergraduate medical education. Furthermore, there is an inherent problem in deciding what aspects of clinical practice

should be emphasized in the general medical education curriculum. Computerized data

bases could help to highlight areas of knowledge and skills that need further emphasis.

They could dynamically link medical education to ongoing patient care. The development of computer course work for learning and testing calls for specificity in defining objectives and in correlating the knowledge and skills to be acquired with the objectives.

Techniques for accomplishing such specificity could help improve the quality of information collected in medical information systems.

This report emphasizes the positive aspects of the application of computer technology. It is beyond the scope of this report to delve into the myriad of sociopolitical issues

in both the use of computers per se and in the medical education and assessment system.

Regarding the former, the issues relate primarily to control and access to electronically

stored data; such issues are not unique to the particular application of computer technology in medicine. Another OTA report is exploring these issues. *

One major sociopolitical issue in the medical education and assessment system that

deserves note, however, is the issue of governance. Medicine, as for other professions

with specialized knowledge, is regulated by licensing boards that are dominated or controlled by the profession. Increasing specialization may lead to changes in the role of licensing boards, which issue general licenses to all physicians under their jurisdiction, and

in the role of specialty boards, which, although voluntarily joined, convey prestige and

economic advantages to their members. If the general medical license is modified to reflect the increasing specialization of medicine, questions will arise on how such licenses

will be determined and regulated.

IMPLICATIONS

The use of computer technology in medical education and assessment activities will

significantly alter these two processes. A number of potential changes are presented and

discussed below:

1. The use of computer technologies to retrieve knowledge and assist in problemsolving will substantially obviate the present need for accumulation and retention of facts by individual physicians. This may require changes in the medical

curricula.

*“Assessment of the Societal Impacts of National Information Systems, ” an OTA study currently in

progress.
——— —————. —

Ch. l—Summary and Analysis ● 7

Electronically stored knowledge is more accurate, consistent, and rapidly retrieved

than the knowledge contained in the physician’s memory. Computers will become more

prevalent and indispensable as tools to aid in diagnosis and other decisionmaking

because of their enhanced memory capabilities. Medical admissions and testing processes

will need to adjust to this change. Since the use of computers will be unavoidable, and

since students and practitioners will need to learn how to use them, courses in computer

operations will need to be incorporated into the medical school and graduate curricula.

Such education is the key to realizing the long-term benefits of computers in the practice

of medicine.

Medicine, however, is not based solely on statistical probabilities; consequently,

computers will act more as aids to, rather than as substitutes for, physician decisionmaking.

2. The improved understanding of disease and health that will occur with the use of

computers will, in turn, change the method and content of physician education.

The computer’s ability to perform statistical correlations between patient characteristics, diseases, treatments, and results will contribute significantly to the medical knowledge base. This knowledge base can be improved and expanded as more data are gathered. New data can be incorporated into the educational and patient care process, leading

to changes in medical school curricula and the practicing physician’s repertoire of supportive services.

3. The accumulation of large data bases on student characteristics and performance, on physician and institutional performance in patient care, and on patient

status will enhance efforts to measure, validate, and improve the quality of medical care and medical education.

The ability to evaluate the quality of medical care has been inhibited by a lack of

understanding about the relationships between both specific structural and process measures and patient outcomes. Although correlation between structural measures (e.g.,

whether a physician is board certified) and patient outcomes may always be tenuous,

systematic correlations between the processs of medical care and patient outcome measures may be possible. As a result of the computer’s ability to provide new knowledge for

diagnostic, prognostic, and therapeutic purposes, computerized data systems will

enhance our current ability to distinguish between medical interventions that make a difference in patient outcomes and those that do not.

The use of computers as aids in defining and monitoring quality of care is limited by

the quality of the data base. The use of computerized data bases may raise the problem of

prematurely legitimizing current diagnostic and therapeutic procedures. If too rigidly applied, norms and standards may impose too much conformity on the practice of medicine, promote the perception of individuals as disease entities, and interfere with the

identification of new syndromes.

4. Computer-based education will allow individualized medical education, perhaps

in different settings. Independent study curricula will provide new opportunities

for students with academic weaknesses or allow students to alter the pace of their

education.

Computer-based education in undergraduate medical education has been demonstrated to be a workable substitute for substantial portions of the traditional lecture/discussion curriculum. Computer-managed and computer-assisted independent study is fea-
8 ● Computer Technology in Medical Education and Assessment

sible for large numbers of preclinical students. Such independent study programs enable

students to control the rate of their progression through the rather intense medical school

curriculum. Students with strong academic credentials, as well as those with academic

deficiencies, benefit from the inherent flexibility of independent study. Students who

choose computer-based independent studies perform comparably to 1ecture/discussion

students on standardized exams. Student attitudes toward independent study are favorable.

A more student-centered and directed form of education could promote more creativity and lead to changes in the way a physician practices. A physician’s ability to

assess his/her own strengths and weaknesses is an important component of his/her functioning capabilities.

Developments in computer technologies may lead to student training in divergent

settings without great variations in the quality of education. Electronic networking and

the proper combinations of computer simulations and real experience are already being

used to meet the continuing education requirements of practicing physicians.

5. The use of computers in medical education will change the role of the faculty

member.

Faculty will spend less time disseminating information in traditional settings.

Rather, they will spend more time orchestrating the learning process and tutoring individuals and small groups. Faculty also will spend more time developing courseware.

6. The evaluation of a student’s problem-solving abilities will be improved through

the use of simulations.

The use of simulations can avoid some of the subjectivity inherent in testing methods

such as essay questions, oral exams, clinical rounds, and the use of patient actors. Simulations can also provide experience with “patients” who have rare or seasonal diseases.

The computer, however, cannot replace subjective assessments that may be essential for

determining physician competence.

The use of computer simulations for credentialing in Canada has demonstrated that

simulations are effective means of measuring portions of the problem-solving components of clinical competence. Physician reaction to these exams generally has been

positive. Initial findings from two U.S. experiments using computers in credentialing are

similar.

7. Continuing medical education will become more individualized and directed

toward remedying identified deficiencies in physician performance.

The continuing education needs of each physician will be more clearly identified

because of the ability to individualize assessments of physician competence. If trends

toward required continuing education continue, physicians may be asked to select a

given number of educational experiences that specifically address their deficiencies. Furthermore, awarding continuing education credit may be linked to evidence of mastery of

the continuing education materials.

Computer-based education not only can provide immediate feedback on performance in continuing education but also can generate and maintain records on the type and

amount of continuing education credit received.
Data on the types of patients and diseases treated by the physician and on the quality of treatment rendered (e. g., professional standards review data) can also be maintained. Coordination and synthesis of this data in developing tests make the test more

individualized by reflecting specific practice patterns and areas of potential concern in

each physician’s practice. As in undergraduate and graduate medical education, computer networking can make educational materials available in divergent settings (e. g.,

home, hospital, or office), without significant variations in quality.

8. Testing and other assessments of physician performance also will become more

individualized.

General tests that are administered to all persons under a particular authority (e. g.,

medical schools, licensing boards, specialty boards) are used as a means of determining

the attainment of a particular level of achievement. These tests, however, do not

necessarily y measure the level of competence needed in patient care because of the increasing specialization in the practice of medicine. Computers will assist in the development

and administration of tests that more accurately reflect the kinds of situations that the

physician faces in his/her practice. By complementing the use of norms and standards,

computer simulations will permit a more precise identification of deficient performance.

9. Individualized testing will accelerate the trend toward testing competence in a

limited area.

The general licenses issued by State medical licensing boards do not limit the scope

of a physician’s practice. However, informal, de facto constraints exist, such as physician

choice in limiting his/her practice to a specific area of medicine. Various institutional

constraints, such as hospital privileges, also act to limit practice. Consequently, broad

licenses to practice hardly exist in fact.

Specialty certification limits assessment to a specific specialty field. Activities are

already underway to reflect the increasing specialization of medical care, The proposed

comprehensive qualifying examination (CQE ) would assess the preparedness of the medical school graduate to enter advanced training. If medical licensing boards eventually

issue a license limited to the supervised setting of graduate medical education, questions

will be raised regarding limiting the subsequent license to practice in an unsupervised setting to a physician’s specialty area.

Individualized testing will make it easier to test competence in more depth and in

particular areas of knowledge, such as in specific specialties. The computer-based exami -

nations being developed by the American Board of Internal Medicine represent one approach to individualized testing in specialty areas.

10. Testing in limited areas of practice will raise questions regarding the issuance of

limited licenses to practice and the governing of the licensure process.

State licensing boards, which now administer or accept common tests applicable to

all medical school graduates, would have to administer different types of tests to the

various specialists. If specialty board certification is accepted in lieu of State medical

board tests, specialty certification would attain comparable legal status to licensure. If

this occurs, the appropriateness of specialty self-regulation and determination of standards will be questioned.

11. Fundamental questions will arise regarding the changing role of medical licensing boards.
10 ● Computer Technology in Medical Education and Assessment

Standard setting, as in the acceptance of board certification in lieu of a licensing

exam, could be separated from professional regulation; the responsibilities for professional regulation could remain with the licensing boards. A variation of this approach

already exists in the acceptance of the National Board of Medical Examiners (NBME) test

results by individual State licensing boards; regulation, however, continues to be a responsibility of the licensing boards.

Licensing boards already have shifted away from standard setting, which has been

largely delegated to NBME in the formulation of both the Federal Licensing Examination

(FLEX) and the NBME exam, toward regulatory activities. If the boards are to continue in

this direction, then more systematic relationships could be formed with the various quality assurance programs now in place or being developed. For example, statistical deviations from accepted standards or norms could identify physicians that should be investigated further by the board.

12. The application of computer technology in the education and assessment of

other health professionals and in the education of patients deserves exploration.

Although this study focuses on physician education and assessment, the conclusions

and implications are directly applicable to the training and evaluation of other health

professionals. Nurses and allied health professionals could benefit from individualized

education in much the same way as physicians.

The development of individualized education and simulations for patient use (e.g.,

living with a colostomy, care and feeding of the newborn) linked with trends toward

home and personal computing could enhance patient involvement in and responsibility

for their own care.

13. The development of integrated data systems in health will raise complex technical, political, and social questions as to control of and access to such data.

Incompatible hardware and software, rapid obsolescence of hardware, security

needs, and the configuration of computer networks and their communication systems

will affect the use, coordination, and sharing of integrated data systems. Traditional organizational differences in objectives (e.g., education, testing, patient care, cost containment, and fraud detection) will also affect coordination and shared use. Storage of personal data in electronic data banks, control of access to data, and centralization of data

bases may create threats, real or imagined, to individual privacy.

Many questions will arise around issues such as access to information and the creation of an information elite. Specifically, as the general public becomes more literate in

computer capabilities and uses, issues will be raised about the types of information and

decisionmaking that can be handled only by physicians.
Chapter 2

COMPUTER TECHNOLOGY IN

EDUCATION AND ASSESSMENT
—

2 ●

COMPUTER TECHNOLOGY IN EDUCATION

AND ASSESSMENT

This chapter presents an overview of computer technology and its uses in education

and assessment. First, the definition and context of such computer uses are discussed, followed by a description of the types of educational computer systems that have evolved.

THE TECHNOLOGY

Description

The use of computers in education and assessment, often ca

education (CBE), involves a process by which an individual uses a typ

led computer-based

ewriter-like device,

a terminal, to communicate ‘with a computer. Information is transmitted into and out of

the computer through the terminal by means of a typed page or a special television

screen. Other forms of communication with the computer (e. g., voice) are available only

on a limited basis in very restricted settings. The terminal is connected by a communication link, such as a cable, microwave, or telephone, to a computer that may be distantly

located. The computer receives, stores, retrieves, processes, and outputs data. The terminal, computer, and the communication links are called hardware.

An educator or evaluator uses a simplified computer programing language, called an

authoring language, to translate learning materials into programs that direct the activities of the computer hardware. Computer programs are generally labeled software; the

subsets of programs that serve as the instructional materials are given the specific name

courseware. Records of individual performance and additional education and assessment

programs are stored in the computer or in storage hardware such as tapes and disks; storage hardware are called auxiliary memories. Taken together, hardware and software

comprise a system that allows the user to interact with educational or evaluative materials stored inside the computer.

Figure 1 portrays a sample interactive hardware configuration. The terminal and a

small computer may be combined with sufficient auxiliary memory to create a self-contained system, or they may be part of a larger system tied by a communication link to a

distant computer. Figures 2 and 3 portray sample interactive computer systems in the

medical education and assessment context.

The most suitable configuration of hardware and software for CBE depends on the

desired features of the instruction or assessment program. The choice of systems is influenced primarily by the availability of equipment at an institution or by the desire to use

specific courseware. Alternative systems vary, for example, in speed, reliability, ease of

programing, ease of use, operating costs, initial costs, capabilities, availability of courseware, and number of simultaneous users permitted by the equipment. Differences in

equipment and in software make courseware transfer from one institution to another less

13
—

14 ● Computer Technology in Medical Education and Assessment

Figure 1.—Sample Interactive Computer Configuration

SOURCE. The Ohio State University College of Medicine

than a routine activity. Smaller machines require a lower capital investment and have

lower operating costs, but they generally can accommodate fewer simultaneous users

and may have limited capabilities. Larger machines require a higher capital investment

and are more expensive to operate, but they provide more capabilities and permit larger

numbers of simultaneous users.

There is no consensus on explicit definitions of the descriptors “small” and “large;”

rather, these are subjective terms. However, three general categories of computers that

reflect these terms are commonly used: microprocessors, minicomputers, and large-scale

computers. These three categories are described briefly below.

Microprocessors are the smallest of computers. They can be used as stand-alone

machines that have a terminal “built-in” and, therefore, require no communication links.

They also can be used as sophisticated terminals that are linked to a larger computer.

They generally serve only one user at a time and have limited storage and processing

capabilities. Their greatest advantage is low cost; their greatest disadvantage is the paucity of existing courseware suitable for use on a microprocessor.

Minicomputers are the middle category of computers. They have more software and

courseware because they are generally easier to program than microprocessors. They can

execute instructional programs that are almost as complex as those in large-scale computers. Minicomputers, however, cannot serve as many simultaneous users as the larger

machines and are more costly than microprocessors.

Large-scale computers are the most costly and powerful computers. A typical largescale system can support hundreds of simultaneous users and the most complex software

and courseware. Clusters of terminals linked to the large system can be used for specialized training or other functions.

From the standpoint of user access, computers are generally categorized in the

following four ways: single user systems; small-scale, time-sharing systems; large-scale,

time-sharing systems; and networking,
Ch. 2—Computer Technology in Education and Assessment ● 15

Figure 2.—Sample Interactive Computerized System

in Which the Computer Simulates the Patient

-\ \

\

I

I

\ /

— . verbal

. . . . . . . . . . . .

. . . . . 

. . .

. sensory

. . . . . . abstract . . . . .

. . .. . . . . . . .

.. . .. .. .. . . . .

.: :. . . .. . .

. . . . . . ,

... . .. . . . . . .

. . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . 

. . . .. . . .

. . diagnostic

.. ..”.-..,. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

i

. . .> .. .. . . management

~: ..-. .

. . .

PATIENT

Live Clinical Interaction

 STUDENT

I

STUDENT

Computerized Clinical Interaction

Adapted from figures 1 and 2, p xiii, Toward the Measurement of Competence in Medicine, John R. Senior, 1976

Single user systems exist in the grey zone between the “home/hobby” computer market and the less sophisticated segment of the “office” computer market. In this category,

the terminal and a microprocessor or minicomputer are combined with sufficient auxiliary memory to create a self-contained system that does not require linkage to a larger

computer. Since the auxiliary memory is usually a flexible diskette that has limited storage capacity, a rather small number of programs or courses can be used at any one time;

the complexity of such programs and courses also may be restricted by storage limita-
16 . Computer Technology in Medical Education and Assessment

Figure 3.—Sample Interactive Computerized System

in Which the Computer Acts as the Teacher/Tutor

/

/

f 1

1

. .

INFORMATION \ \— . verbal

. . . . . . .

. . . . . . . . .“. . . . sensory

. . . . . . abstract . . . . .

. . . . . . . .

. . . . . . .

. . . .. . . . .

. . .

. . . . . . .. .. .

. . . . . . :: :.. . . .. . . . . .

. . . . . . . .

. . . . . . . . .

. .. .. . .. . . . . . . . . . . . . .

.. . . . . . . .. . . .. . .. . . . . .

. . . . . . .

[

. . . .

. . . .

. . .

TEACHER OR TUTOR PHYSICIAN OR STUDENT

Live Tutorial Interaction

Computerized Tutorial Interaction

Adapted from figures 1 and 2, p xiii, Toward the Measurement of Competence m Medlcme, John R Senior, 1976

STUDENT

tions. The lack of standardization in the manufacture of small computers increases the

difficulties of transferring courseware from one type of single user system to another

type. Several computer languages are available, including special purpose educational

languages (Gerhold and Kheriaty, 1978). The initial cost for such systems is usually less
Ch. 2—Computer Technology in Education and Assessment ● 17

than $10,000. * The costs for hardware operation and maintenance also are low. The degree of user control is high; computer professionals usually are not required to program

or maintain the system.

Small-scale, time-sharing systems include microprocessor- or minicomputer-based

machines that support from 1 to 16 simultaneous users (Tidball, 1978 b). No clear distinctions exist between this category of use and the previous one except in the number of

users able to interact simultaneously with the computer. Because of these systems’ larger

auxiliary memory capabilities, it is possible for many programs and courses to be continuously and simultaneously available for use. These systems are flexible, require relatively low initial costs, and include central program libraries. Programing and maintenance for such systems generally require special expertise. Initial costs for these systems

range from $10,000 to $100,000. The cost of additional terminals is highly variable, ranging from $1,000 to over $10,000 apiece, depending on the degree of independent

“intelligence” desired by the user.

Large-scale, time-sharing systems are, for the most part, dedicated only partially to

education and assessment because the full capacity of the minicomputer or large-scale

computer is not required for such uses. In a hospital, for example, excess computing

capacity may be used to support the medical information and hospital payroll systems.

The initial costs for these systems range from $100,000 to over $1 million, although the

costs for one terminal’s use of the system are typically less than $600 per year (Ohio State

University, 1977).

However, some large-scale, time-sharing systems are used exclusively for one purpose, such as education and assessment. Thus, their hardware, courseware, and communication links are used optimally for one activity (Hunter et al., 1975; Trends in

Computer-Assisted Instruction, 1978). This optimization can result in operating efficiencies and reduced costs in systems that are extensively used. The costs of such systems are

still relatively high because efforts have, and continue to be, directed at developing

greater capabilities, not at reducing costs. For example, highly sophisticated graphics,

touch panels, microfiche, voice recognition, and audio response capabilities increase

costs. As mentioned earlier, a typical large-scale system can support hundreds of simultaneous users and may provide sophisticated instruction for large training programs.

IBM offers systems that include education and assessment programs as only part of

their design. Examples of systems designed exclusively for education are time-shared interactive computer-controlled information television (TICCIT) and programed logic for

automated teaching operation (PLATO). Both TICCIT and PLATO have had extensive

funding support from the Federal Government. PLATO was started in 1959 at the University of Illinois (Seidel and Rubin, 1975) and continues to be developed by that institution. The TICCIT system began with a small amount of internal funding at the MITRE

Corporation in 1968 (Nuthmann, 1978) and is being developed currently at Brigham

Young University. In 1972, the National Science Foundation added millions of dollars of

support for completing and field-testing both systems (Stetten, 1972). Results of evaluations of the implementation of both systems have been mixed. Development of additional

courseware and more experience in privately supported use are required before conclusions can be reached regarding the efficacy and costs of these systems.

PLATO is now being marketed commercially by Control Data Corporation, and

TICCIT, by Hazeltine Corporation. The purchase price of TICCIT is approximately

*All figures in this chapter, unless indicated, reflect only computer hardware costs. Communication

links, software, and courseware are all additional costs.
18 ● Computer Technology in Medical Education and Assessment

$750,000 for 128 terminals (approximately $6,000 per terminal); PLATO costs $6,000 to

$20,000 per terminal, depending on the size of the computer supporting the system and

the number of terminals connected. A limited number of each system has been sold to

date.

Networking, the final category of use, involves computer-based learning materials,

developed at one site, that are shared with other institutions through a network of interfacing computers and leased telephone lines. The networking approach can be significantly cheaper since it does not generally require as many support staff at the user site

nor faculty expertise to develop CBE materials. Other advantages in joining a network

include access either to a central computer with a large library of tested courses, or to

multiple computers with libraries of courses; economies of scale; the availability of technical and educational network specialists for particular computer applications that an individual institution or user could not afford; wider exposure, and hence, better quality of

courseware; greatly reduced capital investments; expenditures only for computer services used; and opportunities for communication with a community of users. The network

user has access to a library of existing CBE courses, written in different computer languages by one or more authors, without the costs of conversion, installation, and maintenance of all software. The major disadvantages to networking are the lack of control

over computer resources and courseware and the continuing costs for long-distance communications access to the host computer(s).

The development of distributed networks will further opportunities for use of both

single user and time-sharing systems and minimize the disadvantages of networking. In a

distributed network, an entire computer program can be “down-loaded” from a small-or

large-scale, time-sharing system to a single user system. It can be used by the single user

system without connection to the network until an updated version is desired, user data

need to be transferred, or other functions are desired that require linkage to the program

library or host computer(s).

Trends

Several trends are evident in the future of computers that will affect the availability

and costs of CBE:

The number of computers will continue to rise. The number of computers in the

United States has risen from 1 in 1944, to 12 in 1950, to 6,000 in 1960, and to 50,000 general purpose computers in 1975. These figures do not include the almost unquantifiable

number of microprocessors that are in use. Added to the 40,000 computers in use around

the world, the total value of all computers exceeds $35 billion (Molnar, 1975). The

percentage of the U.S. gross national product spent on computer usage will increase from

2.1 percent in 1979 to 8.3 percent in 1985; per capita expenditures for computing will rise

from $101 to $670 in the same period (Nyborg et al., 1977). It is estimated that by 1980

every hospital with 200 or more beds will have a computer, and most homes and private

offices will have access to a computer of some type (Collen, 1974).

The costs of computing will continue to decrease dramatically. The most important

impetus for the dramatic reductions in the costs of computing is improved performance

of electronic circuitry, which can now move electronic impulses from one circuit to the

next in one-trillionth of a second (Branscomb, 1979). Silicon and integrated-circuit technology provides the basis for these advances, although new technologies, such as liquid

helium, are in development and may result in further advances. Such circuits are packed
Ch. 2—Computer Technology in Education and Assessment ● 19

closely together to enhance speed of transfer; they also account for the continued reduction in the size of computers.

The size of computers will continue to decrease dramatically. The computing power

of a machine that filled a large room 25 years ago is now contained in a machine that can

be held in one’s hand. “An individual integrated circuit on a chip perhaps a one-quarterinch square now can embrace more electronic elements than the most complete piece of

electronic equipment that could be built in 1950” (Noyce, 1977). The microprocessors or

personal computers of today, which cost as little as $300 to $500, have larger computing

capacities than those of the first computers, are 20 times faster, and are much more

reliable. Some predict that personal computers the size of an average dictionary will be

available within the next 5 to 7 years at less than the cost of one of today’s handheld

calculators. These future devices will have more computational power than all but the

largest of today’s machines (U.S. Congress, 1978).

Input and output terminals or mechanisms will become increasingly sophisticated.

Typewriter-like terminals, touch panels, and light pen technologies now are used for

learner input in interactive educational computing. Printed pages or visual screens are

the most commonly used means for the computer to output its messages. Currently, terminals can incorporate slides, soundtracks, microfiche, and similar technologies in the

learner-computer dialog. In some cases, these auxiliary technologies are user-controlled

and in others, they are computer-controlled. Voice recognition for input and voice synthesis for output are in early stages of use. Research is being conducted to explore the use

of thought waves or electronic brain impulses to interact with the computer (Fields,

1979).

Costs of software and courseware development will continue to rise. In light of all

the hardware advances and the decreases in their costs, the development of software and

courseware that take advantage of the capabilities of the machine represents the most

serious future problem regarding the use of computers. Currently, costs for courseware

design and development are much more expensive than hardware costs. The gap between

hardware and software costs will continue to widen as labor costs increase and hardware

costs decrease.

Author time required to create truly sophisticated CBE materials cannot be underestimated. Table 1 shows the increasing time requirements for courseware development

as the complexity of the courseware increases from simple linear tutorial questions to

linear tutorials with a variety of feedback options, to linear tutorials with branching

within questions, to linear tutorials with branching between questions, to fully individTable 1.—Computer-Based Education Course Development

Instructional Programmer

Initial author Author Coding and Author review

Type of CBE effort interaction debugging and revision

Linear. . . . . . . . . . . . . . . . 25 5 10 10

Linear with options. . . . . 30 5 10 15

Linear/within question

branching . . . . . . . . . . 40 10 30 20

Linear/between question

branching . . . . . . . . . . 80 30 50 40

Individualized . . . . . . . . . 100 100 50

Simulations. . . . . . . . . . . 3-1o (4-6:)’ 72-124 2-1o

Additional Course Total develop.

review documentary time

5 5 50

5 5 70

5 10 115

10 15 215

40 20 330

7-14 5-6 94-1 70”

NOTE: Average development times in hours for 1 hour of terminal Interact Ion or one case

—

“Time for generation of a case—this time is somewhat less due to an automatic authoring aid

SOURCE: The Ohio State University, College of Medicine, 1976.
20 ● Computer Technology in Medical Education and Assessment

ualized tutorial development, and to simulations. Times for courseware development

and maintenance range from 50 to over 300 hours; costs range from $1,000 to $15,000

per hour of courseware developed (Ohio State University, 1976).

Communication costs will continue to be high. Costs per hour of use depend on the

usable life of the course, the computer and communications hardware configuration, the

volume of use, and the degree of support services provided. Communication costs account for over 50 percent of hourly costs in most networks (Forman et al., 1978). Other

costs, such as those for user services, user training, billing, research, and development,

must also be considered. When all cost components are included, expenditures per student hour for CBE systems use vary greatly. One national network charges from $4 to

$15 per hour (Tidball, 1978a).

Personal computing will rapidly accelerate, posing new opportunities and problems.

The lower costs and smaller sizes of computers have led to their widespread use for hobby and entertainment purposes. In the near future, the use of microprocessors for personal needs will be as commonplace as the use of handheld calculators. Individuals, families, schools, and businesses all will have terminals containing powerful microprocessors

that can be used for a wide variety of functions, including education and assessment. The

development and availability of software and courseware will not keep pace with the increasing availability of hardware. Creativity and careful planning will be necessary to

fully realize the capabilities offered by the linkage of emerging technologies, such as

video discs, with microprocessor technologies. Microprocessor-based computer systems

serve not only as stand-alone systems but also as linked systems. Microprocessor systems

can be linked to “view data” systems such as the British Post Office System; this system

offers stock market status information, consumer reports, bibliographic indexes, electronic financial transactions, and mail-order catalogs (Institute of the Future, 1979).

Home- and office-based education (especially continuing education) will be available as a

result of linking microprocessor systems with extensive data bases.

COMPUTER-BASED EDUCATION

Methods

Specifying the knowledge to be obtained, the skills to be learned, and the standards

of acceptable performance is a prerequisite of any education and assessment system.

Educators refer to the specification of measurable knowledge and skills as behavioral objectives. Ideally, then, instruction should be designed to teach objectives; tests should be

developed to measure attainment of objectives; and administrative mechanisms should

be developed to monitor, record, and manage activities in the educational process. CBE

methods are similarly divided into three categories to reflect the three components of the

educational process: computer-assisted instruction, computer-assisted testing, and

computer-managed instruction.

Computer-assisted instruction (CAI) involves the computer in the presentation of instructional material. The computer is combined with other media, such as lectures, films,

and labs, in the instructional process and is used ideally for those aspects of the teaching/learning process that can benefit most from CAI capabilities.

Computer-assisted testing (CAT) can be on-line, when the test is generated by the

computer and administered at a terminal connected to a computer, or off-line, when the

test is administered manually by paper and pencil. The computer can support testing in

any of the following ways: (a) store and retrieve banks of test items, allowing for efficient
Ch. 2—Computer Technology in Education and Assessment ● 21

generation of numerous alternate tests; (b) administer tests; (c) score and print results of

tests; (d) maintain testing records for purposes of test validation and/or student diagnosis and prescription; and (e) catalog and reference instructional materials relating to

specified knowledge and skills.

Test items can be selected for presentation according to either a set of rules, statistically determined prior to testing, or the individual’s responses to items in the current

testing situation. Both testing mechanisms take less time to complete than traditional

tests, prevent redundancy in test items, provide consistency and standardization for the

entire target audience, provide equivalent alternative versions of tests, and allow for

more precise measurement of achievement and ability. Learners are not retested on previously mastered knowledge and, hence, receive only those test items that are necessary.

The reliability and validity of the tailored tests are comparable to those of traditional

tests containing twice as many items (Weiss, 1976; Be jar et al., 1976; Patience, 1977), and

immediate information regarding the correctness of the response given after each item

can alleviate test anxiety (Weiss, 1976). Automatic maintenance and availability of performance records assist in test validations and in the recognition of gaps in learner

knowledge; such gaps may be overlooked in conventional testing because an adequate

overall grade may mask important specific deficiencies (Kimberlin, 1973). Generally, the

response of test-takers to all types of computer-assisted and tailored testing is positive.

Computer-managed instruction (CMI) manages the learning process by assessing the

student’s level of skill and knowledge through testing, by maintaining records of student

progress through a course or within individual sections of a course, and by prescribing

the next step a student should take in his/her studies (Seidel and Stolorow, 1975). CMI

can route a student forward to the next section or back to remedial sections, give tests,

and direct the learner to written materials, faculty consultation, or CAI courses. CMI can

coordinate both the computerized and noncomputerized portions of the entire instructional process.

Strategies

CBE employs a variety of educational approaches or strategies, including drill and

practice, tutorials, simulations and games, and inquiry. These strategies have been used

for both individual and group-oriented education. The use of CBE by groups is relatively

new (Bergin et al., 1978a & b).

Drill and practice exercises require the user to complete a series of exercises, for example, balancing equations in acid-base calculations, that relate to a particular area of

knowledge. The computer can use random number generators or complex algorithms to

create an almost endless set of exercises of any given difficulty level that are specifically

tailored to the needs of the individual learner.

Tutorials are instructional situations in which the learner interacts with computer

programs that model a teacher, Such tutorials are designed to meet the individual needs

of each student by allowing “instructors” to be continuously present and, therefore,

responsive to specific learner needs. Student feedback, remediation, as well as testing and

recordkeeping, can be accomplished by a computer instead of by a human instructor. Tutorial programs allow problem solving, question answering, and other similar capabilities. “Intelligent tutors” that are built into these systems offer the student hints, track

student errors, and present alternatives to identify overlooked or misunderstood information. A schematic drawing of a tutorial CBE item is shown in figure 4.

51-541 0 - 79 - 3
22 ● Computer Technology in Medical Education and Assessment

Figure 4.—Schematic of a Tutorial Computer-Based Education Question

QUESTION 1 <

1

CA

W A

I

@

UN

UN

>

\

CORRECTIVE FEEDBACK

BRANCH TO REVIEW UNIT \

/

/

CORRECTIVE FEEDBACK

DEFAULT —> TYP E

NEXT

STUDY PRESCRIPTION

OR SUBSEQUENT QUESTION

\

K EY : C A = C ORRECT A NSWER

WA = W RONG A NSWE R

UN = U NANTICIPATED RESPONS E

Simulation and games are instructional strategies that allow participation in a situation closely resembling an actual experience. The sophistication of the computer allows

for generation and use of simulation rules and enables complex simulated decisionmaking by the student in accordance with these rules. Simulations permit students to perform

“dangerous” and “critical” tasks. They also telescope real-world time, abstract essential

task elements from a potentially confusing total environment, and expose the user to a

wider variety of environmental variations. For example, flight simulators have been used

in aviation since 1939. Real flight accommodates only 6 practice landings per hour; flight

simulations allow 30, and can be conducted despite airport traffic, unfavorable weather,

and initial pilot inexperience without risk to personnel or equipment (Orlansky and
Ch. 2–Computer Technology in Education and Assessment ● 23

String, 1977). Through 1981, the total Department of Defense investment for flight

simulators will approach $2 billion (Allen, 1976). Aircraft control, submarine control,

navigation, automobile control, medical diagnosis and treatment, business management,

population and ecology studies, and dozens of other complex situations also have been

simulated for the purposes of training and testing both individuals and groups. The skills

developed through simulations transfer well to on-the-job performance (Puig, 1976).

Since the 1950’s, there have been efforts to construct computer programs that simulate human thought processes through artificial intelligence. Research groups at Bolt,

Baranek, and Newman; Stanford University; and the Massachusetts Institute of Technology have developed programs to teach logic, symbolic integration, and automatic cri -

tiquing of student achievement. Using artificial intelligence, the computer program itself

tries to simulate a human tutor (Papert, 1975; Brown et al., 1977). Recent work in this

area includes the “sophisticated instructional environment” (SOPHIE), which is being

used by the Navy to teach electronic troubleshooting. This program creates a “reactive”

situation in which the student learns by experimenting with his/her own ideas (Brown,

1975). Currently, the major source of financial support for artificial intelligence development is the Department of Defense, although small projects have been funded by the National Institute for Education and the National Science Foundation (U.S. Congress,

1978).

Computerized games create a cooperative or competitive environment in which the

student interacts with real or artificial participants to achieve specified goals. Simulation

and games are often used together as instructional tools for teaching problem-solving

skills. In electronic troubleshooting, for example, partners can introduce faults into a

simulated machine and the “winner” both diagnoses his/her problem and provides a

solution first. Data requested from the computer by either player may be helpful to both.

Inquiry permits the learner or user to address questions to the computer. To process

the questions and provide answers, the computer uses key words and stored responses.

For example, the computer could be queried as to contra-indications for drugs or serve as

a “consultant” for the diagnosis and management of patient problems. This use differs

from artificial intelligence because queries are compared to specific answers rather than

being interpreted according to a general rule or algorithm.

Funding

During the past 20 years, CBE development and testing have been supported by Federal, State, and local governments; educational institutions; private foundations; the

military; large industrial firms; the computer industry; and, to a lesser extent, the publishing industry (Tennyson, 1977). Table 2 shows the results of a 1974 study of expenditures by developers for CBE programs in all subject areas.

More recent data on funding for health-related CBE projects between 1968 and 1978

show three primary funding sources: the Federal Government, private medical foundations, and medical boards. Their combined support totaled over $9 million (Association

of American Medical Colleges, 1979). Among the Federal agencies, the National Center

for Health Services Research, the National Library of Medicine, and the Bureau of Health

Manpower provided over $5.4 million. Almost .50 percent of these Federal funds went for

direct support to medical school programs that were developing, using, or evaluating

CBE; over 35 percent was directed to improve the cost-effectiveness of computers in
24 ● Computer Technology in Medical Education and Assessment

Table 2.—A Sample Spread of Computer-Based

Education Funding and Development Sources

as of 1974

Percent of CBE programs

Categories of developers developed as of 1974*

. —

National Science Foundation. . . . . . .

Colleges and universities . . . . . . . . . .

Military ., . . . . . . . . . . . . . . . . . . . . . . .

U.S. Office of Education . . . . . . . . . . .

Private sector . . . . . . . . . . . . . . . . . . . .

Public schools . . . . . . . . . . . . . . . . . . .

National Institutes of Health , . . . . . .

Other Federal agencies. . . . . . . . . . . .

Other—foundation

State agencies . . . . . . . . . . . . . . . . .

Community colleges . . . . . . . . . . . .

Professional societies. . . . . . . . . . . .

300/o

29

10

9

9

8

1.8

1.8

1.4

Total, . . . . . . . . . . . . . . . . . . . . . . . 1000/0

— — .

“Is somewhat Inflated since It Includes Iearnlng about computers (e g , comput.

er science and programming) as well as learning wlth computersd (e g , CBE)

SOURCE Human Resources Research Organizatlon, 1974

cost-effectiveness of computers in health education; and the remainder was dedicated to

projects designed to educate physicians in the use of computers.

Among private medical foundations, the National Fund for Medical Education, the

Robert Wood Johnson Foundation, the Kellogg Foundation, and the Merck Foundation

have expended over $2 million on a wide variety of CBE projects. Medical boards, such

as the National Board of Medical Examiners, the American Board of Internal Medicine,

and the American College of Physicians, jointly spent almost $2 million in the development of CBE materials for the assessment of physician competence.

Federal funding levels for CBE research and development have decreased in recent

years (U.S. Congress 1978). This reduction is due partially to the inability of CBE researchers to document initial claims of large cost decreases and proficiency increases in

education and training as a result of CBE use. In many situations where CBE proved

useful, State, local, or private support was used to replace Federal money (Montgomery

County Public Schools, Ohio State University, etc.).

Limitations and Capabilities

Computer-based education is most frequently applied to test knowledge and skills in

areas in which correct responses do not involve an affective component, such as attitudes

or preferences. In situations where an evaluator must observe the performance of certain

skills (e. g., surgery), assessment by computer is ineffective, if not impossible. Additionally, required individual and organizational changes often inhibit effective CBE introduction, diffusion, and utilization (Casburgue, 1978). Despite these and other limitations, CBE can be an improvement over more conventional educational and evaluative

methods. Table 3 summarizes the results of 32 studies that are recent and representative

samples of the many CBE evaluation studies. The findings of these studies are summarized below.

Savings in learner time to complete a course of study were shown in the great majority of the studies, with as much as 50-percent savings in training and testing time. The

reorganization of instruction to an objectives-based form, a prerequisite for CBE use, can

itself cause significant learner timesaving.
Ch. 2—Computer Technology in Education and Assessment ● 25

.

s

s

T

s

s

s

T

s

s

s

T

T

s

T

T

s

s

s

s

s

T

T

s

Table 3.—Thirty-two Studies of Computerized Simulations and Testing

—. . —.

Source

Abernathy and McBride, 1978 . . . . •

Allen, 1976 . . . . . . . . . . . . . . . . . . . . .

Bejar et al., 1977 . . . . . . . . . . . . . . . .

Bentz, 1975 . . . . . . . . . . . . . . . . . . . .

Brown et al., 1977 . . . . . . . . . . . . . . .

Brown, 1977. ....., . . . . . . . . . . . . .

Brown and Weiss, 1977 ... . . . . . .

Buchanan, 1978 ... . . .

Ellis, 1978 . . . . . . . . . . . . . . . . . . . .

Gregory, 1975.......,.. . . . . . . . .

Guerra et al. 1977. . .

Hansen et al.,1974. . . . . . . . . . . . .

Johnson, 1976.. . . . . . . . . . . . . . . . .

Lippey and Partos, 1976. . . . . . . . . .

McLain and Wessels,1975. . .

Misselt and Call-Himwich, 1978 . . .

Mockovak et al.,1974, . . . . . . . . . .

Orlansky and String, 1977 . . . . . . . .

Puig,1976

Tanks . . . . . . . . . . . .

Aircraft carrier. . . . . . . . . .

Weapons trainer. ., . . . . . . . . .

Airtraffic controller . . . . . . . . . .

Automobile . . . . . . . . . . . . . . . .

Airborne Ecosystem. . . . . . . . .

Roberts, 1977 . . . . . . . . . . . . . . . .

Sealy,1975. . . . . . . . . . . . . . . . . . .

Vale, 1977 . . . . . . . . . . . . . . . . . .

Willey,1975

Dartmouth . . . . . . . . . . . . . . . .

Ohio State . . . . . . . . . . . . . . . . .

University of Wisconsin. .

University of lllinois . . . . . . . . .

University of Michigan ...,

Totals. . . . . . . . . . . . . . . .

—————— —

Time saved

+

+

+

+

+

+

+

+

+

+

+

+

o

+

+

—

+

+

o

+

+

+

o

+

+

+

+

+

+

o

+

+

+ = 27

— = 1

0 = 4

Greater Improved

———— Cost saved efficiency ————skills —

+

+

0

—

0

0

+

+

0

+

0

0

0

—

+

+

0

+

0

+

+

+

0

+

0

—

0

+

13

7

12

—

+

+

+

+

+

+

+

+

+

+

+

+

0

+

+

—

+

+

0

+

+

+

+

—

+

+

+

+

+

+

+

+

~8-

2

2

+

+

+

+

+

+

+

+—

+

+

+

+

0

+

—

+

+

0

+

+

+

+

+

+

0

+

+

—

+

+

+

26

3

3

Provision of

training not

previously

available

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

—

(conventional

—

superior)

+

—

+—

—

—

—

+

+

+

+

+

+

+

+

25

7

0

Key S = simulation

T = testing

+ = positive results

— = no slgnificant difference

O = no results

SOURCE Human Resources Research Organization 1978

Cost savings findings were mixed. These findings reinforce the need for further

studies in this area. One such study could examine linking particular educational needs

with the use of different media. For example, instructor provision of tutorial sessions for

three or four students may be more cost-effective than CBE sessions of the same material

largely because of the high developmental cost-to-use ratio for CBE. If there is a need for

frequent repetition of the sessions over an extended period of time, however, CBE may be

more cost-effective. Examples of cost savings are evident in the computer-based simulations of the University of Michigan’s Psychology Department; these simulations are carried out at one-half the cost per student hour ($7 versus $14) of live experimentation

(Willey, 1975). IBM also realized cost savings from CBE use. IBM’s Field Engineering
26 ● Computer Technology in Medical Education and Assessment

Division has used CBE since 1964. As a consequence, the number of their education

centers decreased from 17 to 3, and the costs of training were reduced by 60 percent

(Long, 1978).

Greater efficiency in learner functioning was demonstrated in the majority of these

studies. Learners showed greater achievement per unit of time spent because they could

proceed at their own pace, receive continuous feedback on their individual performance,

take tests at appropriate points, and discuss materials presented with their peers. Tests

could be tailored to precisely assess the individual’s performance, not only by giving a

score, but also by identifying any gaps in knowledge. Since clerical chores of grading and

remediation are part of most CBE programs, the efficient use of instructor time also increased.

Improved skills were shown in 80 percent of the studies in such diverse areas as flying and flight maintenance, merchant marine and submarine operations, personal interviewing, business forecasting, and on-the-job performance of telephone servicepersons

(Puig, 1976; Roberts, 1977).

Provision of training not previously available was characteristic of the large majority of the studies analyzed. Simulation for training purposes is a prime example of this

uniqueness: simulation can provide practice of many kinds that only years of experience

could previously have provided. For example, weather, traffic, time, maintenance, and

fuel costs make training of airline pilots or tanker captains in real craft impractical (Puig,

1976); and financial, political, and physical phenomena cannot be adjusted for training

purposes.
Chapter 3

MEDICAL EDUCATION

AND ASSESSMENT
3 ●

MEDICAL EDUCATION AND ASSESSMENT

INTRODUCTION

Although a clear demarcation exists between the formal education process of physicians in undergraduate and graduate (residency) education and the more informal learning process accompanying active patient care, it is now commonly recognized that medical education is a continuum spanning both processes. This recognition has been manifested in two ways.

First, continuing medical education for physicians, initially implemented on a voluntary basis, now is a formal requirement for reregistration by many State medical licensing boards and for recertification by some specialty boards.

Second, in evaluating the quality of care provided by physicians, criteria are usually

grouped according to structure, process, or outcome measures. Structural measures refer

to the availability of resources, including facilities, equipment, and the numbers and

types of different health care personnel. Structural measures also include qualitative aspects of the personnel providing medical care, such as the extent of educational background and board certification in a specialty area. Process measures usually assess

whether or not diagnosis and/or therapy have been appropriately conducted according

to norms or standards, or the medical profession’s average care or judgment of what that

care should be. Outcome measures reflect the end results of medical interventions and relate to the condition of the patient.

Outcome measures could tell us more precisely when and how medical care can help

patients. However, knowledge is still rudimentary concerning the relationship between

medical interventions and specific health outcomes. Ideally, structure, process, and outcome measures could be related. Specific medical interventions (process) could be correlated with improvements in health status (outcome). This correlation would then be

used to modify the learning process (structure): the content of education would be improved by learning from experiences in the actual practice of medicine.

Medical education and assessment as continuous processes are most observable in

the formal and informal education and assessment activities (continuing education requirements, recertification, peer review systems) that have appeared in clinical practice.

The continuity is also reflected in the trend to revise present educational and assessment

strategies to more accurately reflect the explosion of medical knowledge and the fact that

physicians practice in specialized areas of medicine. The amount of knowledge that is

needed to address all types of medical problems is too large for any physician to consistently retain, recall, and apply.

cialty of medicine, yet knowledge

cialties. The medical information

methods of medical education and

Physicians now need to know more-about every speand medical capabilities are expanding across all speexplosion also results in changes in the content and

in the use of medical knowledge.

29
30 ● Computer Technology in Medical Education and Assessment

Changes are occurring in medical licensure as a result of this reformulation of educational and assessment strategies. The content areas to be tested are changing, and the

types of medical care that individual physicians are authorized to provide also may

change. Such changes would lead to a modification in the role of State licensing boards

and possible involvement of the medical specialty boards in the licensing of physicians.

The continuity of medical education and assessment activities is graphically summarized in figure 5.

UNDERGRADUATE MEDICAL EDUCATION

Admissions

Medical school is the common denominator for all types of physicians. Students are

selected from a pool of applicants on the basis of generally uniform criteria, modified by

the desire and need for diversity. Schools seek diversity in their student bodies voluntarily, as a result of judicial actions, and/or as a condition of Federal and State financial support. Geographic origins and minority/disadvantaged applicants, for example, are now

factors in the admissions process.

The geographic origin of a student is seen not only as a factor in student body diversity, but also as an important determinant of practice location. Consequently, attempts

to influence the future geographic distribution of practicing physicians may be reflected

in medical school admissions criteria. Such criteria are intended to improve access to

medical services.

Favoring minority/economically disadvantaged applicants in the admissions process also addresses access questions, but of two different types. Access to medical services

is one; students selected on the basis of these criteria are assumed to have a greater rate of

practice in the types of communities from which they come. The primary consideration,

however, is access into the medical profession by the disadvantaged.

Factors entering into the decisions of medical school admission committees include:

1) Medical College Admissions Test (MCAT) results; 2) the applicant’s academic background and overall grade point average, difficulty of courses taken, and performance in

related courses such as biology, chemistry, physics, and mathematics; 3) biographical

data, including applicant’s State of residence, ethnic background, and participation in

nonacademic activities such as student government; 4) recommendations and reports

from the applicant’s premedical health profession advisor; 5) information and impressions from the personal interview, which is intended to allow the applicant to demonstrate communication and personal interaction abilities, and includes questions regarding

the applicant’s plans for career and location, as well as motivation for the study of medicine; and 6) results of other assessment instruments such as Miller Analogies, MyersBriggs personality assessments, and any special knowledge tests used by selected medical

schools.

Each school’s admissions committee has its own particular needs and methods of

judging the relative importance of each of the above criteria in choosing among applicants. The mix of criteria used depends largely on a school’s previous experience with

accepted applicants. In addition, most State-supported medical schools favor resident

applicants by limiting the admission of nonresident students to 10 to 15 percent of the

entering class (Association of American Medical Colleges, 1978a).
Ch. 3—Medical Education and Assessment ● 31

Figure 5.— The Medical Education and Assessment Continuum

Stages

Premedical Education Undergraduate Medical Graduate Medical Education Act we Medical Practice

and Admission Education Time spans vary) (Llcensure and Certlfication Periods vary)

Typical Time Spans

Llcensure Period I

(Repeat)

1 2 3 4 1 2 3 to 7

Certification Period (Repeat)

ASSESSMENT

MECHANISMS

Performance I 0I

I Specialty

t

I I

I

Board

Exams

/

I I (Optional)

MCAT I

or I

I

/

I

I

(Vertical arrows I

I Requirements,

refer to I Part I Part II I Part Ill if any

Time Periods) I I

(CME, etc.)

I 1

I

/

Recertification

I or

I

Requirements,

if any

I I

I

(CME, etc )

/

I

! I

I I

/

I I

Leading To I I

I

I

I 1

4

+ I

“CREDENTIALING” Graduation Graduation NBME I t

I Board Elgible

I

/

I

= Licensure

(Upon Completion of

I Approved Residency)

}

:

I

I

, + Practice Time

I

(Some Specialty)

I

I

I

I

/

I

+ Board Exams

I 1 = Certification

I

I

i

I

I

I

I

I

/

Relicensure

/

I

i

I

I

I

I

1

Recertification

I I

I

I

I

I

/

I

/

I !

~

I

I ~ Hospital Prlvileges Etc

I

In 1977-78, there were 15,493 first year positions available in U.S. medical schools.

There were 40,569 applicants competing for admission, of which 15,977 (39.4 percent)

were accepted for first or later year positions (Gordon, 1978). Table 4 summarizes trends

and projections in the numbers of medical schools, graduates, and physicians in practice

between 1960 and 1990.
32 ● Computer Technology in Medical Education and Assessment

Table 4.—Number of Medical Schools, Graduates, and Physicians in Practice,

Selected Years

Number of physicians

Year Number of schools’

————

Number of graduates} M.D. D.O.

1960 . . . . . . . . . . . . . . . . . . . . 85 7,081 247,300 12,000

1970 . . . . . . . . . . . . . . . . . . . . 101 8,367 311,200 12,000

1975 . . . . . . . . . . . . . . . . . . . . 114 12,714 364,500 14,100

1980 (projected) . . . . . . . . . . 122-124 16,086 426,300 17,700

1990 (projected) . . . . . . . . . . — 18,318 564,200 29,800

‘M.D. schools and graduates only

SOURCES “Medical Education in the United States, 1977 -1978,” JAMA 2402822 (December 1978). Bureau of Health Manpower, Health Resources

Administration, DHEW, 1978

Curriculum

The majority of U.S. medical schools conduct traditional 4-year undergraduate

medical education programs. Approximately 6 percent of the schools have programs in

which the student can receive the M.D. degree in 3 years (34 to 36 months) (Association

of American Medical Colleges, 1978b). In this setting, the student takes coursework

essentially on a 12-month basis. Although there are slight reductions in the total hours of

some courses, the basic content of these courses is similar to the courses in traditional 4-

year programs. Some schools accept students into a special program immediately following graduation from high school. Assuming normal academic progress, the students

graduate approximately 6 years later. Other schools accept highly qualified students

after the junior year in undergraduate college and award the M.D. degree 4 years later,

for a total of 7 years.

The general 4-year program of undergraduate medical education is divided into two

major portions—the basic medical sciences and clinical sciences. The basic sciences

generally span the first two academic years of the program and include anatomy, biochemistry, physiology, microbiology, pathology, and pharmacology. These disciplines

may be taught either independently as separate disciplines, or together as separate

aspects of particular human organ systems, such as the cardiovascular system or the central nervous system. In either case, the material is taught via a mix of lectures, laboratory

exercises, and discussion and/or correlative sessions. When the curriculum is taught by

the separate disciplines, the usual sequence is anatomy, biochemistry, and physiology in

the first academic year; microbiology, pathology, and pharmacology are taught in the

second academic year. In interdisciplinary curricula, where the disciplines are taught

together within particular human organ systems, there is no given sequence of courses arranged according to discipline. Other types of nontraditional curricula are offered in

which the student is permitted to study and progress through the curriculum at his/her

own pace.

A variety of educational support services are employed in the medical school program. Although the primary form of presenting didactic information is the lecture, other

associated learning resources include audiovisual aids, case study materials, and computer-assisted instruction (CAI). Sixty percent of U.S. medical schools incorporate CAI into

the curriculum. CAI is generally in the form of independent units of instruction that are

available to the student upon his/her request (Association of American Medical Colleges, 1977).

Attempts to introduce early patient encounters during the two basic science years as

a means of demonstrating the relationship between the basic sciences and patient care are
Ch. 3—Medical Education and Assessment ● 33

increasing. Through these programs, there are opportunities to use the knowledge

learned and to expose students to problem-solving and patient diagnosis exercises.

In the transition from the basic science to clinical science years, learning through formal lectures decreases while learning through working with patients increases. The transition to clinical medicine is aided by the development of physical diagnosis, history taking, and physical examination skills and by introductory courses in the various major

clinical specialties.

There are generally six clinical areas in which the student is required to serve: family

medicine, internal medicine, obstetrics and gynecology, pediatrics, surgery, and psychiatry. Students serve a period of 8 to 12 weeks in each of these six clinical areas. During

each clerkship, students learn via lecture, discussion, and considerable interaction with

clinical faculty on their rounds in the hospital or clinic. Students conduct physical

examinations, take patient histories, and participate in the formulation of treatment

plans and requests for laboratory analyses. As the student progresses through a particular clerkship, he/she is required to assume an increasing share of responsibility for

patient diagnosis and treatment under the supervision of the clinical faculty and house

staff. The student is evaluated in each clinical clerkship by the faculty member(s) supervising the clerkship and by house staff.

Following the required rotations, which usually occur during the entire third year of

medical school, the student may select certain specialty areas according to interest and

career choice. Fourth year electives can be either a mixture of basic science areas and

clinical specialties or totally clinical electives. The student interested in medical research,

for example, might choose to take additional basic science coursework as well as clinical

electives. Regardless of the particular choices, the student spends the majority of the

fourth and last year of medical school in his/her chosen area.

Assessment of Student Performance

The purpose of evaluation in undergraduate education is to assure that students

achieve the educational objectives of the medical school and thereby qualify for the

academic degree. The extent of a student’s knowledge is assessed through objective tests

that are administered throughout the student’s medical school career. Student acquisition

and use of clinical judgment skills are assessed primarily through the use of evaluation

forms in the clerkship phase of medical education.

Within the basic sciences, objective examinations are both written and practical.

They are designed essentially to assess the acquisition of knowledge and understanding

of concepts within each discipline of the basic medical sciences. Faculty use their own

tests and simulations as well as more widely used and standardized ones. The National

Board of Medical Examiners (NBME) provides standardized tests for intramural use by

medical school departments and faculty for student assessment. Based on the student’s

performance on locally developed examinations, he/she either advances in the program

or is required to take remedial and/or tutorial instruction. The particular pass/fail

criteria are a matter of the individual department’s policy, Students often take the comprehensive examinations administered nationally by NBME at the end of the basic science portion of medical school (NBME Part I). Approximately three-fourths of all medical schools also require students to take Part I, and some schools also require Part II

(Association of American Medical Colleges, 1978b). The extent to which medical schools

require passing performance on this exam varies. Some schools require a passing score on
34 ● Computer Technology in Medical Education and Assessment

each discipline section; others, an overall passing score. Some require the student only to

take the exam. The degree to which the NBME Part I scores contribute to the student’s

evaluation varies considerably among schools.

In addition to these methods of evaluation, many programs offer opportunities for

self-evaluation. Often these self-evaluation methods utilize computer-simulated situations and computerized sets of test items and test situations. The availability of these

facilities allows students to continuously assess their performance and progress.

The methods of assessing student performance in the clinical sciences are quite different from the basic sciences. The clinical evaluation contains a high degree of subjectivity and, although a standard institutional form may be used, often the clinical sciences

are designed to determine the student’s ability to integrate and synthesize the basic science information in clinical settings. Written, oral, and practical evaluations occur in

each of the clinical rotations; and emphasis is placed on the student’s ability to interact

with patients, conduct physical examinations, take patient histories, make accurate diagnoses, and properly adapt basic science knowledge to the diagnosis and treatment of patients.

The assessment of student performance and preparedness during the course of

undergraduate medical education is only one of the purposes of the assessment activity in

medical school. A second and equally important purpose is to determine the adequacy

and effectiveness of the curriculum itself. In most medical schools there exists an Office

of Medical Education Research and Development that develops the internal mechanism

for continuous student and curriculum assessment within the medical school.

GRADUATE MEDICAL EDUCATION

Graduate medical education (GME) is a period of training that leads to qualification

in a specialty. Medical students begin interviewing for graduate medical education positions (residencies) in the spring, summer, and fall between their junior and senior years.

Students usually apply through the National Residency Matching Program (NRMP) for

first-year appointments by the end of December of their senior year. Residency appointments are released in the spring.

Although based on different criteria, the same type of assessment that occurs in the

medical school application process also occurs in the residency application process. Program directors review the applicant’s academic performance in undergraduate medical

education, recommendations of the faculty (particularly clinical faculty), and, if applicable, performance on Parts I and II of the NBME test. The results of the program

director’s interview with the applicant, if conducted, are also considered. Finally, the

graduate program director’s past experience with graduates of specific medical schools

enters into the consideration.

The first year of GME is designated as the first year of an approved residency program. Beginning in July 1975, the term internship has not been used in approved graduate training programs (American Medical Association, 1977a). Although an integral part

of the approved residency is a specialty, the first year is designated as either a categorical

or a flexible first year. Such categorization reflects the fact that the first-year residency

program is not necessarily limited to one specialty. One type of categorical first-year residency limits the program’s content to the specialty field of the sponsoring programs. A
.

Ch. 3—Medical Education and Assessment

second type may include two or more specialty fields as determined by a sponsored

● 3 5

program. A flexible first year is sponsored by two or more approved residencies and is jointly planned and supervised by the directors of such residency programs.

Following the first year of GME, additional periods of education are required to

satisfy qualifications for specialty board certification. The years of required training for

each specialty are summarized in figure 6.

During residency, the assessment of professional attitudes, self-discipline, communication skills, and various aspects of clinical judgment is accomplished chiefly through the

subjective evaluations of the attending physicians and program directors. A number of

specialty boards are now encouraging and introducing evaluative techniques for use during the residency training period. These in-training examinations assist the resident and

the program directors to identify the resident’s strengths and weaknesses in diagnostic

techniques, medical procedures, and basic scientific knowledge in the specialty area.

LICENSURE AND SPECIALTY CERTIFICATION

Licensure

States and U.S. territorial jurisdictions (the District of Columbia, Puerto Rico, the

Virgin Islands, and Guam) have the authority to license physicians. A physician must be

licensed in each State or territory in order to practice there, although licensure reciprocity exists between most of them. Each jurisdiction establishes, through laws and regulations, the requirements for eligibility to take the licensing exam.

There are two primary examination pathways for licensure in the United States: 1)

Parts 1, 11, and III of the NBME exam, and 2) the Federation Licensing Examination

(FLEX). FLEX is constructed by the Federation of State Medical Boards and produced

through NBME. It is the examination administered by individual State medical licensing

boards. For foreign physicians, some States accept a Canadian license; others accept specialty board certification.

The NBME examination is composed of three parts. Part I, usually taken upon completion of the basic medical sciences, assesses the student’s grasp of knowledge and concepts within the basic sciences. It is a comprehensive, 2-day examination consisting of approximately 1,000 multiple choice questions equally distributed among the disciplines of

anatomy, behavioral sciences, biochemistry, microbiology, pathology, pharmacology,

and physiology.

Part 11, taken either during the third or fourth year of medical school, is designed to

measure knowledge and comprehension of the clinical sciences. Approximately 900 multiple choice test items are administered over a 2-day period. Questions are derived equally from internal medicine, obstetrics and gynecology, pediatrics, preventive medicine

and public health, surgery, and psychiatry.

Part III, usually taken during the first year of graduate medical education, focuses

on problem-solving abilities. The exam utilizes programed testing in the form of patient

management problems as well as multiple choice questions, many of which require interpretation and analysis of illustrations, graphs, and tables of data.

Since the sequence of the NBME exam begins during undergraduate medical education, the majority of students successfully completing Parts I and II will use the NBME as

their method of achieving licensure. Approximately 80 percent of U.S. medical school
3 6 • Computer Technology in Medical Education and Assessment

Figure 6.— Years of Graduate Medical Education

I

I I

Pediatrics

Allerg~!mmun~logy1 Internal Medicine 1

I

I

I

I

I

I

1 I I I

1 I

‘

b

I

I

) I

I

I

i

1

I

I

* I

I I

I

I

I

I

I I I

I I

I

I

I

I I

I i

I 1

I I

I

I

[

! 1 I

I

I

I

I

;

I ● I I

I I

I

-,”, # !“

‘ - n

J

i I

I

i

I

I

I I

Flex 1 I

I

I

I I

SOURCE: Graettlnger, 1977
—

Ch. 3—Medical Education and Assessment ● 37

graduates achieve initial licensure by successfully completing Parts I, II, and 111 of the

NBME exam (American Medical Association, 1977b). The percentage of graduates

achieving licensure through the NBME exam is influenced by the fact that a number of

schools require their students to take and pass Parts I and/or II of the NBME exam as a

part of their intramural evaluation system. However, the student always has the option

to choose between the two certifying methods, regardless of whether or not parts of the

NBME exam were taken in medical school.

If the physician does not take the NBME tract, he/she is required to take FLEX. Candidates usually take FLEX during the first year of graduate education. The examination is

composed of three parts that must be taken at one time in 3 successive days. FLEX also

differs from the NBME exam in its method of scoring; less emphasis is placed on the basic

sciences. Although FLEX, at any one administration, is the same for all States, and all

boards require a minimum weighted average score of 75, there are interstate variations

on the acceptable minimum scores for individual subject areas. Thus, although the reported scores are uniform, there is some variance among licensing jurisdictions regarding

acceptable passing scores (Merchant, 1978).

The composition of FLEX is derived from test items within those developed and validated by NBME. Questions are selected by committees, the members of which represent

various State medical licensing boards. The 3 days of tests are similar to Parts I, II, and

III of the NBME exam. The first day covers the basic medical sciences, the second day

covers the clinical sciences, and the third day covers clinical problem-solving.

FLEX is the main route to licensure for foreign medical graduates; 78 percent of candidates taking FLEX in 1977 were foreign medical graduates (Merchant, 1978).

None of the State boards require a relicensing examination. However, as of December 1978, 23 States and Puerto Rico had enacted legislation or written regulations requiring continuing medical education (CME) for reregistration of the license to practice medicine. These States are identified in table 5. In the other States, relicensure remains a pro

forma process accomplished by paying a renewal fee.

Specialty Certification

The successful completion of a residency program, and, in a number of specialties,

additional practice, qualify the physician for specialty board certification. Specialty

board certification is not legally required. Rather, it is a voluntary process designed to indicate the type of specialized medical expertise a physician possesses beyond the general

ability to practice medicine. The physician who has completed a residency program and

received a license to practice is legally qualified to practice in that respective specialty

without passing a certification exam. However, a physican will not be considered board

certified.

The requirements necessary to achieve specialty certification are determined by the

boards themselves. The 22 specialty boards and several other organizations are members

of the American Board of Medical Specialties (ABMS). Among its functions, ABMS establishes standards, policies, and procedures for assuring the continued competence of

physicians (Amercian Board of Medical Specialties, 1977). The recognized plans and educational standards for graduate training in each of the specialties are published annualIy

in the Directory of Accredited Residencies (American Medical Association, 1977a).

The certifying exam for specialty board certification is developed by each of the 22

specialty boards. A number of specialty boards are receiving assistance and guidance

51-541 0 - 79 - 4
38 ● Computer Technology in Medical Education and Assessment

Table 5.—Jurisdictions With Authority to Require Continuing Medical Education

for Reregistration of the License to Practice Medicine

Requirement in

State effect

Alaska. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Arizona . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Arkansas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

California . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Colorado . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hawaii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Illinois . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

lowa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Kansas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Kentucky . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Maryland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Massachusetts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Michigan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Minnesota . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Nebraska . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

New Hampshire. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

New Mexico . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Pennsylvania. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Puerto Rico . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Rhode Island . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Utah . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Washington . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Wisconsin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SOURCE” ’’Medical Education ln the United States, 1977.1978,’’JAMA 240:2850(December1978)

No

Yes

No

Yes

Yes

Yes

Yes

No

Yes

No

Yes

Yes

Yes

Yes

No

Yes

Yes

Yes

Yes

No

Yes

Yes

Yes

Yes

from NBME in the development and validation of their exams. The certifying exam for

most specialty boards consists of oral and written tests. The written exam assesses basic

scientific knowledge pertinent to the specialty. The examination of skills necessary to apply basic knowledge to the management of clinical problems is assessed chiefly through

the oral exam. Skills, such as the ability to obtain and interpret information required for

the proper diagnosis, the proper selection of therapy, and general patient management,

are also assessed by the oral exam. Using standard techniques to specify and assess cornpetence in specialty training becomes increasingly difficult, however, as the sophistication ofatient care and management techniques increases.

All 22 medical specialty boards have established policies to provide recertification.

Seventeen boards have established dates on which recertification will begin; six boards

have already administered their first exam. Five boards have made recertification mandatory. Table 6 summarizes these developments.

Emerging Developments

As a result of a report of the Committee on Goals and Priorities of NBME (National

Board of Medical Examiners, 1973), the methods of assessing the readiness of a medical

school graduate to enter GME has received considerable attention. NBME is presently

under contract to the Bureau of Health Manpower, Department of Health, Education,

and Welfare (HEW), to develop and validate methodologies for an examination, the

Comprehensive Qualifying Examination (CQE), tO be administered Prior to graduate

medical education (National Board of Medical Examiners, 1979). NBME is developing a

rating scale and a behavioral checklist for the exam and methods for assessing skills in sequential diagnostic problem-solving.
Ch. 3–Medical Education and Assessment ● 39

Table 6.— Implementation of Recertification by the Specialty Boards

Board Implementation date Mandatory requirement

Allergy and immunology. . . . . . . . . . . . . . . . . . . . . . 1 0/77

Anesthesiology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1984

Colon and rectal surgery. . . . . . . . . . . . . . . . . . . . . . 1985

Dermatology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1985

Family practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10/76

Internal medicine. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10/74

Neurological surgery. . . . . . . . . . . . . . . . . . . . . . . . . None

Nuclear medicine. . . . . . . . . . . . . . . . . . . . . . . . . . . . None

Obstetric-gynecology . . . . . . . . . . . . . . . . . . . . . . . . 12/77

Ophthalmology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1983

Orthopedic surgery . . . . . . . . . . . . . . . . . . . . . . . . . . None

Otolaryngology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1980

Pathology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1983-84

Pediatrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1/80

Physical medicine and rehabilitation . . . . . . . . . . . 1978

Plastic surgery.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1978

Preventive medicine . . . . . . . . . . . . . . . . . . . . . . . . . None

Psychiatry and neurology. . . . . . . . . . . . . . . . . . . . . None

Radiology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1980

Surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1986

Thoracic surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1986

Urology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1980

SOURCE “Medical Education in the Unlted States, 1977.1978,’’ ./AMA 24O: 2850(December. 1978)

No

No

Yes

No

Yes

No

—

—

No

No—

No

Yes

No

No

No—

—

No

Yes

Yes

No

CQE could provide an assessment of the medical school graduate’s readiness for

graduate training; such assessments are not conducted currently. In addition, CQE could

be offered to all those entering graduate training for specialization: graduates of U.S. and

foreign medical schools.

This new examination may lead to: l) a limited license to practice upon attainment

of specialty status, and 2) intermingling of licensure by State medical licensing boards

with specialty certification by the independent specialty boards. First, the graduate

trainee would concentrate on a specialized area of medicine. If CQE is eventually acceptedly State medical licensing boards as a license to practice in the supervised setting,

then the subsequent license to practice in the unsupervised setting may also be limited to

the physician’s specialty area. Second, all physicians who presently wish to practice must

successfully pass the State licensing exam or other recognized tests. Those who qualify

through specialty certification might not be required to be examined by the State if they

successfully pass their specialty board certification exams (Merchant, 1978; Evans,

1978). Widespread adoption by State medical licensing boards of specialty certification,

in lieu of the licensing exam, would further link the governance powers now vested in

public agencies, the State medical licensing boards, with the powers of private organizetions, the specialty boards.

CLINICAL PRACTICE

The process of assessing a physician’s performance during practice is different from

the one used during formal training. Some elements of the process are the same; for example, continuing education requirements and recertification exams are similar to learning and testing in the physician’s formal training years. However, the practicing physician also is subject to the requirements of various programs designed to measure, evaluate, monitor, and/or improve medical services. Many of these programs are commonly

known as “quality assurance” programs because of these objectives. There are generally
40 ● Computer Technology in Medical Education and Assessment

two major categories of such programs. Programs may be designed primarily to assess

and improve the quality of medical care by retrospective analyses of physicians’ treatment of patients. These programs rely on the analysis and interpretation of data about

the medical care process and are designed to evaluate care already delivered. Other programs are designed to provide guidance to the physician during diagnosis and treatment.

Such programs are commonly referred to as concurrent programs. Both retrospective

and concurrent programs frequently have two distinct goals: 1) intervention in the medical care process in order to improve the quality of care rendered, and 2) reduced costs of

medical care through reduction of inappropriate and/or excessive medical services.

Some quality assurance programs are federally mandated, such as the Professional

Standards Review Organization (PSRO) program; others are privately sponsored. The

PSRO program has established a major system of hospital quality assurance through

concurrent review and medical care evaluation studies; efforts are now underway to

identify variations in medical services through the development and analysis of provider

and patient profiles. Recently enacted legislation established State government units to

identify and detect cases of fraud and abuse in the Medicaid Program. A standardized

data processing system, the Medicaid Management Information System (MMIS), has

been installed in several States and is scheduled for full-scale national implementation.

The Joint Committee on Accreditation of Hospitals (JCAH) promotes its own recommended hospital quality assurance programs and has recently entered the area of ambulatory care. Insurance companies have increasingly turned to review of claims data to

isolate cases of flagrant abuse. Descriptions of a number of these review systems follow.

Hospital discharge abstract processing systems use data abstracted from hospital

records by hospital medical records personnel. Abstract forms designed by a processing

service include, at the minimum, the Uniform Hospital Discharge Data Set (UHDDS).

The processor prepares a number of reports, such as the disease and operations index.

Most abstract forms have been modified to include PSRO data elements to accommodate

PSRO requirements for concurrent review information. Hospital discharge abstract

processing systems are intended both to assist the institution in meeting the requirements

imposed by JCAH and to provide the information for the hospital medical staff’s various

committees, such as the Utilization Review, Tissue, Tumor, and Infection Committees.

Peer review and quality assurance activities are performed retrospectively. Reports

derived from the abstracted data can be used by the physician committees to examine institutional and physician care patterns more clearly.

Private health insurance companies, which serve as intermediaries for Federal health

insurance programs and PSROs, use a variety of methods for processing claims data for

quality and utilization review. One such method, employed by Blue Shield of California,

utilizes the electronic data system’s retrospective analysis of medical services (RAMS)

programs (E.D.S. Federal Corporation, 1978). Data submitted on the claims form for

services are used for this retrospective analysis as well as for payment of claims. Systems

are designed to analyze the relationships among patients, medical problems, providers,

and procedures used to assess the appropriateness of care (quality and utilization). This

assessment is performed by employing predefine criteria to evaluate problem and treatment interactions (proper treatment for specific problems) and to report any deviations

for further study. The treatment model concept (analysis of problems versus treatment

procedures) allows evaluation of the health care services delivered for a specific problem

(diagnosis) by comparing the services actually rendered with the ideal or expected patterns of care. The established peer group norm concept is used as a supplement to the

treatment model approach. Treatment analysis profiles are developed using previously

defined models and community norms to evaluate a provider’s practice by diagnostic
Ch. 3—Medical Education and Assessment ● 41

categories. Patterns of care failing these explicit criteria are noted as exceptions for further study and review. Summary profiles that report percent of deviation from the group

norm and any deviation based on high and low parameters, which the user may select,

are available for review. These profiles are used to indicate the necessity of further, more

detailed review of patients and their providers. The computer, with its capability to store

large data bases, allows comparison of criteria, establishment of practice norms, and

production of practice pattern profiles.

The Commission on Professional and Hospital Activities (CPHA), a nonprofit organization based in Michigan, began to provide discharge abstract services as early as 1955

and has developed a data base consisting of approximately 150 million patient records

from 2,200 U.S. and Canadian hospitals. The Commission has developed average

lengths of stay for selected diagnoses or groups of diagnoses; these serve a useful function

in admissions certification and concurrent review activities in the PSRO program. In addition to the Professional Activities Study (PAS), CPHA offers the Quality Assurance

Monitor (QAM) from which three quarterly reports are prepared: 1) priority for investigation, 2) monitor profile, and 3) audit trail listings. These reports can be used by the

various medical care committees to assist their review activities.

The computerized MMIS (U.S. Department of Health, Education, and Welfare,

1973) is a collection of subsystems designed by HEW’s Health Care Financing Administration to assist States in the management of their Medicaid programs. Fifteen States are

currently operating certified MMISs; 11 are expected to have their systems certified; and

23 are developing, or have partially implemented, MMIS.

One of the MMIS modules is the Surveillance and Utilization Review module

(S/UR). The S/UR module is designed to: 1) produce a prescreen set of profiles that have

been compared to the average pattern of care as defined by the State, 2) limit production

of profiles to those providers showing aberrant behavior, 3) perform postpayment utilization review, and 4) offer options in the individual State’s approach to utilization

review. S/UR is a retrospective review mechanism for care that has already been provided and reimbursed. S/UR operates by using an edited paid claims tape to: 1) accumulate totals (for example, total number of providers in class group and total number of office visits); 2) select data items; 3) produce averages and standard deviation reports; 4)

produce exception control limits reports; 5) produce exception summaries; 6) print exception provider profiles; and 7) produce, for hospitals, treatment analyses that relate the

care provided to diagnosis and age group categories.

S/UR was designed to be a statistical reporting system, using means and standard

deviations instead of norms. It is a flexible system because it can modify exception control limits through tables instead of through computer program changes.

PSROs were federally mandated and organized primarily to determine the medical

necessity and appropriateness of care in Medicare, Medicaid, and Maternal and Child

Health. PSROs are required to develop standards based on the care normally provided

within the PSRO region. Major functions of the review process include admissions certification, continued stay review, retrospective review, and Medical Care Evaluation

(MCE) studies (Goran et al., 1975). Determination of the necessity and appropriateness

of medical services required developing criteria and standards for various types of diagnoses referenced to various patient characteristics, such as age and sex. Average length of

stay based on these parameters has been calculated by the Commission on Professional

Hospital Activity, but this data base does not include data from all hospitals in the country. AUTOGRP, an interactive computer system, provides the means by which each in-
42 ● Computer Technology in Medical Education and Assessment

stitution can more clearly investigate its own case mix characteristics, and to further

define its criteria for length of stay (Mills, et al., 1976).

Admissions certification requires assessment of the status of the patient by a review

coordinator to determine whether the diagnosis is appropriate for treatment at the health

facility; the coordinator also determines whether admissions criteria for that diagnosis

are met. If the review coordinator questions that admission, the case is referred to a

physician advisor, who decides whether or not the admission should be certified. If the

admission is appropriate, the coordinator determines the average length of stay for that

particular diagnosis by age and sex, and indicates that a review of the case will be necessary within a prescribed number of days.

Review after admission is termed continued stay review. The coordinator reviews

the patient’s chart to determine when the patient may be discharged and whether the

discharge falls within the prescribed length of stay. If the length of stay needs to be extended, the coordinator may either approve an extended length of stay or refer the case to

the physician advisor. If inappropriate admission or stay occurs, the patient and the admitting or attending physician are notified that the stay may not be certified for payment.

Retrospective review is also performed by PSROs through profiling. Patient, practitioner, and institutional profiles are developed and reviewed to sharpen the length of stay

indicators and to identify potential problems.

Finally, the PSRO’s MCE studies focus on specific administrative and clinical problems and, after appropriate action has been taken, determine whether or not the problem

has been solved.

Computers are principally used in the PSRO program both to schedule admissions

certification and concurrent review activities and to evaluate the patterns of care. Computerized data bases that provide length of stay indicators and norms and standards are

also used in order to properly certify the appropriate length of stay for a particular case.

Computer applications in review functions include profiling and, to some extent, MCE

studies. The computer is particularly useful in aggregating data elements for profiling

because it is able to handle a wide variety and number of cases, and it can perform

statistical analyses to obtain new dimensions of the information in a variety of displays.

The computer is used in MCE studies to process data that focus on the individual problems being studied and to monitor the impact of efforts to solve these problems (Martin,

1978).

The PSRO Management Information System (PMIS) (USDHEW, 1975) is a system

originally designed for use by the Health Standards and Quality Bureau, HEW, to process data required from each PSRO. These data are used to monitor and evaluate the performance of PSRO. By aggregating data from the various PSROs, it is possible to cluster

them to allow self-evaluation. In addition, PMIS aggregates data regarding length of stay

and other indicators of the medical appropriateness and necessity of admissions and

hospital use. An inventory of selected MCE study designs and outcomes is also maintained and forwarded to the PSROs to assist them in conducting these studies.

PSRO projects may also cover ambulatory care. The physician ambulatory care

evaluation (PACE) program (Nelson et al., 1976), operated by the Utah Professional

Review Organization (UPRO), is a physician-directed professional review effort that

utilizes both claims data and an advanced automated system for building histories of ambulatory patients and screening them for compliance with clinical guidelines. Both quality and utilization issues are addressed. Where patterns of variation from peer expecta-
Ch. 3—Medical Education and Assessment ● 43

tions are observed, intervention is directed toward improving patient care. The approach

involves educational contacts with physicians rather than immediate punitive action.

Denial of payment is only employed when other methods fail.

The New Mexico Professional Standards Review Organization’s Revised Medicaid

Ambulatory Care Review System (Health Care Management Systems, Inc., 1976) contains both a prepayment and a retrospective review system component based on claims

that have been processed by the fiscal intermediary for claims payment. A small sample

of New Mexico providers is reviewed before payment as a result of previous analyses of

their practice patterns that have indicated substantial deviations from standard patterns.

All other providers are reviewed retrospectively at least twice a year if their claims

volume justified such a review. Both prepayment and retrospective review employ

criteria developed by physician committees, and claims data are subject to these guidelines. Physicians failing the retrospective guideline review are reported to educational

committees. The educational committee assigns one or more of its members to follow up

the case. The case is researched by an educational committee member, who writes the

physician in question explaining why certain standards should be met. This explanation

is supported by appropriate scientific and medical facts and includes literature citations.

Physicians on prepayment review whose responses fail the guidelines may be denied all

or partial payment and may be referred to the ambulatory care review committee for

possible future educational intervention by the committee.
Chapter 4

COMPUTER APPLICATIONS

IN MEDICAL EDUCATION

AND ASSESSMENT
4 ●

COMPUTER APPLICATIONS IN MEDICAL

EDUCATION AND ASSESSMENT

INTRODUCTION

Computers are being used throughout the continuum of medical education and assessment. The unique capabilities of computers, particularly their ability to rapidly store,

process, and retrieve large volumes of data, provide assistance in and help manage education, assessment, and patient care activities. They allow a much wider range of individualized responses in tutorial sessions, a greater degree of “instructor” responsiveness

to individual backgrounds and needs, and more immediate feedback in the teaching or

testing setting. Also, aspects of the patient-physician encounter can be simulated for

training purposes and for providing a more objective testing mechanism for assessing

clinical problem-solving abilities.

The diagnosis and choice of treatments now depend largely on the subjective recall

of the individual physician. Some of the deductive reasoning processes that take place in

diagnosing and treating patients can be done more consistently, comprehensively, and

instantaneously by using computers as “consultants. ” Computers are comparable to

other medical aids upon which the physician has come to depend; they serve as replacements for less dependable technologies, Physicians no longer reIy on fingertip percussion

of the chest wall to gauge heart size; instead, X-rays, electrocardiograms, echocardiograms, and other more sophisticated technologies are used.

Finally, computers have a unique role in the management of large volumes of data

for reimbursement of health services, utilization review, and the planning, monitoring,

and evaluation of medical care services.

Four different types of computer applications in medical education and assessment

are discussed in this chapter. These four types are: 1) computer-supported independent

study in the basic medical sciences, 2) the use of computerized simulations in training and

testing for clinical competence, 3) computer-based medical consultation systems, and 4)

computer uses in quality assurance/data management systems. These applications were

chosen to illustrate the uses of computers across the education and assessment continuum. Each case study also was chosen to emphasize a particular capability of the computer. Each application may have multiple capabilities both in instruction and in assessment, and the concepts underlying each application are potentially applicable to other

portions of the continuum.

47
48 ● Computer Technology in Medical Education and Assessment

COMPUTER-SUPPORTED INDEPENDENT STUDY PROGRAM

IN THE BASIC MEDICAL SCIENCES

The Ohio State University (OSU) College of Medicine has two programs for completing the basic science portions of the Doctor of Medicine curriculum (Beran et al.,

1976; Merola et al., 1978; Cramblett et al., 1979). The lecture-discussion program follows a traditional format in which students progress through a standard content sequence in a fixed amount of time—is months. In contrast, the independent study program (ISP) follows a self-study format in which students progress through the curriculum

at independent rates. The principal difference between ISP and the traditional system is

the former’s acceptance of time as a variable in the learning process. The student progresses when he/she achieves a prescribed level of competency, not when the next lecture

series begins. Individual progression rates require that students completing their basic

science studies be allowed to enter their clinical clerkships in almost any month of the

year.

ISP features an integrated curriculum, modular student study objectives, computerbased tutorial evaluation systems, and a means of managing a student population progressing at independent rates. In July 1970, the first class of 32 students was accepted into

ISP, followed by classes of approximately 60 students each in 1971 and 1972. The latter

two classes comprised a little over one-fourth of the entering medical students of each

year. Currently, 40 percent of the entering classes select ISP.

The ISP curriculum is designed around two main segments: normal man and pathophysiology. Table 7 outlines the components (units) of the ISP curriculum. Each student

is required not only to master the units, but also to participate in laboratory exercises, attend clinical correlation sessions and psychosocial seminars, and demonstrate satisfactory verbal communicative skills through oral exercises with basic science and clinical

faculty. The student flow through an independent study learning unit is shown in figure

7.

Each student’s progress is monitored by the computer through the computer-based

self-evaluation or Tutorial Evaluation System (TES), a series of interactive computerassisted instruction (CAI) “question and answer” sessions between the student and the

computer terminal. When the student answers a question correctly, he/she is immediately reinforced and then advanced to the next question, If a student answers a question incorrectly, the student must complete corrective feedback and/or remedial work before

reanswering the question. If a student shows a total lack of understanding of a concept,

he/she may be branched to a remedial review unit. If an answer is given that the course

author did not anticipate, the student may be taken to a coaching sequence designed to

guide him/her to the correct answer. If serious weakness in an area is demonstrated, a

study prescription may be given, such as “you lack understanding in area X and should

review pages Y through Z in text W“ (See figure 4 in chapter 2 for a schematic summary

of this process). The faculty controls generation of these study prescriptions. Figure 8

gives a sample student-computer dialog.

ISP presents logistical problems in terms of managing the flow of students through

the program. With students presenting themselves individually and at different times for

tutorial evaluation, lab work, and subject matter consultations, the faculty member must

be assisted in monitoring each student’s progress and performance. To meet this need for

monitoring students, student reports are provided at various intervals. The reports comprise the computer-based management system for ISP. Sample reports are depicted in

figure 9.
Ch. 4—Computer Applications in Medical Education and Assessment ● 49

Table 7.—The Independent Study Program Curriculum Outline

. —

Associated tutorial

evaluation system

Uni t Titl e module(s )

Normal man

1 Cell structure and function ... . . . . . . . . . . . . . . . . . ... A

2 The cardiovascular system . B

Exam 1

3 The respiratory system . . . . . . . . . . . . . . . . . . . . . . . . . . . . c

4 Kidney function, urinary system, body fluids . . . . . . . . . . . . . . . . . . D

5 The gastrointestinal system . . . . . . . . . . . . . . . . E

Exam 2

6 Skin and supporting tissues . . ., . . . . . . . . . . . G

7 Muscle. the spinal cord, and peripheral nervous system. ., H

8 The central nervous system . . . . . . . . . . . . I

Exam 3

9 The endocrine-metabolic system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J

10 Nutrition ....,.... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F

11 Reproduction. . ., . . . . . L

12 Hematology . . . . . . . . . . . . . . . . . . 0

Exam 4

Comprehensive exams-Sections 1.2, and 3 (anatomy, biochemistry and physiology) of the NBME exam.

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

.

Pathophysiology

The immunologlc system . ,

Pathologic mechanisms and genetic s

Mlcrobiology. ,. . . . . . . . . . . . . . . . . . . . .

Epidemiolog y

Exam 5

Drug mechanisms . . . . . . . . . . . . . . . . . . . . . .

The cardiovascular system . . . .

Exam6

Infectious disease s

The respiratory system . . . .

Exam7

Hematology. . . .

The gastrointestinal syste m

The renal system . . . . . . .

Exam8

The central and peripheral nervous system.. . . . . . . . . .

The musculoskeletal syste m

Behaviorial science. . . . .

Exam9

Introduction to pediatrics

The endocrine syste m . . .

The reproductive system . .

Final Comprehensive NBME Part l(all sectionsl

SOURCE Ohio State University 1979

P

Q/K

R

T

s

x

v

Y

u

w

z

1

2

L/3

M / 4

5

The student progress report provides information regarding: 1) units and subunits

completed to date, 2) cumulative time spent on TES, and 3) a summary of the week’s activity, including a listing of study prescriptions. The TES timeis only 1 1/2 hours out of a

40-hour work week. Most student time is spent in independent study of the learning

resources. The computer, however, can easily provide the reporting described herein

since the student must complete a TES exercise within each subunit.
50 ● Computer Technology in Medical Education and Assessment

Figure 7.—Student Flow Through a Typical Module

I

m

IV

A

n

1. Student reviews educational objectives and examines list of available instructional resources.

Il. Student pursues instructional resources in accord with his individual needs.

Ill. Student utilizes self-evaluation system where he receives tutorial feedback as to his mastery of the

educational objectives.

IV. Student confers with faculty member for final review of instruction unit (optional).

(Typically, from one to five submodules are included in each module, with the student proceeding

through steps I-IV for each submodule.)

V. At specified points in the curriculum he must take a written exam covering several modules, and

VI. Demonstrate his ability to transmit information orally in prescheduled oral exams.

SOURCE: Merola et al , 1978

The student response printout can provide every response entered at the terminal

sorted by student and/or by question. The student cluster report is used primarily by the

curriculum directors and faculty to determine the location of the ISP students throughout

the curriculum in order to schedule labs or clinical sessions. The student status report is
Ch. 4—Computer Applications in Medical Education and Assessment ● 51

Figure 8.— Part of a Student-Computer Dialog

Explanation

Computer poses a question.

Student responds (unanticipated

response ) .

Computer poses review question.

Student responds.

Computer offers coaching information and poses another

question.

Student responds (correctly)

Computer gives reinforcement

to correct answer.

Computer re-poses the original

question.

Student answers (wrong answer)

Computer offers tutorial feedback pointing to the correct

answer.

Student responds (correct

answer).

Computer gives reinforcement

to the correct answer.

Student-Computer Dialogue

Computer: Name the pathological conditions

within the brain caused by blockage of the

lateral and medial foramina of the inferior

medullary vellum.

Student: I don’t remember.

Computer: Are you having difficulty placing

the location of the inferior medullary velure

and/or the significance of the foramina?

Student: Yes.

Computer: The inferior medullary velure forms

part of the roof of the fourth ventricle. The

foramina communicate with the cisterna magna

and/or cisterna pontis. What is contained in

these cisternae?

Student: Cerebrospinal fluid (CSF).

Computer: Correct. Now consider blockage of

the foramina and answer the original question.

Computer: Name the pathological conditions

within the brain caused by blockage of the

lateral and medial foramina of the inferior

medullary vellum.

Student: Water on the brain.

Computer: Not acceptable. It’s time to become

more professional in your terminology!

Student: Hydrocephaly.

Computer: Very good! The obstruction will

cause an increased quantity of cerebrospinal

fluid in the ventricular system.

SOURCE The Ohio State University College of Medicine,1978.

an abbreviated version of the student progress report and is available on demand at the

CAI terminal.

Another kind of report provides item statistics information that the faculty can use

to analyze, correct, and revise TES items. This report also may be used to help determine

how well students responded as a group to TES questions. Most importantly, this report

lists all student responses that were unanticipatedly the TES author. Responses that will

improve TES sessions, as determined by faculty members, are added immediately to the

TES program. Figure 10 depicts an item statistics report.

Faculty members in a specific discipline are responsible for the preparation of individual aspects of each unit and may revise and design changes in relevant modules of

TES. The almost infinite variety of question types that involve corrective feedback,
52 ● Computer Technology in Medical Education and Assessment

Figure 9.—Sample Computer-Based Management System Reports

ISP PROGRESS REPORT ON COURSE PILOT FOR JOHN DOE

PRINTED ON MAY 5, 1976 AT 17:44

STUDENT NBR/0316 CLASS/75

FACULTY ADVISOR/AJM TYPE/LINEAR

PERSONAL CHART/75

LAST USAGE /MAY 3

PROGRESS CHART USED FOR THIS REPORT/75

7/09/75

SCHED - - A A B C D D F F G G H H

COMP— A AB BC CD DE EF F FG G GE H H H

10/08/75

SCHED- I J K L M N OOPPPP P

COMP— HI IJK K K KL LM MNO OP P P P P P

1/07/76

SCHED- P P Q Q R R ST T T U U

COMP-- P PQ Q Q QR RS S S S S S S SJ

4/07/7 6 *CURRENT WEEK

SCHED- U V V V W W W X X Y Y Z Z

COMP-- u w v v-w w w w w

*CURREN T

NOTE: CHART SHOWS ONLY MODULES COMPLETED - STUDENT CURRENTLY AT XA

A Student Progress Report. This report gives a linear view of an individual

student’s progress as compared to a projected rate of progress through the entire

curriculum. It also gives the faculty. adviser and the last date of usage.

STUDENT STATUS - PILOT

DATE 04/07/76 TIME 4: 57 SELECTION = ALL

STUDENT NAME STUDENT NO. GROUP AREA TYREC START DATE LAST DATE TIME SEG LABEL-PROB-SEQ 030

CHARLES JONES

PHYLLIS SMITH

DALE WITHE

Scott GREEN

JOANN BROWN

ROSALIND DOE

TONI BLACK

JAMES DILL

MARY HILL

DOUGLAS HALL

JOHN DORN

339

340

341

342

344

171

345

351

352

231

353

75 0 15

75 0 15

75 0 15

75 0 15

75 0 15

73 0 15

75 0 15

75 0 15

75 0 15

74 0 15

75 0 15

7/17/75

7/14/75

7/18/75

7/14/75

7/22/75

7/19/73

7/14/75

7/14/75

7/11/75

7/12/74

7/21/75

3/04 /76

3/20/76

1/ 14/76

4/03/76

3 /04 /76

8 /20/75

3/01 /76

4/02/76

4/01 /76

1/ 16/76

4/03/76

22: 00

32: 14

27: 52

42: 22

15: 57

35: 16

57: 55

36: 50

32: 03

02: 47

19: 03

QW UAOOOTQW UAOOOTQW QAOOOTQW WG999ZAP LAOOOTQW WF1OOSQW UAOOOTQW WAOOOTQW klA060S36 5BUIOSQW QA020S3- 0

3- 0

1- 0

1- 0

1- 0

1- 0

3- 0

1- 0

1- 0

1- 0

1- 0

etc.

A portion of the Student Status Report. (Student names are fictitious.) This report

gives the location of each student in the curriculum (label), when each student started in

the program, and the last date that student had any activity at the terminal.

SOURCE Pengov,1974

coaching statements, and immediate reinforcement permits individualistic approaches to

the same material. The possibility of making constant changes in TES design affords each

faculty member the opportunity to instantly correct errors, update content, or change the

emphasis in the original system’s content.

The authorship of TES changes as new faculty become involved in the program. The

design of individual TES items is modified and revised by new faculty concomitantly
—

Ch. 4—Computer Applications in Medical Education and Assessment ● 53

Figure 10.— Item Statistics Report

ITEM STATISTICS FOR QUESTION, IA1106- 1 COURSE: PILOT

TIME PERIOD: 10/01/75 - 12/29/75 DATA COVERS 47 STUDENTS USAGE

ATTEMPT NUMBER

1 2 3 4

RESPONSE

1- 2 CA , 4 1 19 9 3

1- 3 CA ‘ 1 :

1- 9 WA 2 1

1- 14 WA 1

1- 18 WA ; 2 , 3

1- 24 WA 14 ! 3 \

99- 99 UN 26 15 3

t I I

ATTEMPT 47 42 14 3

TOTALS t 1 .

RESPONSE % OF TIME WA % OF TIME UN

>4 TOTALS LEAD TO UN LEADS TO UN

 . .

35

1

3

1

5

17

44

I —- —

—

33%

100%

60%

71%

—.

. —

UNANTICIPATED RESPONSES MADE ON FIRST AND SECOND ATTEMPTS:

1. NUCLEUS RUBRUM

INFERIOR CEREBELLA PEDUNCLE

MLF

BASES PEDUNCULI

MLF

RED NUCLEEI

MIDDLE CEREBELLA peduncle

***ALL UNS NOT LISTED*AA

2. BRACHIUM PONTIS

RED NUCLEUS

NO

TECTUM

CEREBRAL PEEDUNCLE

CERBRAL PEDUNCLE

RETICULAR FORMATION

TEGMENTUM STATISTICS FOR NUMBER OF RESPONSES

***ALL UNS NOT LIST~*** RANGE 1- 4

MEAN 2

MODE 1

Item Statistics for a Single Question. Correct answer is designated CA, wrong

answer is designated WA, and unanticipated responses are designated UN. In this

example, “red nucleus” was a common unanticipated response, and would be logically

added to the bank of anticipated responses in the computer program.

SOURCE Pengov 1974

with changes in objectives and source materials. Each new faculty member is encouraged

during the -TES revision process to collaborate with instructional programers indesigning

the module. Such collaboration can take the form of correcting obvious problem areas in

TES or discussing and implementing new approaches to TES, such as greater use of problem-solving techniques.

Faculty responsibilities are somewhat different in ISP than in a traditional lecture

curriculum. The emphasis in ISP is placed on integrating material into learning units prepared by an interdisciplinary team of faculty, not on designing and presenting lectures.

Faculty members assess the ability of students to grasp the materials presented in the unit

he/she designed through small group discussion sessions with students working on the

51-541 0 - 79 - 5
54 ● Computer Technology in Medical Education and Assessment

module. Occasional clinical correlation sessions, conducted with both basic science and

clinical faculty, facilitate the extension of concepts of normal function to those of pathophysiology. The emphasis placed on developing the curriculum within an interdisciplinary team and small group discussions, rather than lecture preparation, has created a curriculum that is changing constantly as faculty receive helpful suggestions from both students and other faculty.

The ISP curriculum attracts a wide range of students, and they perform comparably

to lecture-discussion students on both clinical and preclinical portions of standard exams.

ISP appeals to some students who wish to accelerate the completion of their medical

education and to others who wish, or need, to reduce the academic pressure of medical

studies. The flexibility inherent in ISP and the student’s ability to control a rather intense

curriculum benefit students with academic deficiencies, as well as those with strong academic credentials. Students who have had extended absences from a formal education

setting, previous to their admission into the college, also benefit from ISP.

The groups that progress at the slowest rates through the program include a significant number of minority students who need special assistance to complete their medical

studies. However, many students with no apparent academic weaknesses are taking

more time to complete the curriculum than their lecture-discussion counterparts. Despite

a delayed graduation, most of these students prefer more time than the standard 15

months the lecture-discussion program allows. The range of completion time in ISP is 10

to 29 months, the mean is 17 months.

The experience of ISP has illustrated that:

●

●

●

●

●

●

●

Independent preclerkship study is a useful and viable alternative to traditional

programs.

Computer-assisted and computer-managed independent study is feasible for large

numbers of preclinical medical students.

Medical education can be tailored to the individual needs of each learner.

Computer-assisted instructional and management techniques allow each student

to progress independently of others according to his/her own educational background, learning speed, or needs. CAI/CMI allows faculty and administrators to

manage large numbers of students progressing at independent rates. Such monitoring techniques are an essential element for the implementation of any largescale independent study program.

Many faculty have become orchestraters of learning rather than disseminators of

knowledge. The computer does the repetitive tutorial work, thereby providing

faculty members more time for individual student interactions. Overall faculty

time requirements do not change; the nature of their roles and the use of their

time, however, do change.

Student response to ISP has been very positive. They find it enjoyable to learn by

discovering with a faculty member as well as from a faculty member.

Despite the high investment costs for courseware development, overall CAI operational costs compare favorably with the costs for other instructional methods,

such as labs or group discussions.
Ch. 4—Computer Applications in Medical Education and Assessment ● 55

THE USE OF COMPUTER-BASED SIMULATIONS

IN EDUCATION AND ASSESSMENT

Introduction

Several different models of computer simulations used for educational assessment

are described below. The use of computer simulations by the Royal College of Physicians

and Surgeons of Canada, the American Board of Internal Medicine (ABIM), and the National Board of Medical Examiners (NBME) in actual assessment activities is then discussed.

A wide variety of computerized, clinical encounter simulations have evolved in the

last decade. Computers are used to simulate “patients;” physicians or students interact

with typewriter-like computer terminals to solve the “patient’s” problems. Although the

setting does not simulate all aspects of a physician’s interaction with a live patient, the

essential feature of information flow between the two is captured well, particularly if efforts are made to make the simulated interaction more “human” and less machine-like

(Senior, 1976). A more “human” system is characterized by language that is easily understood, by responses that can be corrected quickly, and by courteous and considerate instructions (Sterling, 1975). Only rudimentary typing skills, and no knowledge of computers or programing, are required of the user. Inquiries can be directed to the patient by

typing in actual questions, either as words or sentences, or as code numbers referring to

questions or desired studies. Successful experiments also have used a standard touch-type

telephone to communicate with the computer patient (Friedman et al., 1977). Diagnoses

can be entered, management decisions or actions can be initiated, and data may be reviewed. If needed, special assistance can be provided by attendants at the computer terminal site. However, brief introductory training exercises are usually conducted to familiarize the users with the terminal equipment and mode of operation.

Prior to the use of computers, simulations were primarily paper-and-pencil in format. The best known paper-and-pencil simulation is the patient management problem

(PMP) (Hubbard et al., 1965). A traditional PMP simulates various aspects of a realistic

clinical situation in which the physician or student must manage a patient. After reviewing a paragraph of information describing the patient, the user may request additional

data. On the basis of the information collected, the user diagnoses, treats, or manages the

patient’s problem by choosing from a limited set of options or interventions that he/she

thinks are necessary to reach a proper solution to the patient’s complaints. Appendix I

contains an example of a segment of a conventional paper-and-pencil PMP. Most often,

the branching sequence offers a choice between the physical examination section, the laboratory section, and the history section of the patient management problem. Each section

presents a list of items, including essential, optional, and irrelevant information. The

task of the user is to choose those items that are pertinent to the resolution of the problem. Early PMPs were linear, somewhat inflexible, and offered cues to the user. “The

validation of the PMP has been called into question in that attempts to show that practicing physicians behave similarly in a PMP as they do in the real-life situation have not

been generally successful. This is thought to derive from the fact that in this simulated

situation the physician is given a series of options and clinical investigations from which

selections may be made thereby affording cues to the examinee. In the clinical situation

the options, confirmatory tests, and consultations possible are legion and require an internally generated selection” (Meyer, 1978). Computer simulations can help avoid the

problem of “cueing. ” The computerized simulations described below overcome, in varying degrees of sophistication, the limitations of paper and pencil simulations.
56 ● Computer Technology in Medical Education and Assessment

Sophisticated computerized versions of PMPs contain multiple branches and require

the physician to make choices that lead to any number of strategic routes, several of

which could result in an acceptable solution to the patient’s problem. These computerized

versions allow much more complex branching; provide immediate feedback to the physician; permit the review of previously selected items; utilize audio/visual capabilities; and

provide mechanisms for timing, control, or monitoring of the examination. Additionally, user performance recordings are more detailed, retracing by the user is minimized,

and scoring of the problem is instantaneous. For each problem the user receives both introductory information about the patient, the setting of the problem, and an explanation

of his/her role and tasks.

Sim One is a Iifesize manikin that has several functions, including respiratory activity, pulse rate, blood pressure, skin color, and pupillary size, that are controlled by the

computer. Each function responds to drug administration and other interventions used

by an anesthesiologist to manage patients in surgery. This model simulates a variety of

operating room situations. The clinician can repeatedly experience these situations in credentialing or educational settings without risk to real patients and with immediate feedback regarding the effects of judgments and actions (Abrahamson et al., 1969).

INDEX has a glossary or index list of over 1,000 history questions, physical findings,

laboratory procedures, and diagnoses that physicians use to obtain information from the

patient. A sample report summarizing an interaction between a physician and a patient is

illustrated in figure 11. The physician, identified as “No. 136, ” has examined a patient

who has blood loss anemia that is caused by fibroid tumors of the uterus and von Willebrand’s disease. All inquiries or diagnoses made are listed in order as four-digit numbers.

The final diagnosis and management plan, the time the examinee took to complete the

problem, the dollar costs of studies ordered, the time consumed, and an estimate of the

aggregate discomfort and risk to the patient from the workup are listed in the figure.

Such a record is available for analysis, evaluation, and scoring in printed and computerreadable form.

MATRIX is a modification of a teaching program designed to illustrate that a diagnosis is based on statistical probabilities of the condition in question. These probabilities

are based on factors such as the patient’s age, sex, symptom complex, physical findings,

and results of laboratory tests. The teaching program was adapted to a testing program

on the diagnosis of acute abdominal pain. As in INDEX, the candidate is given an index

of inquiry possibilities and then enters the inquiry by code (Senior, 1976).

Mac Puf is a computerized cardiovascular-pulmonary model in which the computer

program calculates the effects of changing one variable on all the other physiological

variables (Dickinson, 1977). Values of the variables are interrelated by equations, and

the computer solves all the equations simultaneously. The program can be made to

simulate diseases such as acute myocardial infarction, arterial or venous bleeding at any

rate, pulmonary emboli, and obstructive lung disease. It can be designed to respond to

treatments such as digoxin, intravenous fluids, oxygen, respirators, rotating tourniquets,

vasopressor drugs, and bicarbonate. If untreated, the patient may deteriorate and “die”

or stabilize at some lower level of function. Appropriate and timely therapy may reverse

the downward trend of simulated illness, and vital signs will improve. Complications

and new medical crises can be programed to occur at any time. Thus, this model of a

changing and responsive patient represents a potentially useful tool for assessing management skills or the ability of a clinician both to treat crisis problems and to avoid complications.
Ch. 4—Computer Applications in Medical Education and Assessment ● 57

Figure 11 .—Example of an INDEX Case Interaction Printout

#136

CASE: 4 TIME: 14: 50 DATE: 10/31/7 2

HISTORY PHYSICAL STUDIES DIAGNOSES

PROBLEM BEGAN 1010

MAIN PROBLEM 1000

FAMILY BLEEDING 2716

HOSPITALIZATIONS 2340

ALCOHOL 2330

PROLONGED BLEEDING 1963

ANTICOAGULANTS 1564

HEPATITIS 1760

ALLERGIES 2550

PALLOR 3024

MENSTRUATION 1920

MENORRHAGIA 1926

SPLEEN PALPABLE 3748

FECAL BLOOD 3834

LIVER ENLARGED 3747

SMEAR AND DIFFERENTIAL: 5008

ROUTINE STUDIES: 5000

PLATELET COUNT: 5058

PROTHROMBIN TIME: 5066

CAPILLARY FRAGILITY: 3506

BLEEDING TIME: 5060

PETECHIAE 3054 APLASTIC ANEMIA

IRON DEFICIENCY ANEMIA

THROMBOCYTOPEMIC PURPURA

VON WILLEBRAND’S DISEASE

(IMPRESSION LIST)

SPECIAL COAGULATION TESTS: 5086

(FINAL LIST)

IRON DEFICIENCY ANEMIA

VON WILLEBRAND’S DISEASE

PATIENT MANAGEMENT PLAN

FERROUS GLUCONATE 300 MG PO TID

ADVISE PATIENT TO REMAIN UNDER SUPERVISION FOR BASIC DISEASE

DOLLAR COST OF STUDIES: $25

TOTAL TIME TO COMPLETE ALL LAB STUDIES: 1 DAYS 12 HRS O MIN

PATIENT TIME CONSUMED: O DAYS 1 1-HRS 25 MIN

PATIENT RISK: 7 PATIENT DISCOMFORT: 35

ELAPSED TIME FOR PROBLEM SOLUTION: 28 MINUTES.

SOURCE Senlor,1976
58 ● Computer Technology in Medical Education and Assessment

CASE, a computer-assisted simulation of the clinical patient encounter, was developed

in the late-1960’s at the University of Illinois (Harless et al., 1971; 1973). CASE uses an

unprompted and undirected process. The interaction occurs at a computer terminal and

the physician elicits whatever history, physical examination, and laboratory data that

are needed for the diagnosis and management of the patient’s problem by typing unstructured, English language phrases or sentences into the computer. The computer provides

brief introductory materials to the physician, including a description of the clinical setting, the circumstances around the patient’s visit, and some physical characteristics of the

patient. After the introduction, the physician receives no further clues. Candidates must

determine what information they think is important in order to define and manage the

patient’s problem. Appendix I contains a typical undirected interaction between a physician and a CASE patient.

A wide and representative variety of internists’ patients can be simulated by this

computer model, and each simulated, computerized patient (CASE) can be refined to respond appropriately to approximately 95 percent of the physician’s inquiries. Over 50

CASES have been created to date; most are housed at the OSU College of Medicine.

The costs of creating, refining, and field testing each CASE are estimated to range

from $5,000 to $15,000, depending on the degree and type of review provided. The relatively low development cost for a simulation as complex as CASE is made possible by the

existence of a series of computer programs that guide the creation or generation of new

CASES. This generating system (GENESYS) allows the creation of a large number of

CASES in a relatively short time.

GENESYS essentially involves a three-phase process: 1) author preparation, 2) technical creation, and 3) refinement. During the first phase, the author compiles data describing his/her patients for subsequent use in a CASE simulation. The technical creation

phase entails entering the author-supplied information into the computer according to a

standardized format. The final phase, refinement, is designed not only to ensure that the

CASE simulation can respond to the widest possible variety of anticipated questions, but

also to “humanize” the computerized patient.

Computerized PMPs, INDEX, and CASE have been used and are being tested for actual physician assessment activities by the Royal College of Physicians and Surgeons in

Canada, by NBME, and by ABIM. The use of each simulation is described in more detail

below.

Pediatric Fellowship Examination of the Royal College of Physicians and

Surgeons of Canada.

The Royal College of Physicians and Surgeons of Canada uses Computerized Patient Management Problems (CPMP) regularly in their Pediatric Fellowship Examination

process to test the ability of pediatric candidates to manage patients (see appendix I for

an example of a CPMP). The College also uses conventional multiple-choice questions

and oral examinations in the Fellowship exam. The specific objectives of the CPMP section are to assess the candidate’s skill and ability in data gathering, formulating hypotheses regarding the patient’s problem, and resolving the problem. Candidates have a

choice of taking an examination in either English or French.

The computer portion of the examination has been administered annually since 1974

to all candidates eligible for certification in pediatrics. The development of the CPMP examination was precipitated by several factors (Skakun, 1978): 1) concerns about the inability of multiple choice and essay examinations to measure various clinical aspects of
Ch. 4—Computer Applications in Medical Education and Assessment ● 59

medicine; 2) concerns about using a single examination, such as mutiple choice or essay,

to certify specialists; and 3) concerns about the degree of reliability, validity, and standardization of oral examinations.

The types of CPMPs used in the Royal College exam are the linear and the sequential.

The linear uses two response strategies. In the simple choice, the candidate selects six

items or less from a list of possibilities; in the ordered choice, the candidate chooses a

fixed number of options and then rank orders them. In the sequential type of CPMP, the

candidate scales each option on a scale of 1 to 20; the most pertinent information is given

a weight of 20 and the least pertinent a weight of 1. In this type of problem each candidate is presented with the same amount of information and is led, in an identical stepby-step manner, to the next options in the problem.

Officials of the Royal College note that the examination could have been administered using conventional paper-and-pencil PMPs. Use of the computer, however, has advantages in that it prevents the retracing of physician questioning and allows for more

complex branching problems, incorporation of visual and audio components (i. e., Xrays, heart sounds, etc.), and immediate scoring of the problem (Skakun, 1978).

Most of the CPMPs used are developed and tested on the IBM 1500 computing system at the Division of Education Research Services in the Faculty of Education at the University of Alberta. Each member of a five-person committee is asked to develop two

problems in areas relevant to pediatric practice. Each problem is then reviewed by the entire committee. Once the committee reaches agreement on the content, the problems are

programed for presentation by a computer. A group of pediatricians not involved in the

preparation or initial review of the problems takes an examination in which the problems

are presented. On completion of the examination, their comments and suggestions are

used to further refine CPMPs. These updated problems are then used in the Royal College examination. Problems are reviewed annually for accuracy and relevance by practitioners not involved in their initial construction.

CPMPs also have been used since 1974 as part of the examination process for approximately 110 final year medical undergraduates at the University of Alberta. Additionally, CPMPs have been incorporated into a self-assessment package that is available

to fellows of the Royal College during the time of the College’s annual meeting.

The Royal College exam is administered at eight different centers across Canada,

using either available computing facilities in each center, or a network such as the

CHARGEX system of the Canadian Imperial Bank of Commerce.

Each year, approximately 150 candidates take the Royal College’s CPMP examination in addition to taking a multiple choice test, an oral examination, and an in-training

survey. The cost per candidate in 1974 for administering four CPMP problems was approximately $225. These costs have been decreasing gradually since 1974; in 1977, the

cost per candidate was approximately $150. For the self-assessment tests held at the time

of the College’s annual meeting, approximately 200 fellows participate at a cost of significantly less than $100 per fellow.

Evaluation studies of the Royal College Fellowship exam indicate (Fincham et al.,

1976; Berner et al., 1977; Skakun et al., 1977):

● No positive correlations exist between CPMP, multiple choice, and oral examination portions of the Fellowship Examination. This finding suggests that different

aspects of competence are being measured by the various types of tests.
60 ● Computer Technology in Medical Education and Assessment

. There are low interproblem correlations between CPMPs. This suggests that candidate performance might be case- or problem-specific, or that the skills required

to manage the patient, such as problem solving and decisionmaking, differ from

case to case.

● Candidates had no difficulty in using the computer terminal and viewed the problems as realistic and relevant to pediatrics. They suggested that CPMPs were better tests of pediatric skills than the multiple choice questions.

● Nationwide computer-based testing is feasible and computerized patient management problems can be translated to multiple computer languages and hardware

systems and still operate successfully to support such testing.

The Computer-Based Examination (CBX) Project of the American Board of

Internal Medicine and the National Board of Medical Examiners

The CBX project, initiated in 1968, is based on a computer simulation of a patient

with a given disease or diseases. Although various computer simulations have been explored, all involve physician interactions with a computerized patient that is or is not

hospitalized. In the current CBX simulation model (which evolved from INDEX), laboratory tests, medical procedures or consultations, and drug therapies interact with the patient’s disease in a time sequence similar to one occurring in real life. By simulating the effect of physician action on the patient, and by adjusting the patient’s status accordingly,

the computer model provides an almost life-like, dynamic simulation of the patient/physician encounter.

The computer model, through a sampling of the physician’s interaction with a patient, permits the assessment of a number of factors, including cost of workup, risk (pain,

complications, and mortality) to the patient, time taken to initiate corrective therapy,

and length of a patient’s hospital stay. The model also permits a step-by-step evaluation

of the physician’s action in ordering tests and prescribing therapy. Appendix I contains a

sample interaction between a physician and a simulated patient.

The major reservation to implementing this system centers around the ability of the

project: 1) to develop a scoring strategy and to determine whether the competencies

assessed by this model are appropriate to measure, and 2) to develop a method for the

cost-effective production of new simulations. Future plans involve further testing and refinement of the model as well as development of additional patients or scripts. Studies

have been initiated to evaluate the model and determine its economic feasibility for largescale testing and use by NBME and ABIM as part of their certification and recertification

processes (Friedman, 1978).

A Model for Evaluation and Recertification through Individualized Testing

(MERIT) of the American Board of Internal Medicine.

MERIT was begun in 1973 to investigate: 1) an approach to recertification that uses

the specific patient problems of each internist as the basis for the evaluation of his/her

skills, knowledge, and clinical judgment; 2) the use of an advanced computer-assisted

simulation of the clinical patient encounter (CASE) as the examination instrument for the

individualized evaluation; 3) the development of a scoring system for CASE to assess

various dimensions of clinical behavior; 4) the involvement of each participating internist in an evaluation process that will help him/her to identify deficiencies and thus to plan

for more meaningful, individualized continuing education; and, 5) the attitudes of prac-
ticing internists toward

(Harless et al., 1978).

Ch. 4—Computer Applications in Medical Education and Assessment ● 61

a recertification process that embraces the preceding components

The MERIT process has undergone extensive evaluation during the past 5 years with

450 Board-certified internists in 3 field studies. The first study was conducted in San

Francisco with 28 physicians in 1973-74; the second study, held in 1976, included 90 physicians located in 9 health care and training institutions throughout the State of Ohio;

and the third study was conducted in January 1978 at 15 centers nationwide and involved

332 physicians.

The MERIT process was designed to incorporate individualized evaluations of a candidate’s performance in patient care as a major part of the recertification procedure. It is

meant to supplement the multiple choice questions and paper-and-pencil PMPs used in

current examinations. MERIT incorporates a number of steps, the first of which is the

gathering of information from each candidate about the types of patients he/she sees in

practice. This patient information is summarized into an individual practice profile that

is used to select the three most representative computerized patient simulations for that

physician’s exam. Table 8 depicts a typical practice profile.

The MERIT process uses the CASE simulation and generates six separate scores (a

seventh, history and physical exam minimum data base score, is under consideration):

1.

2.

3.

4.

5.

Diagnosis Score. Diagnoses for each CASE that a candidate is required to list

have been identified. Each diagnosis is then assigned some proportion of a total

100 percent. This weight is based on the degree to which the disease can be

treated and the degree to which it threatens life. An individual’s diagnosis score

is the total of the weighted value for the correct diagnoses listed; the possible

range is O to 100 percent.

Management Score. The management steps (medications, procedures or consultations, and patient instructions) that a candidate is required to list have been

identified. Each management step listed as a scoring criterion is weighted differently, totaling 100 percent across all three therapy areas. An individual’s score is

the sum of the weights of each correct therapeutic step he/she lists in the interaction.

Critical Concept Score. Information from the history, physical examination, and

laboratory workup for each CASE that is considered critical in determining both

the patient’s problems and an appropriate treatment plan has been identified.

Each piece of information (“critical concept”) is then weighted nine, four, or one

to reflect its importance in the patient’s workup. An individual’s critical concept

score is either the total weighted value of critical concepts he/she elicited, divided by the weighted total of critical concepts in the CASE, or the percentage of

the total possible critical concept value elicited.

Efficiency Score. This score is applied in the laboratory section only. In addition

to identifying critical laboratory information, procedures or tests that are neither

critical nor excessive (0-weight) have been identified. An individual’s efficiency

score is the total number of critical laboratory results elicited, divided by the sum

of critical and inefficient procedures (or divided by total procedures minus 0-

weighted procedures). An individual’s efficiency score is, then, the percentage of

total laboratory results elicited that are critical to the CASE.

Danger/Discomfort (D/D) Score. Laboratory procedures that entail some risk

and/or discomfort are weighted ( – 9, –4, and – 1) to reflect the degree of risk

or discomfort. An individual’s D/D score is a total of the negative weight received. This is not a percentage score; zero is perfect.
62 ● Computer Technology in Medical Education and Assessment

I

(

I

. . .

. . .

. . .

. . .

I

I

I I

I I

I

I

I

I

. .
Ch. 4—Computer Applications in Medical Education and Assessment . 63

6. Cost Score. Every laboratory procedure or test has a dollar cost value associated

with it. An individual’s cost score is the sum of the value of every procedure

and/or test ordered.

The CASE model, when used in an examination, requires that the examinee use

his/her own intellectual resources to resolve the problems of the simulated patient. No

direction is provided, such as in a multiple choice list, to resolve the problem. Differences

between CASE and a multiple choice list are analogous, though more pronounced, to the

difference between a multiple choice and an essay exam; in both CASE and an essay

exam, the uncued behavior and deeper thinking of the examinee are being assessed.

Patterns of behavior can be determined for each physician being examined because

each is acting without cues. Behavior patterns that are dysfunctional or potentially harmful to a patient can be corrected. For example, a physician may demonstrate a tendency

to use some drugs excessively or some procedures unnecessarily. Alternatively, patterns

that are creative and effective can be rewarded and shared with the physician community.

Dramatic errors can be uncovered since the examinee is not cued or restricted in

what he/she does to the simulated patient. An example of this occurred during MERIT

field trials. A phyician dealing with a simulated patient determined that exploratory surgery should be performed on his patient. As it turned out, the patient was in congestive

heart failure, but the physician had not discovered this during his workup. A cued or

multiple choice exam would never uncover this situation, since the treatment plan suggested by the physician was so radical it probably would not have been a multiple-choice

option.

In the course of the MERIT project a variety of research questions have been addressed with the following results:

●

●

●

●

●

●

●

It is feasible to individualize an examination based on the kinds of patients an internist sees in practice.

It is technically and logistically feasible to reliably examine physicians simultaneously at different sites nationwide.

Internists gather information and initiate patient management steps similarly with

CASE as with real patients.

Internists generally found the MERIT process to be an acceptable and desirable

form of evaluation.

The need for reliability suggests that any implementation of MERIT include the

use of five to seven, as opposed to three, CASES.

Comparison of MERIT scores with other component scores in the recertification

examination confirmed that MERIT measures substantially different aspects of

competence.

The validity of MERIT CASE scores compares favorably with other examination

methods employed in the study.

ABIM is improving the current scoring system and examining the logistics and costs

of full-scale implementation of this methodology (Meskauskas and Norcini, 1978).

COMPUTER-BASED MEDICAL CONSULTATION SYSTEMS

Computer-based consultation systems range from those used in specific disease or

problem areas to those used for more general purposes. A number of these systems are
64 ● Computer Technology in Medical Education and Assessment

briefly summarized below. More general purpose computer systems also are used to help

guide the physician in diagnosing and managing patient problems. Some of these will be

described in the next section on quality assurance/data management systems.

Most of the consultation systems accumulate data on patient characteristics, histories, physical findings, laboratory data, treatments, and outcomes. These data are then

analyzed for correlations that might be found between certain patient characteristics and

treatments with specified outcomes. This information is then used to help guide the physician in managing the patient and in discovering new knowledge regarding patient care.

Duke University Cardiovascular information System. This system provides physicians with a large data base regarding clinical experiences with coronary artery disease.

The data describe outcomes of patients with various sets of attributes (Rosati et al.,

1975). The patient attributes, laboratory and physical findings, history, and outcome of

a large number of patients are stored in a computer, which then classifies the information. When attributes of a new patient are entered, the computer selects the most closely

matched subgroup. The disease courses and outcomes of all patients previously categorized in the same subgroup are displayed. The computer’s memory is accurate, unbiased by recent or dramatic events, and enhanced by the great number of entries derived

from the entire institution, rather than from one physician’s experience. Therefore, the

physician’s management decision can be based on far more accurate and unbiased information than could be possible without the computer.

Electrolyte and Acid-Base Consultation System. Beth Israel Hospital in Boston,

Mass., has constructed a computer consultation program to help physicians manage patients with electrolyte and acid-base disorders (Bleich, 1971). The program directs a

dialog in which the physician, or other user, enters clinical and laboratory information.

On the basis of the abnormalities detected, the program asks traditional questions to further characterize the electrolyte and acid-base disturbance. During or after the completion of the dialog, an evaluation note is produced that contains a list of diagnostic

possibilities, an explanation of underlying pathophysiology, therapeutic recommendations, precautionary measures, suggestions for further studies, and references to the medical literature.

HELP. The HELP system is the core of a complex group of computerized subsystems

developed at the Latter Day Saints Hospital in Salt Lake City, Utah (Giebink and Hurst,

1975). A variety of findings on symptoms data and patient status information is incorporated into the system after being acquired through laboratory tests, blood gas analysis,

intensive care monitoring units, multiphasic health testing stations, and medical record

abstracts. Decision logic that uses a variety of statistical techniques is employed to develop probabilities for certain diagnoses or treatment selections based on the historical data

base. In addition, the system provides warnings of patient conditions that may require

intervention in several areas of the hospital, including the emergency room. The rapidly

growing data base and the decision logic are also being used in evaluation and research.

INTERNIST. The INTERNIST was developed at the University of Pittsburgh as a

computer-based diagnostic consultation system for problems in internal medicine (Lawrence, 1978). It is based on assigning rough estimates of the likelihood of the association

of a disease, given a finding, and a similar estimate of the likelihood of the finding, given

a disease. It currently covers about two-thirds of internal medicine. Internist represents

an attempt to model, within a computer program, the thinking processes that the author

uses to evaluate a case and make a diagnosis. It is not a model based on probabilities in

the statistical sense and, therefore, is subject to the individual perspective brought to the

model. The system mimics the diagnostic behavior of the “excellent clinician” and is an
Ch. 4—Computer Applications in Medical Education and Assessment ● 65

example of a number of decision analysis types of systems currently being developed and

tests.

lndiana University Medical Center Computer Reminders System. The computer reminders system is based on the assumption that the physician must apply a few simple

rules to a few items of information many times (McDonald, 1976 and 1978). Using a very

simplified computer language, developed especially for this use, the physician writes the

rules he/she wishes applied to the data and then lets the computer repeatedly apply them.

The system provides reminders for a large percentage of the simple clinical decisions; it assures that baseline screening tests have been done, checks that abnormal test

findings are examined further, and assures that treatments are followed with appropriate

measures. The system occasionally suggests a diagnosis when a particular pattern of abnormalities is evident, but more often suggests treatments that might correct a pattern of

abnormalities. Studies of the system’s impact on patient care are currently underway.

COMPUTER USES IN QUALITY ASSURANCE/DATA

MANAGEMENT SYSTEMS

As described in the previous chapter, the practicing physician is subject to the requirements of various “quality assurance” programs designed to measure, evaluate,

monitor, and/or improve medical services. Computers provide the basis for the data

processing components of these programs. Some of these activities have been described

earlier using examples from both the private and governmental sectors. These examples

included activities sponsored by the Joint Commission on Accreditation of Hospitals,

private health insurance companies (e.g., Blue Shield) in payment of claims and quality

and utilization review, hospital discharge abstract services, the Medicaid Management

Information System (MMIS), and activities related to Professional Standards Review

Organizations (PSROs). The latter two programs are briefly summarized below, followed by short descriptions of two multipurpose systems.

Medicaid Management Information System

MMISs are used to assist States in the management of their Medicaid programs. The

Surveillance and Utilization Review (S/UR) module of MMIS: 1) produces a pre-screen

set of profiles that have been compared to the average pattern of care as defined by the

State, 2) limits production of profiles to those providers demonstrating aberrant behavior, 3) performs postperformance utilization review, and 4) offers options in the individual State’s approach to utilization review. *

Professional Standards Review Organizations

PSROs are federally mandated programs organized primarily to determine the medical necessity and appropriateness of medical services provided to patients in federally financed programs. Major functions of the review process include admissions certification,

continued stay review, retrospective review, and medical care evaluation studies.

Computers are used principally to schedule both admissions certification and concurrent review activities and to evaluate patterns of care. Computer applications that

*See ch. 3 for a more detailed discussion.
66 . Computer Technology in Medical Education and Assessment

provide length of stay indicators and norms and standards are also used to properly certify the appropriate length of stay for a particular case. Review function computer applications include profiling and, to some extent, evaluating medical care. Since profiling

is an aggregation of data elements, the computer is particularly useful because it handles

a wide variety and a large number of cases. Statistical analyses are performed to display

the information in a variety of ways. The computer is used in medical care evaluation

studies not only to process data that focus on the individual problems being studied, but

also to monitor those problems after solutions have been implemented to determine

whether or not the action taken has solved the problem. *

Computer-Stored Ambulatory Record System

The Computer-Stored Ambulatory Record System (COSTAR) developed at Massachusetts General Hospital (Barnett, 1976) is used in the Harvard Community Health Plan

to provide automated medical records and business office support. The data base is also

available for retrospective quality assurance review and to preprogram reminders or

alerts within the system for patient followup and selection of preferred therapies.

Problem Oriented Medical Information System

The Problem Oriented Medical Information System (PROMIS), developed at the

University of Vermont (USDHEW, 1977) is unique in two respects. It not only radically

restructures the medical record, but also directs the process of clinical care. The PROMIS

laboratory staff developed these capabilities in order to address problems hindering the

provision of medical care: dependence on the physician’s memory, ineffective organization for massive amounts of medical data, and lack of meaningful feedback about the appropriateness of care.

In PROMIS, data is organized by patient problems. The computer record is structured around four phases of medical action: an initial data base on each patient, including medical history and physical examination; a list of the patient’s problems;

diagnostic and treatment plans for each problem; and progress notes on each problem indicating how the patient is progressing during therapy. Except for the initial data base,

every entry into the computer record is associated with a particular problem of the patient. Thus, when a technician enters the result of a laboratory test, the data is entered

under the problem for which the test was initially ordered. By structuring the record in

this way, all information pertinent to a problem is organized logically for review by the

physician and other medical care professionals.

Personnel enter data about patients through video terminals. The videoscreen of the

terminal displays an array of choices, and the provider makes a selection by touching the

screen. Data is entered by the medical care professionals who originate them. For example, physicians and nurses enter notes about the patient’s progress, radiologists enter

notes as they read films, and technicians in the clinical laboratory enter results of tests. In

addition, patients enter their own medical histories. Each staff member has a unique

identification code that allows entry and access only to those parts of the computer

record necessary for the provision of care.

*See ch. 3 for a more detailed discussion, including the application to ambulatory care in New Mexico

and Utah.
Ch. 4–Computer Applications in Medical Education and Assessment ● 67

SUMMARY

Medical education and assessment and patient care could be integrated, interactive

subsystems of the medical care system. Computer technology not only has shown promise in helping achieve this integration, but also has aided advances in this direction. The

capabilities of computer-assisted instruction, testing, and management to provide individualized assistance in the medical school curriculum are being extended toward individualized examination on entry to and during practice. In the future, some of these

examinations will be based on data extracted from the actual practice experience of examinees, thereby allowing both for better measurement of practice strengths and weaknesses and for better definition of continuing education needs.

Computer capabilities to maintain and analyze institutional, regional, and national

data bases on actual patient care will allow better correlation between the condition of

the patient (outcome) and patient care techniques (process). These capabilities also will

help provide better indicators both of patient care techniques that are successful or unsuccessful for specific diseases, individuals and situations, and of the costs, for example,

of alternative methods of diagnosis and treatment. This type of information will enhance

an outcomes approach to quality of care measurement and reduce the present emphasis

on process measures. Better definition and correlation also will restructure the content

and emphasis of undergraduate and graduate medical education.

The creation and use of computer-based simulations enable better assessments of

aspects of clinical competency. These simulations may hasten consensus formation

regarding the definition of certain dimensions of competent care because of the requirement for specificity in therapeutic responses to simulated illnesses. Since patterns of performance behavior can be recorded in detail by the computer, cumulatively, they can be

used to create diagnostic and therapeutic profiles for particular diseases. These

cumulative profiles of care can be used in PSRO and other quality review situations as

more detailed standards against which to measure appropriate care. For example, the extent to which a repertoire of tests or a particular test is indicated or required can be used

in addition to the rougher measures of quality of care, such as length of hospital stay.

Inversely, using computerized recordkeeping profiles, appropriate care for disease

entities can be defined and verified in great detail within norms and ranges. These profiles of appropriate care can be added to the medical knowledge base and create computer

simulations for teaching and testing. Such simulations could expose students to patients

and diseases not experienced in the clinical setting. Through individualization of instruction and the use of computers for patient scheduling, students in clinical rotations can be

matched to patients to provide them with the proper, rather than random, exposure to

different diseases. These experiences can be recorded so that simulated patients may be

used appropriately to fill voids in experience.

The computer’s information storage and retrieval capabilities can increase the accessibility of the medical knowledge base. Textbook as well as clinical data can be quickly updated and restructured to provide the same data organized in a variety of ways.

This ability to reorganize, restructure, and analyze data also facilitates creation of new

medical knowledge.

The computer can be a viable and cost-effective resource if used in a coordinated

manner. Having one computerized data base for education, another for assessment,
68 ● Computer Technology in Medical Education and Assessment

another for patient care, and another for administration, not only is inefficient in most

cases, but also prohibits reaping of combined benefits. Thus, one of the challenges facing

the increased use of computer technology in medicine is the design of a system that

facilitates and enhances data and information flow among the subsystems described in

this chapter.
Chapter 5

DEVELOPMENT AND DISSEMINATION

OF COMPUTER-BASED EDUCATION
DEVELOPMENT AND DISSEMINATION OF

COMPUTER-BASED EDUCATION

OVERVIEW

This chapter describes the extent of development, use, and dissemination of the

computer-based educational materials discussed in the previous chapter. The institutions

described in this chapter were selected to represent a variety of institutional developers

and users of health sciences computer-based education (CBE) materials. Each institution

applies computer technologies differently and possesses a range of computer hardware,

software, and mechanisms by which users access computer-based materials.

Three centers that have different types of CBE are described in this chapter: the Ohio

State University College of Medicine, the University of Illinois School of Basic Medical

Sciences, and the Massachusetts General Hospital. The Ohio State CBE system serves

most health areas, including nursing and allied health. The University of Illinois center

uses one of the most sophisticated CBE hardware systems and concentrates CBE use in

the basic medical sciences. The Massachusetts General Hospital has a hospital-based program focusing on clinical applications.

The methods of disseminating CBE materials are described in the second section of

this chapter. These methods represent both on-line and off-line methods of sharing. In

off-line dissemination, CBE courseware is transferred to the users’ computer; in on-line

dissemination, CBE courseware is made available to the user by networking or some

other form of communication. The Health Education Network, Inc., is one example of

on-line dissemination; the Computer-Assisted Teaching Systems Consortium and

Milliken Communications Corporation are examples of two different off-line sharing

methods.

INSTITUTIONAL DEVELOPMENT AND USE

During the past decade, the development and dissemination of computer-based educational and assessment materials in the health sciences have been rapid. In 80 American

and 4 Canadian institutions that responded to a 1975 survey of the 135 U.S. and Canadian medical schools (Lefever and Johnson, 1976; Tidball, 1976), there was at least a 60-

percent annual growth rate, since 1967, in the number of medical school courses using

CBE. Figure 12 shows the cumulative number of such courses versus the year in which

they were first taught. The 351 courses taught in 1975 were divided almost evenly between the basic and the clinical sciences, These courses were developed using more than

15 different types of computers and programing languages.

Of the 84 schools that responded to the 1975 survey, 69 percent generated programs

at their own institutions, and 64 percent used programs from other institutions via networking or time-sharing arrangements (an institution can both create its own programs

71
72 ● Computer Technology in Medical Education and Assessment

Figure 12.—Cumulative Number of Courses Involving CBE Versus the Year

in Which They Were First Taught

400

350

300

250

200

150

100

50

351

65 70 75

YEAR FIRST TAUGHT

SOURCE Tldball, 1976

and use those of any number of other institutions). Fifty-one percent of the institutions

reported having permanent facilities for computer-based instruction.

The critical factors that facilitate or inhibit the development or use of CBE in the institutions described in this chapter are the following (Casburgue, 1978):

● the high financial investment required for the development of CBE is a primary inhibiting factor to use;

● a reliable computer system is an essential prerequisite for CBE development or

use;
●

●

●

Ch. 5—Development and Dissemination of Computer- Based Education ● 73

an institutional commitment must be made not only to the development and use

of CBE, but also to the provision of support staff;

educational programs must be available for faculty administrators and technical

personnel; and

the degree of CBE integration with other instructional modalities is an important

determinant of its use.

Table 9 ranks the top 10 medical institutions according to both the number of CBE

course hours and the number of CBE programs or units they have produced. * Three of

the institutions are described below.

Table 9.—lnstitutions Producing Computer-Based Materials

Ten most productive institutions Ten most productive institutions

(by course hours) (by number of programs)

Institution Course hours Institution Programs

Ohio State. . . . . . . . . . . . . . . . . . . . . .. ....229 Ohio State. . . . . . . . . . . . 63

Illinois . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Cornell. , . . . . . . . . . . . . . . . . . . . . . . . . . 33

Pennsylvania State. . . . . . . . . . . . . . . . . . . . 58 Massachusetts General Hospital. . . . . . . . 27*

Kansas. . . . . . . . . . . . . . . . 52 Purdue. . . . . . . . . . . . . . . . . . . . . . . . . 21

U.S. Army . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 California (SF). . . . . . . . . . . . . . . . . . . . . . . . 16

Massachusetts General Hospital. . . . . . . . 36 Michigan . . . . . . . . . . . . . . . . . . . . . . . . 14

Florida. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Harvard . . . . . . . . . . . . . . . . . . . . . . . 10

British Columbia . . . . . . . 25 Northwestern . . . . . . . . . . . . . . . . . . . . . . 10

California (Los Angeles) . . . . . . . . . . . . . . . 19 Connecticut . . . . . . . . . . . . . . . . . . . . . . . . . 9

Cornell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Missouri (Kansas City) . . . . . . . . . . . . . . . . . 9

. Each of these basic programs can create multiple cases so that over 200 different learning situations are available

SOURCE Kamp, 1975

The Ohio State University College of Medicine

The largest collection of available CBE courseware in medicine and health-related

areas resides at The Ohio State University (OSU) College of Medicine (Kamp, 1975).

This CBE system has grown from a one-terminal, one-course system in 1967 to a system

that provides 22-hours-per-day, 7-days-per-week CBE access for students and health professionals throughout the United States; maintains a program library of over 425 interactive hours of CBE materials; logs approximately 3,000 usage hours per month; and adds

CBE course materials to its library at the average rate of 5 interactive hours per month

(Pengov, 1978).

The computer has been an important element at OSU in undergraduate medical education, nursing education, allied health education, continuing health sciences education,

and patient and nonmedical support staff education. It is used both to assist in instruction and to help manage the instructional process. Every medical and nursing student in

the college uses the computer. A smaller but significant percentage of allied health students also uses the computer.

In the traditional lecture-discussion curriculum, as in the independent study program (ISP), computer-assisted instruction programs are rather extensively employed.

Fifty-two CBE programs are used by medical students in this curriculum; each student

spends a minimum of 25 interactive hours at the computer terminal. The most extensive

use of CBE is in physiology; CBE tutorial units parallel class lectures and readings on

physiology. Since 1973, the college has refined, developed, and used the computer-

*There are other producers of CBE medical materials (e.g., the Air Force) that are not described in this

table but are described elsewhere (Deland, 1978).
——

assisted simulation of the clinical patient encounter (CASE) (Pengov, 1978). The model,

a natural language simulation of the patient-physician encounter, is used on an elective

basis by students in the clinical areas.

The School of Nursing’s Bachelor of Science curriculum includes more than 35 interactive hours of CBE materials that are used in a variety of ways by over 600 nursing

students. Eight CBE programs, representing 20 interactive hours, are required of all nursing students; 7 CBE programs, 15 interactive hours, are used as optional, supplemental

learning materials (Mourad and Forman, 1976).

The degree of CBE integration into the curriculum of the 11 divisions in the School

of Allied Medical Professions varies by division. Six divisions do not use CBE; others use

40 CBE programs in approximately 20 required courses. Student involvement with CBE

varies from 4 hours in radiation therapy to over 100 hours in medical dietetics (Breeze

and Schimpfhauser, 1976).

During 1968, a pilot network, designed to provide continuing education to Ohio

physicians, was established and placed in four community hospitals chosen to represent

various geographical areas of the State. The goals of this project were revised in 1970;

other health professionals were included, and the network was expanded to 10 hospitals

within the State. Federal subsidies to the hospitals for program development declined

between 1970 and 1975, and, as a result, 18 institutions began to pay the full costs of

using CBE through OSU's Computer-Assisted Instruction Regional Education Network

(CAIREN) (Forman et al., 1978).

Throughout its history, CAIREN has attracted some 29 institutions to its membership roles. The CAIREN membership has fluctuated according to the educational needs

of each member institution; membership has also changed for reasons of cost. Membership has ranged from 1 to 7 years, from small community hospitals of less than 300 beds

to large teaching hospitals of nearly 1,000 beds, and from mental institutions to healthrelated educational institutions and to technical schools offering nursing and allied health

training programs. Table 10 summarizes the membership profile during CAIREN’S first

10 years.

The institutional libraries of most CAIREN users have a terminal that provides

access to CBE materials. CAIREN avails CBE to hospital personnel on all shifts, every

day of the week, for both in-service training and orientation programs. CBE continuing

education courses are accredited for use in Ohio by the professional societies listed in

table 11.

Table 10.–Computer-Assisted Instruction Regional Education Network (CAIREN)

Membership Profile

Institution type

Health-related Mental

Year Number of Community educational health Hours of

ending members hospitals facility institution use

1969 . . . . . . 4 4 ——

1970 . . . . . . 12 12 ——

1971 . . . . . . 13 12 1 ——

1972 . . . . . . 19 15 4 8,536

1973 . . . . . . 16 12 4 13,300

1974 . . . . . . 20 10 4 6 11,374

1975 . . . . . . 15 8 3 4 8,430

1976 . . . . . . 9 8 1 6,220

1977 . . . . . . 8 7 1 4.164

1978 . . . . . . 12 7 5 4;326

SOURCE: Ohio State University College of Medicine.
Table 11 .—Accreditation of Computer-Based Continuing Education Courses by

Professional Societies

Approximate hours of CBE

Professional society credit available

American Dietetic Association. . . . . . . . . . . . . . 108

American Medical Association. . . . . . . . . . . . . . 70

American Osteopathic Association . . . . . . . . . . All programs eligible for Category II-A credit

Ohio Council of Medical Technology. . . . . . . . . 60

Ohio Nurses Association . . . . . . . . . . . . . . . . . . 6

State of Ohio Board of Pharmacy . . . . . . . . . . . . 106

SOURCE Pengov. 1978

As a direct result of the continuing education efforts, courses have been developed

for use by nonmedical support staff and patients. Courses on the medical record and on

medical terminology are examples of two support staff courses that have been useful in

CAIREN hospitals. Examples of CBE patient courses include those that offer instruction

to mothers in the care and feeding of newborns, allow patients to assess the probability

of heart attacks, and provide guidance on proper menu planning. The variety of audiences that use the college’s CBE materials is shown in table 12.

The college’s CBE materials are available to users via two routes. In the first, “offline, ” the CBE programs are removed from the OSU computer and put in the user’s computer. Over 100 C-BE programs have been transferred “off-line” and are used largely on

compatible IBM equipment using COURSEWRITER III programing language. In the second and more heavily used route, the user has a terminal onsite but links via telephone

lines to the CBE data base at OSU. The extent of CBE use at OSU has been influenced by

the ability of one integrated data base and CBE hardware system to service a wide variety

of users. CBE use also has been influenced by the large numbers of support staff whose

existence minimizes the extent to which faculty authors must learn about computer operations. Figure 13 delineates the three major user groups: the college’s own educational

programs, the statewide sharing network (CAIREN), and the nationwide Health Education Network. * Table 13 provides data on the three groups of users. Figure 14 shows the

breakdown of the use by the College of Medicine itself, and table 14 summarizes this use

over 6 years.

Table 12.—The Ohio State University College of Medicine Computer-Based

Education Programs Indexed by Audience

— —- —— —— —

Audience Number of programs

Clinical laboratory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Dietary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Dental. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Emergency medical. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Environmental services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Medical records. ., . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Medical (physicians, residents, interns, students) . . . . . . . . . . . 113

Nursing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Occupational therapy ., . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Optometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Patients and families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Pharmacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Physical therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Respiratory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Radiology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Secretarial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

SOURCE Pengov, 1978

*This is a network for interinstitutional sharing of CAI materials described later in this chapter.
College of

Medicine

Figure 13.—Data Base and CBE System User Group Overview

(CAIREN)

SOURCE. The Ohio State University, 1978

Table 13.—College of Medicine Computer-Based Education User Group Summary

Number of Average usage Total

Category terminals hrs./month usageb

College of Medicine. . . . . . . . . . . . . . . . . . . . . . . 24 1,728 20,738

Computer Assisted Instruction Region

Education Network (CAIREN) . . . . . . . . . . . . . 9 347 4,164

Health Education Network . . . . . . . . . . . . . . . . . 39a

348 4,174

Totals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 2,423 29,076

a

Thls number represents the monthly average number of institutions throughout the country accessing OSU CBE materials

Each institution may have a number of terminals which are used for accessing the system

b

January-June 1977

SOURCE Ohio State University College of Medicine, 1978.

Although early CBE efforts at Ohio State received considerable outside funding, current operational support comes primarily from college funds (68 percent) and from income generated through National and State network use (32 percent). Users of the Health

Education Network pay $6 to $10 per connect hour depending on volume of use. Users in

the CAIREN network pay approximately $650 per month for unlimited access through a

single terminal. Average user dollar expenditures for CAIREN are shown in figure 15. All

cost figures include expenditures for CBE courseware development.

The University of Illinois School of Basic Medical Sciences

The University of Illinois was one of the first institutions to use the programed logic

for automatic teaching operations (PLATO) system for computer-based education in the

health sciences. The PLATO system, developed at the University of Illinois ComputerBased Education Laboratory (Bitzer and Boudreaux, 1969), is primarily committed to
—

Figure 14.–CBE Usage Profile for the College of Medicine (1977)

Independent Study Program Lecture Discussion

(Medical Curriculum) Program (Medical

Review

LIBRARY

Services S

School of Allied

Medical Professions

School of \ \

Nursing

/

Visitors

SOURCE The Ohio State University College of Medicine, 1978

Table 14.— Computer-Based Education Utilization Summary for the

College of Medicine

Computer

Year Hours Percent increase/decrease

1972 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9,693 —

1973 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16,884 +42.6

1974 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19,159 +11.9

1975 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17,140 –11.8

1976 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21,711 +21.1

1977 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20,737 – 4.7

SOURCE Ohio State University, 1978

research in CBE. However, it also is used to support accredited instruction in a variety of

subjects during at least 80 hours of a typical week at the University (Hody and Avner,

1978). Currently some 950 terminals are linked to the system (Bloomfield et al., 1978).

The unique graphic, audio, tactile, and other capabilitiesof the PLATO terminal distinguish it as one of the most advanced CBE terminals available (Sherwood and Stifle,

1975).

The University of Illinois’ PLATO system and its several years of operational experience provided the requisite groundwork for the School of Basic Medical Sciences to

develop a computer-based core curriculum. This effort began in 1973 with funding from

both the U.S. Department of Health, Education, and Welfare and the State of Illinois.

Lessons used in CBE provide both primary and supplementary instructional materials,

drills, vocabulary builders, and simulations. Building upon the initial CASE work (Harless et al., 1973), simulated exercises now involve social counseling, differential diagno-
Figure 15.—Average User Dollar Expenditures*

Computer Assisted Instruction Regional Education Network (CAIREN)

Communications

cost s

Computer

cost s

ARE NOT INCLUDED

Ad m i n i

Terminal 

Term i n al

Field Se

Staff T r

Office S

Program

Develo

Visual U

Etc .

Charges

stratio n

Insurance

Pa per

rvices

avel

uppli es

pmen t

pd a t es

SOURCE: The Ohio State University College of Medicine, 1978

sis, physician attitudes toward the dying patient, as well as traditional case studies of diseases (Bloomfield et al., 1978).

A computer-based diagnostic examination system, used in the undergraduate medical curriculum, allows students to evaluate themselves. Students are encouraged to take

an exam via the PLATO terminal after they have completed each of 10 clinical problems.

Each exam lasts about 2 1/2 hours, allows the student an opportunity to retry questions,

and provides literature references for incorrect responses. Combining all types of CBE

materials, there are about 150 modules of basic medical science materials.

In 1974, PLATO terminals were made available to a number of medical schools

upon their agreement to author lessons and to make the materials available to their students. Illinois provided the terminals and computer time free; participants paid a share of

the service costs and all of their communication costs. Seven institutions have taken advantage of this opportunity. These seven and Illinois have logged over 17,000 student instructional hours on 80 terminals, or over 4,000 to 5,000 hours per year of use. This level

of use accounts for approximately 10 percent of the total University of Illinois PLATO

system use (Bloomfield et al., 1978). The number of PLATO users is growing as more

courseware is made available.
Ch. 5–Development and Dissemination of Computer-Based Education ● 79

Massachusetts General Hospital

The Massachusetts General Hospital (MGH) Laboratory of Computer Science developed its first computer simulations approximately 10 years ago. The programs were developed to teach and test clinical problem-solving skills and serve as a supplement to

traditional methods of medical education.

The MGH data base now contains simulations in approximately 30 subject areas;

most of the topics are presented in a case-oriented format (see appendix I). Each case requires approximately 10 to 45 minutes to complete, depending on the program strategy,

the format employed, and the pace preferred by the user. The user can select whatever

content material is of particular interest and a time and location that are most convenient. Programs are updated frequently. User comments, analyzed daily by the MGH staff,

are often the source of program revision.

MGH became the first host computer on the experimental Health Education Network in 1972. Since May 1975, the programs have continued to be available through a

national communications network with MGH assuming administrative and financial

responsibilities. Under contractual arrangements, user institutions are charged for program use on a monthly basis.

In the past 7 years, MGH has served 20,000 individuals in over 150 institutions

throughout the United States and in several foreign countries with approximately 70,000

hours of computer-based medical education. Monthly usage has ranged from 20 to more

than 1,500 hours. While institutions using the MGH materials vary in size and type, most

user groups are medical schools or university medical centers (68 percent of total groups)

and hospitals (16 percent).

Through local and hospital program use as well as the network activity, MGH provides access to its computer simulations for medical students, physicians, and other

health professionals. In undergraduate medical education, some students use the programs as required or recommended course material; others use them on an ad hoc basis

for independent study. Programs are used by individuals and by student or student/

teacher groups to discuss alternate actions and likely outcomes at each point requiring

user input. To enhance the learning experience, most of the simulations have HELP or

CONSULTATION options to offer medical or program advice in addition to the feedback to a particular response.

In graduate medical education, practicing physicians use the simulations for private

self-assessment, for continuing medical education, and for certification and recertification requirements. Acceleration in the growth of medical knowledge, diversity of training programs, and growing State and specialty society requirements for continuing medical education activity have contributed to the need for new sources of continuing education credit.

The Harvard Department of Continuing Education has approved many of the MGH

programs for over 70 hours of Category I credit toward the Physician’s Recognition

Award of the American Medical Association (AMA). Also, CME approval has been

received by two specialty groups: The American Academy of Family Physicians (AAFP)

for elective credits and The American College of Emergency Physicians (ACEP) for

Category I credit. Table 15 lists the program names and shows the maximum number of

hours of continuing education credit for which each program can be used. Physicians

using the MGH programs can register for credits on-line at the time of program use. At

the end of the registration period, a certificate listing total hours earned is forwarded to
80 ● Computer Technology in Medical Education and Assessment

Table 15.—Massachusetts General Hospital: Computer-Assisted Continuing

Medical Education Program (1979)

Maximum hours approved by:

Program module AMAa AAFPb

ACEP C

Abdominal pain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Arterial blood gas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Cardiac arrhythmias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Cardiopulmonary resuscitation. . . . . . . . . . . . . . . . . . . . . . . . . . .

Coma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Cough and fever (Pediatrics) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Digitalis usage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Diabetic ketoacidosis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Fluid and electrolyte management . . . . . . . . . . . . . . . . . . . . . . . .

Gastrointestinal bleeding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hypertension diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hypertensive emergencies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hypertension treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Joint pain..... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Orthopaedic problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Respiratory distress—newborn. . . . . . . . . . . . . . . . . . . . . . . . . . .

Trauma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

3

3

10

10

5

5

—

2

3

5

3

10

5

3

3

3

5

3

3

10

10

5

5

—

2

3

5

3

10

5

3

3

3

5

2

4

6

14

6

—

3

2

5—

3

—

5

1 —

4

aAmerlcan Medical Association

bAmerican Academy of Family Physicians

cAmerican College of Emergency Physicians.

SOURCE Health Education Network information Packet, 1979

the participating physician. With a computer terminal and a telephone call (usually local)

to the network, a physician can have a personal, interactive method of education from

home, office, or hospital without leaving his/herpractice to attend acourse orlecture.

The programs have proven to be quite popular with a large number of medicaI

students and physicians. Users have expressed a very positive attitude to the computer

teaching experience as a valuable supplement to other educational formats. This unique

approach to medical education is judged as entertaining and productive. The continuing

survivaI of the network with no

given the programs by thousands

NETWORKS, CONSORTIA,

outside funding also attests to the favorable appraisaI

of users.

AND OTHER

DISSEMINATION MECHANISMS

Interinstitutional sharing of CBE resource materials is necessary if CBE is to continue

to be used in the health sciences. The impetus for resource sharing is provided by a variety of factors. For example, the high developmental costs of CBE materials make sharing

a necessity; no one institution can afford to develop all the courseware to meet its needs.

The costs of maintaining hardware are also high. It is not cost-effective or even possible

for faculty authors, who must ensure the ongoing relevance and accuracy of the materials, to perform maintenance functions or distribute lesson updates to each institution that

uses them. Via networking, the most scarce resource, the author, is effectively shared,

because all users are on-line and interact with the same copy of the CBE courseware.

Via “off-line” sharing, author resources also are shared, but updated and new

courseware must be added continually to the computer at each user site. Hence, “off-line”

sharing of CBE materials will depend on the availability of mechanisms to develop and

provide access to modified versions of each program. “On-line” sharing in contrast, is

subject to the costs of long distance communication. Another obstacle to sharing CBE is
the lack of cooperation among schools in developing and distributing learning resource

materials.

In this section, programs that utilize the three mechanisms for sharing CBE materials

are summarized: the Health Education Network, Inc., on-line use; the Computer-Assisted Testing Consortium (CATS), transfer of CBE programs off-line from one institution to another; and Milliken Communications Corporation (MCC), transfer of programs off-line to a single individual.

The Health Education Network

The Health Education Network, Inc., evolved from a federally supported and federally directed experimental network to a user-supported and user-directed operational

network (Wooster, 1976). In 1972, the Lister Hill National Center for Biomedical Communications of the National Libary of Medicine established an experimental network to:

1) assess the technical feasibility of interinstitutional sharing of CBE materials through

networking, and 2) determine whether CBE materials developed at one institution would

be accepted and used by others. Both the technical feasibility of networking and user acceptance of sharing CBE materials were established in 1974, after 18 months of network

operation. Federal support continued until May of 1975. Currently, the Health Education

Network enables institutions to share CBE materials at a fraction of the cost of developing and maintaining these materials individually.

The federally supported experimental network began as a result of a 1968 conference

sponsored by the Council of Academic Societies of the Association of American Medical

Colleges and the Lister Hill Center. Conference participants concluded that there was a

need for a national network and that there were many services that might be provided.

As a result, the Lister Hill Center convened a meeting in 1971 of potential hosts to discuss

providing programs for networking. The center thought it could obtain programs from

the hosts, put them on a central computer, and distribute them to users. Hosts were willing to share the programs, but they insisted that the programs reside on their own computers for continued courseware control and maintenance by the author team. Potential

hosts disagreed with the center’s demonstration plans. The center wanted to demonstrate

programs to a wide group of users by employing a small number of programs; the hosts,

on the other hand, wanted to work extensively with a small number of users. This issue

was resolved by devising two classifications of users, trial and operational.

In June 1972, MGH became the first hospital to offer materials on the Network. Two

months later, The OSU College of Medicine was officially connected to the Network, and

in January 1973, the University of Illinois Medical Center was joined to the Network.

Connection involved installation of a TYMNET minicomputer and development of software to interface with the host computers. MGH was an original test site for TYMNET

node connection to Digital Equipment Corporation hardware, and OSU was an original

test site for TYMNET node connection to IBM 360/370 series equipment. Given the

“stat e-of-the-art,” a period of some months was required to achieve smooth transmission

of technical messages and to establish a stable and reliable network connection for users.

The University of Illinois Medical Center ceased to be a host on the Network in May

1974. The Lister Hill Center made the decision to transfer a portion of that data base, the

CASE simulation materials, to the OSU College of Medicine.

Initially, the experimental network provided free service to institutions willing to

support their own computer terminal(s) and connection to the nearest TYMNEI” network
82 ● Computer Technology in Medical Education and Assessment

node. The center did not anticipate the high user acceptance which was, in fact, so great

that the center was forced to recover some of its costs from participating institutions. In

February 1974, user charges of $2.50 per connect hour were instituted. Network usage

continued to outreach budget allocations. Therefore, a plan evolved both for gradual increases in user charges and for beginning the transition from a federally supported network experiment to a user-supported network.

Less than a year after the first user fee announcement, the Lister Hill Center increased the usage fee to $5.00 per connect hour. Still, the fee covered less than one-third

of the total cost for Network operation. This circumstance is perceived as a major reason

for the decision and subsequent announcement by the Lister Hill Center in January 1975,

to withdraw support from the Network in May 1975. In May of 1975, the Lister Hill

Center officially terminated its support of the Network. The next day the operational,

user-supported Network began. By June 1976 the Network had become an incorporated,

tax-exempt organization.

The Health Education Network is managed by a board of directors elected by the

user communities of various hosts who coordinate their sharing under the Network umbrella. The Network is comprised of: 1) host institutions with well-developed healthrelated CBE materials on their own computers, 2) a communications carrier with a

nationwide network, 3) institutions that wish to access and pay for available CBE

courseware, and 4) a management structure to preserve and enhance the interinstitutional sharing (Tidball, 1978).

Currently, the Network’s hosts are: the OSU College of Medicine, MGH’s Laboratory of Computer Sciences, The University of Texas Health Sciences Center at Dallas,

and Washington University (St. Louis) Department of Surgery, and Milliken Communications Corporation (as joint hosts). Through these hosts, the Health Education Network

provides access to the largest library of health-related CBE materials in the world—over

600 interactive hours. The Network’s CBE materials are comprised of the cumulative

data bases of the several hosts.

Access to the library of programs is provided through TYMNET, a commercial telecommunications company that operates a nationwide network with approximately 165

connected sites. Using almost any computer terminal, the user need only to call the

nearest connected city on the Network to access any Network host.

To date, over 150 institutions have used the Network. The largest groups of users

have been medical schools (65 percent) and teaching hospitals (10 to 15 percent); other

categories of users include professional societies, group practices, allied health schools,

Federal agencies, industry, and individuals. Some 60 institutions are now members of the

Network. Cumulatively, the Network has provided over 250,000 interactive user sessions, with on-line usage ranging from 600 to 2,000 hours per month. Composite, annual

use of the Network is presented in table 16.

Member institutions use the Network for a variety of purposes. Many use the Network primarily for demonstrations, faculty education, and as an optional learning resource for students. Several institutions have integrated the Network’s CBE materials

into their teaching programs; some have required CBE courses that are offered by the

Network, especially anatomy and physiology. The University of Washington modified

the OSU independent study tutorial programs to meet the needs of their own independent

study program. Before a local system was available, the medical students in Seattle used

CBE programs based in Columbus, Ohio, to pursue their medical studies.
Ch. 5–Development and Dissmination of Computer-Based Education ● 83

Table 16.—Health Education Network

(annual hours of use)*

Year Hours

1972 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1,340

1973 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18,140

1974 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19,390

1975 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15,600

1976 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14,800

1977 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12,820

1978 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12,940

“Source of CBE materials only from Ohio State Unversity and Massachusetts General Hospital

The Network has supported computer-based continuing education activities by such

groups as the Connecticut and Ohio Academies of Family Physicians, and the American

Academy of Orthopedic Surgeons. Without the Network, many such activities could not

have been undertaken.

The current cost to the user of the Health Education Network ranges from $4 to $10

per connect hour depending on the time of day (prime time v. nonprime time) and the

level of user commitment. The higher the commitment in usage hours, the lower the unit

cost. The Network supports a wide variety of terminals that can be purchased for less

than $2,000 or rented for $50 per month. The user institutions provide their own computer terminals and phone connections to the nearest Network node. Hourly user fees include local host computer costs, costs for personnel to support and maintain the hardware and software systems, costs for personnel to provide support to users, and communication costs. Charges for development and evaluation of CBE materials and for

assisting users in curricular incorporation of the CBE materials are not included. These

costs are generally assumed by the hosts, which have developed the CBE materials for

their own use.

The Network has stimulated the development of educational computing in the

health professions and increased awareness of the potential of CBE. Faculty, student, and

staff users have had an opportunity to use a new learning resource without a large initial

investment in hardware, software, or personnel. They have had access to recognized

materials and to the work of accomplished CBE authors. FacuIty aIso have had the opportunity to develop new lessons or alter existing lessons from their own terminals. Finally, they have had access to computer and educational consultants at the host sites.

The quality of CBE materials has improved as a result of national exposure. Widespread peer review, although still an emergent phenomenon, has accelerated review and

refinement of programs. Remote authoring has further refined course development and

added new materials to the CBE library. Reliable, multi-institutional access to, and use

of, lessons has fostered developments in CBE materials analogous to publication in the

print media.

Via the Network, the hosts have a relatively trouble-free mechanism for sharing

their CBE materials with others. Sharing materials has forced other changes that are of

long-term benefit to hosts and users, including: the development of simplified procedures

for user access; the implementation of standards for courseware documentation; improvements in remote authoring; and the refinement of user services to address the problems of consultations on the use of CBE courseware. In addition, reporting systems and

program development procedures were generalized, extensive management reporting

systems were installed, and methods of billing and contracts management were implemented.
84 ● Computer Technology in Medical Education and Assessment

The Computer-Assisted Teaching Systems Consortium

The Computer-Assisted Teaching Systems (CATS) Consortium is a CBE sharing

mechanism whereby participating medical schools cooperate in the development of new

approaches to medical education, exchange educational and CBE materials, and generally encourage the development of innovative teaching in pharmacology (Doull and

Walaszek, 1978). The system primarily uses computer assisted and managed testing

(CAT); however, the system also uses some computer-assisted instruction (CAI).

The Health Education Network, Inc. shares materials “on-line” via network link of

the user to the host computer. CATS, in contrast, shares materials “off-line” by sending

computer tapes, documentation, and other materials to the user for operation on his/her

own computer. This choice of sharing mode was consciously planned to enable users to

control the program’s uses and modifications and to minimize user costs by avoiding expensive communication links. Although developed on IBM computers, the system may

be used on several types of large computers and on minicomputers.

Similar computerized test-item banks exist in several basic science areas, most

notably pathology (Group for Research in Pathology Education, 1974) and physiology

(Dennis, 1978). The pathology materials are maintained in Iowa and shared through a

consortium similar to CATS. The physiology materials are maintained at OSU and are

shared through the Health Education Network and CAIREN,

In 1970, the Department of Pharmacology began developing a system that would

both teach pharmacology more effectively to medical and nursing students and attempt

to meet the needs of medical technicians, hearing and speech therapists, graduate

students, pharmacists, house staff, and other health professional groups. Pharmacology

course material was subdivided into a number of modules or units, and CAI and computer-managed instruction /computer-assisted testing (CMI/CAT) materials were developed

to help assess student competence in each of these modules or units.

CATS was used on a pilot basis in 1971 and 1972 to teach pharmacology to medical

students and was fully implemented and extended to other teaching areas in pharmacology in 1973. The CATS Consortium was formed in 1974 and includes over 5 0

American and 12 European medical schools. They all use the CATS CMI/CAT materials

developed at the University of Kansas Medical Center’s Department of Pharmacology. A

much smaller number use the CAI materials.

The CMI/CAT portion of CATS contains over 25,000 test items, along with software, to create and maintain new questions, to generate tests from the test-item bank,

and to grade and post scores from the exams. Only objective type, versus open-ended or

essay type, questions are included in the CMI/CAT item bank. The questions come from

recent exams given by the Kansas pharmacology faculty, various Consortium members,

and other pharmacologic colleagues of the Kansas faculty. New questions are screened

for appropriateness and acceptability of question-type format. Each question is classified

and tagged so that it can be automatically retrieved by content and difficulty when a test

is generated. Questions are reviewed for currency and correctness when they are selected

from the file for inclusion in the exams, rather than when they are added to the file.

Faculty responsible for teaching each unit in pharmacology decide how many exams

they wish to prepare and how many questions from each category they wish to include in

their exams. The faculty can personally select test items from a large set produced by the

computer, or they can use statistical information or a random number generator to produce a test for their review.
Ch. 5—Development and Dissemination of Computer-Based Education ● 85

At Kansas, exams containing 100 questions in each of the 5 units of the pharmacology course are offered weekly throughout the 20-week teaching period. Each student

who registers for the independent study program is given a packet of prepunched cards

containing his/her name, number, mail box number, and course number and section.

Whenever a student wishes to take an exam, he/she submits one of the cards, and an optical scanning answer sheet with individual information is generated. The sheets are automatically scored and reports for the faculty and the department are prepared. Item analysis information is added to the question so that cumulative data can be kept on the difficulty and use, for example, of each question.

The CAI portion of CATS serves primarily as an enrichment or adjunct to the medical student pharmacology course. Although only a few optional CAI programs were

available in 1972, 98 percent of the students in a pilot program used all of the programs,

and virtually all were in favor of expanding this approach. A significant increase was

demonstrated in student performance in both of the pharmacology units in which CAI

material was available (Norton et al., 1972). Currently over 120 simulations and

tutorials are used, for an average of 20 hours by each of the 160 medical students at Kansas. These CAI programs are available to CATS Consortium members, although most

use only the CMI/CAT portion of CATS. This is due in part to the difficulties in translating the CAI programs to other computer languages and to the costs of CAI operation as

compared to CMI/CAT operation (Doull and Walaszek, 1978).

Consortium growth indicates that faculty are willing to use materials developed by

others if they have some editorial control over the materials. Also, the availability of a

body of materials that covers a defined subject area facilitates incorporation of the

materials into ongoing educational programs rather than into optional, adjunct programs. Another important aspect of the CATS Consortium is the membership requirements; each school must contribute questions to the item bank and its data must be made

available to others for revision of questions and for peer review purposes. This facilitates

sharing because each school considers itself an active, rather than a passive, participant.

Milliken Communications Corporation

Milliken Communications Corporation (MCC), a for-profit publisher of CBE materials, is a wholly owned subsidiary of Milliken Publishing Company formed specifically

to develop and market CBE continuing education materials to physicians. Its courseware

is entirely oriented to continuing education provided through the use of microprocessors

and the home computing market.

In 1975 the Department of Surgery, Washington University School of Medicine,

began the experimental development of a CBE system to provide students with individualized instruction in general surgery and to determine the feasibility of CAI as a means

of providing postgraduate continuing medical education (Halverson and Ballinger,

1978). The work was initially supported by the Institute of Medical Education and

Research, St. Louis, Me., and by Milliken. CBE courseware was authored by experts

from across the country, and the Department of Surgery coordinated the effort. A series

of computer programs called “drivers” were prepared so that authors could write and

edit CBE seminars without learning computer programing. The standard MUMPS

language and PDP computers were used for the development of the CBE programs

(Bowie and Barnett, 1976).

The seminars were initially used by medical students as a supplement to the usual

third-year general surgery curriculum. Although use of the CBE materials was not com51-541 0 - 79 - 7
86 ● Computer Technology in Medical Education and Assessment

pulsory, about two-thirds of the students took advantage of these materials. Eighty-five

percent of the 59 students felt the experience was beneficial and wanted more opportunities to take CBE seminars (Halverson and Ballinger, 1978).

As an outgrowth of the success of the Department of Surgery’s efforts and with its

help, Milliken Publishing Company formed MCC to adapt the undergraduate seminars

to continuing education courses. Milliken, in conjunction with Washington University, is

now developing continuing education series in internal medicine, primary care, and psychiatry. Experts author the materials, and MCC provides programing and computer support. An editorial board consisting of Washington University and MCC personnel

review all materials. MCC pays a varying honorarium, but no royalty, to its authors. To

date approximately 70 authors from 30 medical schools have participated (Milliken

Communications Corporation, 1979).

Milliken’s CBE seminars are designed to simulate a conversation between the physician and the author; the mode is primarily tutorial. The computer presents a concept or

basic information and asks the physician a question that requires recall, inference, or

judgment. The physician responds by typing in his/her answer; the computer responds

using the preprogrammed responses provided by the author. Table 17 shows the seminars

currently being marketed by MCC. Over 50 interactive hours are now available, with

more in- development. The ‘number of Category 1 Continuing Education Credit hours

which can be earned is noted following each seminar title. To receive credit, the physician places the computer-generated completion code on a completion card and returns

the card to Milliken.

Table 17.—Continuing Education Seminars Currently Marketed by

Milliken Communications Corporation

AMA category 1 AMA category 1

Surgery credit hours Internal medicine credit hours

Surgical infection. . . . . . . . . . . . . . . . . . . . . . . . . 2.0 ● Management of cardiac arrhythmias-1 . . . . . . . 1.5

Diagnosis and treatment of abdominal ● Management of cardiac arrhythmias-n . . . . . . 1.5

injuries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 ● Management of cardiac arrhythmias-ill. . . . . . 1.0

Immunology for surgeons. . . . . . . . . . . . . . . . . . 1.5 ● Athletic injuries (jogging and tennis). . . . . . . . 1.5

● Surgical treatment of peptic ulcer disease . . . 1.5 ● Clinical approach to patients with

● Gastrointestinal inflammatory disease chronic obstructive lung disease. . . . . . . . . . 1.5

(Crohn’s disease)-1 . . . . . . . . . . . . . . . . . . . . . . 1.5 ● Asthma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5

Acute and chronic pancreatitis. . . . . . . . . . . . . . 1.5 ● Chronic renal failure. . . . . . . . . . . . . . . . . . . . . . 1.5

Surgical parathyroid disease . . . . . . . . . . . . . . . 1.0 ● Gonorrhea. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5

Gastroesophageal reflux. . . . . . . . . . . . . . . . . . . 1.5 ● Hodgkin’s disease . . . . . . . . . . . . . . . . . . . . . . . 1.5

● Clinical management of advanced ● Thyroid dysfunction. . . . . . . . . . . . . . . . . . . . . . 1.5

breast disease . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 *Understanding nutrition . . . . . . . . . . . . . . . . . . 1.5

● Multiple endocrine neoplasis . . . . . . . . . . . . . . 1.5 ● Evaluation of cardiac sounds . . . . . . . . . . . . . . 1.5

Malignant melanoma 1.5

Peripheral arterial disease-1 . . . . . . . . . . . . . . . . 1.5

Peripheral arterial disease-n . . . . . . . . . . . . . . . 1.5

● Colorectal polyposis . . . . . . . . . . . . . . . . . . . . . 1.0

Cutaneous thermal burns . . . . . . . . . . . . . . . . . . 1.5

Surgical nutrition . . . . . . . . . . . . . . . . . . . . . . . . . 1.5

Colorectal cancer. . . . . . . . . . . . . . . . . . . . . . . . . 1.5

● Gastrointestinal bleeding . . . . . . . . . . . . . . . . . 1.5

● Gastrointestinal inflammatory disease-n . . . . 1.5

● Shock. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5

● Cancer of the thyroid . . . . . . . . . . . . . . . . . . . . . 1.5

● Trauma. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5

● Portal hypertension . . . . . . . . . . . . . . . . . . . . . . 1.0

Extrahepatic biliary surgery . . . . . . . . . . . . . . . . 1.5

● Thrombosis, thrombolysis and

thrombophlebitis. . . . . . . . . . . . . . . . . . . . . . . 1.5

“Also qualifies for primary care.

SOURCE: Milliken Communications Corporation, St. Louis, Mo,, 1979,
Ch.. 5—Development and Dissemination of Computer-Based Education ● 87

Seminars are programed and stored on courseware diskettes that provide approximately 45 minutes of interaction each and can be used hundreds of times. The user can

start, stop, exit, and reenter the seminar at will. To use the CBE seminars, the physician

must purchase or lease an APPLE microprocessor (the same model available in hobby or

home computing stores), and the diskettes which contain the CBE materials. MCC seminars are also available for demonstration through the Health Education Network, but,

thus far, usage has been limited to less than 200 hours (Milliken Communications Corporation, 1979).

MCC both sells and leases all materials and equipment. The purchase price of the

total system ranges from $2,525 to $2,850, depending on the number of diskettes purchased; the lease price ranges from $2OO to $3OO per month, depending on the length of

the lease; and individual diskettes cost $2o to $22, depending on the number purchased.
APPENDIX
Appendix I

EXAMPLES OF SIMULATIONS OF

THE PATIENT-PHYSICIAN ENCOUNTER

INTRODUCTION

This appendix contains abbreviated examples of different methods of simulating the

patient-physician encounter. The paper-and-pencil patient management problem (PMP)

presents a brief description of a patient, then offers the user a series of choices. The user

rubs out the block accompanying his/her choice(s) to obtain information, which is then

used to help in choosing among subsequent choices.

A computerized patient management problem (CPMP) is designed similarly to a

paper-and-pencil PMP. The advantages of the CPMP include: 1) more complex branching of the decisionmaking process; 2) immediate feedback to the user; 3) review of previously selected items; 4) easier access to adjunct use of audio/visual materials; and 5) better timing, control, monitoring, and scoring of the exam.

Simulations in approximately 30 subject areas are contained in the data base of the

Massachusetts General Hospital. The user can select whatever content material is of particular interest and a time and location that are most convenient. In addition to the feedback to a particular response, most simulations offer medical or program advice through

HELP or CONSULTATION, which provide probabilities for certain diagnostic or treatment selections based on historical data bases.

In the simulation used in the Computer Based Examination (CBX) Project, the laboratory tests ordered by the user, the procedures or consultations which are initiated,

and the amount and route of drug administration interact with the patient’s disease in a

realistic time sequence. By simulating the effect on the patient of each action taken by the

user and adjusting the patient’s status accordingly, the computer model provides a much

more life-like, dynamic simulation of the patient-physician encounter than does the

CPMP. Through a sampling of the interaction, assessment of a number of patientoriented factors can be accomplished, such as costs of the workup, risk (pain, complications, mortality), time taken to initiate corrective therapy, and amount of time the patient is kept in the hospital (if hospitalized). In addition, a step-by-step evaluation of the

user’s actions in test and therapy ordering can be done.

In the computer-assisted simulation of the clinical patient encounter (CASE), the

patient-physician encounter is unprompted and undirected. After a brief introduction to

the patient, the user receives no further clues. He/she must determine what information

is important. The user elicits whatever history, physical examination, and laboratory

data are needed for the diagnosis and management of the “patient’s” problem.

91
PART 1

A SEGMENT OF A PAPER AND PENCIL

PATIENT MANAGEMENT PROBLEM (PMP)

(Note: In this example, the examinee is presented with a short description of

the problem and given a series of choices. Here, he has chosen number 3, followed by number 9. These choices are reflected by “rubbing out” the appropriate

number.)

A 42-year-old man with known diabetes is brought to the hospital in a

comatose state. There is no obvious evidence of trauma. There is Kussmaul

breathing and the breath has an acetone odor. The skin is dry. The eyeballs

are soft to palpation. Examination of the heart and lungs shows nothing abnormal except for labored respiration and rapid, regular heart rate of 120

per minute. The abdomen is soft. There is no evidence of enlarged liver or

spleen or abnormal masses. Deep tendon reflexes are somewhat hypoactive bilaterally. The rectal temperature is 36.7 C (98.0 F). Blood pressure is

100/70 mm Hg.

You would immediately

1. Order serum calcium determination

2. Order serum bicarbonate determination

3. Measure venous pressure

4. Order urinalysis (catheterized specimen)

5. Perform lumbar puncture

6. Order blood glucose determination

You would now

7. Administer digitalis

8. Administer morphine

9. Administer insulin

10. Administer coramine

11. Start intravenous infusion with normal

saline

7. xxxxxxxxx x

8. xxxxxxxxx x

9. Ordered

10. xxxxxxx

11. xxxxxxx

SOURCE: Skakun et al., 1978
PART 2

A SEGMENT OF A COMPUTERIZED

PATIENT MANAGEMENT PROBLEM (CPMP) WITH FEEDBACK

(Note: In this example the computer has been asking a series of questions

on a particular pediatric problem with each question followed by a list of

choices. Here, the user has chosen number 101, “Type of feeding,” and the

computer has answered, "Breast feel...Taken well." Note that this could have

been done manually. By pressing the space bar, the user is automatically

branched to the appropriate next series of questions, based upon his “101”

response.)

Computer: What points in the history obtained from the mother or from

the nursery notes would be particularly pertinent:

101 Type of feeding

102 Mother and baby blood groups

103 Health of siblings

104 Maternal use of drugs or medication

105 Family history of blood or liver diseas e

106 Any vomiting?

107 Increased sweating

108 Color and nature of stools

109 Color of urine

User: 101

Computer: Breast fed ...Taken well

Press Space Bar to Continue

Source: Skakun et al., 1978

51-541 0 - 79 - 8
PART 3

EXAMPLES OF SIMULATIONS FROM THE

MASSACHUSETTS GENERAL HOSPITAL

(Note: Four different types of simulations are illustrated here: (1) a static

patient model, (2) a dynamic patient model, (3) a physiologic model, and (4) a

disease process model. User’s entries are underlined. The numbered entries

correspond to the inquiries that are reproduced by the computer, with the

corresponding answers, immediately following the number inquiries. For example,

in the first illustration, the typed entry "#l" inquires as to the "Characte r

of Pain."

f The first illustration also includes an estimate of disease probabilities provided by the HELP program.)

Illustration Number One:

A 45-YEAR OLD WHITE

ROOM BY HIS ANXIOUS

WHICH STARTED ABOUT

(At this point, the

ITEM #l:

ITEM #8:

ITEM #18:

ITEM #4:

CHARACTER

ON FIRE.

VOMITING:

WIFE

Static Patient Model.

MALE USED-CAR SALESMAN IS BROUGHT TO THE EMERGENCY

BECAUSE HE COMPLAINS OF MODERATE EPIGASTRIC PAIN

A WEEK AGO AFTER A NIGHT OUT WITH THE BOYS PLAYING POKER.

user begins his/her interrogation of the “patient.”)

OF PAIN: IT’S A HOT, BURNING PAIN, LIKE MY STOMACH IS

I OFTEN VOMIT, AND THEN THE PAIN IS SOMETIMES BETTER.

DOES EATING A MEAL AFFECT YOUR PAIN: I FEEL BEST RIGHT AFTER A MEAL,

ESPECIALLY WHEN I HAVE ICE CREAM FOR DESSERT.

WHEN DID YOU FIRST HAVE THIS PAIN: I’VE BEEN HAVING PAINS LIKE THIS

FOR A NUMBER OF YEARS. IT SEEMED TO START WHEN I WAS IN KOREA. I T

GETS PARTICULARLY BAD WHEN THE NEW MODEL CARS COME OUT IN THE FALL.

(If the user requests the HELP function at this point, he/she receives the

following information.)

ITEM #HELP

***AT THIS POINT***

ESTIMATED DISEASE PROBABILITIES

(3) PEPTIC ULCER (GASTRO/DUODENA.L) 0.648

(6) ACUTE GASTRITIS 0.087

(7) ACUTE GASTROENTERITIS (VIRAL OR BACTERIAL) 0.063

WOULD YOU LIKE SUGGESTIONS FOR FURTHER WORKUP? YES
Appendix I ● 95

COMPUTER’ S SUGGESTED TESTS

(4) WHEN DID YOU FIRST GET THIS PAIN

(3) HOW LONG DO THE PAINS LAST

(33) TENDERNESS TO PALPATION

(15) ANTACID RELIEF

WHAT IS YOUR IMPRESSION? GIVEN THE DATA YOU HAVE THUS FAR, WHAT DIAGNOSIS

DO YOU THINK SHOULD BE MOST SERIOUSLY CONSIDERED?

DIAGNOSIS: ULCER

BE MORE SPECIFIC

DIAGNOSIS: ULCER OF DUDENUM

WHEN YOU TYPED ‘DUDENUM’, DID YOU MEAN ‘DUODENAL’? YES

(At this point, the computer has interrupted to request the user to state his/her

leading diagnostic impression. Note that the computer can recognize non-specific

diagnoses and simple mis-spellings.)

YOU MAY NOW COLLECT MORE INFORMATION IF YOU CHOOSE TO DO SO

ITEM #7: HEARTBURN: THAT'S WHAT I’VE BEEN TELLING YOU, DOC. I HAVE TERRIBLE

HEARTBURN, PARTICULARLY WHEN MY WIFE NAGS ME.

(At any time, the user may terminate the interaction by indicating that he/she

wishes to make a final diagnosis; i.e., by entering the letter "D.” )

ITEM#D:

DIAGNOSIS: PEPTIC ULCER OF DUODENUM

AN EXCELLENT CHOICE: BUT PERHAPS A DIAGNOSIS IS SLIGHTLY PREMATURE.

WOULD YOU ADMIT THIS PATIENT TO THE HOSPITAL? YES

WOULD YOU SCHEDULE A LAPAROTOMY FOR THIS PATIENT? NO
96 ● Computer Technology in Medical Education and Assessment

Illustration Number Two: Dynamic Patient Model.

THE PATIENT IS A 41-YEAR-OLD WOMAN WHO WAS ADMITTED 2 WEEKS AGO WITH

SEVERE BIVENTRICULAR FAILURE. SHE HAD BEEN SLOWLY IMPROVING WITH BEDREST

AND DIURETICS BUT NOW, ACCORDING TO THE AIDE, “LOOKS TERRIBLE. ” WHAT DO

YOU DO FIRST?

1. TAKE AN EKG

2. START ON IV

3. CHECK MAJOR VESSELS FOR PULSES, SEE IF PATIENT IS BREATHING

4. GIVE A SHOT OF INTRACARDIAC EPINEPHRINE

5. IMMEDIATE ELECTRICAL CARDIOVERSION

6. GIVE A SHARP BLOW TO THE PRECORDIUM

#3

THERE ARE NO PALPABLE CAROTID OR FEMORAL PULSES; THE PATIENT IS APNEIC AND

DOES NOT RESPOND TO YOUR SHOUT.

WHAT IS YOUR NEXT MOVE?

1. START CHEST COMPRESSION

2. GIVE A SHARP BLOW TO THE PRECORDIUM WITH YOUR FIST

3. QUICKLY GO OUT AND YELL FOR HELP

4. TAKE AN EKG

5. TILT BACK THE HEAD AND GIVE FOUR BREATHS

#2

THE AMERICAN HEART ASSOCIATION DOES NOT RECOMMEND USE OF THE PRECORDIAL THUMP

IN UN-WITNESSED ARRESTS. THE DANGER IS THAT OF PRODUCING VENTRICULAR FIBRILLATION

IN THE ANOXIC HEART. THERE ARE REPORTS OF SUCCESSFUL USE OF “THUMP-VERSION” IN

SITUATIONS OTHER THAN WITNESSED ARRESTS, BUT THESE ARE STILL LARGELY ANECDOTAL.

THE OFFICIAL POLICY IS THAT PRE-CORDIAL THUMP SHOULD ONLY BE USED WITHIN THE

FIRST MINUTE OF AN ARREST. YOU TILT BACK THE HEAD AND GIVE FOUR QUICK BREATHS.

THE MOST IMPORTANT STEP IS TO ENSURE AN ADEQUATE AIRWAY AND RESPIRATIONS. THIS

ALONE IS SOMETIMES SUFFICIENT TO RESUSCITATE A PATIENT. NO RESPONSE TO PROVIDING

AN AIRWAY AND GIVING SEVERAL QUICK BREATHS.

WHAT DO YOU DO NOW?

1. START CHEST COMPRESSION

2 . GIVE A SHARP BLOW TO THE PRECORDIUM

3. YELL FOR HELP WHILE STARTING EXTERNAL CHEST COMPRESSION

4 . TAKE AN EKG

#3

GOOD. YOU WISH TO INITIATE CARDIOPULMONARY RESUSCITATION AS QUICKLY AS POSSIBLE,

BUT MUST NOT OVERLOOK THE FACT THAT IT IS MORE EFFECTIVELY PERFORMED BY TWO PEOPLE

THAN ONE. IN THE HOSPITAL, TRAINED HELP SHOULD BE JUST A SHOUT AWAY. THE PATIENT

REMAINS PULSELESS. ONE OF THE FLOOR NURSES HAS JOINED YOU.

WHAT SHOULD YOU DO?
Appendix I ● 97

1. INTUBATE THE PATIENT

2. TAKE AN EKG

3. START VENTILATION AND CHEST COMPRESSION

4. GIVE THE PATIENT AN INJECTION OF INTRACARDIAC EPINEPHRINE

#3

The interaction continues to completion.)
Illustration Number Three: Physiologic Model.

(Note : In this illustration, the patient has a tube draining to the outside

from the gallbladder area, probably following gallbladder surgery. The patient

is also on anticoagulant therapy, and the purpose of this exercise is to illustrate

the effect on clotting time from this related surgery.)

INITIAL CONDITIONS: PATIENT IS DRAINING MATERIAL FROM A T-TUBE.

CONTROL IS 12.5 SECONDS (This refers to the normal clotting time, and the user

is attempting to slow the clotting time with the anticoagulant drug, warfarin.)

DAY 1: PATIENT’S CLOTTING TIME IS 12.5 SECONDS.

WARFARIN DOSE: 20–MG

DAY 2: PATIENT’S CLOTTING TIME IS 14.5 SECONDS.

WARFARIN DOSE: 7.5 MG

PATIENT NOW ON DISULFIRAM

DAY 3: PATIENT’S CLOTTING TIME IS 22.7 SECONDS.

WARFARIN DOSE: HELP

THE FOLLOWING POINTS MAY BE HELPFUL:

DISULFIRAM POTENTIATES WARFARIN. IT HAS NOT BEEN WELL-STUDIED BUT IS

PROBABLY DUE TO INHIBITION OF HEPATIC METABOLISM. A PATIENT WITH A

BILIARY FISTULA OR T-TUBE DRAINAGE (ESPECIALLY WITH ASSOCIATED PANCREATIC

DISEASE) IS MORE SENSITIVE TO WARFARIN BECAUSE OF A DECREASED VITAMIN K

ABSORPTION.

WARFARIN DOSE: 7.5 MG

DAY 4: PATIENT’S CLOTTING TIME IS 31 SECONDS.

WARFARIN DOSE: 5 MG

DAY 5: PATIENT’S CLOTTING TIME IS 44.3 SECONDS.

***HEMORRHAGE HAS OCCURRED BECAUSE OF EXCESSIVE ANTICOAGULATION***

...WARFARIN DISCONTINUED.. .VITAMIN K GIVEN.

***NEW PATIENT** *

INITIAL CONDITIONS. PATIENT RECEIVING ADRENOCORTICOSTEROIDS.

CONTROL IS 13.1 SECONDS.

DAY 1: PATIENT’S CLOTTING TIME IS 13.1 SECONDS.

WARFARIN DOSE: HELP

DO YOU HAVE A GENERAL IDEA OF DOSES USED? YES
THE FOLLOWING POINTS MAY BE HELPFUL:

PATIENTS ON ADRENOCORTICOSTEROIDS ARE REPORTED TO REQUIRE A LARGER DOSE

OF WARFARIN. THE EFFECT IS USUALLY NOT LARGE. THE MECHANISM IS POORLY

UNDERSTOOD BUT MAY BE RELATED TO THE WELL-KNOWN BUT LITTLE UNDERSTOOD

TENDENCY OF THE ADRENOCORTICAL STEROIDS TO PRODUCE A ‘HYPERCOAGULABLE

STATE’ BY INCREASING CLOTTING FACTOR SYNTHESIS.

WARFARIN DOSE: 35 MG

DAY 2: PATIENT’S CLOTTING TIME IS 15.4 SECONDS.

WARFARIN DOSE: O MG

DAY 3: PATIENT’S CLOTTING TIME IS 16.7 SECONDS.

WARFARIN DOSE: 10 MG

(The interaction can be terminated here or subsequently.)
100• ● Computer Technology in Medical Education and Assessment

Illustration Number Four: Disease Process Model.

THE PATIENT IS A 44 YEAR OLD FEMALE WHO HAS BEEN REFERRED

HOSPITAL FOR TREATMENT OF DIABETIC ACIDOSIS. THEY HAD NO

IS NOT SURE ABOUT HER INSULIN DOSE.

PHYSICAL EXAM: MENTAL STATUS

FROM A COMMUNITY

BEDS. HER FAMILY

WHEN YOU TYPED ‘MENTAL STATUS,’ DID YOU MEAN ‘MENTAL STATE’?

YES OBTUNDED

PHYSICAL EXAM: ORTHOSTATIC VITAL SIGNS. BLOOD PRESSURE: 92/58 PULSE: 103

AFTER SITTING THE PATIENT UP APICAL PULSE RISES BY 23, SYSTOLIC BLOOD PRESSURE

FALLS BY 32

PHYSICAL EXAM: SKIN. COLOR NORMAL. TURGOR VERY POOR.

PHYSICAL EXAM: (The blank tells the computer to move on to

TEST: BLOOD SUGAR. SENT TO THE LAB.

TEST: URINALYSIS. SPECIFIC GRAVITY: 1.040; SUGAR: 4+; ACETONE:

MICRO: 10-20 WHITE BLOOD CELLS, FEW BACTERIA

TEST: ELECTROLYTES. SENT TO LAB.

TEST: URINE CULTURE. SENT TO LAB.

the next section.)

LARGE:

TEST: .

THERAPY ORDERS FOR NEXT HOUR.

Rx:

Rx:

Rx:

Rx:

ONE LITER NORMAL SALINE.

CZ INSULIN 50 UNITS INTRAVENOUS.

CZ INSULIN 50 UNITS SUBCUTANEOUS.

KCL 20 MEQ IN THE INTRAVENOUS SOLUTION.

Rx: ●

***ENDING THERAPY FOR TIME O***

ONE HOUR WILL NOW PASS.

LAB TEST RESULTS FROM PREVIOUS HOUR:

BLOOD SUGAR: 567 MG %.

NA: 137 MEQ/L: K: 3.1 MEQ/L: C02: 8 MEQ/L.

PHYSICAL EXAM: VITAL SIGNS. BLOOD PRESSURE: 95/57; PULSE: 102; T: 99 PO; R: 31.

PHYSICAL EXAM: ABDOMEN. BOWEL SOUNDS PRESENT. NO ENLARGED ORGANS.

PHYSICAL EXAM: .

TEST: EKG. PROMINENT U WAVES.

TEST: ELECTOLYTES. SENT TO LAB.

TEST: BLOOD SUGAR. SENT TO LAB.

TEST: BLOOD GAS ANALYSIS. P02: 92; PC02: 23; PH: 7.08 .

TEST: .

THERAPY ORDERS FOR NEXT HOUR.

Rx: HELP .

DO YOU WANT l-MEDICAL ADVICE OR 2-HELP WITH THE MECHANICS OF USING THE COMPUTER? 1
Appendix ● 101

YOU USED AS INITIAL THERAPY 50 UNITS CZ INSULIN AND 50 UNITS SUBCUTANEOUS CZ

INSULIN . YOUR INITIAL INSULIN DOSE WAS ADEQUATE BUT THE BLOOD SUGAR IS STILL

HIGH. CONTINUE WITH ANOTHER 100 UNITS INTRAVENOUS CZ INSULIN. THE AVERAGE

PATIENT IN KETOACIDOSIS REQUIRES 500 UNITS OF INSULIN (TOTAL), WITH A RANGE

OF 100-1500 IN LARGE SERIES!

YOU GAVE 1 LITER INTRAVENOUSLY. THIS IS REALLY ADEQUATE INITIAL HYDRATION.

YOU WILL FIND THAT PATIENTS IN KETOACIDOSIS REQUIRE LARGE VOLUMES OF FLUID --

AVERAGING 6 - 8 LITERS, AND SOMETIMES OVER 15 LITERS.

BICARBONATE IS PROBABLY NOT NEEDED IN MOST PATIENTS. BECAUSE C02 DIFFUSES

QUICKLY OUT OF THE CEREBROSPINAL FLUID AND HC03 LESS QUICKLY IN, YOU MAY GET

A PARADOXICAL CEREBROSPINAL FLUID ACIDOSIS IF YOU CORRECT THE SYSTEMIC ACIDOSIS

TOO QUICKLY WITH BICARBONATE.

YOU GAVE 20 MEQ KCL. APPROPRIATE TO START. WITH THE SERUM K FALLING, NOW IS

THE TIME TO START ADDING KCL TO EACH BOTTLE. I SUGGEST 60 MEQ OVER THE NEXT

HOUR.

Rx: 2 LITERS HALF NORMAL SALINE.

RX: 60 MEQ KCL.

Rx: 100 UNITS CZ INSULIN INTRAVENOUSLY.

Rx: .

***ENDING THERAPY FOR TIME 1 HOUR***

ONE HOUR WILL NOW PASS.

(Several simulated hours later.)

YOU SEEM TO HAVE THINGS WELL IN HAND. THE

BLOOD SUGAR IS 90 AND THE SERUM ACETONE IS

AND HAS NORMAL VITAL SIGNS.

LET US STOP NOW. YOU MAY BEGIN AGAIN WITH

PH IS 7.33, THE K IS 3.57, THE

NEGATIVE. THE PATIENT IS ALERT

ANOTHER PATIENT IF YOU WISH.
102 ● Computer Technology in Medical Education and Assessment

PART 4

A SIMULATED PATIENT - PHYSICIAN ENCOUNTER

IN THE

COMPUTER-BASED EXAMINATION (CBX) PROJECT

(Note: In this interaction, the underlined items indicate the user inputs;

all other materials are generated by the computer.)

A 58 YEAR OLD WHITE FEMALE COMES TO YOUR OFFICE REQUESTING A PHYSICAL EXAMINATION

REQUEST? HISTORY RESULTS DUE TUESDAY AT 8:12 A.M.

  ORDERED ** **

HISTORY REQUIRES 0.2 HOURS

REQUEST? RESULT

RESULTS OF HISTORY

A 58 YEAR OLD WELL DRESSED WHITE WOMAN ENTERS YOUR OFFICE. SHE IS IN

NO APPARENT ACUTE DISTRESS AND INDICATES THAT SHE IS SEEKING MEDICAL ASSISTANCE

BECAUSE HER “HUSBAND IS WORRIED THAT SHE MAY HAVE HAD A HEART ATTACK THIS

MORNING.”

SHE INDICATES THAT SHE DOES NOT INDULGE IN MUCH PHYSICAL EXERCISE BUT

DID NOTE THAT TWO DAYS AGO WHEN SHE WENT JOGGING WITH FRIENDS FOR THE FIRST

TIME IN QUITE A FEW YEARS, SHE DEVELOPED A DULL PAIN IN HER CHEST WHICH

WENT AWAY AFTER 3 MINUTES OF REST.

THIS MORNING SHE ATTEMPTED TO PLAY TENNIS WITH FRIENDS FOR THE FIRST

TIME IN MANY YEARS AND NOTED A SHARP CHEST PAIN LOCATED IN THE MIDSTERNAL

LINE HALF WAY BETWEEN THE UMBILICUS AND THE TOP OF THE STERNUM. THIS PAIN

PERSISTED FOR TEN MINUTES AND WAS GREATLY AGGRAVATED BY ANY MOTION OF THE

ARMS. SHE DENIES RADIATION DOWN EITHER ARMOR EITHER LEG. SHE NOTED A

SLIGHT SHORTNESS OF BREATH WHICH DISAPPEARED AS THE PAIN SUBSIDED. SHE DID

NOT RESUME HER TENNIS GAME AND A DULL PAIN STILL PERSISTS OVER THE ANTERIOR

CHEST.

THE PATIENT HAD BEEN IN GOOD HEALTH ALL HER LIFE. SHE RECENTLY RETIRED

AFTER 29 YEARS IN THE POST OFFICE AND WAS GIVEN A CLEAN BILL OF HEALTH.

1 WEEK PTA THE PATIENT HAD A PHYSICAL EXAMINATION FOR LIFE INSURANCE

COVERAGE. AT THAT TIME SHE WAS TOLD THAT HER BLOOD PRESSURE WAS 135/90. SHE

WAS SOMEWHAT CONCERNED ABOUT THIS SINCE ON HER YEARLY PHYSICAL ONE YEAR AGO

HER BLOOD PRESSURE WAS 110/60 AND ON RETIREMENT TWO MONTHS BEFORE SHE WAS TOLD

THAT HER BLOOD PRESSURE WAS 125/80.

SHE DENIES HEADACHES, CHEST PAIN, SHORTNESS OF BREATH, DECREASED

TOLERANCE TO ACTIVITY OR A FAMILY HISTORY OF HEART DISEASE. NO OTHER

SIGNIFICANT MEDICAL HISTORY. NO RHD, KNOWN MURMURS, CONGENITAL ANOMALIES,

HYPERTENSION, OR EARLY DEATHS IN FAMILY. SHE DOES NOT TAKE MEDICATION,

USE DRUGS, SMOKE CIGARETTES, OR HAVE ANY EMOTIONAL PROBLEMS.

THE PATIENT IS NEW TO YOUR PRACTICE SINCE SHE RECENTLY MOVED TO THIS

TOWN. SHE CURRENTLY WORKS AS A STOCK BROKER (WHICH SHE ENJOYS BUT FINDS

STRESSFUL). SHE IS MARRIED AND HAS THREE TEENAGE CHILDREN IN GOOD HEALTH.

SHE HAS BLUE CROSS INSURANCE COVERAGE. REQUEST? PHYSICAL EXAMINATION

.
8:12 A.M. TUESDAY DAY 1

PHYSICAL EXAMINATION REQUIRES 0.6 HOURS

RESULTS DUE TUESDAY AT 8 :48 A.M.  ORDERED** **

REQUEST? RESULT

RESULTS OF PHYSICAL EXAMINATION

GENERAL: A SLIGHTLY OBESE 58 YEAR OLD WHITE WOMAN IN NO ACUTE DISTRESS.

SEE SLIDE 1-72.

PULSE 80 /MIN

PULSE QUALITY GOOD

RESPIRATION 20 /MIN

BP SYSTOLIC 135

BLOOD PRESSURE DIASTOLIC 88

HEIGHT 70 INCHES

WEIGHT 70 KILOGRAMS

TEMPERATURE 99.4 F

HEENT: NORMAL EXAM

CHEST: CLEAR A&P. DIAPHRAGMS MOVE EQUALLY AND SYMMETRICALLY.

CARDIAC: PMI IS 1 CM. LATERAL OF THE LMCL. HEART SOUNDS ARE OF GOOD

QUALITY. THERE IS A SOFT GRADE II/VI APICAL SYSTOLIC

MURMUR AND A DEFINITE S3. NO RUB.

ABDOMEN: SLIGHTLY OBESE, LIVER IS NOT FELT BELOW THE COSTAL MARGIN. NO

MASSES, NO LESIONS, NO FLUID WAVE. SPLEEN IS NOT PALPABLE.

GU: NORMAL EXAM

RECTAL: GOOD SPHINCTER TONE, NO MASSES OR ABNORMALITIES. STOOL BROWN.

EXTREMITIES: SYMMETRICAL, WITHOUT DEFORMITIES, CYANOSIS, CLUBBING OR

EDEMA.

SKIN: SLIGHTLY MOIST. NORMAL TEXTURE AND TEMPERATURE. NO LESIONS.

NEURAL: NERVOUS WOMAN. ALL CRANIAL NERVES ARE INTACT. POSTURE, MOTOR

AND SENSORY EXAMINATION IS WNL.

REQUEST? CHEST X-RAY

CHEST X-RAY REQUIRES 0.5 HOURS LAB NOT YET OPEN:

RESULTS DUE TUESDAY AT 9:30 A.M. (THIS TEST IS AVAILABLE STAT)

 ORDERED** **

REQUEST? EMERGENCY ELECTROCARDIOGRAM

EMERGENCY ELECTROCARDIOGRAM REQUIRES 0.2 HOURS

RESULTS DUE TUESDAY AT 9:00 A.M. ***ORDERED 

REQUEST? RESULT
104 • Computer Technology in Medical Education and Assessment

9:00 A.M. TUESDAY DAY 1

RESULTS OF EMERGENCY ELECTROCARDIOGRAM

RATE: 100/MIN. NORMAL SINUS RHYTHM. AXIS : O

MODERATE ST SEGMENT ELEVATION IN LEADS 11, III, AND AVF;

MODERATE ST SEGMENT DEPRESSION IN 1, AVL AND V2-V4 .

SEE SLIDE 363.

REQUEST? ICU

PATIENT ADMITTED TO INTENSIVE CARE UNIT

AN XV WILL BE STARTED, PATIENT WILL BE AT BEDREST, PATIENT WILL BE ON A

CARDIAC MONITOR, AND VITAL

NOTIFIED OF ANY CHANGES.

DO YOU WISH THE PATIENT TO

THE PATIENT HAS

THE APPROPRIATE

DO YOU WISE THE PATIENT TO

THE PATIENT HAS

THE APPROPRIATE

SIGNS WILL BE TAKEN HOURLY. YOU WILL BE

RECEIVE PARENTERAL FLUIDS? YES

NOW BEEN PLACED ON PARENTERAL FLUIDS

DOSE WILL BE GIVEN.

RECEIVE BEDREST? YES

NOW BEEN PLACED ON BEDREST

DOSE WILL BE GIVEN.

REQUEST? CREATINE PHOSPHOKINASE, SGOT, CBC, SMA-12, ELECTROLYTES

CREATINE PHOSPHOKINASE REQUIRES 4.0 HOURS

RESULTS DUE TUESDAY AT 1:00 P.M.

SGOT REQUIRES 8.0 HOURS

COMFLETE BLOOD COUNT REQUIRES 2.0

RESULTS DUE TUESDAY AT 11:00 A.M.

SMA12 REQUIRES 2.5 HOURS

ELECTROLYTES REQUIRES 6.0 HOURS

** **ORDERED*** *

RESULTS DUE TUESDAY AT 5:00 P.M.

**** (ODERED 

HOURS

** *ORDERED****

RESULTS DUE TUESDAY AT 11:30 A.M.

 ORDERED** **

RESULTS DUE TUESDAY AT 3:00 P.M.

 ORDERED** **

REQUEST? RESULT

RESULTS OF CHEST X-RAY

SEE SLIDES 141 & 142.

REQUEST? RESULT
NURSE’S NOTES:

INTENSIVE CARE

VITAL SIGNS

UNCHANGED:

PULSE QUALITY

APICAL PULSE:

.

Appendix•10 5

10:00 A.M. TUESDAY DAY 1 (PATIENT ADMITTED)

UNIT NOTE

P 72 BP 156/101 T 99.4 R 20

GOOD

NOT TAKEN

RESULTS OF RHYTHM STRIP

NO ARRHYTHMIA

REQUEST? RESULT

RESULTS OF COMPLETE BLOOD COUNT

HEMATOCRIT 45.0 %

HEMOGLOBIN 14.0 G%

RBC 5200000 PER CU.MM

WBC 8200.0

MCV 88 CU U3

MCHC 35 %

63.0 %

LYMPH 30 %

MONO 5 %

EOS 2 %

BLAST O %

ATYP LYMPH O %

BANDS .0 %

MYELOCYTES .0 %

JUVENILE NEUTROPHILS .0 %

BASOPHILES .0 %

PLAT FROM CBC 300000

REQUEST? RESULT

RESULTS OF SMA12

CALCIUM 9.5 MG%

SERUM PHOSPHORUS 3.8 MG/100 ML

GLUCOSE 80 MG%

BLOOD UREA NITROGEN 14 MG%

URIC ACID 4.0 MG%

SERUM CHOLESTEROL 200MG%

TOTAL PROTEIN 7.0 GMS %

ALBUMIN 4.5 GMS %

TOTAL BILIRUBIN .70 MG%

ALKALINE PHOSPHATASE 4.0 BODANSKY UNITS

LACTIC DEHYDROGENASE 130.0 u/ML

SGOT 10 u/ML
NURSE’S NOTES :

PATIENT IN NO ACUTE DISTRESS.

INTENSIVE CARE UNIT NOTE

VITAL SIGNS P 72 BP 156/101 T 99.4 R20

UNCHANGED:

PULSE QUALITY GOOD

UNCHANGED:

APICAL PULSE: NOT TAKEN

RESULTS OF RHYTHM STRIP

UNCHANGED:

NO ARRHYTHMIA

REQUEST? RESULT

RESULTS OF CREATINE PHOSPHOKINASE

CREATINE PHOSPHOKINASE 260.0 UNITS

REQUEST? RESULT

RESULTS OF ELECTROLYTES

SODIUM 140

POTASSIUM 4.0 MEQ/L

CHLORIDE 103 MEQ. /L

TOTAL CARBON DIOXIDE 27.0 MEQ. /L

REQUEST? RESULT

RESULTS OF SGOT

SCOT IOU/ML

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? &5

NURSE'S NOTES:

INTENSIVE CARE UNIT NOTE
8:00 P.M. TUESDAY DAY 1 (PATIENT ADMITTED)

VITAL SIGNS P 72 BP 139/9 2 T 99.4 R 18

UNCHANGED :

PULSE QUALITY GOOD

UNCHANGED :

APICAL PULSE: NOT TAKEN

RESULTS OF RHYTHM STRIP

UNCHANGED:

NO ARRHYTHMIA

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 12

MORNING ROUNDS:

NOW ON PARENTERAL FLUIDS IV [3173]

OW ON BEDREST [7181]

VITAL SIGNS P 72 BP 139/92 T 99.4 R 18

UNCHANGED:

PULSE QUALITY GOOD

UNCHANGED:

APICAL PULSE: NOT TAKEN

THE PATIENT IS IN THE INTENSIVE CARE UNIT.

REQUEST? EMERGENCY ELECTROCARDIOGRAM

EMERGENCY ELECTROCARDIOGRAM REQUIRES 0.2 HOURS

RESULTS DUE WEDNESDAY AT 8:12 A.M. ***ORDERED 

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 24

RESULTS OF EMERGENCY ELECTROCARDIOGRAM

RATE: 100/MIN. NORMAL SINUS RHYTHM, AXIS: O

Q WAVES PRESENT IN II, 111, AND AVF WITH T-WAVE INVERSION AND SLIGHT ST

SEGMENT ELE’VATION IN THESE LEADS. SEE SLIDE 358.

NURSE’S NOTES:

PATIENT RESTING COMFORTABLY.

INTENSIVE CARE UNIT NOTE
108 ● Computer Technology in Medical Education and Assessment

NOON WEDNESDAY DAY 2 (PATIENT ADMITTED)

VITAL SIGNS P 72 BP 138/92 T 99.4 R 18

UNCHANGED :

PULSE QUALITY GOOD

UNCHANGED :

APICAL PULSE: NOT TAKEN

RESULTS OF RHYTHM STRIP

UNCHANGED :

NO ARRYTHMIA

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 20

NURSE’S NOTES:

INTENSIVE CARE uNIT NOTE

VITAL SIGNS P 72 BP 138/92 T 99.4 R 18

UNCHANGED :

PULSE QUALITY GOOD

UNCHANGED :

APICAL PULSE: NOT TAKEN

RESULTS OF RHYTHM STRIP

UNCHANGED :

No ARRHYTHMIA

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 16

MORNING ROUNDS :

NOW ON PARENTERAL FLUIDS IV [ 3173]

NOW ON BEDREST [7181]

VITAL SIGNS P 72 BP 138/92 T 99.4

UNCHANGED:

PULSE QUALITY GOOD

UNCHANGED :

APICAL PULSE: NOT TAKEN

THE PATIENT IS IN THE INTENSIVE CARE UNIT.

R 18

REQUEST? WARD

PATIENT DISCHARGED FROM ICU TO WARD
REQUEST?

EMERGENCY

Appendix ● 109

8:00 A.M. THURSDAY DAY 3 (PATIENT ADMITTED)

EMERGENCY CREATINE PHOSPHOKINASE, EMERGENCY ELECTROCARDIOGRAM

CREATINE PHOSPHOKINASE REQUIRES 2.0 HOURS

RESULTS DUE THURSDAY AT 10:00 A.M. ** *ORDERED****

EMERGENCY ELECTROCARDIOGRAM REQUIRES 0.2 HOURS

RESULTS DUE THURSDAY AT 8:12 A.M. ***ORDERED 

REQUEST? RESULT

RESULTS OF EMERGENCY ELECTROCARDIOGRAM

UNCHANGED:

RATE: 100/MIN. NORMAL SINUS RHYTHM. AXIS: O

Q WAVES PRESENT IN II, III, AND AVF WITH T-WAVE INVERSION AND SLIGHT ST

SEGMENT ELEVATION IN THESE LEADS. SEE SLIDE 358.

REQUEST? RESULT

RESULTS OF EMERGENCY CREATINE PHOSPHOKINASE

CREATINE PHOSPHOKINASE 208.0 UNITS

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 24

NURSE’S NOTES:

PATIENT IN NO DISTRESS AND INDICATES SHE WISHES TO BE DISCHARGED.

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 24

MORNING ROUNDS:

NOW ON PARENTERAL FLUIDS IV [3173]

NOW ON BEDREST [7181]

VITAL SIGNS p 61 BP 183/92 T 99. 2 R 18

UNCHANGED:

PULSE QUALITY GOOD

UNCHANGED:

APICAL PULSE: NOT TAKEN

THE PATIENT IS IN THE WARD.

REQUEST? ADVANCE

HOW MANY HOURS DO YOU WANT TO ADVANCE THE CLOCK? 9

51-541 0- 79 - 9
NURSE’S NOTES:

PATIENT THREATENS TO SIGN HERSELF OUT OF HOSPITAL.

REQUEST? ORDER THERAPY

NOW ON PARENTERAL FLUIDS IV [3173]

NOW ON BEDREST [ 7181]

DO YOU WANT TO TAKE THE PATIENT OFF ONE OF THESE? NO

ENTER THERAPY NUMBER (OR PUSH 'ENTER') 186

DO YOU WISH THE PATIENT TO RECEIVE REASSURANCE? YES

THE PATIENT HAS NOW BEEN PLACED ON REASSURANCE

THE APPROPRIATE DOSE WILL BE GIVEN.

ENTER THERAPY NUMBER ( PUSH 'ENTER' )

REQUEST : ADVANCE

HOW MANY HOURS IN YOU WANT TO ADVANCE THE CLOCK? 24

NURSE’S NOTES :

PATIENT CONTINUES TO HAVE UNEVENTFUL RECOVERY.

DO YOU WISH TO KNOW THE CORRECT DIAGNOSIS? YES

** *CORRECT DIAGNOSIS***

NOT YET AVAILABLE

NUMBER OF DAYS IN HOSPITAL 5

COST OF DIAGNOSTIC WORKUP $ 181.00

COST OF THE HOSPITAL ROOM ($90/DAY) $ 450.00

COST OF INTENSIVE CARE UNIT ($8/HOUR) $ 376.00

TOTAL COST OF MEDICAL CARE $ 1007.00

DO YOU WANT TO MANAGE ANOTHER PATIENT? NO

GOOD BYE
Appendix •II I

PART 5

AN EXAMPLE OF A COMPUTER ASSISTED SIMULATION OF THE

CLINICAL PATIENT ENCOUNTER (CASE)

(Note: The question marks identify the user’s inquiries . The user’s input.

is indicated by asterisks . Everything else is generated by the computer.)

YOU ARE THE EVENING ATTENDING PEDIATRICIAN IN A BUSY METROPOLITAN

EMERGENCY ROOM. THE EMERGENCY SQUAD WHEELS IN A 15 YEAR OLD, WELLBUILT MALE ADOLESCENT ON A STRETCHER. HE APPEARS TO BE IN MODERATE

DISTRESS, LOOKING ABOUT ANXIOUSLY. HIS SPEECH IS SOMEWHAT GARBLED,

AND HE IS DROOLING FROM THE CORNER OF HIS MOUTH.

? * WHY DID THEY BRING YOU HERE

I HAVEN’T BEEN ABLE TO MOVE FOR THE PAST THREE HOURS. I’M REALLY

SCARED. MY AUNT CALLED THE EMERGENCY SQUAD BECAUSE SHE GOT WORRIED,

SO THAT’S WHY I’M HERE.

? * HAS THIS HAPPENED BEFORE

IT HAPPENED THE FIRST TIME WHEN I WAS 14. IT’S WIERD TO WAKE UP ONE

MORNING AND NOT BE ABLE TO MOVE A MUSCLE.

? * OTHER ILLNESSES (Note: The user has begun to us e

abbreviated inquiries. )

I’VE ALWAYS BEEN PRETTY HEALTHY. I HAD THE MEASLES AND ALL THAT AS A KID,

AND SOMETIMES I GET A COLD, BUT IT’S NEVER VERY SERIOUS.

? * GO OUT FOR SPORTS

I LIKE JUST ABOUT ANYTHING YOU DO OUTDOORS. I SPEND LOTS OF TIME ON MY

BIKE. I REALLY LIKE SWIMMING TOO. WE SOMETIMES GO FOR HIKES WHEN THE

WEATHER IS NICE - BERNADINE AND ME. WE'LL TAKE A PICNIC LUNCH SOMEWHERE.

? *WHO IS BERNADINE

I THINK MY BEST FRIEND ALL THE WAY AROUND IS BERNADINE. WE SPEND A LOT

OF TIME TOGETHER AND CAN TALK ABOUT ANYTHING.

? * SISTERS

PAUL IS 18 AND FINISHING HIGH SCHOOL. ELLEN IS 9.

? *

I’M

?*

GO TO SCHOOL

IN THE NINTH GRADE NOW.

ANY OTHER MEMBERS OF YOUR FAMILY HAVE THE SAME PROBLEM

I DON’T KNOW OF ANYONE WHO’S BEEN PARALYZED LIKE I AM.

? * TRAVEL

I’VE BEEN TO CANADA BEFORE. ..LAST SUMMER ON A BIKING TRIP. I HAVEN’T

BEEN ANYWHERE ELSE, THOUGH.
? * HAVE SEIZURES

I‘VE NEVER BEEN IN ANY ACCIDENTS OR ANYTHING. I MAY SKIN MYSELF UP

SOMETIMES , BUT THAT ‘ S ABOUT IT.

? * HOW IS YOUR VISION

I HAVEN' T HAD ANY TROUBLE WITH MY VISION. I 'VE NEVER EVEN WORN GLASSES

OR ANYTHING LIKE THAT.

? * MOODY

1‘M PRETTY HAPPY MOST OF THE TIME.

IT’ S JUST WHEN I‘M PARALYZED. I CAN ‘ T GET MY WORDS OUT CLEARLY.

? * FAMILY HISTORY OF ILLNESSES

THERE AREN’T ANY DISEASES THAT RUN IN THE FAMILY, THAT I’M AWARE OF. I

DO KNOW THERE ARE AN AWFUL LOT OF RELATIVES WHO HAVE JUST DIED IN THEIR

SLEEP . THAT’S SCARY.

I AM AFRAID OF DYING. I MEAN, NOBODY KNOWS WHY MY RELATIVES HAVE DIED

IN THEIR SLEEP . . ● HOW DO I KNOW THE SAME THING WON’T HAPPEN TO ME? I

HOPE WE CAN FIGURE THIS ALL OUT.

I HAVEN’T HAD ANY COORDINATION PROBLEMS, EXCEPT FOR WHEN I'M PARALYZED,

OF COURSE.

? * DIZZY

I'VE NEVER HAD A DIZZY SPELL.

I PASSED OUT ONCE FROM DRINKING TOO MUCH BEER WITH FRIENDS ONCE, BUT THAT’S

ALL .

? * SMOKE

I’VE SMOKED CIGARS AND CIGARETTES BEFORE. SOMETIMES I LIGHT UP IF I’M

WITH A LOT OF GUYS WHO SMOKE. AND THE CIGARS -WELL - THE’Y ARE AWFUL.

? * DRUGS
I DON’T TAKE ANY MEDICINE.

ALL OF OUR FRIENDS LIKE DRINKING BEER. WE FIND SOMEONE OLD ENOUGH TO BUY

US BEER AND THEN TAKE IT OUT TO PARTY. BEER’S ALL WE EVER HAVE.

? * EATING HABITS

I LIKE JUST ABOUT EVERYTHING, SO I EAT WHATEVER IS PUT IN FRONT OF ME.

I EAT OKAY WHEN I’M AT HOME. I LIKE FRUIT, GREEN VEGETABLES, MEATS. . .

ALL THAT STUFF. IT’S JUST THAT WHEN I’M OUT WITH MY FRIENDS THAT I

LIKE PIZZAS AND BIG MACS. I WOULDN’T SAY I’M MALNOURISHED OR ANYTHING.

I’VE NEVER HAD ANY STOMACH TROUBLE.

? * HEADACHES

I VERY SELDOM HAVE HEADACHES. ASPIRIN ALWAYS GETS RID OF THEM ANYHOW.

? * PAIN

I’M NOT IN PAIN . . . I JUST CAN’T MOVE.

? * p (Note: The user asks to enter

the physical examination

section “p” of the CASE.)

GENERAL APPEARANCE--

WELL DEVELOPED, WELL NOURISHED, WELL HYDRATED WHITE MALE ADOLESCENT WHO

APPEARS TO BE IN MODERATE DISTRESS, LOOKING ANXIOUSLY ABOUT. HE OCCASIONALLY

DROOLS FROM THE CORNER OF HIS MOUTH.

? * H (Note: The user asks to reenter

the history section “h”

of the CASE.)

** HISTORY **

? * HOW LONG HAVE YOU BEEN DROOLING

I JUST CAN’T CONTROL IT. I HATE IT WHEN I DROOL LIKE THIS. AS SOON AS I

CAN'T MOVE, THIS HAPPENS. IT'S BEEN A COUPLE HOURS NOW SINCE THIS HAPPENED.

? * DO YOU ALWAYS DROOL WHEN YOU HAVE THESE PROBLEMS
(Note: The user asks to reenter

the physical examination

sect ion “P” of the CASE.

** PE * *

TEMPERATURE - 37 c. (ORAL)

PULSE - 150 AND REGULAR

RESPIRATION - 30/MINUTE

BLOOD PRESSURE - 110/70 SUPINE; 100/56 SITTING WITH ASSISTANCE

? * CHEST INSPECTION

CHEST - APPEARANCE--

SYMMETRICAL, WITH FULL EXPANSION. NORMAL A-P DIAMETER. NO DEFORMITIES OR

VISIBLE PULSATIONS.

CHEST - PERCUSSION --

NORMAL RESONANCE. DIAPHRAGM MOVES NORMALLY WITH RESPIRATION.

? * CHEST AUSCULTATIO N

CHEST - AUSCULTATION--

NORMAL BREATH SOUNDS IN ALL LUNG FIELD. VOCAL FREMITUS NORMAL.

HEART - INSPECTION—

PMI VISIBLE IN 5TH INTERCOSTAL SPACE NEAR MID-CLAVICULAR LINE. THE

PRECORDIUM IS SOMEWHAT HYPERACTIVE.

HEART - PALPATION--

VERY ACTIVE PRECORDIUM. NO THRILLS OR HEAVES. PROMINENT SYSTOLIC IMPULSE.

HEART - AUSCULTATION--

150/MINUTE. GRADE 2/6 SYSTOLIC EJECTION MURMUR AT LOWER LEFT STERNAL BORDER.

? * HEART PERCUSSION

HEART - PERCUSSION--

NORMAL HEART SIZE TO PERCUSSION.
EXTREMITIES--

NORMAL COLOR WITHOUT CYANOSIS, CLUBBING OR EDEMA. NO VARICOSITIES. NO

TREMOR. PERIPHERAL PULSES EQUAL AND BOUNDING BILATERALLY. JOINTS NORMAL.

SHOULDERS NORMAL.

DEEP TENDON REFLEXES--

ABSENT

MUSCLE STRENGTH--

FLACCID. WHEN ARM IS HELD OVER FACE AND DROPPED, IT FALLS ON PATIENT’S FACE.

? * CRANIAL NERVES

CRANIAL NERVES --

1 INTACT SMELL. II, III, IV, VI INTACT. PUPILS ARE EQUAL, ROUND AND

REACTIVE TO LIGHT AND ACCOMMODATION. FUNDOSCOPIC NORMAL. VII FACE

APPEARS DROOPY. FACIAL MOVEMENT WITH DIFFICULTY. V SLIGHT WEAKNESS OF

MASSETERS. SENSATION INTACT. VIII INTACT. IX, X DECREASED BUT PRESENT

GAG REFLEX. SENSATION INTACT. UVULA ELEVATES SLIGHTLY. REGURGITATES ON

SWALLOWING. SOME HOARSE TONE TO VOICE. XI DECREASED STRENGTH OF

TRAPEZIUS AND STERNOCLEIDOMASTOID. XII DECREASED STRENGTH OF TONGUE

MOVEMENTS.

? * GAIT

GAIT--

NOT ABLE TO WALK.

? * MOTOR SYSTEM

MOTOR SYSTEM

DECREASED MUSCLE TONE, STRENGTH. MUSCLE MASS NORMAL.

BABINSKI REFLEX--

ABSENT BILATERALLY (NO MOVEMENT OF TOES UP OR DOWN).

? * EYES
EYE EXAMINATION--

CONJUNCTIVA AND SCLERA CLEAR - NO INJECTION, PETECHIAE, JAUNDICE, PTOSIS,

LID LAG, EDEMA. PUPILS EQUAL AND ROUND AND NORMAL REACTION TO LIGHT AND

ACCOMMODATION.

? * JOINTS

EXTREMITIES--

NORMAL COLOR WITHOUT CYANOSIS, CLUBBING OR EDEMA. NO VARICOSITIES. NO

TREMOR. PERIPHERAL PULSES EQUAL AND BOUNDING BILATERALLY. JOINTS NORMAL.

SHOULDERS NORMAL.

? * L (Note: The user asks to enter the

laboratory section “L” of

the CASE.)

COMPLETE BLOOD COUNT--

R N 5,200,000

WBC 15,000

HGB 15 GM/loo ML

HCT 45%

DIFFERENTIAL -

POLY 70%

STAB 10%

LYMPH 12%

MONO 4%

EOS 3%

BASO 1%

NRBC/100 WBC NONE

ANISOCYTOSIS NONE

POIKILOCYTOSIS NONE

CHROMICITY NORMAL

PLATELETS ADEQUATE

NORMALS

4,6-6.2 X 1,000,00 0

4.8-10.8 X 1,000

14-18

42-52

40-70

0-6

20-45

2-8

0-5

0-1

URINALYSIS--

APPEARANCE: AMBER YELLOW, CLEAR

REACTION (PH): 6.5 (NORMAL REACTION (PH): 4.6-8.0)

DIACETIC ACID: NEGATIVE

BILE PIGMENT: NEGATIVE

UROBILINOGEN: NEGATIVE

REDUCING SUGAR: NEGATIVE
PROTEIN : NEGATIVE

SPECIFIC GRAVITY : 10,010 (NORMAL: 1. 001-1• 035)

ACETONE : NEGATIVE

MICROSCOPIC URINALYSIS--

BACTERIA: NEGATIVE

CRYSTALS: NEGATIVE

EPITHELIAL CELLS: 0-3/HPF

LEUKOCYTES: O-3 WBC/HPF

ERYTHROCYTES: O-3 RBC/HPF

CASTS: NEGATIVE

? * SMAL

THE SPECIFIED TEST IS NOT AVAILABLE IN THE LABORATORY SECTION.

? * SMAL,2:

SMA 12-- NORMALS

CALCIUM 10 MG/100 ML 9-10. 8

INORG. PHOSPH. 4 MG/100 ML 3.0-4. 5

FASTING GLUCOSE 82 MG/100 ML 60-100

BUN 7 MG/100 ML 8-25

URIC ACID

CHOLESTEROL

TOTAL BILIRUBIN

ALBUMIN

TOTAL PROTEIN

ALK. PHOSPH.

LDH (WACKER)

SGOT (KARMEN)

6 MG/100 ML

210 MG/100 ML

0.7 MG/100 ML

4 GM/100 ML

7 GM/loo ML

9 KA UNITS

95 UNITS/ML

28 UNITS/ML

? * SERUM ELECTROLYTES

SERUM ELECTROLYTES--

NORMAL

CL 99 MEQ/L 100-106

NA 140 MEQ/L 135-145

K 1.8 MEQ/L 3.5-5.0

TOTAL C02 27 MEQ/L 24-30

? * EKG

ELECTROCARDIOGRAM:

RATE “ 150

RHYTHM - SINUS TACHYCARDIA

QRS AXIS - +60

TAXIS-+4 0

PR INTERVAL - 0.17

QRS DURATION - 0.1O

3.5-7. 0

150-27(3

0.3-1. 1

3.5-5. 5

6 - 8

5-13

80-120

15-40
118 ● Computer Technology in Medical Education and Assessment

QT INTERVAL - 0.34

T WAVES ARE INVERTED AND ST SEGMENTS ARE DEPRESSED. U WAVES ARE PROMINENT.

? * CHEST XRAY

CHEST --

PA AND LATERAL CHEST FILMS SHOW WELL EXPANDED LUNG FIELDS WITHOUT EVIDENCE

OF CONGESTION OR ACTIVE INFECTION. DIAPHRAGMS ARE SMOOTH, COSTOPHRENIC

ANGLES ARE CLEAR AND THE CARDIOVASCULAR SHADOW IS NORMAL FOR THE AGE.

BONES APPEAR INTACT AND OF GOOD DENSITY.

?*SED RAT E

ERYTHROCYTE SED RATE (WESTERGREN) --

10 MM/HR (NORMAL: 0-15)

? * D (Note: The user asks to enter the diagnosis

“d” section .

AFTER YOU BEGIN THE DIAGNOSIS AND MANAGEMENT SECTIONS YOU WILL BE UNABLE TO GO

BACK TO THE HISTORY, PHYSICAL OR LABORATORY SECTIONS OF THIS CASE. IF THERE

IS ANYTHING MORE YOU WISH TO ASK THE PATIENT OR OTHER TESTS YOU’D LIKE TO

ORDER, TYPE H, P OR L TO RETURN TO THESE SECTIONS. IF YOU ARE READY TO LIST

YOUR DIAGNOSES AND TREATMENT FOR THE PATIENT, TYPE D .

* D

** DIAGNOSIS AND TREATMENT **

PLEASE TYPE THE DIAGNOSES FOR THE PROBLEM THAT YOU PERCEIVE ED HARRISON TO HAVE.

ENTER ONLY ONE DIAGNOSIS PER LINE. SECONDARY DIAGNOSES SHOULD BE LISTED

SEPARATELY FROM PRIMARY DIAGNOSES -- E.G., DO NOT LIST ‘ACUTE PYELONEPHRITIS

WITH SEPSIS’ ON ONE LINE. PLEASE LIST THE DIAGNOSES SEPARATELY SO THAT YOU

WILL RECEIVE CREDIT FOR BOTH. HIT THE RETURN KEY AFTER EACH DIAGNOSIS AND

WAIT FOR THE ‘DX:’ TO APPEAR BEFORE ENTERING ANOTHER.

YOUR LAST DIAGNOSIS.

** NOTE - THE DOLLAR SIGN ($) INDICATES THAT YOU HAVE

TAKES YOU ON TO THE NEXT SECTION. PLEASE BE SURE YOU

INFORMATION BEFORE TYPING ‘$’ AS YOU WILL NOT BE ABLE

THE SECTION YOU HAVE JUST COMPLETED. **

TYPE A ‘$’ AFTER

COMPLETED A SECTION AND

HAVE ENTERED ALL THE

TO MAKE ADDITIONS TO
.—

Appendix ● 1 1 9

DX: *

DX: *

PLEASE

DX: *

DX: *

DX: *

HYPOKALEMIA

FAMILIAL INTERMITTENT (PERIODIC) PARALYSIS

BE MORE DESCRIPTIVE ABOUT THE TYPE OF PARALYSIS YOU FEEL ED HARRISON HAS.

FAMILIAL PERIODIC PARALYSIS

NEUROMUSCULAR PARALYSIS

$

IN DESCRIBING YOUR MANAGEMENT PLAN FOR ED HARRISON, YOU WILL BE ASKED TO

INCLUDE THE FOLLOWING INFORMATION:

1. MEDICATIONS (USING GENERIC NAMES FOR DRUGS WHEN POSSIBLE;

DOSAGES AND METHOD OF ADMINISTRATION ARE NOT NECESSARY)

2. PROCEDURES, REFERRALS AND/OR CONSULTATIONS (INCLUDE

HOSPITALIZATION HERE, IF NECESSARY)

3. INSTRUCTIONS TO THE PATIENT

** NOTE - YOU ARE EXPECTED TO GIVE MANAGEMENT STEPS ONLY FOR THE SHORT-TERM

CARE OF THIS PATIENT.**

LIST ONLY ONE DRUG, PROCEDURE, INSTRUCTION ON EACH LINE AND HIT THE RETURN

KEY AT THE END OF EACH LINE. WHEN YOU HAVE COMPLETED A SECTION, TYPE A

DOLLAR SIGN ($).

MEDICATION: * IV POTASSIUM CHLORIDE

MEDICATION: * IV RINGERS

MEDICATION: * $

PLEASE LIST THE PROCEDURES, REFERRALS AND/OR CONSULTATIONS YOU WOULD RECOMMEND

FOR ED HARRISON.

PROCEDURE: *

PROCEDURE: *

PROCEDURE: *

PROCEDURE: *

PROCEDURE: *

HOSPITALIZE

MONITOR POTASSIUM INTAKE

REPEAT EKG

REFER TO NEUROLOGY

$

PLEASE LIST ANY INSTRUCTIONS YOU WOULD GIVE THE PATIENT REGARDING HIS

RESPONSIBILITIES IN THE MANAGEMENT OF HIS HEALTH PROBLEMS.
120 ● Computer Technology in Medical Education and Assessment

INSTRUCTION : * COME TO SEE ME IN A WEEK AFTER DISCHARGE

INSTRUCTION : * $

SCORES AND FEEDBACK (Note: At this point the

CASE is complete

and the user receives

feedback on his/her

performance.)

YOUR SCORE FOR DIAGNOSTIC CORRECTNESS: 100

AUTHOR’S DIAGNOSTIC STATEMENT VALUE VALUE OBTAINED

1 . HYPOKALEMIA 40 40

2. NEUROMUSCULAR PARALYSIS 40 40

SECONDARY TO HYPOKALEMIA

3. FAMILIAL PERIODIC PARALYSIS 20 20

YOUR SCORE FOR MANAGEMENT:

AUTHOR’S SUGGESTED MANAGEMENT

1. MEDICATION

A. IV POTASSIUM CHLORIDE

B. IV HYDRATION

2. PROCEDURES, REFERRALS, CONSULT

A. HOSPITALIZATION - EKG

MONITORING & .FOLLOW SERUM

POTASSIUM

B. CONSULT NEUROLOGY

3. INSTRUCTIONS TO THE PATIENT

A. AVOID HIGH CARBOHYDRATE

MEALS

B. FOLLOW-UP (COME IN A WEEK OR

LESS AFTER DISCHARGE)

75

VALUE

70

60

10

15

15

0

15

10

5

VALUE OBTAINED

60

60

0

0

10

0

5

0

5
DR. STEPHEN M. FRIES (UNIVERSITY OF MICHIGAN), AUTHOR OF THE ED HARRISON

CASE, FEELS THE FOLLOWING HISTORICAL INFORMATION AND PHYSICAL AND LABORATORY

TESTS ARE CRITICAL FOR THE DIAGNOSIS AND MANAGEMENT OF THIS CASE.

PLEASE PRESS THE RETURN KEY TO CONTINUE . . . *

** HISTORY ** (Note: A notation is made beside each o f

the author’s critical concepts a s

to whether the user did or did not

obtain the information.)

CANNOT MOVE BUT ABLE TO TALK - OBTAINED

SPEECH SLURRED - OBTAINED

DROOLING - OBTAINED

MEDICATIONS - OBTAINED

SEIZURE HISTORY - OBTAINED

PAST HOSPITALIZATIONS - OBTAINED

FAMILY HISTORY OF MEMBERS DYING IN SLEEP - OBTAINED

FAMILY HISTORY OF PARALYSIS - NOT OBTAINED

ACCIDENT OR INJURIES - OBTAINED

COORDINATION PROBLEMS-OBTAINED

EYESIGHT DIFFICULTIES - OBTAINED

PSYCHIATRIC problems- OBTAINED

INJESTION OF HIGH CARBOHYDRATE MEAL PRIOR TO ONSET - NOT OBTAINED

PULSE - OBTAINED

RESPIRATION - OBTAINED

CHEST PERCUSSION - OBTAINED

HEART INSPECTION - OBTAINED

HEART PALPATION - OBTAINED

HEART AUSCULTATION - OBTAINED

EXTREMITIES - OBTAINED

CRANIAL NERVES - OBTAINED

MOTOR SYSTEM - OBTAINED

CEREBELLA TESTS - NOT OBTAINED

GAIT - OBTAINED

MUSCLE STRENGTH - OBTAINED

VIBRATORY, POSITIONAL, TEMPERATURE, TOUCH AND PAIN SENSATION - NOT OBTAINED

BABINSKI REFLEX - OBTAINED

SERUM ELECTROLYTES - OBTAINED

EKG - OBTAINED
122 ● Computer Technology in Medical Education and Assessment

THERE ARE A NUMBER OF TESTS IN THE LABORATORY SECTION THAT WOULD BE CONSIDERED

DANGEROUS OR UNNECESSARY TO PERFORM ON THIS PATIENT BECAUSE OF HIS CONDITION.

YOU REQUESTED NONE OF THESE.

THE TOTAL COST OF TESTS YOU ORDERED IN THE LABORATORY SECTION IS 66 DOLLARS.

THERE ARE TWO LABORATORY TESTS WHICH SHOULD BE ORDERED FOR EVERY PATIENT YOU SEE

FOR THE FIRST TIME. THESE TESTS ARE THE URINALYSIS AND COMPLETE BLOOD COUNT.

YOU REQUESTED BOTH OF THESE.

DO YOU DESIRE PRINTED EVIDENCE THAT YOU HAVE COMPLETED THE FOREGOING CASE SO THAT

YOU MAY OBTAIN CONTINUING EDUCATION CREDIT? (YES/NO)

* Y

IN ORDER TO VALIDATE YOUR COMPLETION OF THIS ACTIVITY, WE MUST MAINTAIN A RECORD

OF YOUR NAME, DATE, AND CASE COMPLETED.

PLEASE ENTER YOUR FIRST AND LAST NAME AT THIS TIME

EXAMPLE: JOHN

LAST NAMES.)

* EARLA MYER

EARLA MYER, IS

*Y

DOE (PLEASE REMEMBER TO PUT A SPACE

(NO MIDDLE NAME OR INITIAL).

BETWEEN YOUR FIRST AND

YOUR NAME ENTERED CORRECTLY - NO MIDDLE NAME OR INITIAL? (YES/NO)
5/01/79

EARLA MYER HAS COMPLETED CASE 16, ED HARRISON.

THIS IS A CAI PROGRAM OFFERING OF THE OHIO STATE UNIVERSITY COLLEGE OF MEDICINE

DIVISION OF COMPUTING SERVICES FOR MEDICAL EDUCATION AND RESEARCH. AS AN

ORGANIZATION ACCREDITED FOR CONTINUING MEDICAL EDUCATION, THE OHIO STATE

UNIVERSITY COLLEGE OF MEDICINE CENTER FOR CONTINUING MEDICAL EDUCATION

CERTIFIES THAT THIS CONTINUING MEDICAL EDUCATION OFFERING MEETS THE CRITERLA FOR

UP TO 1.5 CREDIT HOURS IN CATEGORY I OF THE PHYSICIAN’S RECOGNITION AWARD OF

THE AMERICAN MEDICAL ASSOCIATION PROVIDED IT IS USED AND COMPLETED AS DESIGNED.

ALL PARTICIPANTS ARE REMINDED TO LIST THE TOTAL HOURS SPENT USING THIS CASE (UP

TO 1.5 HOURS) IN THEIR INDIVIDUAL LOG AND TO NAME THE OSU COLLEGE OF MEDICINE

CCME AS CO-SPONSOR.

(Note: Record of participation is provided fo r

continuing education credit.)
REFERENCES
REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

Abernathy, W., and E. McBride, 1978, “A Computer Simulation for Teaching Client Referral Skills, ” Proceedings of the 1978 Conference on Computers in Undergraduate Curricula (CCUC/9), Denver, Colo.: University of Denver, Department

of Mathematics, pp. 356-363.

Abrahamson, S., J. S. Denson, and R. M. Wolf, 1969, “Effectiveness of a Simulator

in Training Anesthesiology Residents, ” ]ournal of Medical Education 44:515-519.

Allen, J. L., 1976, “The Future of Simulators in Skills Training, ” Proceedings of the

First International Learning Technology Congress and Exposition in Applied

Learning Technology, vol. IV. Buffalo, N. Y.: Calspan Graphics, pp. 2-6.

American Board of Medical specialties, 1977, 1976-1977 Annual Report, Evanston,

111.

American Medical Association, 1977a, Directory of Accredited Residencies,

2977-78, Chicago, Ill.

—; 1977b, Physician Distribution and Medical Licensure in the U. S., 1976,

Chicago, Ill.

Anderson, H., 1969, “Statistical Surveillance of a Title XIX Program, ” American

Journal of Public Health 59:275-289.

Ashbaugh, D. G., and R. S. McKean, 1976, “Continuing Medical Education, the

Philosophy and Use of Audit, ” Journal of the American Medical Association

236:1485-1488.

Association of American Medical Colleges, 1977, 1977-78 Curriculum Directory,

Washington, D.C.

; 1978a, Medical School Admission Requirements, 1978-79, Washington,

D.C.

; 1978b, 1978-79 Curriculum Directory, Washington, D.C.

__1978c, A Study of Three-Year Curricula in U.S. Medical Schools, Washington, D.C.

 1978d, “A Description of the Formal Medical Education Process, ” paper

prepared for the Office of Technology Assessment, U.S. Congress.

 1979, personal communication from Robert Beran to the Office of Technology Assessment, U.S. Congress.

Barnett, G. O., 1976, Computer-Stored Ambulatory Record (COSTAR), NCHSR,

Research Digest Series, USDHEW, Pub]. No. (HRA) 76-3145.

Barr, D. M., and C. R. Gaus, 1973, “A Population Based Approach to Quality

Assessment in Health Maintenance Organizations, ” Medical Care 11 :523-528.

Bejar, I. I., D. J. Weiss, and G. G. Kingsbury, 1977a, Calibration of an Item Pool

for the Adaptive Measurement of Achievement (Research Report 77-5), Minneapolis, Minn.: University of Minnesota Psychometric Methods Program, Department of Psychology.

; D. J. Weiss, and K. A. Gialluca, 1977b, An Information Comparison of Conventional and Adaptive Tests in the Measurement of Classroom Achievement (Research Report 77-7), Minneapolis, Minn,: University of Minnesota Psychometric

Methods Program, Department of Psychology,

Bentz, W. F., and A. D. Luzi, 1975, “Learning in Interactive Mode: How Financial

Events Affect Financial Statements, ” Proceedings of the 1975 Conference on Computers in the Undergraduate Curricula (CCUC/6), Fort Worth, Tex.: Texas Christian University, Department of Computer Sciences, pp. 171-177.
128

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

● Computer Technology in Medical Education and Assessment

Beran, R. L., R. L. Folk, J. V. Griesen, and J. S. Camiscioni, 1976, Individualizing

the Study of Medicine, The Ohio State University College of Medicine Independent

Study Program, New York, N. Y.: Westinghouse Learning Corporation.

Bergin, R., P. G. Lundquist, and P. O. Wester, 1978a, “Computers in Education

(1): Problem-Oriented Medical Education–CASE in Sweden and in the USA,”

Lakartidningen 75:3139-3141 (Swedish).

— ., and P. 0. Wester, 1978b, “Computers in Education (2): CASE at the

Serafimer Hospital—an Evaluation, ” Lakartidningen 75:3142-3143.

Bergner, M., R. A. Bobbitt, W. E. Pollard, D. P. Martin, and B. S. GiIson, 1976,

“The Sickness Impact Profile: Validation of a Health Status Measure, ” Medical

Care14:57-67.

Berkanovic, E., 1974, “An Appraisal of Medicaid Records as a Data Source, ” Medical Care 12:590-595.

Berner, E. S., T. J. Bligh, and R. O. Guerin, 1977, “An Indication for a Process

Dimension in Medical Problem Solving,” Medical Education 11:324-328.

Bitzer, D. L., and H. G. Slottow, 1968, “The Plasma Display Panel—A New

Device for Direct Display of Graphics,” in D. Secrest and J. Nievergelf (eds.),

Emerging Concepts in Computer Graphics, New York, N. Y.: W. A. Benjamin.

Bitzer, M. D., and M. D. Bourdreaux, 1969, “Using a Computer to Teach Nursing,” Nursing Forum 8:234-254.

Bleich, H. I., 1971, “The Computer as a Consultant, ” New England Journal of

Medicine 284:141-146.

Bloomfield, D. K., G. L. Hody, and A. H. Levy, 1978, “Some PLATO Applications in Health Sciences Education, ” in E. C. Deland (cd.), Information Technology

in Health Science Education, New York, N. Y.: Plenum Press.

Bork, A., 1978, “Machines for Computer-Assisted Learning, ” Educational Technology 18:17-20.

Boutwell, R. C., 1978, “Medical Education and Instructional Design, ” ]ournal of

Educational Technology Systems 6:317-339.

Bowie, J., and G. O. Barnett, 1976, “MUMPS-An Economical and Efficient Time

Sharing System for Information Management,” Comp. Prog. Biomed, 6:11-22.

Boyle, D., 1978, “Developing S/UR: Strategy and Usage, ” Journal for Medicaid

Management 1:58-64.

Braithwaite, W. R., 1976, A Computerized Medical Data Base in Medical Care Review, San Francisco, Calif.: University of California, San Francisco.

Branscomb, L. M., 1979, “Information: The Ultimate Frontier, ” Scienc e

203:143-147.

Braunstein, M., J. R. Cohn, L. Chansky, and D. James, 1974, “Family Practice

Residents Learn ‘Computer,’ “ Patient Care, Mar. 1, 1974, pp. 111-120.

Breeze, M. S., and F. Schimpfhauser, 1977, “Computer-Simulated Clinical Encounters Part I: Development, Utilization and’ Evaluation, ” Journal of the

American Dietetic Association 70:382-388.

Brenner, M. H., and H. Paris, 1973, “Record Systems for Hospital Outpatient

Clinics, ” in J. H. Murnaghan, (cd.), Ambulatory Care Data: Report on Ambulatory Medical Care Record, Philadelphia, Pa.: Lippincott Co.

Brian, E., 1973, “Foundation For Medical Control of Hospitalization: CHAPPSRO Prototype,” New England Journal of Medicine 288:878-882.

Brook, R. H., 1973, “Critical Issues in the Assessment of Quality of Care and Their

Relationship to HMOs, ” Journal of Medical Education 48:114-134.

. and A. D. Avery, 1975, Quality Assurance Mechanisms in the United States:

From There to Where? Santa Monica, Calif.: Rand Corporation, Publ. No. P-5520.
42. ; and K. N. Williams, 1976a, “Effect of Medical Care Review on the Use of Injections, ” Annals of Internal Medicine 85:808-817.

43. —; K. N. Williams, and A. D. Avery, 1976b, “Quality Assurance Today and

Tomorrow: Forecast for the Future, ” Annals of Internal Medicine 85:808-817.

44. ; and K. N. Williams, 1977, Evaluation of the New Mexico Peer Review System, 1971 to 1973, Santa Monica, Calif.: Rand Corporation, Publ. No. R-2110-

HEW/RC.

45. Brown, J. M., and D. J. Weiss, 1977, An Adaptive Testing Strategy for Achievement Test Batteries (Research Report 77-6), Minneapolis, Minn.: University of

Minnesota Psychometric Methods Program, Department of Psychology.

46. Brown, J. S., 1975, “Uses of Artificial Intelligence and Advanced Computer Technology in Education, “ in R. Seidel and M. Rubin (eds.), Computers and Communications: Implications for Education, New York, N. Y.: Academic Press.

47. ; 1977, Diagnostic Models for Procedural Bugs in Basic Mathematical Skills

(ICAI Report 8), Cambridge, Mass.: Bolt, Beranek & Newman.

48. ; A. Collins, and G. Harris, 1977, Artificial Intelligence and Learning Strategies (ICAI Report 6), Cambridge, Mass.: Bolt, Beranek & Newman.

49. Brown, M. D., and W. B. Sale, 1974, “QAP, the AHA’s Program, ” Hospitals

48:59-62.

50. Buchanan, C. C., 1978, “Association for the Development of Computer Based Instructional Systems, Special Interest Group on Computer Based Training, ” Newsletter. July 1978, Clarion, Iowa: Allen Printing Co.

51. Buck, C. R., Jr., and K. R. White, 1974, “Peer Review: Impact of a System Based

on Billing Claims, ” New England Journal of Medicine 291:877-883.

52. Bukoski, W. J., and A. L. Korotkin, 1976, “Computing Activities in Secondary

Education, ” Educational Technology 16:9-23.

53. Casburgue, J. C., 1978, “Computer Assisted Instruction in Health Professions Education: Guidelines for Utilization, “ in E. C. Deland (cd.), lnformation Technology

in Health Science Education, New York, N. Y.: Plenum Press.

54. Christofell, T., and M. Loewenthal, 1977, “Evaluating the Quality of Ambulatory

Health Care: A Review of Emerging Methods,” Medical Care 15:877-897.

55. Clark, R. R., 1974, “The Impact of a Computerized Peer Review System on Continuing Medical Education, ” The Ohio State Medical ]ournal. December 1974, pp.

748-75.

56. Cohen, M. F. (cd.), 1974, Hospital Computer Systems, New York, N. Y.: Wiley

Biomedical-Health Publications.

57. Combs, B., 1973, “TYMNET: A Distributed Network, ” Datamation 19:40-44.

58. Cotner, L. E., and R. D. Vidmar, 1977, “A Hospital Peer Review System, ” Proceedings of the First Annual Symposium on Computer Applications in Medical

Care, Washington, D. C.: Institute of Electrical and Electronics Engineers, Inc., pp.

117-122.

59. Cramblett, H., R. E. Pengov, and G. L. Trzebiatowski, 1979, Computers in Health

Care, testimony for the Subcommittee on Domestic and International Scientific

Planning, Analysis and Cooperation, House Committee on Science and Technology, U.S. Congress. Washington, D. C.: U.S. Govt. Printing Office, No. 106.

60. Darby, C. A., A. L. Korotkin, and T. Romashko, 1970, Survey of Computing Activities in Secondary Schools (Final Report AIR (852)10/70), Washington, D. C.:

American Institutes for Research.

61. Davis, R. W., 1977, “Computer Instruction Basic Hardware Alternatives, ” NSPI

Journal 16:7-11.

62. Decker, B., and P. Bonner, 1973, PSRO: Organization for Peer Review, Cambridge, Mass.: Ballinger Publ. Co.
130

63.

64.

65.

66.

67.

68.

69.

70.

71.

72.

73.

74.

75.

76.

77.

78.

79.

80.

81.

82.

83.

● Computer Technology in Medical Education and Assessment

Deland, E. C. (cd.), 1978, Information Technology in Health Science Education,

New York, N. Y.: Plenum Press.

Dennis, V. E., 1977, “Interim Progress Report–The Phase II Physiology Self-Evaluation Unit’s ‘Physeu’ Program, ” Columbus, Ohio: The Ohio State University.

Denson, P. H., 1965, “The Quality of Medical Care, ” Yale ]ournal of Biology of

Medicine 37:523-536.

Dick, W., and R. Latta, 1969, Comparative Effects of Abiliyty and Presentation

Mode in CAI and Programmed Instruction (AD-694021), Springfield, Va.: Clearinghouse for Federal Scientific and Technical Information.

Dickinson, C. J., 1977, A Computer Model of Human Respiration, Lancaster, England: MIP Press Limited.

Dobson, A., J. G. Greer, R. H. Carlson, F. A. Davis, L. E. Kucken, B. J. Steinhardt, T. P. Ferry, and G. S. Adler, 1978, “PSROS: Their Current Status and Their

Impact to Date, “ Inquiry 15:113-128.

Donabedian, A., 1966, “Evaluating the Quality of Medical Care, ” Millbank

Memorial Fund Quarterly 44:166-206.

; 1976, “Measuring and Evaluating Hospital and Medical Care, ” Bulletin of the

New York Academy of Medicine 52:51-59.

; 1978, “The Quality of Medical Care, ” Science 200:856-864.

Doull, J., and E. Walaszek, 1978, “The Use of Computer-Assisted Teaching

Systems (CATS) in Pharmacology, ” in E. C. Deland (cd.), Information

Technology in Health Science Education, New York, N. Y.: Plenum Press.

E.D.S. Federal Corporation, 1976, Retrospective Analysis of Medical Services,

Systems Manual, Dallas, Tex.

Eisenberg, J. M., S. V. Williams, L. Garner, R. Viale, and H. Smits, 1977, “Computer-Based Audit to Detect and Correct Overutilization of Laboratory Tests, ”

Medical Care 15:915-921.

Ellis, J. A., 1978, “A Comparative Evaluation of CMI and IMI, ” Proceedings of the

Association for the Development of Computer Based Instructional Systems, pp.

393-397.

Etnyre, V. A., and E. M. Wolf, 1975, “Measuring the Educational Effectiveness of

Business Policy Courses Using Variable Modes of Computer Interface, ” Proceedings of the 1975 Conference on Computers in the Undergraduate Curricula

(CCUC/6), Fort Worth, Tex: Texas Christian University, Department of Computer Sciences, pp. 37-45.

Evans, L., 1978, personal communication from the Chairman, FLEX Committee, to

the Office of Technology Assessment, U.S. Congress.

Farquhar, B. B., E. P. Hoffer, and G. O. Barnett, 1978, “Patient Simulations in

Clinical Education, “ in E. C. Deland (cd.), Information Technology in Health Science Education, New York, N. Y.: Plenum Press.

Federspiel, C. S., W. A. Ray, and W. Schaffner, 1976, “Medicaid Records as a

Valid Data Source: the Tennessee Experience, ” Medical Care 14:166-172.

Fielden, J., and P. K. Pearson, 1978, Cost of Learning With Computers, London,

England: CET.

Fields, C., 1979, Department of Defense, Advanced Research Project Agency, personal communication to the Office of Technology Assessment, U.S. Congress.

Fincham, S. M., M. Grace, W. C. Taylor, E. N. Skakun, and F. C. Davis, 1976,

“Pediatric Candidates’ Attitudes to Computerized Patient Management Problems

in a Certifying Examination, “ Medical Education 10:404-4O7.

Flashner, B. A., S. Reed, R. W. Coburn, and P. R. Fine, 1973, “Professional Standards Review Organizations: Analysis of Their Development and Implementation
84.

85.

86.

87.

88.

89.

90.

91.

92.

93.

94.

95.

96.

97.

98.

99.

100.

101,

102.

References ● 1 3 1

Based on a Preliminary Review of the Hospital Admission and Surveillance Program in Illinois, “ journal of the American Medical Association 223:1473-1484.

Forman, M. H., R. E. Pengov, and J. L. Burson, 1978, “CAIREN: A Network for

Sharing Health Care Learning Resources With Ohio Health Care Facilities and

Educational Institutions, “ in E. C. Deland (cd.), information Technology in Health

Science Education, New York, N. Y.: Plenum Press.

Friedman, R. B., S. E. Newsom, S. Cheung, and J. Schaltz, 1977, “A Simulated

Patient-Physician Encounter Using a Talking Computer, ” Journal of the American

Medical Association 238:1927-1929.

01978, personal communication from the project director, Computer Based

Examination (CBX) Project, to the Office of Technology Assessment, U.S. Congress.

Gall, J. E. Jr., 1977, “The Patient Care Quality Assurance System, ” Proceedings of

the First Annual Symposium on Computer Applications in Medical Care,

Washington, D. C.: Institute for Electrical and Electronic Engineers, Inc., p. 116.

Garner, H., M. Anger, and W. Trigg, 1975, “Instructional Support Systems, ” in L.

Golubchick and B. Perski (eds.), Innovations in Education, Dubuque, Iowa:

Kendall-Hunt Pub]. Co.

Gaston, G. W., 1969, “Oral Cancer Recognition (ORALCA), ” in The Users’ Guide

to CAI at the Ohio State University College of Medicine, Columbus, Ohio: Ohio

State University.

Gerhold, G., and L. Kheriaty, 1978, “Common PILOT Progress Report, ” ADCIS

News 10(2) :36-40.

Giebink, G. A., and L. L. Hurst, 1975, Computer Projects in Health Care, Ann Arbor, Mich.: Health Administration Press.

Glantz, S. A., 1978, “Computers in Clinical Medicine: A Critique, ” Compute r

11(5):68-77.

Goldstein, R. L., J. J. Roberts, B. Stanton, D. B. Maglott, and M. J. Horan, 1975,

“Data for Peer Review: Acquisition and Use, Results in the Experimental Medical

Care Review Organization Program, “ Annals of Internal Medicine 82:262-267.

Gonnella, J. S., D. Z. Louis, and J. J. McCord, 1976, “The Staging Concept—An

Approach to the Assessment of Outcome of Ambulatory Care, ” Medical Care

14:13-21.

Goran, M. J., J. W. Williamson, and J. S. Gonnella, 1973, “The Validity of Patient

Management Problems,” Journal of Medical Education 48:171-177.

; J. S. Roberts, M. Kellogg, J. Fielding, and N. Jessie, 1975, “The PSRO

Hospital Review System, ” Medical Care 13(4):1-33 (Supplement).

Gordon, T. L., 1978, “Applicants for 1977-78 First Year Medical School Class

(Datagram), ” journal of Medical Education 53:696-698.

Graettinger, J., 1977, National Residency Matching Program Annual Report,

Washington, D. C.: Association of American Medical Colleges.

Grayson, L. P., and J. B. Robbins, 1972, U.S. Office of Education Support of Computer Projects, 2965-1972, Washington, D. C.: U.S Govt. Printing Office, Publ.

No. HEW 5.212:12044-71.

Green, R. (cd.), 1976, Assuring Quality in Medical Care: The State of the Art,

Cambridge, Mass.: Ballinger Publ. Co.

Gregory, R. J., 1975, “Outcome: A Computer Simulation of the Results of Psychotherapy and Its Use in the Psychology Research Course, ” Proceedings of the 1975

Conference on Computers in the Undergraduate Curricula (CCUC/6), Fort Worth,

Tex.: Texas Christian University, Department of Computer Sciences, pp. 213-217.

Group for Research in Pathology Education (GRIPE), 1974, Annual Meeting

Report to Members, Omaha, Nebr.
1030

104.

105.

106.

107.

108.

109.

110.

111.

112.

113.

114.

115.

116.

117.

118.

119,

120.

121.

Guerra, C., H. McDaniel, J. Kaufman, and D. Fuller, 1977, “CASPER: An Interactive Approach to Computer-Managed Testing, ” Proceedings of the Association

for the Development of Computer Based Instructional Systems, pp. 39-54.

Halverson, J. D., and W. F. Ballinger, 1978, “Computer Assisted Instruction in

Surgery,” Surgery 83:633-638.

Hamblen, J. W., 1972, Inventory of Computers in U.S. Higher Education,

1969-1970: Utilization and Related Degree Programs, Washington, D. C.: U.S.

Govt. Printing Office.

Hansen, D. N., B. F. Johnson, R. L. Fagan, P. Tam, and W. Dick, 1974, ComputerBased Adaptive Testing Models for the Air Force Technical Training Environment,

Phase Z: Development of a Computerized Measurement System for Air Force Technical Training (AFHRL.-TR-74-48), Bethesda, Md.: Educational Resource Information Center (ERIC, ED 094 759).

Harless, W. G., G. G. Drennon, J. J. Marxer, J. A. Root, and G. E. Miller, 1971,

“CASE: A Computer-Aided Simulation of the Clinical Encounter, ” Journal of

Medical Education 46:443-448.

—; G. G. Drennon, J. J. Marxer, J. A. Root, L. L. Wilson, and G. E. Miller,

1973a, “CASE-A Natural Language Computer Model,” Computers in Biology

and Medicine 3:227-246.

 G. G. Drennon, J. J. Marxer, J. A. Root, L. L. Wilson, and G. E. Miller,

1973b, “GENESYS-A Generating System for the CASE Natural Language

Model,” Computers in Biology and Medicine 3:247-268.

; 1975, “MERIT-A Model for Evaluation and Recertification Through Individualized Testing, ” Proceedings of the 14th Annual Conference on Research in

Medical Education, Washington, D. C.: Association of American Medical Colleges.

; N. A. Farr, M. A. Zier, and J. R. Gramble, 1978a, “MERIT-An Application

of CASE, ” in E. C. Deland (ed.), Information Technology in Health Science Education, New York, N. Y.: Plenum Press.;

— ,; N. A. Farr, M. Z. Zier, and J. R. Gramble, 1978b, “A Method for Physician

Recertification, ” Technological Horizons in Education Journal 5(6):51-55.

Health Care Management Systems, Inc., 1976, Interim Evaluation Report: New

Mexico Professional Standards Review Organization Revised Medicaid Ambulatory Care Review System, app. A. La Jolla, Calif.: Health Care Management

Systems, Inc.

; 1977, Physician Ambulatory Care Evaluation Program, vols. 1-8. La Jolla,

Calif.: Health Care Management Systems, Inc.

; 1978, “Computer Technology in Medical Care Quality Assurance, ” paper

prepared for the Office of Technology Assessment, U.S. Congress.

Hickey, A. E., 1975, Computer-Assisted Instruction, Newburyport, Mass.:

ENTELEK.

Hody, G. L., and R. A. Avner, 1978, “The PLATO System: An Evaluative Description, ” in E. C. Deland (ed.), Information Technology in Health Science Education, New York, N. Y.: Plenum Press.

Hoffer, L. P., G. O. Barnett, B. B. Farquhar, and P. A. Prather, 1975, “Computer

Aided Instruction in Medicine, ” Annual Review of Biophysics and Bioengineering

4:103-117.

Holloway, D. C., L. J. Wiczai, and E. T. Carlson, 1974, “Evaluating an Information System for Medical Care Evaluation Studies, ” Medical Care 13:329-340.

Hubbard, J., E. Levit, C. Schumaker, and T. Schnabel, 1965, “An Objective Evaluation of Clinical Competence, “ New England Journal of Medicine 272:1321-1328.

Huber, G. A., H. Wolfe, and C. P. Hardwick, 1974, “Evaluating Computerized

Screening as an Aid to Utilization Review, ” Inquiry 11:118-195.
References ● 1 3 3

122.

123.

124.

125.

126.

127.

128.

129.

130.

131.

132.

133.

134.

135.

136.

137.

138.

139.

140.

141.

142.

Human Resources Research Organization, 1974. Project data base on computer

programs that had been designed for teaching and testing. Of 4,900 computer programs that had been developed for the purpose of teaching or testing (CAI and

CAT), table 1 includes 2,750 of these programs.

; 1978, “Use of Computers to Improve Professional Assessment, ” paper prepared for the Office of Technology Assessment, U.S. Congress.

Institute of the Future, 1979, Teletext in the United States: A Study of Public Service Potential and Policy Issues, informal paper presented to the Office of Technology Assessment, U.S. Congress, Menlo Park, Calif.

Jacobs, C. M., T. H. Christoffel, and N. Dixon, 1976, Measuring the Quality of

Patient Care: The Rationale for Outcome Audit, Cambridge, Mass.: Ballinger

Publ. Co.

Johnson, M. E., and L. L. Leveridge, 1978, The Significance of the Optical Videocisc to Education, internal document, American Medical Association, Division of

Educational Policy and Development.

Johnson, S. L., 1976, “Simulators for Sensor Operator Training,” Proceedings of

the First International Learning Technology Congress and Exposition in Applied

Learning Technology, v. IV. Buffalo, N. Y.: Calspan Graphics, pp. 19-23.

Kamp, M., 1975, Index to Computerized Teaching in the Health Sciences, San

Francisco, Calif.: University of California, San Francisco.

Kessner, D. M., C. E. Kalk, and J. Singer, 1973, “Assessing Health Quality—The

Case for Tracers, ” New England Journal of Medicine 288:189-194.

Kimberlin, D. A., 1973, A Preliminary Instructional Model for a Computerized

Training System, Army Signal Center and School, Ft. Monmouth, N. J., Bethesda,

Md.: Educational Resource Information center (ERIC ED 083 822).

Kopstein, F., and R. Seidel, 1968, “Computer-Administered Instruction Versus

Traditionally Administered Instruction: Economics, ” Audio Visual Communications Review 16:147-175.

LaPerle, E. S., and E. G. McNulty, 1974, “Data Set Provides Uniformity, Flexibility, ” Hospitals 48(12):115-120.

Lawrence, S. B., 1978, “INTERNIST,” Forum on Medicine 1(4):44-47.

Lefever, R. D., and J. K. Johnson, 1976, Survey of the Use of Computers in Instruction, Washington, D. C.: Association of American Medical Colleges.

Lembcke, P. A., 1956, “Medical Auditing by Scientific Methods, ” Journal of the

American Medical Association 162:646-655.

; 1967, “Evolution of the Medical Audit, ” Journal of the American Medical

Association 199:111-118.

Lewis, C. E., 1974, “The State of the Art of Quality Assessment—1973, ” Medical

Care 12:799-806.

Lippey, G., and N. Partos, 1976, “Improvements to a Computer-Assisted Test

Construction System, “ Educational Technology 16(12):26-29.

Long, H., 1978, personal communication from the IBM Corporation to the Human

Resources Research Organization in preparation for a paper for the Office of Technology Assessment, U.S. Congress.

Lyons, T. F., and B. C. Payne, 1974, “The Relationship of Physicians’ Medical Recording Performance to Their Medical Care Performance, ” Medical Care

12:463-469.

Magraw, R. M., 1976, “Formal Quality Assessment and Utilization Review Programs: Their Effects on the Basic Transactions in Medical Education, ” Bulletin of

the New York Academy of Medicine 52:105-118.

Martin, J. B., 1978, “Toward Effective Computer-Aided Medical Care Evaluation

Studies, ” Inquirv 15:139-141.

“
143.

144.

145.

146,

147.

148.

149.

150.

151.

152.

153.

154.

155.

156.

157.

158.

159.

160.

Massachusetts General Hospital, 1978, memorandum from the Laboratory of

Computer Sciences to the Board of Directors, Health Education Network, Inc.

McClain, D. H., S. W. Wessels, and K. M. Sando, 1975. “IPSIN-Additional Systems Enhancement Utilized in a Chemistry Application, ” Proceedings Of the 1975

Conference on Computers in the Undergraduate Curricula (CCUC/6), Fort Worth,

Tex.: Texas Christian University, Department of Computer Sciences, pp. 139-145.

McDonald, C. J., 1976, “Protocol Based Computer Reminders: The Quality of

Care and the Nonperfectability of Man, ” New England Journal of Medicine

295:1351-1355.

; R. Murray, D. Jeris, B. Bhargava, J. Seegar, and L. Blevins, 1977, “A

Computer-Based Record and Clinical Monitoring System for Ambulatory Care, ”

American Journal of Public Health 67:240-245.

Merchant, F. T., 1978, personal communication from the Permanent Secretary of

the FLEX Examination Board to the Office of Technology Assessment, U.S. Congress.

Merola, A. J., R. E. Pengov, and B. T. Stokes, 1978, “Computer-Supported Independent Study in the Basic Medical Sciences, “ in E. C. Deland (cd.), lnformation

Technology in Health Science Education, New York, N. Y.: Plenum Press.

Mesel, E., and D. D. Wirtschafter, 1976a, “Automation of a Patient Medical Profile From Insurance Claims Data: A Possible First Step in Automating Ambulatory

Medical Records on a National Scale, ” Milbank Memorial Fund Quarterly/Health

and Society 54:29-45.

. and D. D. Wirtschafter, 1976b. “Automating Ambulatory Medical Records:

A Claims-Based Medical Profile, ” Computers and Biomedical Research 9:89-91.

Meskauskas, J., and J. J. Norcini, 1978, Progress Report on the Data Analysis of

the MERIT—Recertification Study, Philadelphia, Pa.: American Board of Internal

Medicine.

Meyer, T., 1978, State of the Art of Measurement of Physician Competence, draft

of a paper presented to the Coordinating Council on Medical Education.

Milliken Communications Corporation, 1979, personal communication from

Bodie Marx to the Office of Technology Assessment, U.S. Congress.

Mills, R. R. B. Fetter, D. C. Riedel, and R. Averill, 1976, “AUTOGRP: An Interactive Computer System for the Analysis of Health Care Data, ” Medical Care

14:603-615.

Misselt, A. L., and E. Call-Himwich, 1978, Analysis of the Shepard Air Force Base

Computer-Based Education Project (MTC Report No. 21), Urbana, Ill,: University

of Illinois, Computer-Based Education Research Laboratory.

Mitzel, H. E. (cd.), 1974, An Examination of the Short-Range Potential of

Computer-Managed Instruction, conference sponsored by the National Institute of

Education, Chicago, Ill., November 1974.

Mockovak, W. P. et al., 1974, The Application and Evaluation of PLATO IV in Air

Force Technical Training, Lowry Air Force Base, Bethesda, Md.: Education

Research Information Center (ERIC Ed 096 957).

Molnar, A. R., 1975, “Viable Goals for New Educational Technology Efforts: Science Education and the New Technological Revolution, ” Educational Technology

15:4.

; 1978, “The Next Great Crisis in American Education: Computer Literacy, ”

The Journal 5(4):35-38.

Morehead, M. A., 1976, “Ambulatory Care Review: A Neglected Priority, ”

Bulletin of the New York Academy of Medicine 52:60-69.
161.

162.

163.

164.

165.

166.

167.

168.

169.

170.

171.

172.

173.

174.

175.

176.

177.

178.

179.

180,

181.

References ● 1 3 5

Morgan, J. P., 1977, “Watching the Monitors: “Paid” Prescriptions, Fiscal Intermediaries and Drug-Utilization Review, ” New England Journal of Medicine

296:251-256.

Mourad, L., and M. Forman, 1976, The Role of CAI in Nursing Education at The

Ohio State University, internal document, Ohio State University School of Nursing.

Murnaghan, J. H., and K. L. White, 1971, “Hospital Patient Statistics, Problems,

and Prospects, “ New England ]ournal of Medicine 284:822-828.

; 1974, “Health-Services Information Systems in the United States Today, ”

New England Journal of Medicine 290:603-610.

National Academy of Sciences, Institute of Medicine, 1976, Assessing Quality in

Health Care: An Evaluation, Washington, D. C.: National Academy of Sciences.

National Board of Medical Examiners, 1973, Evaluation in the Continuum of Medical Education. Report of the Committee on Goals and Priorities of the National

Board of Medical Examiners. Philadelphia, Pa.: National Board of Medical Examiners.

; ;1975, 1975 Bulletin of Information, Philadelphia, Pa.: National Board of

Medical Examiners.

— , ;. 1979, Annual Report—1978, Philadelphia, Pa.: National Board of Medical

Examiners.

Nelson, A. R., 1976, “Relation Between Quality Assessment and Utilization Review in a Functioning PSRO, ” New England Journal of Medicine 292:671-675.

 — ; and J. Q. Cannon, 1976 , “The PACE Project: Computer-Assisted Peer

of Ambulatory Services, ” Proceedings of a Conference Assessing Physician Performance in Ambulatory Care, San Francisco, Calif.: American Society of Internal

Medicine.

Nevison, J. M., 1970, The Computer as Pupil: The Dartmouth Secondary School

Project, Hanover, N. H.: Kiewit Computation Center.

Norton, S., J. Doull, and E. J. Walaszek, 1972, “A Computer Assisted Teaching

System for Medical Pharmacology, ” presented at the Fifth International Congress

of Pharmacology, San Francisco, Calif.

Norwood, D. D., 1977, “A Patient Care Quality Assurance System Overlayed on a

Hospital Wide Medical Information System, ” in D. B. Shires and H. Wolff (eds.),

Med. Info. ’77, Amsterdam, The Netherlands: North Holland Publ. Co.

Noyce, R., 1977, “Microelectronics,” Scientific American 237:62-69.

Nuthmann, C. F., 1968, Time-Shared Interactive Computer-Controlled Information Television (TICCIT), MTP-307, Washington, D. C.: The MITRE Corporation.

Nyborg, P. S., P. M. McCarter, and W. E. Erickson, (eds.), 1977, Information

Processing in the United States: A Quantitative Summary, Montvale, N. J.: American Federation of Information Processing Societies, Inc.

Ohio State University, 1976, “CAI Course Development Times, ” internal document, The Ohio State University College of Medicine.

 ;1977, “Rate Determination for CAI Access, ” The Ohio State University College of Medicine.

— ; 1978, MERIT—A Model for Evaluation and Recertification Through Individualized Testing: A Final Report, The Ohio State University College of Medicine,

Division of Computing Services for Medical Education and Research.

Orlansky, J., and J. String, 1977, Cost-Effectiveness of Flight Simulators for Military Training, Arlington, Va.: Institute for Defense Analysis.

Papert, S., 1975, “A Learning Environment for Children, ” in R. Seidel and N.

Rubin (eds.), Computers and Communications: Implications for Education, New

York, N. Y.: Academic Press.
136 • Computer Technology in Medical Education and Assessment

182.

183.

184.

185.

186.

187.

188.

189.

190.

191.

192.

193.

194.

195.

196.

197.

198.

199.

200.

Patience, W. M., 1977, “Description of Components in Tailored Testing, ” Behavior Research Methods & Instrumentation 9:153-157.

Pengov, R. E., 1974, “A Modularly Designed Computer-Assisted Instruction Reporting System (CAIRS),” in Med. Info. ’74, Amsterdam, The Netherlands: North

Holland Publ. Co.

 ; 1978, “The Evolution and Use of Computer-Assisted Instruction (CAI) in

Health Sciences Education at The Ohio State University College of Medicine, ” in E.

C. Deland (cd.), Information Technology in Health Science Education, New York,

N. Y.: Plenum Press.

Peters, H. W., 1976, “The Electronic Aristotle,” Computer Decisions 8:42-46.

Pierleoni, R. G., 1978, “Clinical Evaluation Techniques for the Health Professions, ” Improving Human Performance Quarterly 7:204-216.

Pine, S. M., and D. J. Weiss, 1976, Effects of Item Characteristics on Test Fairness

(Research Report 76-5), Minneapolis, Minn.: University of Minnesota Psychometric Methods Programs, Department of Psychology.

Practical Concepts, Inc., 1975, Practical Concepts for Using Interactive TV Applications in the Medical Community, Washington, D. C.: Practical Concepts, Inc.

Puig, G. A., 1976, “The Training Effectiveness of Simulation, ” Proceedings of the

First International Learning Technology Congress and Exposition in Applied

Learning Technology, v. IV, Buffalo, N. Y.: Calspan Graphics, pp. 28-32.

Reichert, A. K., and N. J. Greenberg, “The Application of Simulation Techniques

to Enhance Student Understanding, ” Proceedings of the Association for the Development of Computer Based Instructional Systems, pp. 364-378.

Rimoldi, H. J. A., 1961, “The Test of Diagnostic Skills, ” Journal of Medical Education 36:73-79.

Roberts, E. A., 1977, “In-House Computer for Training?” NSPI Journal 16(9):7-11.

Rosati, R. A., J. F. McNeer, C. F. Starmer, B. S. Mittler, J. J. Morris, Jr., and A. G.

Wallace, 1975, “A New Information System for Medical Practice, ” Archives of Internal Medicine 135:1017-1024.

Sanazaro, P. J., R. L. Goldstein, J. S. Roberts, D. B. Maglott, and J. W. McAllister,

1972, “Research and Development in Quality Assurance, the Experimental Medical

Care Review Organization Program, ” New England Journal of Medicine

287:1125-1131,

. and R. M. Worth, 1978, “Concurrent Quality Assurance in Hospital Care, ”

New England Journal of Medicine 298:1171-1177.

Sayetta, R. B., 1976, “Critique of an Earlier Study of the Sacramento Medical Care

Foundation’s Certified Hospital Admission Program (CHAP),” Medical Care

14:80-90.

Schoolman, H. M., and L. M. Bernstein, “Computer Use in Diagnosis, Prognosis,

and Therapy, “ Science 200:926-931.

Schumacher, C. F., F. D. Burg, and W. G. Taylor, 1975, “Computerization of a

Patient Management Problems Examination to Prevent “Retracing’ ’,” British Journal of Medical Education 9:281-285.

Seely, O., and V. Willis, 1975, “Socrates Test Retrieval at the California State University and Colleges, ” Proceedings of the 1975 Conference on Computers in the

Undergraduate Curricula (CCUC/6), Fort Worth Tex.: Texas Christian University,

Department of Computer Sciences, pp. 135-138.

Segall, A., D. Benor, and O. Susskind, 1978, “Consonance Between instruction

and Evaluation in Medical-Education—Beersheva Experience, ” Improving Human

Performance Quarterly 7:189-196.
201. Seidel, R. J., B. Hunter, C. Kastner, and M. Rubin, 1974, “Obstacles to Widespread Use of Computer-Based Curricula Innovations: A Review, ” Journal of

Computer-Based Instruction 1:28-31.

202. “ and M. Rubin (eds.), 1975, Computers and Communications: Implications

for Education, New York, N. Y.: Academic Press.

203. ; — , and L. Stolorow, 1975, A CMI Heuristic for the Services and ARPA, paper

submitted to Educational Testing Services, Princeton, N.J.

204. — ; and H. Wagner, 1977, Cost Effectiveness Specification for Computer Based

Training Systems, III, Alexandria, Va.: Human Resources Research Organization.

205. : — , R. Rosenblatt, H. Wagner, R. Schultz, and B. Hunter, 1978, Evaluation of a

Prototype Computerized Training System (CTS) in Support of Self-Pacing and

Management of Instruction (Final Report), Alexandria, Va.: Human Resources Research Organization.

206. Senior, J., 1976. Toward the Measurement of Competence in Medicine, The

Computer-Based Examination (CBX) Project, Philadelphia, Pa.: National Board of

Medical Examiners and the American Board of Internal Medicine.

207. Sherwood, B. A., and J. Stifle, 1975, The PLATO IV Communications System

(CERL Report X-44), Urbana, Ill.: University of Illinois.

208. Simborg, D. W., B. H. Starfield, S. D. Horn, and S. A. Yourtree, 1976, “Information Factors Affecting Problem Follow-Up in Ambulatory Care, ” Medical Care

14:848-856.

209. Skakun, E. N., W. C. Taylor, and D. R. Wilson, 1977, “A Follow-Up Study of the

Computerized Patient Management Problem Examination in Pediatrics, ” in D. B.

Shires, and H. Wolff (eds.), Med. Info., ’77, Amsterdam, The Netherlands: North

Holland Publ. Co.

210. — ; W. C. Taylor, and D. R. Wilson, 1978, “Computerized Patient Management

Problem as Part of the Pediatric Certifying Examination, ” manuscript submitted to

the Journal of Computer-Based Instruction.

211. ; 1978, personal communication to the Office of Technology Assessment, U.S.

Congress.

212. Slee, V. N., 1970, “Professional Activity Study, ” Medical Care 8(4):34-40.

213. Sorlie, W. E., and L. A. Jones, 1975, “Descriptions of a Computer-Assisted Testing

System in an Independent Study Program, Journal of Medical Education 50:81-83.

214. St. John, D. B., D. R. Dobin, and B. H. Flashner, 1973, “An Automated System for

the Regulation and Medical Review of Long-Term Care Facilities and Patients, ”

American Journal of Public Health 63:619-630.

215. Starkweather, J. A., 1971, “Educational Networks and Interfacing With the University Computing Center, ” in Computer Applications in Dental Education, San

Francisco, Calif.: University of California, San Francisco.

216. Sterling, T. D., 1975, “Humanizing Computerized Information Systems, ” Science

190:1168-1172.

217. Stetten, K. J., 1972, Toward a Market Success for CAI: An Overview of the TICCIT Program (M72-73), Washington, D. C.: The MITRE Corporation.

218. Suppes, P., and M. Morningstar, 1969, “Computer-Assisted Instruction, ” Science

166:343-350.

219. Tennyson, C. L., 1977, “A Look at the CAI Systems Market,” NSPI Journal

16(9):10-12.

220. Tidball, C. S., 1976, survey summary (mimeo) of R. D. Lefever and J. K. Johnson,

1976, Survey of the Use of Computers in Instruction, Washington, D. C.: Association of American Medical Colleges.
138 ●

221.

222.

223.

224.

225.

226.

227.

228.

229.

230.

231.

232.

233.

234.

235.

236.

237.

238.

239.

Computer Technology in Medical Education and Assessment

; 1978a, “Health Education Network, ” in E. C. Deland (cd.), lnformation

Technology in Health Science Education, New York, N. Y.: Plenum Press.

; 1978b, “Using Minicomputers for Courseware Delivery, ” in E.C. Deland

(cd.), lnformation Technology in Health Science Education, New York, N. Y.:

Plenum Press.

“Trends in Computer-Assisted Instruction, ” 1978, Educational Technology

18(4) :April issue.

U.S. Congress, Office of Technology Assessment, 1977, Policy Implications of

Medical lnformation Systems, Washington, D. C.: U.S. Govt. Printing Office.

; House of Representatives, 1978, Computers and the Learning Society, report

prepared by the Subcommittee on Domestic and International Scientific Planning,

Analysis, and Cooperation of the Committee on Science and Technology, 95th

Cong., 2d sess.

U.S. Department of Health, Education, and Welfare, 1973, Medicaid Management

Information System, General System Design for Title 19, vols. 1-5, Washington,

D.C.

 ;1975, Federal Reports Manual, PSRO Management Information System,

1975, Health Care Financing Administration, Washington, D.C.

; ;1977, Automation of the Problem-Oriented Medical Record, Washington,

D. C.: National Center for Health Services Research, Health Resources Administration, DHEW Publ. No. (HRA) 77-3177.

 ;1978a, Federal Support for Health Professions Education, options paper, Office of the Assistant Secretary for Health, Office of Health Policy, Research, and

Statistics.

 ;1978b, Advance Data, Vital and Health Statistics, no. 30, National Center

Health Statistics.

Vale, C. B., 1977, Applications of Computerized Adaptive Testing (Research Report 77-1), Minneapolis, Minn.: University of Minnesota Psychometric Methods

Program, Department of Psychology.

Van Matre, N., 1978, “The Navy’s CMI System, ” Proceedings of the Association

for the Development of Computer Based Instructional Systems, pp. 269-270.

Voss, G. et al., 1978, “Computer-Assisted Instruction in Medicine: A German

View, ” Proceedings of the Association for the Development of Computer Based Instructional Systems, pp. 63-72.

Votaw, R. G., and B. B. Farquhar, 1978, “Current Trends in Computer-Based Education in Medicine, ” Educational Technology 18:54-56.

Webb, S. B., Jr., M. C. Robson, and A. L. Lanckton, 1975, “A Monitoring Strategy for Quality Assurance Programs, “ Connecticut Medicine 39:253-256.

Weiss, D. J., 1974, Strategies of Adaptive Ability Measurement (Research Report

74-5), Minneapolis, Minn.: University of Minnesota Psychometric Program,

Department of Psychology.

—; (cd.), 1975, Computerized Adaptive Trait Measurement: Problems and Prospects. Proceedings of a Symposium Presented at the Convention of the American

Psychological Association (Research Report 75-5), Minneapolis, Minn.: University

of Minnesota Psychometric Methods Program, Department of Psychology.

; 1976, Final Report: Computerized Ability Testing, 1972-1975, Minneapolis,

Minn.: University of Minnesota Psychometric Methods Program, Department of

Psychology.

Welch, C. E., 1973, “Professional Standards Review Organizations—Problems and

Prospects, ” New England Journal of Medicine 289:291-295.
240.

241.

242.

243.

244.

245.

246.

White, N. H., 1976, Ambulatory Care Quality Assurance Projects, vols. 1-3, La

Jolla, Calif.: Health Care Management Systems.

Willey, L. V., 1975, Computers and Instructional Productivity, Bethesda, Md.:

IBM Corporation.

Williamson, J. W., 1977, Improving Medical Practice and Health Care: A Bibliographic Guide to Information Management in Quality Assurance and Continuing

Education, Cambridge, Mass.: Ballinger Publ. Co.

; 1978, “Formulating Priorities for Quality Assurance Activity, Description of

a Method and Its Application, ” Journal of the American Medical Association

239:631-637.

Wismar, B. L., and G. R. Christopher, 1969, “Computerized History, ” in Proceedings of the 8th Annual Conference on Research in Medical Education, Washington,

D. C.: Association of American Medical Colleges.

Wooster, H., 1976, “An Experiment in Networking: The LHNCBC Experimental

CAI Network, 1971 -1975,” Journal of the Americal Society for Information Science

27:329-338.

Zimmerman, J., 1978, “Physician Utilization of Medical Records: Preliminary

Determinations, ” Medical Informatics 3:27-35.

U s. GOVERNMENT PRINTING OFFICE : 1979 0 - 51-541