Evidence Report/Technology Assessment 

Number 132 

Costs and Benefits of Health Information Technology 

Prepared for: 

Agency for Healthcare Research and Quality 

U.S. Department of Health and Human Services 

540 Gaither Road 

Rockville, MD 20850 

www.ahrq.gov 

Contract No. 290-02-0003 

Prepared by: 

Southern California Evidence-based Practice Center, Santa Monica, CA 

Center Directors 

Paul Shekelle, M.D., Ph.D. 

Sally C. Morton, Ph.D. 

Mathematician 

Emmett B. Keeler, Ph.D. 

Content Experts 

Jerome K. Wang, M.D. 

Basit I. Chaudhry, M.D. 

Cost Effectiveness Analyst 

Shin-Yi Wu, Ph.D. 

Physician Reviewers 

Walter A. Mojica, M.D., M.P.H. 

Basit I. Chaudhry, M.D. 

Project Director 

Margaret Maglione, M.P.P. 

Programmer 

Elizabeth A. Roth, M.A. 

Staff Assistants 

Cony Rolon, B.A. 

Di Valentine, J.D. 

Librarian 

Roberta Shanman, M.L.S. 

Medical Editor 

Sydne J. Newberry, Ph.D. 

AHRQ Publication No. 06-E006 

April 2006
This report is based on research conducted by the Southern California Evidence-based 

Practice Center (EPC), under contract to the Agency for Healthcare Research and 

Quality (AHRQ), Rockville, MD (Contract No. 290-02-0003). The findings and 

conclusions in this document are those of the author(s), who are responsible for its 

content, and do not necessarily represent the views of AHRQ. No statement in this 

report should be construed as an official position of AHRQ or of the U.S. Department of 

Health and Human Services. 

The information in this report is intended to help clinicians, employers, policymakers, 

and others make informed decisions about the provision of health care services. This 

report is intended as a reference and not as a substitute for clinical judgment. 

This report may be used, in whole or in part, as the basis for the development of clinical 

practice guidelines and other quality enhancement tools, or as a basis for reimbursement 

and coverage policies. AHRQ or U.S. Department of Health and Human Services 

endorsement of such derivative products may not be stated or implied. 
 ii

This document is in the public domain and may be used and reprinted without permission except 

those copyrighted materials noted for which further reproduction is prohibited without the 

specific permission of copyright holders. 

Suggested Citation: 

Shekelle PG, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. 

Evidence Report/Technology Assessment No. 132. (Prepared by the Southern California 

Evidence-based Practice Center under Contract No. 290-02-0003.) AHRQ Publication No. 

06-E006. Rockville, MD: Agency for Healthcare Research and Quality. April 2006. 

No investigators have any affiliations or financial involvement (e.g., employment, 

consultancies, honoraria, stock ownership or options, expert testimony, grants or 

patents received or pending, or royalties) that conflict with material presented in the 

report.
 iii

Preface 

 The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based 

Practice Centers (EPCs), sponsors the development of evidence reports and technology 

assessments to assist public- and private-sector organizations in their efforts to improve the 

quality of health care in the United States. This report was requested and funded by the Office of 

the Assistant Secretary for Planning and Evaluation (ASPE) with additional funding from the 

Office of Disease Prevention and Health Promotion (ODPHP), U.S. Department of Health and 

Human Services. In addition, the report was requested by the Leap Frog Group and the Centers 

for Medicare & Medicaid Services (CMS). The reports and assessments provide organizations 

with comprehensive, science-based information on common, costly medical conditions and new 

health care technologies. The EPCs systematically review the relevant scientific literature on 

topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to 

developing their reports and assessments. 

 To bring the broadest range of experts into the development of evidence reports and health 

technology assessments, AHRQ encourages the EPCs to form partnerships and enter into 

collaborations with other medical and research organizations. The EPCs work with these partner 

organizations to ensure that the evidence reports and technology assessments they produce will 

become building blocks for health care quality improvement projects throughout the nation. The 

reports undergo peer review prior to their release. 

 AHRQ expects that the EPC evidence reports and technology assessments will inform 

individual health plans, providers, and purchasers, as well as the health care system as a whole, 

by providing important information to help improve health care quality. 

 We welcome written comments on this evidence report. They may be sent by mail to the 

Task Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither 

Road, Rockville, MD 20850, or by e-mail to epc@ahrq.gov.

Carolyn M. Clancy, M.D. 

Director 

Agency for Healthcare Research and Quality 

Jean Slutsky, P.A., M.S.P.H. 

Director, Center for Outcomes and Evidence 

Agency for Healthcare Research and Quality 

CAPT Penelope Royall, P.T., M.S.W. 

Director 

Office of Disease Prevention and 

 Health Promotion 

Office of Public Health and Science 

Office of the Secretary 

Michael J. O’Grady, Ph.D. 

Assistant Secretary for Planning and Evaluation

U.S. Department of Health and Human Services

Beth A. Collins Sharp, R.N., Ph.D. 

Acting Director, EPC Program 

Agency for Healthcare Research and Quality 

Margaret Coopey, R.N., M.G.A., M.P.S. 

EPC Program Task Order Officer 

Agency for Healthcare Research and Quality

 v

Structured Abstract 

Objectives: An evidence report was prepared to assess the evidence base regarding benefits and 

costs of health information technology (HIT) systems, that is, the value of discrete HIT functions 

and systems in various healthcare settings, particularly those providing pediatric care.

Data Sources: PubMed®, the Cochrane Controlled Clinical Trials Register, and the Cochrane 

Database of Reviews of Effectiveness (DARE) were electronically searched for articles 

published since 1995. Several reports prepared by private industry were also reviewed. 

Review Methods: Of 855 studies screened, 256 were included in the final analyses. These 

included systematic reviews, meta-analyses, studies that tested a hypothesis, and predictive 

analyses. Each article was reviewed independently by two reviewers; disagreement was resolved 

by consensus. 

Results: Of the 256 studies, 156 concerned decision support, 84 assessed the electronic medical 

record, and 30 were about computerized physician order entry (categories are not mutually 

exclusive). One hundred twenty four of the studies assessed the effect of the HIT system in the 

outpatient or ambulatory setting; 82 assessed its use in the hospital or inpatient setting. Ninetyseven studies used a randomized design. There were 11 other controlled clinical trials, 33 studies 

using a pre-post design, and 20 studies using a time series. Another 17 were case studies with a 

concurrent control. Of the 211 hypothesis-testing studies, 82 contained at least some cost data. 

We identified no study or collection of studies, outside of those from a handful of HIT leaders, 

that would allow a reader to make a determination about the generalizable knowledge of the 

study’s reported benefit. Beside these studies from HIT leaders, no other research assessed HIT 

systems that had comprehensive functionality and included data on costs, relevant information 

on organizational context and process change, and data on implementation. 

A small body of literature supports a role for HIT in improving the quality of pediatric care. 

Insufficient data were available on the costs or cost-effectiveness of implementing such systems. 

The ability of Electronic Health Records (EHRs) to improve the quality of care in ambulatory 

care settings was demonstrated in a small series of studies conducted at four sites (three U.S. 

medical centers and one in the Netherlands). The studies demonstrated improvements in provider 

performance when clinical information management and decision support tools were made 

available within an EHR system, particularly when the EHRs had the capacity to store data with 

high fidelity, to make those data readily accessible, and to help translate them into contextspecific information that can empower providers in their work. 

Despite the heterogeneity in the analytic methods used, all cost-benefit analyses predicted 

substantial savings from EHR (and health care information exchange and interoperability) 

implementation: The quantifiable benefits are projected to outweigh the investment costs. 

However, the predicted time needed to break even varied from three to as many as 13 years. 

Conclusions: HIT has the potential to enable a dramatic transformation in the delivery of health 

care, making it safer, more effective, and more efficient. Some organizations have already 

realized major gains through the implementation of multifunctional, interoperable HIT systems 

built around an EHR. However, widespread implementation of HIT has been limited by a lack of 
 vi

generalizable knowledge about what types of HIT and implementation methods will improve 

care and manage costs for specific health organizations. The reporting of HIT development and 

implementation requires fuller descriptions of both the intervention and the 

organizational/economic environment in which it is implemented. 
vii

Contents 

Executive Summary...........................................................................................................................1 

Evidence Report.............................................................................................................................11

Chapter 1. Introduction ....................................................................................................................11 

 A Framework for Considering the Costs and Benefits of Health Information Technology......12 

 Elements of the Business Case ..................................................................................................12 

 What Is Generalizable Knowledge Regarding Health Information Technology?.....................14 

Chapter 2. Methods..........................................................................................................................17 

 Original Proposed Key Questions..............................................................................................17 

 Technical Expert Panel ..............................................................................................................17 

 Literature Search........................................................................................................................18 

 Additional Sources of Evidence.................................................................................................19 

 Article Review ...........................................................................................................................19 

 Selection of Articles and Data Elements for Interactive Database......................................20 

 Synthesis of Results ...................................................................................................................21 

 Peer Review ...............................................................................................................................22 

Chapter 3. Results ............................................................................................................................23 

 Figure 1. HIT Literature Flow ...................................................................................................24 

 Description of the Studies..........................................................................................................24 

 The Costs and Benefits of Health Information Technology in Pediatrics .................................26 

 Introduction..........................................................................................................................26 

 Literature..............................................................................................................................26 

 Summary of Evidence..........................................................................................................27 

 Summary..............................................................................................................................30 

 Conclusion ...........................................................................................................................30 

 Electronic Health Records and Quality of Ambulatory Care ....................................................31 

 Introduction..........................................................................................................................31 

 Research Study Inclusion Criteria .......................................................................................31 

 Analytic Framework ............................................................................................................32 

 Analysis................................................................................................................................33 

 Conclusions..........................................................................................................................41 

 Economic Value of an Electronic Health Record System .........................................................41 

 Summary of Key Findings from Non–Financially Focused Studies ...................................42 

 Analytic Methods to Assess the Economic Value of an EHR System ................................43 

 Evidence of Economic Costs and Benefits of an EHR System ...........................................45 

 Table 1. Summary of Studies Assessing the Economic Impact of an EHR System............51 

 Health Information Technology and Patient Centeredness .......................................................54 

 Barriers to Implementation ........................................................................................................56 
viii

Chapter 4. Discussion ......................................................................................................................59 

 Limitations .................................................................................................................................59 

 Conclusions................................................................................................................................60 

 General Conclusions ............................................................................................................60 

 Pediatrics....................................................................................................................................60 

 EHRs and the Quality of Ambulatory Care ...............................................................................61 

 Economic Value of an EHR System..........................................................................................61 

Recommendations for Future Research...........................................................................................62 

 Recommendations......................................................................................................................62 

 Recommendations Regarding Public and Private Types of Organizations to Perform 

 the Proposed Research and/or Analysis...............................................................................63 

References........................................................................................................................................65 

Appendixes 

Appendix A: Technical Expert Panel 

Appendix B: Information Technology in Health Search Strategies 

Appendix C: HIT Round Two Screener 

Appendix D: List of TEP Members Who Provided Comments 

Appendix E: Peer Reviewers Comments 

Appendix F: Articles Included in HIT Interactive Database 

Appendix G: List of Excluded Articles 

Appendixes and Evidence Tables for this report are provided electronically at 

http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. 
1

Executive Summary 

Introduction 

The United States health care system is at risk due to increasing demand, spiraling costs, 

inconsistent and poor quality of care, and inefficient, poorly coordinated care systems. Some 

evidence suggests that health information technology (HIT) can improve the efficiency, costeffectiveness, quality, and safety of medical care delivery by making best practice guidelines and 

evidence databases immediately available to clinicians, and by making computerized patient 

records available throughout a health care network. However, much of the evidence is based on a 

small number of systems developed at academic medical centers, and little is known about the 

organizational changes, costs, and time required for community practices to successfully 

implement off-the-shelf systems. 

An analysis of the usefulness of implementing HIT must take into consideration several 

factors: 

 The potential of this technology to improve health care quality, safety, and patient 

satisfaction—and how this potential has been demonstrated. 

 The cost-effectiveness of the technology—the business case for adoption of the 

technology—including the total costs of implementation (both financial and in terms of 

resources) and any cost savings that accrue. Concerns exist that those who bear the 

greatest share of such costs are not able to recoup those costs. 

 The ability to generalize the effects of an HIT intervention on costs and benefits in 

existing systems (using published experience with or research on these systems) to the 

technology’s use by other health care organizations. 

The Leap Frog Group and a number of components of the U.S. Department of Health and 

Human Services (HHS)—the Centers for Medicare & Medicaid Services (CMS), the Office of 

the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention 

and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality 

(AHRQ)—requested a review of the research on HIT to compile and evaluate the evidence 

regarding the value of discrete HIT functions and systems in various health care settings. This 

Evidence-based Practice Report on the costs and benefits of health information technology 

systems, along with an accompanying interactive database that catalogs and assesses the existing 

evidence was prepared by the Southern California Evidence-based Practice Center (EPC). This 

report systematically reviews the literature on the implementation of HIT systems in all care 

settings and assesses the evidence in four specific circumstances: 

1. The costs and benefits of HIT for pediatric care. 

2. The ability of one aspect of HIT, the electronic health record (EHR), to improve the 

quality of care in ambulatory care settings. 

3. The costs and cost-effectiveness of implementing HER. 

4. The effect of HIT on making care more patient-centered. 
2

Methods

An electronic search of PubMed, the Cochrane Controlled Clinical Trials Register, and the 

Cochrane Database of Reviews of Effectiveness (DARE) was conducted for articles published 

from 1995 to January 2004. Additional references were obtained by reviewing the references in 

several major reports prepared by private industry and by RAND Health. Two reviewers, each 

trained in the critical analysis of scientific literature, independently reviewed each study and 

resolved disagreements by consensus. The principal investigator resolved any disagreements 

that remained unresolved after discussions between the reviewers. 

Studies selected for review had to be either: 

 A meta-analysis. 

 A systematic review. 

 Original research that tested a hypothesis (that is, a report that compared data between 

groups or across time periods, assessing a specific question and using statistical tests to 

assess differences). 

 Original research that conducted predictive analyses (a report that used modeling 

techniques and simulations to predict the effects of an HIT implementation). 

Of 855 articles screened, 256 were accepted for review. Descriptive studies of HIT 

implementations were identified and classified according to the categories listed below, but were 

not reviewed in more detail. 

The contents of each selected article or report were abstracted using electronic dataabstraction forms prepared especially for this analysis. Abstracted data included the system’s 

capabilities, interventions used, study design, implementation processes, evaluation methods, 

outcomes, costs, and barriers to implementation. A structured abstract was created for each 

report; these abstracts can be accessed in an online, interactive database created for this evidence 

report. (This database can be accessed at http://healthit.ahrq.gov/tools/rand.) 

Results 

Overall Results 

Of the 256 studies reviewed, 156 were about decision support, 84 assessed the electronic 

medical record, and 30 were about computerized physician order entry (CPOE). One hundred 

twenty-four of the studies assessed the effect of the HIT system in the outpatient or ambulatory 

setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a 

randomized design. There were 11 controlled clinical trials, 33 studies that used a pre/post 

design, 20 studies that did a time series, and another 17 that were case studies with a concurrent 

control. Among the 211 hypothesis-testing studies, 81 contained at least some cost data. 

Many of the studies concerned HIT systems developed and evaluated by academic and 

institutional leaders in HIT. 

 Regenstrief Institute in Indianapolis, IN (18 studies) 

 Partners/Brigham and Women’s Hospital in Boston, MA (19 studies) 
3

 Intermountain Health in Salt Lake City, UT (11 studies) 

 Kaiser Permanente health care system (5 studies) 

 Vanderbilt University in Nashville, TN (2 studies) 

 U.S. Department of Veterans Affairs (VA) health care system (15 studies) 

Studies from these institutions have contributed greatly to our knowledge about the 

usefulness of particular HIT functionalities (such as CPOE or computerized electronic alerts), 

and are examples of what can be realized by the implementation of broadly functional HIT at 

these specific institutions. But these studies also have limitations, in terms of their usefulness to 

inform decisions about the adoption of HIT elsewhere. The primary limitation is that these HIT 

systems were developed over the course of many years by technology champions at these 

institutions and, in a process of co-evolution, were adapted particularly to the working 

environment and culture of their respective institutions. Consequently, the “intervention” at these 

sites consists not only of the HIT system but also the local champions, who were often also the 

evaluators in published studies. Furthermore, it is challenging to calculate the cost of the 

development of the HIT system as a whole, since this process occurred over many years at each 

institution. In addition, these systems are not commercially available from a vendor—and 

vendors supply most HIT systems in use in the U.S. 

We were able to identify only 15 studies that used a randomized or controlled clinical (RCT 

or CCT) design, included cost data, and assessed HIT systems that were not from one of the 

leading academic and institutional HIT institutions or the United Kingdom (another setting that 

has limited generalizability to U.S. health care institutions). When these 15 studies were 

examined for their HIT functionality using the classification system developed by the Institute of 

Medicine, 4 of them concerned only decision support and 4 assessed HIT systems with decision 

support and administrative processes. The remaining seven studies addressed other single 

functionalities or combinations of up to three functionalities. We were not able to find a single 

study that used a randomized or controlled clinical trial design, that did not report data from one 

of the leading academic or institutional HIT systems or the U.K., that reported cost outcomes and 

that assessed an HIT system including at least four of the eight IOM categories of functionality. 

For the 103 hypothesis-testing studies that used a design other than a randomized or 

controlled clinical trial, 45 reported cost data. Of these 45 studies, 23 assessed systems that were 

not one of the leading academic or institutional HIT systems or that came from the U.K. An 

examination of these 23 studies for their functionalities showed, as in the studies using an RCT 

or CCT design, that most studies did not evaluate systems with a broad level of functionality. 

Five studies assessed only decision support, and three studies each assessed only administrative 

processes or order entry management. Three studies assessed HIT systems with two 

functionalities, order entry management and decision support. The remaining nine studies 

assessed various combinations of two or three functionalities. No study evaluated an HIT system 

with at least four of the eight categories of functionality. 

The literature is even sparser regarding information about the organizational context of an 

HIT implementation. Of the hypothesis-testing studies, we identified only 3 studies that provided 

information about the financial context of the organization, such as the degree of managed 

care/capitation penetration; 6 studies with information about system penetration; 2 studies about 
4

facilitators to implementation; 1 study explicitly discussing sustainability of the HIT 

intervention; 12 studies reporting extrinsic factors in valuing costs and benefits such as the health 

care market competitiveness; and 6 and 9 studies, respectively, reporting the initial costs of the 

HIT system and costs of implementation. 

In summary, we identified no study or collection of studies—outside of those from a handful 

of HIT leadership institutions—that would allow a reader to make a determination whether the 

study's reported benefit was generalizable. Besides these studies from HIT leaders, no other 

research assessed HIT systems that had comprehensive functionality while including data on 

costs, relevant information on organizational context and process change, and data on 

implementation. This limitation in generalizable knowledge is not only a matter of study design 

and internal validity. Even if further randomized, controlled trials are performed, the 

generalizability of the evidence would remain low unless additional systematic, comprehensive, 

and relevant descriptions and measurements are made regarding how the technology is utilized, 

the individuals using it, and the environment it is used in. 

The Costs and Benefits of HIT in Pediatric Settings

Early evidence shows that stand-alone clinical decision-support systems (CDSS) (such as 

drug dosing calculators) can reduce medication dosing errors, and CPOE plus CDSS can reduce 

the incidence of harmful medication errors in the inpatient pediatric and neonatal intensive care 

settings. Other HIT systems, such as electronic medication administration records, pharmacybased robots, smart infusion pumps/devices, and medication bar-coding, are predicted to reduce 

medication errors, but need further study. 

The use of CPOE plus CDSS has been demonstrated, in separate studies, to (1) reduce the 

frequency or duration of antibiotic use for common pediatric illnesses such as pharyngitis and 

otitis media, and (2) improve completeness and reduce variation in clinical documentation. In the 

ambulatory setting, a single study showed that an appointment reminder system is cost-effective 

and significantly reduces missed appointments, while in the neonatal intensive care unit, another 

study showed that CPOE can reduce medication and radiology turnaround times. Therefore, the 

evidence for HIT cost-savings in pediatrics is limited, but appears optimistic. 

Electronic Health Records and the Quality of Ambulatory Care 

Adoption of EHR systems is widely believed to be critical to the delivery of consistent, highquality health care, although the current use of EHRs is limited. Seven studies were identified on 

the use of EHR in four ambulatory care settings (three in the United States and one in the 

Netherlands). The findings reported in all of these studies were primarily related to the 

implementation processes and to changes in clinical processes. 

With the exception of one study that examined the effects of incorporating HIV care 

guidelines and alerts on quality of care for HIV-positive patients, all the studies assessed the 

effects of adding various types of information related to laboratory test and prescription ordering 

to EHR ordering screens. In general, these studies showed that providing laboratory test 

guidelines and related information on test-ordering screens was associated with a decrease in 
5

orders for overused tests and an increase in orders for underused tests; provision of formulary 

guidance was associated with increased adherence to a formulary for at least one class of 

medication; and addition of HIV care guidelines and alerts was associated with improved quality 

of care. 

The Economic Value of an EHR System 

While EHR systems may be essential for improving efficiency and quality of health care, 

implementation of an EHR system requires substantial capital investments and organizational 

change. Consequently, many health care organizations are seeking evidence from previously 

implemented systems about the costs and benefits of EHR adoption in order to better inform 

decisions about the optimal timing and strategy for implementation. 

Not all of the costs and benefits reported when implementing new systems or making 

changes to existing systems were financial. EHRs were associated with improvements in service 

and other resource utilization, provider productivity, care efficiency, documentation quality, 

clinical decisionmaking, guideline compliance, and costs of care. 

Despite considerable variation among the few studies that modeled financial costs and 

benefits, all predicted substantial cost savings from EHR implementation. However, these studies 

each made a number of assumptions, and the predicted break-even points ranged from as short a 

time as 3 years to as long as 13 years. 

HIT and Patient-Centered Care 

The evidence is sparse for the ability of HIT systems to make health care more patientcentered. The best evidence of such a change is the beneficial effect on preventive care of using 

computerized reminders to patients. Telemedicine and consumer health informatics also have 

limited evidence of benefit in specific contexts. The evidence is much more limited about the 

health effects of more general, interactive health information technologies such as the Internet or 

e-mail, or the effect on patient trust and satisfaction of implementing HIT systems such as the 

electronic health record. 

 

Barriers to HIT Implementation 

Studies identified a large number of barriers to the implementation of HIT. These barriers 

can be classified as situational barriers (including time and financial concerns), cognitive and/or 

physical barriers (including users’ physical disabilities and insufficient computer skills), liability 

barriers (including confidentiality concerns), and knowledge and attitudinal barriers. Cutting 

across all of these categories, however, may be the need for a major structural and ideological 

reorganization of clinical medicine as it is now practiced in the majority of settings to be able to 

integrate itself with and enjoy the benefits of HIT. 
6

Conclusions 

Limitations of the Review 

 The primary limitation of this review is the quality, quantity, and generalizability of the 

available (published) studies. Substantially more information regarding implementation may 

have been obtained by contacting leading HIT implementers and conducting structured 

interviews with them. 

 Many of the costs and financial benefits of EHR will change over the years, because they 

depend on the changing price of such factors as hardware, software, labor, and 

pharmaceuticals and medical devices. Consequently, the costs reported in some of the older 

articles are of limited relevance. 

General Conclusions 

 Predictions based on statistical models suggest that HIT has the potential to assist in 

dramatically transforming the delivery of health care, making it safer, more effective, and 

more efficient. However, the experimental evidence supporting benefits from HIT is more 

limited. 

 A number of large health care organizations have realized some of these major gains 

through the implementation of multifunctional, interoperable HIT systems built around an 

electronic health record. 

 The impact of HIT implementation on the cost and quality of care is not going to be 

consistent across institutions, independent of context. However, the specific context within 

which HIT is implemented, including the setting, the clinical issues, and the patient 

populations, greatly influences its use and effects. 

 More widespread implementation of HIT is limited by the lack of generalizable knowledge 

about what types of HIT and methods for its implementation will prove most useful for 

specific health organizations, especially for small practices and small hospitals. 

 The reporting of HIT developments and implementations needs to be improved, with greater 

attention given to descriptions of both the intervention and the organizational/economic 

environment in which the technology is implemented. 

 A high priority must be placed on establishing standards for the information that needs to be 

measured and reported in studies of HIT implementation, similar to the CONSORT 

standards developed for reporting clinical trials of therapeutics. 

 Using existing published evidence, it is not possible to draw firm conclusions about which 

HIT functionalities are most likely to achieve certain health benefits—and the assessment of 

costs is even more uncertain. 

 Existing evidence is not sufficient to clearly define “who pays for” and “who benefits from” 

HIT implementation in any health care organization—except those, such as Kaiser and the 

VA, that are responsible for paying for and delivering all the care for the defined population. 
7

 Statistical models can be built to estimate the costs and benefits of interoperable HIT 

systems within and across health care provider settings, payers/purchasers, and cumulatively 

across the health care continuum, but these models are based on many untested assumptions. 

 Implementation of HIT faces many barriers, including institutional barriers, cognitive and/or 

physical barriers, liability barriers, and knowledge and attitudinal barriers.

Evidence Report 
 11

Chapter 1. Introduction 

The use of health information technology (HIT) has been promoted as having tremendous 

promise in improving the efficiency, cost-effectiveness, quality, and safety of medical care 

delivery in our nation’s healthcare system. The realization of these benefits is especially 

important in the context of reports that show five years of consecutive annual double-digit 

increases in healthcare costs and increases in the numbers of adverse health events.1,2 At the 

same time, reports have suggested that 50 percent of all healthcare dollars are wasted on 

inefficient processes. Legislators and organizational leaders at the federal and state levels have 

emphasized the need for healthcare to follow the example of many non–healthcare industries, in 

which implementation of computer information technology has been critical in increasing the 

accessibility of mission-critical information, automating labor-intensive and inefficient 

processes, and minimizing human error. 

The most important use for HIT may be to help reduce medical errors. This technology-based 

strategy has proven effective in reducing the effects of human error in industries such as banking 

and aviation. Clinical HIT systems may make a substantial impact on medical quality and safety 

by integrating relevant automated decisionmaking and knowledge acquisition tools into the 

practices of medical providers, thereby reducing errors of omission that result from gaps in 

provider knowledge or the failure to synthesize and apply that knowledge in clinical practice. 

These systems, when integrated within larger HIT systems, may improve medical 

decisionmaking and appropriate use of diagnostic tests and therapeutic agents. 

In the ambulatory healthcare environment, the use of HIT offers a variety of benefits. First, it 

can improve the efficiency and financial health of the practice. For years, many offices have used 

computerized scheduling and financial systems to streamline office processes by tracking 

practice productivity and automating reimbursement processes. Second, the use of ambulatory 

electronic health records (EHRs) also offers an opportunity to monitor and improve clinical 

quality by improving information access and reducing duplicative documentation. And 

technology-based “e-prescribing” tools may improve the efficiency and safety of prescribing 

practices in the outpatient setting just as they have done in the hospital setting. Finally, the 

widespread adoption of HIT will allow the achievement of system connectivity and information 

exchange among providers of the same organization, among organizations, and ultimately 

regionally and nationwide. 

However, the majority of medical organizations and providers have been slow to adopt HIT. 

Recent surveys of computerized physician order entry (CPOE) use show that only 9.6 percent of 

hospitals have CPOE completely available for use, and only half of these hospitals require use of 

CPOE.3

 In the ambulatory setting, recent estimates place the use of electronic health records at 6 

to15 percent of office-based physicians.4,5 The potential advantages of widespread adoption of 

HIT in our nation’s healthcare system make it vital to examine the scientific evidence that 

currently supports the relative costs and benefits of HIT, and the barriers to implementing 

various types of HIT systems across the spectrum of healthcare environments. 
 12

A Framework for Considering the Costs and Benefits of 

Health Information Technology 

Private organizations deciding whether to invest in HIT must weigh the costs and benefits of 

doing so. Although the primary goal of nonprofit healthcare organizations may be to provide 

high-quality care, these organizations still need to watch the bottom line to survive, which 

includes understanding the costs of measures designed to improve quality. Such private returnon-investment (ROI) calculations can provide results that are quite different from those of 

societal cost-benefit analysis, which are often reported in clinical journals. 

For example, one study showed that a hospital that installed a computerized reminder system 

to alert providers when patients were not up-to-date on their immunizations increased 

pneumococcal vaccine orders by 8 percent.6

 Another study showed that, among the elderly, each 

$12 vaccination averts $20.27 in hospital costs and increases life expectancy an average of 1.2 

days.7

 From society’s point of view, the reminder system saves money and improves health, so it 

is a win-win program. However, from a financial perspective, the hospital has spent money on a 

system that had no effect on the costs or revenues of current stays because the pneumococcal 

vaccine is not delivered in the hospital. To benefit from this intervention, the hospital must make 

a reputation for higher quality and convert it into profits. This is one example of the potential for 

a mismatch between who pays for and who accrues cost savings from HIT use. A more extreme 

example would be a hospital’s implementation of a HIT intervention that averts future 

hospitalization. In this case, HIT implementation both costs the hospital money and decreases 

hospital revenues, even if the HIT implementation has a net cost-savings from a societal (or 

Medicare) perspective. 

Elements of the Business Case 

The business case for investing in HIT must consider both financial and nonmonetized1

consequences. The financial aspect deals with the effect on the organization’s bottom line. Any 

HIT investment has immediate costs in purchase, adaptation to the local organization, and staff 

training. So the business case for HIT depends on the downstream financial benefits exceeding 

the immediate costs. Because profits = revenue – costs = (revenue per patient – costs per patient) 

× (number of patients), long-term profits can come from increases in (profitable) patients, 

increases in revenues per patient, or decreases in cost per patient. The easiest of these to 

understand is costs per patient. All organizations benefit from becoming more efficient and 

reducing the costs of providing particular services. HIT can reduce the waste involved in 

collecting information and getting it to where it is needed for better decisionmaking. This 

increase in efficiency can streamline health care and billing processes, and avoid the costs of 

unnecessary services and of dealing with errors, both in patient care and in billing. Also, working 

in high quality organizations has some intangible benefits to staff, which may lead to better 

retention and productivity at equal levels of pay. 

 

1

 Nonmonetized consequences are merely costs and benefits that are not expressed in dollar terms. It may be easy to 

express some of them in dollars but difficult to realize the corresponding cash flows. (For example, the time you 

spend in traffic may be worth $100/hour, but who is going to pay you for it?) Others may resist expression in 

dollars. 
 13

However, if the HIT is used to raise the quality of care or change the mix of services 

provided, the resulting financial costs and benefits depend on how the organization is paid and 

what expenses it bears. These factors can greatly affect what kind of return on investment is 

likely and when it will be realized. The next three paragraphs provide some examples. 

A reputation for higher quality should increase the demand for an organization’s services in a 

competitive market, but it is difficult to prove that you are better than your competition or better 

than you used to be. HIT can raise quality and can also generate the statistics to prove you have 

done so. Perceived higher quality allows organizations to increase market share and to negotiate 

higher prices from payers whose members demand access to those organizations, even if they 

have to pay slightly higher premiums to get it. In a competitive fee-for-service environment, 

greater market share increases revenues and may also permit some economies of scale. 

HIT can also be used to increase reimbursable services per patient, such as covered 

immunizations and exams. HIT pays if it reduces waste, but it reduces profit if it reduces current 

or downstream services. Hospitals whose payments are set by DRGs (a fixed payment that 

depends on the diagnosis of the patient but does not vary with actual costs) benefit somewhat 

from shorter length of stay (although the last days of a hospitalization are the cheapest), but not 

from reduced readmissions (except those where a Medicare patient bounces back into the 

hospital before sufficient time lapses post-discharge to qualify the readmission for 

reimbursement as a “new” episode of care). A hospital also will not benefit financially from 

interventions that shift care to physicians’ practices. 

The biggest gains from quality and HIT come when providers are paid by means of a 

capitated fee system. Under such a system, any investment that reduces the total costs of care for 

these patients can be recouped, so it pays to reduce unnecessary services and to provide care in 

the most efficient setting. HIT may help to share the information needed to do so. Such reasoning 

was behind the Department of Veterans Affairs’ (VA’s) decision to develop its HIT system. 

Most published examples of cost-saving quality projects come from health maintenance 

organizations (HMOs)—for example, better diabetes or heart failure care that keeps patients out 

of the hospital. Also for HMOs, high quality can offset other undesirable features—such as poor 

access or amenities—or can justify higher premiums. The gains to HMOs of better care will be 

more certain when capitation payments are adequately risk adjusted. Without risk adjustment, 

providing high quality chronic illness care, an area where HIT is particularly useful, may have 

the unprofitable side effect of attracting more-expensive patients. 

Because some of the financial gains from high quality may go to purchasers (employers) 

rather than providers, particularly in noncapitated, fee-for-service environments, some 

purchasers have started to pay directly for quality. If the case for HIT were strong enough, 

insurers might want to subsidize it in part (i.e., based on the insurer’s share of the provider’s 

caseload). However, unless an insurer covers most of the patients in a particular health care 

organization or insurers agree to collaborate, it does not pay one insurer to subsidize HIT for an 

entire provider or organization because a substantial portion of the cost savings accrue to other 

payers (the “free rider” problem). 

Non-healthcare businesses that are selecting investments might consider only financial return 

on investment (ROI), but providing health care is a business with an unusual emphasis on 

nonmonetized goals. The nonmonetized part of the business case includes all nonmonetary 

arguments that the organization feels will influence the decision to adopt or reject the 

intervention. Examples include the following: 
 14

 Maintaining credentials 

 Satisfying reporting requirements 

 Satisfying a requirement to do a quality improvement project 

 Avoiding exposure to liability 

 Building goodwill or reputation 

 “Because it’s the right thing to do.” 

Many of these nonmonetized items have financial aspects. For example, the intervention may 

reduce the cost of meeting a preexisting reporting requirement. Also, many organizations, 

particularly nonprofits, have nonfinancial goals—such as providing high quality care—in 

addition to financial goals.2

What Is Generalizable Knowledge Regarding Health 

Information Technology? 

In this report, we use the term generalizable knowledge to mean published evidence of the 

effects of a HIT intervention on costs and benefits that other health care organizations can use to 

implement HIT and reasonably expect benefits similar to those reported in the original study. 

Therefore, generalizable knowledge from a study has two components: (1) the internal validity of 

the study and (2) the utility of the information to others considering implementing HIT. We can 

illustrate differences in generalizable knowledge by considering some examples. 

The simplest example is that of a particular pharmaceutical therapy for patients with a certain 

condition. In this case, a randomized, placebo-controlled trial of the new pharmaceutical agent 

would be a study with good internal validity. Because pharmaceuticals are manufactured for 

consistency in strength and are given according to specified dosing schedules, another health 

care organization examining the results of such a study could reasonably assume that 

administration of the new pharmaceutical in the same doses and to patients with similar 

characteristics would result in benefits similar to those reported in the original study. 

A second example would be the assessment of a new surgical therapy. In such a case, the 

evidence would not come from a randomized, double blind, placebo-controlled trial, since this 

design is not generally feasible for tests of surgical therapy. Data may come from studies 

comparing patients randomly assigned to surgical therapy or to an alternate therapy or 

nonrandomized studies comparing surgically treated patients with historical controls or even case 

series. As the confidence in the equivalence of the comparison groups at baseline diminishes, the 

difference in benefit must become greater for the reader to conclude that beneficial effects on 

outcomes are due to differences in therapy and not other differences between groups at baseline. 

Even after accepting that a particular study reports a real difference in outcomes between 

groups, the healthcare organization or practitioner contemplating offering surgery must consider 

more factors than when contemplating the prescription of a new pharmaceutical agent. Surgical 

therapies are not as standardized as pharmaceutical agents, and outcomes depend upon such 

factors as the skill of the surgical team and hospital. There is no reason to expect that every 

surgeon and hospital delivers equivalent care the way physicians and patients can expect a 

 

2

 Nonprofits may explicitly have commitments to provide the highest quality care, but for-profits also share medical 

ethics and culture to do the best they can for their patients. 
 15

standard dose of a pharmaceutical to have equivalent potency. Hence, a study describing the 

effects of a surgical therapy needs to give more detail than a study describing the effects of a 

pharmaceutical drug, namely, enough description of the surgeon and hospital that other 

healthcare organizations or providers can determine whether the reported outcomes are likely to 

be achieved in their own clinical situation. 

When considering HIT evaluation, the situation becomes even more complex. Both the 

intervention and the subjects of the intervention are qualitatively different in a study of HIT than 

in a study of a pharmaceutical or surgical intervention. HIT implementation consists of a 

complex organizational change undertaken to promote quality and efficiency. Studies of 

organizational change are fundamentally different from studies of medical therapies. 

Organizational interventions interact with a wide range of organizational system components. To 

be successful, they must address these components in a locally effective way. Thus, in a sense, 

these interventions are by nature not widely generalizable, in contrast to studies of narrow 

interventions such as pharmaceuticals, which aim to identify treatment effectiveness that is 

operator-independent, or generalizable across settings or providers. This difference has several 

important consequences. First, randomized controlled trials are not always feasible for assessing 

organizational change. The risks and benefits of reliance on controlled trials for evidence about 

interventions involving organizational change has been debated.8,9 However, reliance only on 

randomized clinical trials for evidence of the effect of HIT on costs and outcomes risks 

restricting the focus to narrow and tightly defined elements of HIT. In many real-world 

applications, complex organizational change interventions are implemented as a series of steps, 

with each step dependent on the organizational response to the previous step. Therefore, we 

judge that generalizable knowledge must and can come from many types of studies. However, 

we also judge that these studies must report details of the intervention and the organizational 

characteristics of where the intervention was implemented to allow other organizations to make 

judgments about the applicability of the results. 

We consider the intervention in HIT studies to have at least four components: 

 Technical—including the system components being tested (which may consist of CPOE, 

clinical charting, or electronic prescribing); the preexisting technology infrastructure 

(e.g., clinical and financial systems, network); and the existing electronic interfaces and 

integration. 

 Human factors (machine-person interface)—system usability (e.g., “user-friendliness,” 

system response time, intuitive user interface, support for workflow processes), support 

for specialty or context-specific actions (e.g., clinical content, order-sets, and level and 

acceptability of clinical decision-support). 

 Project management—effecting complex sociotechnical process change around HIT 

implementation, aligning IT and organizational resources to achieve project milestones, 

and controllership of IT budgets. 

 Organizational and cultural change, which may include a partnership of medical staff 

and administrative leadership to govern, align incentives, and mobilize organizational 

inertia to achieve desired outcomes through process change. 

Cutting across all four of these components is effective communication. Most organizational 

change and IT projects have a strong but unrecognized communication component, which 

encompasses, among other things, the sharing of vision, values, and information about the 

components of HIT system selection, as well as its implementation and use. 
 16

Without an adequate description of all of these components in a study of HIT costs and 

benefits, it is difficult for others to be able to infer how, or even whether, they can reproduce the 

results. Omitting such information would be analogous to omitting the strength or dosing 

schedule from the report of a study of a pharmaceutical intervention.3 

Similarly, the analogue of the patient in a study of HIT is the organization. No consensus 

exists regarding what aspects of the organization are most important to report, but some aspects 

are clearly important. Aspects that have been proposed as important include size, staffing, the 

organization’s prior experience with quality improvement initiatives, processes expected to be 

influenced by the intervention and how these work currently, and the financial context of the 

organization. These characteristics may well determine which types of HIT interventions work in 

a given setting. For this review, we assessed (a) whether studies measured some key 

organizational characteristics and (b) what those characteristics were. Such characteristics might 

be considered key organizational demographics, just as gender, age, and illness severity would 

be considered key demographic characteristics for an efficacy and safety study of a new 

pharmaceutical. 

However, knowing even these characteristics may not be enough to understand why a HIT 

intervention did or did not work. An organization has to do more than simply buy the software to 

be successful. It must also invest in adapting the software to the organization, developing new 

policies and procedures, and training staff. The extent to which the organization is willing and 

prepared to perform these and other critical additional functions to embed the HIT into all 

relevant systems determines organizational readiness for change. There is unfortunately little 

scientific knowledge about which organizational characteristics are essential, and which, like the 

color of the patient’s eyes when assessing the effect of taking a new pill, are unimportant. Thus, 

even if the description of a successful intervention includes many of the details described above, 

without information about organizational readiness, readers cannot know whether or not the 

same intervention is likely to work in their own organization and how long and expensive the 

transitional process might be. 

 

3

 However, we recognize that there are barriers to providing this level of specification: For example, prior to that 

advent of the internet, journals might have been reticent to devote limited space to such descriptions, and the 

knowledge of what variables need to be included changes over time. 
 17

Chapter 2. Methods 

Original Proposed Key Questions 

An evidence report on the costs and benefits of HIT systems was requested by the Leap Frog 

Group, the Centers for Medicare and Medicaid Services (CMS), the Office of the Assistant 

Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health 

Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ). The 

purpose of the report was to develop an evidence base regarding the value of discrete HIT 

functions and systems in various healthcare settings. 

Original key questions for the report were: 

1. What does the evidence show with respect to the costs and benefits of inter-operable 

electronic HIT data exchange for providers and payers/purchasers? 

2. What is a framework that could be used in this study to describe levels/bundles of EHR 

functionality and to estimate the costs and benefits by such levels/bundles of functionality 

by payer/purchaser and percentage of provider penetration? 

3. What knowledge or evidence deficits exist regarding needed information to support 

estimates of cost, benefit and net value with regard to HIT systems? Discuss gaps in 

research, including specific areas that should be addressed, and suggest possible public 

and private organizational types to perform the research and/or analysis. 

4. What critical cost/benefit information is required by decision makers (at various levels) in 

order to give a clear understanding of HIT Systems value proposition particular to them?

 

5. What analytic methods (e.g., sources of data, algorithms, etc.) could be used to produce 

evidence of the costs and benefits within and across health care provider settings, 

payers/purchasers, and cumulatively across the health care delivery continuum and 

payers, of deploying electronic health information technology functions examined in this 

study? 

6. What are the barriers that health care providers and health care systems encounter that 

limit implementation of electronic health information systems? 

Technical Expert Panel 

Each AHRQ evidence report is guided by a Technical Expert Panel (TEP). We invited a 

distinguished group of scientists, clinicians, and information technology experts, including 
 18

individuals with expertise in medical informatics, Internet health, and telecommunications to 

participate in the TEP for this report. A list of panel members is included as Appendix A∗. 

The TEP’s participation in the preparation of the report began with a meeting that was 

conducted via conference call at the start of the project; the purpose of this meeting was to get 

TEP input on the scope of the project, especially the specific information technology applications 

to address. We were also seeking input on what constitutes evidence because most of the data on 

HIT implementation derive from interventions that are not RCTs, which are the usual backbone 

of EPC evidence reports. This particular meeting was held at two separate times in order to 

accommodate scheduling conflicts; TEP members were asked to participate on the date that was 

more convenient for them. The meetings were held on March 19 and March 26, 2004.

At this meeting, we also discussed the framework for how to conduct our research. Many 

TEP members were interested in HIT implementation issues, for example, what can be learned 

from others who have implemented HIT in various settings, including both community and 

academic settings. They also emphasized that HIT is often implemented through multicomponent 

interventions, of which IT is just one aspect. 

Based on the comments received during the TEP conference calls and numerous discussions 

with AHRQ, it was determined that the report would focus on reviewing the evidence from 

existing published articles regarding the costs, benefits, and barriers to implementing HIT. Many 

other excellent suggestions were received during the conference calls, such as performing new 

cost-benefit analyses or collecting unpublished information on barriers, but the decision was 

made that a review of existing published evidence should precede any other analyses. 

Literature Search 

At the time this report was undertaken, another team at RAND was working on a project 

entitled “Leveraging Modern Information Technology to Transform Medical Care Delivery.” 

This project, funded by private industry, aimed to suggest policy changes that are likely to 

increase the rate of adoption of HIT in the United States. One part of the project involved 

assessing the effects of information technology on costs, health outcomes, and adverse events. 

We were given the list of titles from the team’s November 2003 search of PubMed, which sought 

systematic reviews published in English from 1995 to 2003. PubMed, which is maintained by the 

U.S. National Library of Medicine, is widely recognized as the premier source for bibliographic 

coverage of biomedical literature. It encompasses information from Index Medicus, the Index to 

Dental Literature, and the Cumulative Index to Nursing and Allied Health Literature (allied 

health includes occupational therapy, speech therapy, and rehabilitation), as well as other sources 

of coverage in the areas of health care organization, biological and physical sciences, humanities, 

and information science as they relate to medicine and health care. 

Our own search for studies of HIT began with an electronic search of PubMed on January 6, 

2004 for reports of original research as well as any additional articles about HIT published since 

1995. We ordered all articles on the HIT topics, regardless of study design or language. 

 

∗ Appendixes and Evidence Tables for this report are provided electronically at 

http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. 
 19

Appendix B shows our specific search strategies. We also searched the Cochrane Controlled 

Clinical Trials Register Database and the Cochrane Database of Reviews of Effectiveness 

(DARE). The Cochrane Collaboration is an international organization that helps people make 

well-informed decisions about health care by preparing, maintaining, and promoting the 

accessibility of systematic reviews on the effects of heath care interventions. In December 2004, 

we also conducted a specific search of the journal Health Affairs, developing a list of all articles 

with “information technology” or “information systems” as keywords. Health Affairs has 

published special editions on this topic in recent years. 

Additional Sources of Evidence 

Several other sources of evidence were considered, based on the recommendations of the 

TEP. Advanced Technologies to Lower Health Care Costs and Improve Quality was published in 

fall 2003 by the Massachusetts Technology Collaborative in partnership with the New England 

Healthcare Institute. Research was conducted by the First Consulting Group and was sponsored 

by several Massachusetts companies involved in healthcare and health insurance. The report 

focuses on seven advanced technologies (including examples of HIT, such as computerized 

physician order entry and electronic prescribing in the inpatient and ambulatory care setting) that 

have demonstrated both financial benefits and improved quality of care. It also includes 

discussions of barriers to implementation. 

The Value of Computerized Provider Order Entry (CPOE) in Ambulatory Settings was 

published in 2003 by the Center for Information Technology, also located in the Boston area. 

This group conducted an international search for both academic and commercial sources of 

literature and also contacted 35 vendors regarding their currently available health information 

technology packages. The report found that CPOE can significantly improve quality while 

lowering costs.

Meta-Analysis on Computer-Based Clinical Reminder Systems reports on a 1996 metaanalysis of 16 trials by Shea, DuMouchel, and Bahamonde published in the Journal of the 

American Medical Informatics Association (JAMIA). The authors found that computer 

reminders in the ambulatory care setting improved utilization of vaccinations, breast cancer 

screenings, and colorectal cancer screenings, but not pap smears or other preventive care. 

Personal files were contributed by project staff, consultants, and technical expert panel members 

in response to a request for any applicable unpublished literature on the costs and benefits of 

HIT. 

Articles could have been identified in more than one way (for example, the PubMed search 

and personal files might contain some of the same articles). 

Article Review 

We reviewed the articles retrieved from the various sources against our exclusion criteria to 

determine whether to include them in the evidence synthesis and in the special interactive 

database tool we created to accompany this report (see below). A screening review form that 
 20

contains a series of categorization questions was created to track the articles (see Appendix C∗). 

Two reviewers, each trained in the critical analysis of scientific literature, independently 

reviewed each study, and resolved disagreements by consensus. The principal investigator 

resolved any disagreements that remained unresolved after discussions between the reviewers. 

As previously indicated, this report includes evidence from articles with many different study 

designs. Our initial search was unrestricted by study design. The resulting articles were divided 

into four categories: reviews, descriptive reports, hypothesis testing-studies, and predictive 

analysis studies. 

Review articles identified by the search were classified as either systematic (including metaanalyses) or nonsystematic. The determination of systematic versus nonsystematic was made by 

reading the methods section of the article to see whether an acceptable method was employed to 

identify evidence. This assessment was made by the Center directors working independently 

with consensus resolution. Only systematic reviews were considered for further inclusion. 

Articles were classified as descriptive if they primarily described the workings or 

implementation of a HIT system. We further classified these as qualitative or quantitative, based 

on the presentation of information regarding such factors as number of tests ordered and costs of 

implementation. 

A third category of articles was classified as hypothesis-testing studies, indicating that 

researchers attempted to answer a study question by comparing data between groups or across 

time periods and using statistical tests to assess differences. Hypothesis testing studies were 

further classified as (1) those containing an intervention with a concurrent comparison group, 

which included randomized and nonrandomized controlled trials and controlled before-after 

studies; and (2) studies with an intervention but without a concurrent comparison group, which 

included pre-post studies, time-series studies with more than two measurement points, and 

studies that used a historical control group. Additional classifications of hypothesis testing 

studies included those without an intervention, which were cross-sectional in nature, and “other” 

hypothesis testing studies. 

The fourth category of studies was predictive analyses, which included studies that used 

modeling techniques to predict what might happen with a HIT implementation rather than what 

did happen. Predictive analyses include cost-effectiveness and cost-benefit analyses. They 

typically use data from multiple studies and depend upon several assumptions, some of which 

are not always explicitly stated. 

 Selection of Articles and Data Elements for Interactive Database 

Articles that were classified as systematic reviews, meta-analyses, hypothesis-testing, or 

predictive analyses went on to more detailed review. For reasons discussed below, we created 

structured abstracts for these articles and placed them in an interactive database of HIT studies 

(http://healthit.ahrq.gov/tools/rand). 

 

∗ Appendixes and Evidence Tables for this report are provided electronically at 

http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. 
 21

We looked for the following data in each article: a description of the HIT system; the purpose 

of the study; the year or years the study was performed; the study design; the outcomes reported; 

a description of the study settings; the intervention and control arm; the evaluation method; a 

description of the HIT system, including how the system was acquired, the year the system was 

installed, the capability and comprehensiveness of the system; the integration of guidelines or 

decision support; the interoperability; the HIT implementation strategy; the financial context, 

such as whether this is a managed care or capitation environment, pay for performance, or area 

of public accountability; the system penetration; facilitators and barriers; evidence of the HIT 

system sustainability; extrinsic factors in valuing costs and benefits; the cost of the HIT system 

or systems, including initial costs of the hardware and the software; the cost of implementation, 

including planning, hiring, training, temporary productivity loss, data entry, and other 

organizational resources; anything about long-term cost; and outcomes, in terms of changes in 

healthcare utilization, changes in quality of care and patient safety, changes in healthcare costs, 

changes in efficiency and productivity, changes in revenue, and time needed to accrue the 

benefit. These data were judged to be important—and, in some cases, vital—to an understanding 

of the study’s results as generalizable knowledge. 

Synthesis of Results 

Based on considerations about a framework for considering costs and benefits of HIT and 

what constitutes generalizable knowledge, we determined that a synthesis of the results of the 

included studies could not be meaningfully accomplished using conventional EPC methods for 

such syntheses. In other words, because the interpretation of the results of HIT studies is quite 

context-specific, meta-analysis would not be appropriate. No studies were really homogeneous 

or similar enough to consider together. 

Similarly, a narrative review needs an organizing construct, such as “studies about 

CPOE,” or “studies of HIT in rural hospitals,” or even “studies of HIT that incorporate decision 

support and report benefits and costs for patient safety in the capitated ambulatory environment.” 

However, the possible combinations of key variables is so vast that any limited number of 

narrative syntheses we might produce for this evidence report would inevitably not meet the 

needs of many potential users. Therefore, we decided that the most useful synthesis of this 

evidence would be in the form of structured abstracts of the included studies, presented in the 

interactive searchable database, which can be used by interested readers of this report to identify 

those HIT studies that meet their own particular contextual requirements. We also present four 

narrative reviews of studies in particular contexts, to illustrate the uses of the interactive database 

and also as a mechanism to discuss the strengths and limitations of the evidence regarding HIT. 
 22

Peer Review 

A draft of this report was prepared in April 2005 and sent to the TEP members and others for 

review. We received comments from the persons listed in Appendix D∗. Each comment received 

was tracked in an electronic spreadsheet and addressed in preparing the final report. Peer review 

comments and our responses to them are listed in Appendix E. Service as a reviewer of this 

report should not in any way be construed as agreeing with or endorsing the content of the report. 

 

∗ Appendixes and Evidence Tables for this report are provided electronically at

http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. 
 23

3 1 S y s t e m a t ic R e vie w s / M e t a - a n a ly s is

1 3 0 H y p o t h e s is t e s t in g w it h n o c o s t o u tc o m e s

8 2 H y p o t h e s is t e s t in g w it h c o s t o u tc o m e s

1 3 P r e d i c t i ve a n a l y s is

R e fe re n c e L is ts

(n = 31 5 )

I X L ib ra ry S e a rc h

(n = 242)

L ib ra ry S e a rc h 

(N = 3,8 5 0 )

H e a lth A ffa irs J o u rn a l

(n = 98)

E x p e rt s L ib ra ry

(n = 78 )

2 5 6 A r ti c le s Q u a l i ty R e v ie w e d a n d i n cl u d e d in th e H I T in te ra c ti v e d a ta b a se

8 7 0 A rticle s R e q u e ste d

8 5 5 A rticle s S c re e n e d

1 5 R e je c te d :

2 N o t F o u n d

3 D u p lic a te d ata

2 D u p lic a te a rtic le

8 U n a b le to fin d a tra ns la to r

1 2 6 R e je c te d o n A bs tra c t R e vie w

9 9 6 T i tle s C o n s i d e re d R e le v a n t

4 , 5 8 3 T i tle s Id e n ti fie d fo r T i tle R e v ie w

3 ,5 8 7 R e jec te d o n Title R e vie w

5 9 9 R e je c te d :

1 2 4 T o p ic (N o t H IT )

4 O u tco m es n ot re le va n t

2 8 8 D e s c riptive Q u a litative 

(e x c lu d in g R e vie w s )

1 8 3 D e s c rip t ive Q u a n t it a t ive 

(e x c lu d in g M e t a -A n a ly s e s )

Chapter 3. Results 

We screened 855 articles, of which 599 were rejected: 124 did not have HIT as the subject; 4 

did note report relevant outcomes; 288 were descriptive qualitative studies; and 183 were 

categorized as descriptive quantitative studies. A total of 256 articles was included in the HIT 

interactive database. (Figure 1 presents this information pictorially.) 

Figure 1. HIT Literature Flow 
 24

Description of the Studies 

Of the 256 studies included in the database, 156 pertained to decision support, 84 assessed 

the electronic medical record, and 30 were about CPOE (categories are not mutually exclusive). 

One hundred twenty four of the studies assessed the effect of the HIT system in the outpatient or 

ambulatory setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven 

studies used a randomized design; 11 were other controlled clinical trials, 33 used a pre-post 

design, 20 used a time series, and another 17 were case studies with a concurrent control. Among 

the 211 hypothesis-testing studies, 82 contained at least some cost data (or data on utilization or 

efficiency, that could be converted to costs). 

Many of the studies concerned HIT systems developed and evaluated by academic and 

institutional leaders in HIT: the Regenstrief Institute, Partners/Brigham and Women’s Hospital, 

Intermountain Health, Kaiser, Vanderbilt, and the VA health care system. The HIT systems at 

the Regenstrief Institute and Partners were each assessed in 18 and 19 separate studies, 

respectively; 15 assessed the VA health information system; 11 studied Intermountain Health; 5 

studied Kaiser; and 2 assessed the HIT system at Vanderbilt. Studies from these institutions have 

contributed greatly to our knowledge about the usefulness of particular HIT functionalities (such 

as CPOE or computerized electronic alerts) and are examples of what can be realized by the 

implementation of broadly functional HIT at these specific institutions. But these studies also 

have limitations in terms of their usefulness to inform decisions about the adoption of HIT in 

other locations. The primary concern is that these HIT systems were developed over the course 

of many years by champions at these institutions, and, in a process of coevolution, were specially 

adapted to the working environment and culture of their respective institutions. Consequently, 

the “intervention” consists of not only the HIT system but also its local champions, who were 

often also the evaluators in published studies. Furthermore, it is challenging to calculate the cost 

of the development of the HIT system as a whole, since this process has occurred over many 

years. Finally, these systems are not commercially available from vendors, whereas most HIT 

systems in the United States are commercial systems. 

We were able to identify only 15 studies that used a randomized or controlled clinical design, 

included cost data, and assessed HIT systems that were not located at one of the leading 

academic and institutional HIT institutions or in the United Kingdom (UK), another setting that 

has limited generalizability to U.S. health care institutions. When these 15 studies were 

examined for their HIT functionality using the classification system developed by the Institute of 

Medicine,4

 four of them concerned only decision support; four assessed HIT systems with 

decision support and administrative processes; and one study each assessed HIT systems with 

health information and data storage; health information and data storage with decision support; 

order entry management alone; order entry management with reporting and population health 

management; decision support with patient support and administrative processes; and health 

information with data storage decision support and administrative processes. In other words, we 

were unable to find a single study that used a randomized or controlled clinical trial design, 

 

4

 The eight functionalities are documentation (health information and data storage); results management; order entry 

management; decision support; electronic communication and connectivity; patient support; administrative 

processes; and reporting and population health management. 
 25

reported data from a site other than one of the leading academic or institutional HIT systems or 

the UK, reported cost outcomes, and assessed a HIT system that included at least four of the 

eight IOM categories of functionality. 

Of 103 hypothesis-testing studies that used a design other than a randomized or controlled 

clinical trial, 45 reported cost data. Of the 45 studies that reported cost data, 23 assessed systems 

that were not one of the leading academic or institutional HIT systems or UK systems. An 

examination of these 23 studies for their functionalities showed, as in the studies using an RCT 

or CCT design, that most did not evaluate systems with a broad level of functionality. Five 

studies assessed only decision support, and three studies each assessed only administrative 

processes or order entry management. Three studies assessed HIT systems with two 

functionalities: order entry management and decision support. The remaining nine studies 

assessed various combinations of two or three functionalities. No study evaluated a HIT system 

with at least four of the eight categories of functionality. 

Regarding information about the organizational context of a HIT implementation, the 

literature is even more sparse. Of the hypothesis-testing studies, we identified only three studies 

that provided information about the financial context of the organization, such as the degree of 

managed care/capitation penetration; six studies with information about system penetration; one 

study about facilitators to implementation; one studies explicitly discussing sustainability of the 

HIT intervention; twelve studies reporting extrinsic factors in valuing costs and benefits, such as 

the healthcare market competitiveness; and six studies and nine studies, respectively, reporting 

on the initial costs of the HIT system and costs of implementation. No studies explicitly 

discussed sustainability of the HIT intervention. 

In summary, we identified no study or collection of studies, outside of those from a handful 

of HIT leaders, that would allow a reader to make a determination about the generalizable 

knowledge of the system’s reported benefit. Besides these studies from HIT leaders, no other 

research assessed HIT systems with comprehensive functionality while also including data on 

costs, relevant information on organizational context and process change, and data on 

implementation. This limitation in generalizable knowledge is not simply a matter of study 

design and internal validity: Even if more randomized controlled trials are performed, the 

generalizability of evidence will remain low unless more systematic, comprehensive, and 

relevant descriptions and measurements are made regarding how the technology is utilized, the 

individuals using it, and the environment it is used in. 

As is apparent from the preceding discussion, the interpretation of studies of HIT is highly 

context-specific and is not amenable to the techniques of meta-analysis frequently used in other 

evidence reports to summarize results across studies. Certain functionalities of HIT systems have 

been the subject of recent reviews, such as CPOE,10 computer-based clinical decision support 

systems,11-13 and the use of computer-based guideline implementation systems.14 We will not 

summarize these reviews here. Readers are referred to the interactive database of HIT studies to 

select those studies that are most relevant to their own situation in terms of functionalities, 

clinical settings, outcomes reported, and other factors. The remainder of this chapter presents 

four examples of syntheses of the literature for specific situations: the effect of HIT in the field 

of pediatrics; evidence regarding the effect of the electronic health record on quality of 

ambulatory care; studies that report and predict the potential benefits and costs of 
 26

implementation of the electronic health record; and health information technology and patientcentered care. 

The Costs and Benefits of Health Information Technology in 

Pediatrics 

Introduction 

A decision to implement health information technology should carefully weigh the costs and 

benefits of incorporating it into the clinical environment. This is especially true in settings 

involved in the healthcare of infants and children, where patterns of practice and the needs of 

clinicians are unique. A recent report issued by the medical informatics taskforce of the 

American Academy of Pediatrics (AAP) cited a number of special requirements for the effective 

use of electronic medical record (EMR) systems in pediatrics.15 The practice of primary care and 

subspecialty pediatrics requires specialized collection of growth data, immunization history, 

longitudinal developmental inventories, parent education, age- and weight-based norms and 

dosing of therapeutics, specialized terminologies, and unique school-based forms and reports. 

In the area of pediatric patient-safety, a growing number of studies have described the 

frequency of medication errors and adverse drug events (ADEs) in both the inpatient and 

ambulatory settings.16-19 For a number of reasons—including weight- and age-based medication 

dosing, medication unit-doses designed for adult patients, and the limited ability of children to 

communicate or self-check medications before they are administered,20, 21—infants and children 

are at higher risk for serious medication errors and resultant ADEs than are adults. HIT is 

believed to be a vital component in the quest to improve medication safety in pediatrics. 

 These special requirements, combined with a small commercial market for pediatric HIT 

systems relative to the adult population, make the implementation of HIT in the pediatric setting 

challenging and perhaps costly. Clearly, more must be known about the relative costs and 

benefits of HIT implementation and use in pediatrics and evidence of its impact on the six 

quality aims identified in the IOM report, Crossing the Quality Chasm,22 to deliver safe, 

effective, efficient, patient-centered, timely, and equitable healthcare. 

Literature 

Of the 256 articles included in the database, 14 articles were determined to contain 

quantitative data on the costs and/or benefits of HIT use in the pediatric healthcare setting. 

Because of a paucity of evidence, we also included descriptive quantitative studies in this 

section. 
 27

Summary of Evidence 

Medication Use and Patient Safety. Given recent insight into the prevalence of medication 

errors in the pediatric population, health information technology is believed by most to be an 

important tool in reducing the rate of medication errors that occur in the care of infants and 

children. 

Mullett et al.23 enhanced an existing adult antiinfective computerized decision-support 

system for use in an academic pediatric intensive care unit (PICU) and measured its impact on 

medication-related outcomes. The study reported a 59-percent decrease in pharmacist 

interventions for erroneous drug doses and a decreased number of patient days of subtherapeutic 

(p<0.001) or excessive (p<0.001) antiinfective doses. In addition, the surveyed physicians 

reported that the use of the system improved their antiinfective choices and perhaps reduced the 

likelihood of ADEs. The authors also reported a decreased number of orders per patientantiinfective course as well as decreased robust estimated costs of antiinfective use by 9 percent 

in the intervention group vs. control ($86.60 vs. 78.43). 

A study by Fortescue and colleagues24 examined and characterized 616 medication errors 

occurring in the pediatric inpatient units of two academic tertiary referral medical centers. In a 

hypothetical experiment, physician experts determined what percentage of these errors could 

potentially have been prevented by the implementation of safety systems. Specifically, this 

hypothetical experiment determined that basic CPOE would avert 60 percent of potentially 

harmful errors, while CPOE with clinical decision-support systems (CPOE +CDSS) would 

increase the prevention of harmful errors to 75.8 percent. Other HIT systems identified by the 

report as being important for averting medication errors in pediatrics settings included 

computerized/electronic medication administration record (e-MAR) (19.2 percent of potentially 

harmful errors), robots in pharmacy (2.5 percent), smart intravenous infusion devices (4.2 

percent), medication and patient and staff bar-coding (4.2 percent), and an automated bedside 

medication dispensing device (5.8 percent). 

A number of studies have directly measured the benefit of CPOE using a variety of errorcapture methodologies and study designs in different pediatric clinical environments. In a 

prospective cohort study, the authors documented medication prescribing errors (MPEs) and 

potential adverse drug events (PADEs) in a pediatric intensive care unit before and after 

implementation of a “home-grown” CPOE system.25 The data showed a significant reduction of 

both MPEs (30.1 to 90.2 percent, p< 0.001) and PADEs (2.2 to 1.3 percent, p<0.001). A study by 

Cordero and colleagues in the neonatal intensive care setting (NICU) showed that CPOE could 

eliminate gentamicin prescribing errors as well.26 The sum of this early evidence indicates that 

CPOE +CDSS has significant potential to reduce harmful medication errors, but the relative 

costs and complexities of achieving these beneficial outcomes need to be examined further. 

Immunizations. Although a growing body of literature suggests that the use of HIT in 

pediatrics may be an important ingredient in reducing medication errors, a key challenge for 

pediatric providers lies in the area of maximizing adherence to vaccination recommendations. 

Paper-based immunization records do not allow for rigorous population-based monitoring or 

quality control. Therefore, computerized immunization registries, as separate or integrated 
 28

systems and with clinical decision-support or reporting capabilities, offer tremendous potential in 

tracking and improving the rates of adherence to recommended immunization guidelines. 

Ornstein et al.27 evaluated a computer-based preventive services alerting system integrated 

into an electronic medical record system in an academically affiliated family practice clinic. In 

addition to surveying patient and physicians regarding their perceptions of the reminder system, 

the researchers performed before-and-after audits of adherence to recommended preventative 

services including childhood immunizations. Of the five immunization services tracked, only the 

administration of diphtheria and tetanus booster showed a small but significant improvement 

(48.8 to 50.6 percent, p=0.02). Adherence to the other recommended vaccinations did not show a 

significant improvement. 

Szilagyi, and colleagues,28 in an academically affiliated pediatric urban clinic, used a 

computerized database system to generate reminder letter for influenza vaccination to patients 

identified with moderate to severe asthma. Eligible patients were randomized into an 

intervention group, which received the reminders, and a control group. After four months, a 

review of the medical chart revealed a significant difference in influenza vaccination rates (30 

percent intervention vs. 7 percent control, p<0.01). This study demonstrated that computerized 

disease registry systems could serve as an important tool in improving vaccination rates in 

pediatrics. 

Effective Disease Management. In addition to providing a potential means to influence 

prescribing and immunization practices in pediatrics, HIT systems also hold tremendous promise 

in improving clinical decisionmaking and disease management. 

Medication Dosage and Delivery. Chiarelli et al.29 evaluated a microprocessor device with 

computerized algorithms for insulin dose adjustment for pediatric patients with insulindependent diabetes mellitus, based on self-monitored blood glucose (SMBG) levels. This 

prospective randomized on-off-on study revealed that although the mean glycosylated 

hemoglobin levels and pre-meal SMBG levels did not improve, control patients were more likely 

to experience episodes of hypoglycemia than were patients using the device, and patients using 

the device used less insulin than during their corresponding baseline phase (p<0.0001) and less 

insulin than the control group. 

Disease-Based Clinical Decision Support. Schriger and colleagues30 implemented an 

electronic medical record in a university hospital emergency department that provided 

documentation advice and recommendations for laboratory testing and treatment. Using an onoff-on interrupted study design, the authors measured appropriateness of care for febrile children 

less than three years of age, when measured against an evidence-based guideline. No evidence 

was found for improvements (or worsening) in appropriateness of care during the intervention 

phase compared to the baseline phase. However, use of the system was found to increase 

documentation of essential elements of the history and physical examination by 13 percent (95% 

confidence interval, 10 to 15 percent) as well as documentation of after-care instructions by 33 

percent (95% confidence interval, 28 to 38 percent). 

The appropriate course of antibiotic treatment for acute otitis media (AOM) is an area of 

concern in pediatrics. A study by Christakis et al.31 measure the impact of HIT on the antibiotic 

prescribing behavior of pediatric providers in an academic pediatric residency training clinic and 

compared cohorts during the pre-intervention and post-intervention phases. During the post-
 29

intervention phase, providers were randomized to receive point-of-care advice recommending a 

course of antibiotics of less than 10 days duration (primary outcome) or delayed initiation of 

antibiotics (secondary outcome) for the treatment of AOM. Measurement of adherence to this 

computerized alert showed that providers in the intervention group had a 34-percent increase 

compared to the control group in the proportion of antibiotic prescriptions that were for less than 

10 days (p<0.01). However, during the intervention period, both the intervention and control 

groups became more likely to prescribe antibiotics, with the intervention group deteriorating less 

than the control group (p<0.095). The results demonstrate that the prescribing practices of 

pediatricians for treatment of a common pediatric illness can be affected by a computerized 

reminder system. 

Using a similar study design, Margolis et al.32 developed a computerized algorithm system 

that mandated structured input of data by providers for common pediatric problems. In return, 

the system provided recommendations for disease management and correct use of antibiotics. 

The investigators demonstrated decreased use of antibiotics for OM (p<0.001) and pharyngitis 

(p<0.01) as well as increased adherence to protocol recommendations for these two disease 

processes in the intervention group compared with the control group. However, the use of 

antibiotics for upper respiratory infections (URIs) did not change. The authors noted that the 

structured algorithms in the HIT system did improve the documentation of clinical elements 

important to ideal clinical care of pharyngitis, otitis media, and upper respiratory infections. It 

must be noted however, that this system’s rigid requirements for physician documentation also 

made the HIT system unusable, and the physicians refused to use the system after five weeks. 

Improved Documentation. Because many studies have reported an impact of HIT use on the 

quality and completeness of medical documentation, a study by Carroll and colleagues focused 

on the impact of a personal digital assistant (PDA) on documentation discrepancies in a NICU.33

In this before-and-after study, all the NICU resident physicians used a PDA-based charting 

system during the intervention phase, comparing their progress notes against a predefined 

reference standard during both phases. The authors demonstrated that after adjustment for 

covariates, PDA-based charting did reduce discrepancies in patient weights in the charts but did 

not affect the number of medication or vascular line discrepancies. 

Timeliness, Efficiency and Cost-Effectiveness of Care. Quattlebaum et al.34 studied a 

scheduling/practice management system that automatically generated reminder postcards for 

appointment the following week. In this randomized controlled trial, the authors demonstrated a 

reduction of the no-show rate in their pediatric ambulatory practice from 19 to 10 percent. A 

cost-benefit analysis of the HIT system and its impact on missed appointments revealed that for 

each $1 spent on reminders, an additional $7.50 of revenue was captured. 

In the inpatient setting, the previously discussed study of CPOE in the NICU by Cordero and 

colleagues26 measured not only CPOE’s effect on gentamicin dosing errors but also the time 

from medication prescription to administration for initial doses of a single medication and 

radiology tests during the pre- and post-CPOE phases. The authors documented significant 

reductions in the average turnaround time for both medications (10.5 to 2.8 hours, p<0.01) and 

radiology tests (42 to 32 minutes, p<0.001). 
 30

Summary 

Early evidence shows that stand-alone CDSS can reduce medication dosing errors, and 

CPOE + CDSS can reduce the incidence of harmful medication errors in the inpatient pediatric 

and neonatal intensive care settings. However, other HIT systems, such as electronic MAR, 

pharmacy-based robots, smart infusion pumps/devices, and medication bar coding, are predicted 

to reduce medication errors but need to be studied further. 

HIT also has tremendous potential to improve vaccination rates and disease management in 

pediatric outpatients. CDSS and registries have been shown to be effective in increasing 

vaccination rates in targeted populations, but only a limited HIT impact on general pediatric 

immunization rates has been demonstrated. Similarly, a patient clinical decision-support device 

that assists insulin dosing in children with diabetes reduces episodes of hypoglycemia and 

overall insulin requirements, but does not affect traditional measurements of glycemic control. 

And the use of computerized documentation systems with integrated CDSS has been 

demonstrated, in separate studies, to 1) reduce the frequency or duration of antibiotic use for 

common pediatric illness such as pharyngitis and otitis media, and 2) improve completeness and 

somewhat reduce variation in clinical documentation. 

In the ambulatory setting, a single study showed that an appointment reminder system is 

cost-effective and significantly reduces missed appointments. Another study showed that CPOE 

in the NICU can reduce medication and radiology turnaround times. Therefore, the evidence for 

HIT cost-savings in pediatrics is limited but deserving of optimism. 

Conclusion 

A small body of literature supports the assertion that HIT use in pediatrics is beneficial in the 

areas of medication safety, adherence to immunization and disease-based guidelines, patient 

decision-support in diabetes management, clinical documentation, patient appointments, and inhospital order processing. No data on the costs or cost-effectiveness of implementing these 

systems were found, except in one case. In addition, because many of these HIT systems were 

tested and/or developed in academic settings, the ability to generalize these findings to other 

organizations is uncertain. 
 31

Electronic Health Records and Quality of Ambulatory Care 

Introduction 

Despite rapid advances in the biomedical sciences, a growing body of evidence shows 

serious shortfalls in the quality of care Americans receive,35,36 and significant longstanding 

shortfalls in performance have persisted despite recent increases in attention to quality.37-40

International comparisons have demonstrated similar problems in quality.41-43 If the United 

States is to realize the full value of biomedical knowledge and of financial investments made in 

healthcare, the mechanisms through which that knowledge is operationalized and care is 

delivered must be radically redesigned. 

Although the content of healthcare continues to change dramatically, the methods of health 

care delivery have not. In particular, a vast majority of the healthcare industry continues to 

deliver care, manage information, and conduct clinical transactions through the use of paper 

records. 

Although the use of electronic health records (EHRs) is limited in healthcare, there is a 

renewed conviction by the government, provider groups, and healthcare purchasers that 

widespread adoption is critical to the delivery of consistent, high-quality care. However, EHR 

implementation, without other important changes in the way healthcare services are provided, is 

unlikely to improve quality. Such process redesign and reengineering is difficult and resourceintensive and is also hampered by the complexity and fragmentation of our current healthcare 

system. Therefore, despite the potential benefits of widespread EHR use, better empirical 

evidence is needed to confirm that EHR use does in fact improve quality and—perhaps more 

fundamentally—to understand what capabilities EHRs need to have for quality to be improved. 

At present, the depth and breadth of the empirical evidence regarding EHR use and its 

attributable impact on the quality of care remains unclear. 

The purpose of this review is to examine and synthesize the available research evidence for 

the impact of EHR on quality of care in the outpatient setting. The review will also attempt to 

differentiate the direct impact of EHRs as point-of-care and workflow tools from how EHRs 

have been used to indirectly achieve those results, by measuring clinical and process outcomes. 

We elected to focus on ambulatory care because of the large volume of health services delivered 

in this arena. In addition, because the vast majority of outpatient practices comprise fewer than 

ten providers—many of whom lack technical infrastructure and resources—it is unclear whether 

widespread implementation of EHRs will be feasible in this environment. 

Research Study Inclusion Criteria 

From our database of 256 articles, we selected all 84 papers that related to EHRs. We then 

screened these articles against the following inclusion criteria: (1) the study reported quality-ofcare data as study outcomes, (2) the EHR was documented to have the following minimal 

functionality—electronic documentation (viewing, entry, or both), results management, CPOE, 

and some form of decision support, (3) the study was conducted in the ambulatory setting. The 
 32

criteria for functionality were chosen based on the IOM’s “White Paper on Key Capabilities of 

an Electronic Health Record.” Given the rapid technical advances in EHR systems, we reviewed 

additional functional criteria to provide decisionmakers with the most relevant and forwardlooking information available. 

Analytic Framework 

The Donabedian “Structure—Process—Outcome” model for quality was used as a 

framework for this review.44 In this model, structure is defined as the resources and factors 

involved in producing care and the manner in which those resources and factors are organized. 

Examples of structural quality include the number of beds in a hospital, the number of physicians 

in an emergency room per shift, the budget for a clinic, and the presence of disease management 

program for diabetes. Process of care is defined as the activities that constitute health care. 

Examples include screening for breast cancer, ordering laboratory tests, and prescribing a 

medication. Outcomes are the end results of healthcare delivery processes. They are the 

consequences of health services or can be logically attributed to the act of providing those 

services. Whereas structure relates to the environment in which healthcare is delivered and 

process relates to the provisions of care, outcomes are events that occur with patients and 

consumers—as individuals, groups, or populations. 

Two aspects of this model are particularly relevant to EHRs. First, to fully assess quality of 

care, there need to be links from structure to process to outcomes. The technical and functional 

capabilities of EHRs form a structure for care. In order to derive value from the EHR structure, 

new clinical processes need to be designed to utilize the EHR functional structure. These EHRmediated processes should in turn lead to a specific set of better outcomes. Second, the 

distinctions among structure, process, and outcome are somewhat arbitrary in the model. Health 

care delivery is viewed as an interconnected series of structure—process—outcome 

relationships. For example, in a primary care clinic of three physicians (structure 1), a patient 

may have an electrocardiogram performed (process 1), which shows an abnormality (outcome 1). 

This abnormal result necessitates a referral to a cardiologist (structure 1) who orders a stress test 

(process 2), which comes back suggestive of coronary artery disease (outcome 2) and so forth. 

This structure—process—outcome chain is central to the role of EHRs in quality, because an 

EHR is a tool that explicitly links the three. An EHR with decision support for diabetes 

management (structure) allows a physician to order a hemoglobin A1C (process) and check the 

results (outcome). Because this outcome is stored in the EHR database, it in turn becomes part of 

the structure of care. The EHR can allow or even remind the physicians to act on that result 

(process 2), e.g., modify the patient’s insulin dose, which will, in turn, lead to a lowering of the 

patient’s blood sugar (outcome 3) or a reduction in the likelihood of a long-term diabetic 

complication. 

In addition to imposing the Donabedian Structure—Process—Outcome model, we organized 

deficits in quality by means of a conceptual framework that divides quality problems into three 

types: (1) the underuse of appropriate health services, (2) the misuse or inappropriate use of 

health services, and (3) the overuse of health services.36 
 33

Analysis 

Seven research studies were identified using the search criteria outlined earlier.45-51 Four of 

the seven were conducted at academic medical centers,45-47, 51 and three of those four were 

conducted at a single institution, Regenstrief Institute (the fourth was conducted at Beth Israel 

Hospital in Boston).45-47 Two studies were conducted at a large, integrated healthcare delivery 

network, Kaiser,48, 49 and one was conducted in the Netherlands.50 All four studies from the 

academic medical centers assessed internally developed HIT systems, rather than a commercially 

available system. One study48 assessed two different systems at two sites, one of which was 

internally developed by the organization and the other a commercially developed product. 

All studies included data on structural quality. These varied highly and were largely 

qualitative in nature. In particular, reporting on the organizational and workflow changes needed 

to implement an EHR or a new EHR functionality was limited. All seven studies analyzed 

quality with respect to process of care. Six of the seven45-49, 51 assessed quality with respect to 

some type of outcome. 

In terms of the types of problems the interventions were trying to address, six of the seven45-

50 included data on the effects of EHRs on decreasing overused or redundantly used healthcare 

services. Two48, 51 included measures of the effects of EHRs on appropriate but underused care. 

None used explicit methods to evaluate the impact of EHRs on inappropriate use of care. 

 

EHR Systems in Use at Regenstrief 

Structure. Three studies that met our criteria were conducted at the Regenstrief Institute, 

which includes a research institute and an ambulatory care practice affiliated with both a 

university medical school and a large public hospital.45-47

The development of the EHR at the Institute began in the mid-1970s. Subsequent system 

enhancement and implementation broadened its functionality and scope of use. In 1984, CPOE 

was added to the EHR capabilities and became uniformly used in the outpatient setting. In the 

three studies covered in this analysis, the system included electronic documentation, results 

management, CPOE, and decision support. 

All three studies examined the effects of incorporating new information elements into the 

process used by physicians to order diagnostic tests. Each involved the integration of EHR-stored 

data into physician decisionmaking at the point of care. 

The first paper45 reported on the effect of a structural change in care delivery: the addition of 

diagnostic test cost data to the EHR order function. Thus, physician workflow was altered 

through the inclusion of cost data in the order entry process, to be shown at the point of care. 

After physicians ordered tests, the charges for each test and the total charges for all tests were 

provided automatically in a new window. Physicians were then offered the option to cancel any 

or all tests. 

This study used a complex randomized design to test the effect of the intervention. First, 

baseline utilization data were collected during a 14-week observation period. Second, physicians 

were randomized either to receive the cost data during order entry or to use the usual EHR 
 34

functional interface where no cost data were provided. Finally, data collection continued for a 

19-week post-intervention period. 

The second paper reported on the effect of an intervention47 in which the investigators 

created statistical models to predict the likelihood of abnormal results for commonly ordered 

diagnostic tests. These pretest probabilities were displayed to physicians immediately prior to 

test ordering. Data needed for the models were obtained through EHR mediated prompts to 

physicians and from patient-specific data already electronically stored. Physician workflow was 

altered by the need to enter data during the test-ordering process and by the incorporation of the 

pretest probability into their decisionmaking process. The number of data prompts given to 

physicians was not reported. 

The study used a randomized design in which patients were the unit of randomization. The 

EHR sorted patients automatically by the predetermined allocation. When physicians cared for 

intervention patients, the pretest probability function was activated during the electronic ordering 

process. When physicians ordered tests for the control patients, no additional decision support 

was provided. 

The third paper reported on an intervention47 in which care delivery was modified through an 

intervention in which past diagnostic test results were automatically displayed as physicians 

ordered new tests. The last three results for a test, the time interval between tests, and the total 

number of times the test had been ordered for the patient were displayed at the point of care. No 

additional data entry was required of physicians. Physician workflow was altered by the need to 

incorporate past test results during decisionmaking. 

The study used a complex multiphased, randomized design. First, during a 13-week preintervention period, baseline data were collected regarding physician test ordering patterns. 

During the 16-week intervention period, patients were the unit of randomization. Finally, test 

ordering was monitored during an 8-week post-intervention period. 

Process. Each of these studies examined the impact of the EHR-related structural 

improvements described previously on physician diagnostic test ordering practices. In each 

study, the EHR-based intervention decreased the number of diagnostic tests ordered by 

physicians, suggesting that quality of care was improved through the decrease in overused health 

services. 

The first study,45 in which test charge data was displayed, showed an overall 14-percent 

decrease in the number of tests ordered by physicians per visit in the intervention group. 

Decreases were observed for both scheduled and for unscheduled visits. The multiphase design 

allowed additional conclusions to be made regarding the importance of maintaining the decisionsupport element as part of the structure of care. In the pre-intervention period, no differences in 

test ordering were noted. During the intervention period, physicians randomized to the decision 

support tool ordered 17 percent fewer tests. In the post-intervention period, after the decision 

support was removed from the EHR, physicians who had been in the intervention group ordered 

only 7 percent fewer tests than during baseline. This effect decrement suggests that the 

knowledge of costs the test physicians gained during the intervention was not sufficient to alter 

practice over time. Instead, it was the presence of the additional cost information within the 

structure of care that most affected performance. 
 35

The second study,46 in which abnormal test result probabilities were displayed to providers, 

showed a 9 percent overall decrease in the number of tests ordered by physicians. Two tests, 

urinalysis and urine culture, which had been underused prior to the intervention, showed 

increases in ordering frequency (+14 percent and +27). However, for the other six, overused tests 

examined, decreases ranged from 4 percent (electrocardiogram) to 14 percent (chest x-ray). 

These findings suggest that providing point-of-care pretest probabilities via an EHR improves 

the quality of care processes by decreasing overused testing and by increasing the use of 

previously underused care. 

The third study,47 in which past abnormal test results were displayed, showed an overall 9-

decrease in the number of tests ordered by physicians. As in the first study, data were analyzed in 

the post-intervention period to assess the persistence of the effect. After the EHR intervention 

was discontinued, the researchers observed a non-statistically significant 11-percent increase in 

the number of tests ordered (the post-intervention period time frame was not long enough for this 

trend to reach statistical significance). 

Outcomes. In the first study,45 the primary outcome was diagnostic test-related charges. 

Charges were 13 percent ($6.68 in 1988 dollars) lower per visit for the intervention group 

physicians than for the control group. Decreases in charges were directly due to the decrease in 

the number of tests ordered, i.e., the improvements in quality of care processes. Given the 

likelihood that the intervention reduced overused care, this outcome increases the efficiency of 

care delivery. 

In the second study,46 which used statistical models to predict whether any of eight 

commonly ordered diagnostic tests were likely to be abnormal, the primary outcome was 

financial charges for tests. Overall, charges decreased 9 percent ($1.09 in 1986 dollars). A 

technical outcome of the study was the operating characteristics of the statistical models used to 

predict lab test abnormalities, which were based on data collected and stored in the EHR. All 

predictive models for the study tests performed well, with receiver operating curve areas 

generally over 0.75 (range 0.66 to 0.92). 

In the third study,47 efficiency outcomes were also observed due to the decrease in the 

number of tests ordered. Charges for tests in the intervention group were 13 percent lower per 

visit (approximately $1.82 per visit in 1986 dollars). 

EHR Systems in Use at Kaiser Permanente 

Structure. Two studies came from regional medical centers in the Kaiser Permanente 

network, an integrated, not-for-profit, nonacademic healthcare delivery system. 

In the first of these two studies,48 comprehensive EHRs were implemented in two regions 

(Northwest and Colorado) of the enterprise. One EHR was internally developed (Colorado) and 

the other externally developed by a commercial vendor (Northwest). Although the EHR systems 

were different, both were reported to have comparable functionality, including the following: 

documentation, clinical results management, CPOE for both diagnostics and medications, 

administrative data management, and decision support. Specific decision support functions 

varied between the two sites in both content and scope. 
 36

The second of these two studies provided a brief qualitative description of structural changes 

associated with and supporting EHR implementation. Implementation was carried out gradually, 

in phases, beginning in discrete areas of the ambulatory care clinics. The majority of system 

implementation was completed within one year of initiation. The authors note that because of 

phased implementation, it was “some time” before changes in health care delivery were noted. 

(Data related to time course of impact will be discussed in the Outcomes section of this analysis.) 

No data on the specific organizational drivers for adoption were included. 

Structural reorganization and improvements in workflow were described briefly. Prior to the 

implementation of EHR in each region, the presence of multiple ambulatory sites required paper 

records to be physically delivered. Paper charts were warehoused and had to be delivered 

“several miles.” For same-day and unscheduled ambulatory visits, availability of paper records 

was “unreliable.” After EHR implementation, use of paper charts was “essentially eliminated” 

and electronic patient charts became available for emergency room visits, unscheduled visits, and 

same-day appointments. Charts also became available for telephone contacts, and the resulting 

improvement in clinical workflow led to more effective utilization of telephone-based care, with 

physicians reporting that they were better able to address patient health issues over the phone 

when provided with access to electronic records. The authors cite this outcome as a primary 

reason for decreased office visits, one of the primary outcomes of the study. 

In terms of time frame of impacts, little difference in services was noted during the first year 

of implementation (the authors reported system implementation was mostly completed one year 

after implementation began). 

Evaluating the effects of EHR adoption was itself a form of structural change in this study. In 

order to determine appropriate utilization measures to assess the effect of EHR implementation, 

interviews were conducted with 100 individuals with a broad array of organizational roles. 

Interviews led the investigators to hypothesize that ambulatory care delivery had become more 

efficient by making needed information available during the initial episode of care, thus 

decreasing the need for follow-up visits and redundant services. Interviews also suggested that 

quality had improved. These hypotheses formed the basis for the selection of metrics and the 

quantitative evaluation done in the study (discussed in the Process and Outcome section of this 

analysis). No further details were provided regarding the data acquired from these interviews or 

the methodology used to conduct them. 

The study design was a retrospective time-series analysis with data analyzed at one-year 

intervals before and after implementation. Baseline data were used from the three years prior to 

implementation. For the Kaiser Northwest site, four years of post-implementation data were 

available, whereas for Colorado, only two years of post-implementation data were available 

because of later implementation. 

The second of the two studies, conducted in the Kaiser Northwest system,49 examined the 

incorporation of guidelines through the EHR to support the decisionmaking process for ordering 

radiology tests and medications. The two-phase implementation process was described briefly. In 

the first, a read-only results reporting system that integrated data from departmental systems was 

implemented. In the second phase, the commercially developed EHR described above for the 

first Kaiser study was implemented. Together, both phases took approximately three years to 

complete. Per the authors, attempts were made to present guidelines to providers as efficiency 
 37

aids that would streamline their workflow; the electronic guidelines were kept simple and 

integrated smoothly into existing procedures. Provider adherence to guidelines was not 

mandated; however, the electronic ordering system was designed to make adherence simple, and 

the guidelines were presented in text form without requiring explicit interaction. No further 

specific implementation-related information was provided in the paper for either the EHR or the 

guideline tools. 

The study design was a time-series analysis that examined utilization patterns at multiple 

time points before and after guideline implementation. 

Process. The first Kaiser study,48 which examined the effects of EHR implementation at two 

sites in the Kaiser network, examined multiple processes of care. 

Three quality indicators from the Health Plan Employer and Data Information Set (HEDIS) 

were chosen to assess quality. These items were chosen in part because their definitions 

remained consistent over the time period of the analysis. Each was a process-of-care measure: 

advice on smoking cessation, cervical cancer screening, and retinal eye examination. No 

statistically significant differences were found in performance on these process measures from 

the pre- to the post-implementation period. 

However, multiple utilization-related processes were examined and showed considerable 

change after EHR implementation. In general, they suggest improvements in quality of care 

through a decrease in redundant health services. Age-adjusted rates of radiology test utilization 

decreased overall by 4 percent after EHR implementation. The authors note that over this same 

period, radiology service use increased within the Kaiser system as a whole and nationally as 

well (quantitative data not provided for either increase). Laboratory test utilization in one site 

decreased 18 percent four years post-implementation. However, utilization rates subsequently 

increased 5 to 7 percent annually. In the other site, the rate of laboratory test utilization had risen 

14 percent prior to EHR implementation, but decreased by 3 percent over the two postimplementation years included in this study. Comparisons of laboratory utilization with other 

non-EHR sites in the network were not included in the analysis. The number of telephone 

encounters physicians scheduled with patients increased substantially after EHR implementation, 

rising from 1.3 telephone encounters per member per year to 2.1 telephone encounters per year. 

Per the authors, physicians qualitatively reported that telephone encounters were more effective 

because their capacity to resolve patient issues was enhanced by accessing the EHR. 

The second Kaiser study49 focused on processes of care–related adherence for two radiology 

tests and on formulary adherence for one medication, after guidelines were incorporated into the 

EHR. 

Use of upper gastro intestinal (UGI) radiology testing decreased from 11 UGI per thousand 

members to 6 UGI per thousand members after guideline implementation (40-percent relative 

decrease). The number of chest x-rays ordered also decreased 20 percent. Prescription of a 

nonformulary medication for depression decreased from 4.7 percent of all selective serotonin 

reuptake inhibitors (SSRIs) to 2.4 percent (SSRIs are the most widely used class of medications 

for depression and multiple agents are available for prescription in this class). Noted effects were 

sustained over time. The analysis made no attempt to control for other factors that may have 

affected utilization of radiology testing or formulary adherence. 
 38

Outcome. Outcomes in the first Kaiser study related to efficiency and utilization.48 The ageadjusted total office visits decreased by 9 percent in year 2 after initial implementation. Primary 

care visits decreased by 11 percent and specialty care visits decreased by approximately 5 

percent, both of which were statistically significant. Reductions in visits held across patient 

cohorts, including those with the greatest baseline rates of visits. The number of patients making 

three or more visits decreased by approximately 10 percent between year 1 and year 2 postimplementation in the Northwest region and by 11 percent in Colorado. In year 4, a further 

decrease of 2 percent was noted in the Northwest region. No comment was made on whether 

these decreases in the high-volume use category were statistically significant. Direct 

comparisons with utilization at non-EHR sites were not possible because of inconsistent 

definitions of office visits. However, in three other network regions (all of which used 

independent definitions of a visit) for which visit utilization data were available for the same 

time period, no similar decreases were noted. 

In terms of the statistical analysis, no strict control variables were included in the analysis. 

The following structural measures were reviewed separately to examine possible confounding: 

rates of ER visits, ratio of primary care providers to members, ratio of referrals to outside 

providers. Per the authors report, none changed significantly over the study time frame. 

Appointments made for patients after doctor-managed telephone encounters decreased by 7 

percent after the EHRs were implemented. However, when telephone contacts reverted to nurses, 

these appointments “rose” (no quantitative data provided). 

EHR Systems in Use in the Netherlands 

Structure. A single report details a large multisite study in the Netherlands in which the 

effects of two different types of EHR laboratory test order interfaces were examined.50 Both 

sought to decrease the number of laboratory tests ordered by providers by presenting a limited 

set of tests on the primary laboratory order screen in the EHR. While all available tests in a 

laboratory system cannot usually be presented at once on a computer screen, these interventions 

did not allow screen size or human factor constraints to dictate which test options were initially 

made available to providers. Instead, they presented considerably smaller sets of choices. Thus, 

both interfaces changed provider workflow considerably when compared to paper or to 

nonrestrictive EHR order interfaces. Although providers could order any tests they wanted, any 

test not explicitly present on the EHR laboratory screen required additional search time to find 

and call up. 

In one experimental condition, statistical probability was used to select the fifteen most 

commonly ordered tests overall to present to a provider on the initial order interface. In the other 

condition, the tests presented to providers depended on the patient’s specific diagnosis. 

Diagnosis-specific tests were presented electronically, based on recommendations from existing 

guidelines. This intervention altered provider workflow to a greater extent than did the first 

intervention. First, a menu of guidelines/indications was presented, from which the provider had 

to select those most relevant to the patient’s conditions. Based on the indications for testing 

entered by the provider, the EHR picked the most relevant to present as possible options for 

ordering. The guideline set was not comprehensive for all possible tests, and all possible 

indications for a test were not included in the electronic guidelines. Physicians could override 
 39

recommended tests and order nonrecommended tests at their discretion by entering, “other 

indication.” 

All physician practices in the sample were already using EHRs at the time of the proposed 

laboratory-ordering intervention. However, prior to the intervention, lab tests were ordered 

through structured paper-based order forms. All practices in the region using EHRs were offered 

the opportunity to participate in the experiment and add one of the electronic lab ordering 

functionalities to their systems. Of 64 practices, 46 (72 percent) agreed. Sixty-two general 

practitioners worked at those 46 sites. A three-month implementation period was included to 

familiarize the physicians with the software. Over the course of the study, four practitioners 

withdrew: one solo practitioner withdrew because the software decreased the performance of his 

computer, another withdrew because of dissatisfaction with the system, and two other physicians 

in the same practice withdrew for unspecified reasons. Thus, complete data were available for 44 

practices, representing 60 physicians. 

Physicians were still left with the option of using paper order forms during the study. In the 

non–guideline specified cohort, 88 percent of all orders were entered through the software. In the 

guideline-based electronic order cohort, 71 percent of all tests were ordered through the 

software. Final data analysis included total lab tests ordered both electronically and through 

paper forms. 

Process. This study50 focused primarily on process change: examining the effect of changes 

in information presentation on test ordering. Physicians randomized to the guideline-based 

interface ordered 1.4 percent fewer lab tests (5.5 vs. 6.9) than did physicians presented with the 

list of most commonly ordered tests. This difference translated into a relative decrease of 20 

percent in tests ordered. The 20 most commonly ordered tests accounted for 80 percent of all 

tests. No data on human factors issues or usability were reported. Such data may have been 

informative, given the different workflows created by each intervention. Further supporting the 

potential utility of such data are the different rates of use for each software package (in the 

guideline cohort, 71 percent of all tests were ordered through the software and 29 percent 

through paper; in the other cohort, 88 percent of all tests were ordered through the software and 

the remaining 12 percent through paper). 

Outcomes. No outcomes were reported for this study.50 

EHR Systems in Use at Beth Israel Deaconess Medical Center (Boston MA) 

Structure. The last EHR study was conducted at the ambulatory care medicine practice at the 

Beth Israel Deaconess Medical Center in Boston, an academic medical center.51 Development of 

their clinical computing system began in the 1970s and was internal. System functionality at the 

time of the study included documentation, results management, order entry, decision support 

administrative data management, and electronic communication through email. Electronic 

documentation and results management capabilities were available through the Internet. 

The goal of the study, which began in 1990, was to improve quality of outpatient HIV care 

by incorporating guideline-based alerts and alarms into the system. At the time of the study, no 

national consensus guideline on HIV care existed. Thus, as a first step, a set of guidelines was 
 40

developed internally by a panel of local experts. The guidelines were then automated and 

incorporated into the EHR. 

The alerts and alarms created new provider workflows when compared to a paper-based 

system. Decision support was given to providers on-line and without provider prompting. 

Clinicians were given the opportunity to act on the alerts and reminders as they appeared, by 

sending electronic messages for orders to be executed. The system also allowed providers to 

decline recommendations; and space was included in the EHR to document the reason. The 

workflow for each of those options differed. Alerts popped up each time a provider logged on, 

regardless of the patient being seen or reason for accessing the system. Reminders were shown 

only at the time of the patient visit. This study, which was conducted over 18 months (from 1992 

to 1993), used a controlled clinical trial design. Five practice sites were involved. Coin flips were 

used to assign practices to the intervention or the control condition. All clinicians at a site were 

assigned to the same condition over the course of the study. The total sample included 22 

providers. 

Process. The purpose of this study51 was to assess the effects on processes of care of 

incorporating electronic guidelines for outpatient HIV care into the EHR. One year after 

implementation of the EHR guidelines, the number of eligible patients receiving recommended 

HIV care in the alerts intervention group was 85 percent vs. 64 percent in the control group. At 

three months post-implementation enhanced utilization of appropriate services was noted for all 

measures, including ordering CD4 counts (82 percent vs. 60 percent), starting AZT or DDI when 

appropriate (86 percent vs. 65 percent), modifying AZT dose (76 percent vs. 62 percent), PCP 

prophylaxis (88 percent vs. 42 percent), and complete blood counts (89 percent vs. 65 percent). 

All findings were statistically significant to a p value of 0.05 except for starting AZT/DDI or 

changing the AZT dose. The median response time for a provider to order appropriate services in 

response to new clinical information was 11 days in the intervention group and 52 days in the 

control group. 

At one year, the number of eligible patients receiving recommended HIV care in the 

reminders intervention group was significantly greater than in the control group (68 percent 

versus 46 percent). Processes of care examined included pneumovax receipt (82 percent vs. 38 

percent), TB skin testing (78 percent vs. 62 percent), H. influenza vaccination (41 percent vs. 25 

percent), tetanus vaccination (31 percent vs. 17 percent), and referrals to ophthalmology (75 

percent vs. 46 percent) (p values were less than 0.05 for all results except TB testing, for which 

p=0.07 and tetanus vaccination, for which p=0.1). At one year, toxoplasmosis titers were drawn 

on an equivalent percent of patients (82 percent vs. 81 percent). However, the median response 

time in the intervention group was 8 days vs. 168 days for the control group. No differences in 

cervical cancer screening were noted. In the intervention group, the median time for a provider to 

act on clinical information to order appropriate services was 114 days vs. more than 500 days in 

the control group. 

In the intervention patient cohort, 303 alerts and 432 reminders were generated and sent to 

clinicians. In the control group, 388 alerts and 360 reminders would have been sent to providers. 
 41

Outcome. This study51 examined rates of visits to primary care, rates of hospitalizations, 

visits to emergency rooms or walk-in clinics. No statistically significant differences were 

observed. Rates of pneumocystic disease and one-year mortality also showed no differences. 

Conclusions 

The studies reviewed in this analysis illustrate a range of ways in which ambulatory EHRs 

can serve to improve quality of care. In particular, they demonstrate how provider performance 

can be improved when the clinical information management and decision support tools are 

available within an EHR system. A recurrent theme in these studies was the capacity of EHRs to 

store data with high fidelity, to make those data readily accessible, and to help translate them into 

context-specific information that can empower providers in their work. 

This analysis is limited by a number of factors. The small number of studies included in the 

sample was largely a function of the search criteria. In particular, few systems with the core EHR 

functionalities of documentation, results management, provider order entry, and decision support 

have been examined, particularly for commercially developed products. These functional criteria 

were chosen to make the analysis more pertinent to decisionmakers currently considering EHR 

adoption. Because of the rapid technical advances in EHR, many of the studies of EHR systems 

are out of date. This review has focused on EHRs with these core functionalities in order to 

provide decisionmakers with an overview of the evidence that is most likely to be pertinent to the 

choices they are making now. Another major limitation is the lack of description (and data) 

pertaining to the workflow reengineering and organizational change that were required for EHR 

use. As discussed earlier, the “intervention” in these studies is not only the EHR system but also 

the manner in which these systems change the way healthcare professionals work, organizations 

function, and consumers receive care. This information is highly context-specific, and for the 

findings of research on EHRs to be more widely generalizable, this part of the “intervention” 

needs to be characterized, described, and measured more accurately and comprehensibly. 

Without such process implementation data, the applicability of findings from one context to 

another will be a barrier to informed decisionmaking. 

Economic Value of an Electronic Health Record Systems and 

Health Information Technology Applications 

Consumers of the healthcare system, including government in the United States, employers, 

and patients, are demanding higher quality, safety, consistency, efficiency, and value. In order to 

meet these demands, interoperable computerized health information technology, especially an 

EHR system that documents patient care processes and outcomes across the continuum of care, 

is widely believed to be a critical tool. Ideal use of an EHR system enables improved capture and 

integration of patient information from diverse sources and allows clinicians to access 

longitudinal patient-specific information for clinical decisionmaking and disease management. 

Other commonly used terms referring to aspects of an EHR system include personal health 

record and electronic medical records. In this review, EHR refers to a HIT element that performs 
 42

the functions of electronic recording, storage, accessing, and viewing of patient medical 

information.52-55 An EHR system is a computer application with EHR functionality at minimum. 

Often, financial data are also included in such a system. Since the system is designed to be used 

institution-wide to replace paper-based medical records and to aid the efficiency of healthcare 

processes, many EHR applications also contain other system functions, including prescription 

and test ordering, care management reminders, and other clinical decision-support capabilities. 

While the EHR is considered essential technology for improving efficiency and quality of health 

care, implementation of an EHR system requires substantial capital investments and 

organizational change. Consequently, many health care organizations are seeking evidence and 

lessons learned about the costs and benefits of EHR adoption in order to better inform decisions 

about the timing and strategy for implementation. 

EHR is the second most common HIT element among the articles identified that contain 

economic data. Our literature search identified 92 hypothesis-testing or predictive analysis 

articles containing information on costs, utilization, or efficiency. Of these, 32 studies assessed a 

HIT system in which EHR was one of the major system elements. However, only nine articles 

quantitatively assessed the economic value of an EHR system as a whole. We discuss these 

articles in further detail below. Most of the remaining studies were tests of certain nonfinancial 

hypotheses or examination of a subset of functionality, such as decision support, instead of the 

entire EHR system. Although these studies do not assess the costs and benefits of the entire 

system, they provide indirect, often empirical, evidence that can support the economic appraisal 

of the value of an EHR system. Before we begin the review of the nine articles, we first 

summarize the main findings of the remaining studies. Interested readers are referred to our 

interactive evidence database to learn more about these studies 

(http://healthit.ahrq.gov/tools/rand). 

Summary of Key Findings from Non–Financially Focused Studies 

Among its other functions, an EHR system can facilitate automatic generation of patient 

reminders for preventive services, screening, and disease management. Five Canadian studies 

used the same EHR system to generate patient reminders and compared the effectiveness and 

cost-effectiveness of three strategies—physician reminders, telephone reminders, or letter 

reminders—to remind patients to get preventive services.56-60 All forms of reminders were 

effective, with reminders delivered directly to patients being somewhat more effective than 

reminders to physicians. Another study used computerized pharmacy records to generate patient 

feedback and compared the effectiveness and cost-effectiveness of two depression care 

programs.61 Feedback with care management was significantly better than feedback alone. 

Electronic charting is a feature of EHR that has been reported to affect provider 

productivity.62-65 These studies found that the time needed for development of care plans and 

documentation initially increased, but preparation time decreased subsequently. The initial loss 

of productivity was associated with the baseline computer skills of the users (clinicians). Two 

studies assessed computerized documentation systems used for the ICU.66, 67 The authors of the 

first study asserted that addition of their computer-based nursing documentation required no 

specific ICU software or bedside workstations because it was implemented in a well-networked 
 43

information technology environment.66 Thus, they found that compared with paper charting, their 

electronic charting system was relatively inexpensive (although the figures were not provided). 

In addition, the documentation was more complete because of the presence of reminders for 

missing entries, and data quality remarkably improved. The other study used work sampling and 

cost analysis methodologies to show net savings of a vendor-developed bedside documentation 

system specifically designed for an ICU.67 

Another potential benefit of EHR systems is avoidance of morbidity because of improved 

patient safety. One study of ADEs found that building ADE detection and reporting capability 

into EHR can improve detection and potential reduction of ADE in hospital settings because of 

the ability of the EHR system to easily identify and confirm patients experiencing ADEs and 

thus enabled early intervention.68 Several studies have shown that severe ADEs were associated 

with longer hospitalizations and higher hospitalization costs (over $2,000 in 2005 dollars).68-70

Several studies investigated the impact of point-of-care alerts and reminders imbedded within 

an EHR system during the process of documentation or entry of orders into the system.45, 47, 71-76

These decision-support functions within an EHR system, when accompanied by the required 

changes in process and communication, altered physicians’ ordering behaviors by facilitating 

appropriate resource utilization and reducing unnecessary charges.45, 47, 71-76 For example, one 

study estimated that reducing the ordering of redundant clinical laboratory tests could produce an 

annual savings of $35,000 in laboratory charges.72 A randomized controlled trial that tested the 

effect of immediately printed summaries of a computerized medical record on physician test 

ordering rates in an emergency room setting77 showed significant improvement in the costeffectiveness of internists’ ordering behaviors. The impact on surgeons’ ordering patterns was 

positive but not significant. 

Some EHR systems offer sophisticated CPOE and decision-support functionality. A 

randomized controlled trial found positive effects of an EHR with integrated CPOE on resource 

utilization, provider productivity, and care efficiency.78 Two additional studies showed that an 

EHR with integrated decision support helped providers improve the quality of documentation, 

clinical decisionmaking, and guideline compliance, and resulted in reduced utilization of services 

and costs of care.76, 79 However, another study found that implementation of clinical guidelines 

via an EHR had no significant effect on clinical outcomes or healthcare costs.30 The benefits of 

information technology seem to depend greatly on the quality of the implementation and the 

level and type of decision-support technology. 

Analytic Methods to Assess the Economic Value of an EHR System 

This section provides a more detailed review of the nine articles that quantitatively assessed 

the economic value of an EHR system, including summaries of the analytic methods used in the 

studies. Two articles by the same author described the same ambulatory EHR system, although 

the economic estimates differ slightly between articles.53, 54 Therefore, we refer to them as one 

study described in two articles. Of the remaining seven articles, four report on evaluations of the 

economics of an EHR system: two in the ambulatory care setting52, 80 and two in an integrated 

delivery network (IDN) (an IDN comprises providers—both inpatient and outpatient—as well as 

payers and purchasers, in one connected managed care organization).81,55 Another study concerns 

health care information exchange and interoperability,82 and the remaining two are 
 44

methodological papers.83, 84 A brief summary of the methods and findings of these studies is 

presented in Table 1. 

To assess the value of EHR, one methodological paper described a return-on-investment 

framework to evaluate the costs and benefits of implementing an ambulatory EHR system, and 

another presented a spreadsheet tool to help family physicians estimate the costs of 

implementing an EHR system. Both reports provided illustrative examples. Of the remaining six 

studies, five used cost-benefit analysis52-55, 81, 82 and one used cost-consequence analysis.80 All 

the cost-benefit analyses adopted the ROI framework, which assesses the difference between the 

costs of an EHR investment and the benefits reaped from it. In these studies, both costs and 

benefits are quantified in monetary terms to the extent the authors determined was feasible. For 

example, one study aimed to justify the cost of EHR by first identifying the goals of the system 

in order to determine benefits.55 To quantify benefits, the authors then performed an extensive 

literature review, surveyed other institutions that were implementing EHR, interviewed EHR 

vendors, and conducted process-mapping sessions to identify potential cost savings on work 

processes affected by EHR. However, cost estimation was based largely on vendor response to a 

request for information or on current information system costs in the healthcare organization.52,80

The cost-consequence study showed costs in monetary terms but did not quantify the benefits of 

EHR except for time saved from chart pulling.80 All studies except two used the perspective of 

an organization, either outpatient settings52-54, 83, 84 or IDN.55, 81 Of the other two studies, one 

adopted a societal perspective,80 and the other used multiple perspectives, from organization 

level to national level.82

Six studies reported their data sources.52, 55, 80-83 All used multiple data sources, including 

primary data collected from an existing EHR system,52 published data,52, 55, 82, 83 workplace and 

demonstration site observations,80, 81 and surveys or interviews of key informants or EHR users. 

55, 80 Experts were a primary source of data for some studies,52, 81, 82 as were vendors.55, 81 One 

study also used process-mapping sessions.55

Cost and benefit variables used in each study are listed in Table 1. The cost variables 

included in most studies included hardware and network acquisition, software licensing, ongoing 

technical support and maintenance, and training and other implementation costs. Costs 

associated with temporary productivity loss due to the EHR implementation were captured by 

two studies,52, 80 but a third study assumed no cost associated with loss of productivity, given a 

long-term EHR implementation strategy.81 Other cost variables include installation (which was 

not quantified in greater detail),83 data entry,84 printing,53, 54 system integration in IDN setting,55

personnel,53, 54 and institutional and project management.55, 80 The health care information 

exchange and interoperability study also included interface development cost. 

The benefit variables included the following: savings from chart pull and transcription;52, 80-83

time saved to document diagnostic codes;81, 83 prevention of ADEs;52 reduction in drug52, 81

laboratory,52, 81 or radiology costs;52, 81 improvement of charge capture;52, 81 decreased billing 

errors;52 personnel and space savings from reduced existing and future medical record storage 

requirements;53, 54 as well as automated generation of clinical forms,53, 54 pharmacy 

information,81 and billing data generation.53, 54 One study provided a comprehensive list of 

potential benefits grouped into four categories: data capture and access, decision support, 

business management, and streamlining patient flow.55 The health care information exchange 
 45

and interoperability study reported quantifiable benefits from health information connectivity for 

providers, payers, and other stakeholders, including radiology centers, laboratories, pharmacies, 

and public health departments.82 Three studies showed the net ROI.55, 82

Various analytic designs were used, although the reference strategy was always the 

traditional paper-based medical record. One study assessed the economic value of an EHR 

system concurrently with its implementation and reported the first-year economic consequences 

of the system.80 Another study predicted a fixed annual savings and expenses based on seven 

years of EHR implementation experiences in a HMO.53, 54 Yet another study constructed a 

hypothetical primary care provider patient panel using average statistics from one of the nation’s 

leading organizations in EHR implementation.52 Both studies of EHR systems in IDN settings 

are analytic predictions because EHR had not been implemented in the studied organizations.55,81

The health care information exchange and interoperability study for a fully standardized 

nationwide system is also an analytic prediction based on a conceptual framework describing 

how health care entities share information and a functional taxonomy reflecting the amount of 

human involvement required, the sophistication of IT, and the level of standardization.82

Sensitivity analyses were performed in three studies.52, 80, 82

Evidence of Economic Costs and Benefits of an EHR System 

Our interactive evidence database (http://healthit.ahrq.gov/tools/rand) provides a structured 

abstract for each of the nine identified studies regarding the costs and benefits of EHR. Main 

findings are highlighted in Table 1 and summarized below. 

Costs of Implementing an EHR System. Five studies quantitatively assessed the costs of 

implementing an EHR system.52-55, 80, 81 The costs can be divided into two categories: (1) cost of 

the system itself (hardware, software, license, maintenance, and support) and (2) implementation 

cost (training, temporary loss of productivity, etc.). The costs vary significantly by the scale of 

the healthcare organization and the functionality of the EHR system. 

(1) Cost of an EHR System. One study estimated the system costs for an ambulatory EHR to 

be $9,700 per provider (in 2002 dollars), which included $1,600 for the annual software license, 

$1,500 for annual support and maintenance, and $6,600 for hardware (three computers and 

network, refreshed every three years).52 The estimate was for a hypothetical primary care 

provider office and was modeled after a well-developed and widely used EHR system at a 

leading IDN health care system. The component parts of the EHR system include online patient 

charts, electronic prescribing, laboratory order entry, radiology order entry, and electronic charge 

capture. 

In a Swedish primary health care setting with 50 staff, the system cost of a vendor-developed 

EHR system was estimated to be $240,000 in the first year (in 1995 U.S. dollars).80 The EHR 

system supported full-text patient records and included a controlled medical terminology, a 

structured patient database, and tools for the analysis and reporting of patient data. The hardware 

at the sites comprised one server supporting approximately 40 workstations and 20 printers. 

In a large HMO with 13 outpatient care locations in Ohio, a homegrown ambulatory EHR 

was estimated to have had a system development cost of $10 million (in 1996 dollars) and 
 46

additional annual expenses of $630,000 (in 1996 dollars) for printing, network expenses, 

memory, and license renewals.53, 54 The EHR system was used routinely by 220 physicians and 

110 allied health professionals. It implemented an encounter system that collected and presented 

such medical data elements as diagnoses, allergies, medications prescribed, immunizations, vital 

signs, and smoking status at the time of an encounter. The system also generated physician 

reminders for guideline compliance and patient reminders for preventive services and was linked 

to centralized clinical data on the mainframe, such as laboratory results, radiology reports, 

emergency department notes, and hospital discharge summaries. 

An academic cancer center with a staff of about 8000 and facilities that included a hospital, 

outpatient clinics, and remote patient-care sites was interested in implementing an EHR for its 

IDN as both a clinical and financial information management tool in 1994. The cost estimates for 

vendor-developed EHR systems to meet the center’s organizational needs ranged from $15.8 

million to $21 million (in 1994 dollars), which included costs of hardware, software, interface 

development, network cost, data conversion, training, and annual maintenance.55 Additionally, 

the annual support costs were estimated to range from $3.8 million to $5.3 million. 

A 2002 study estimated the costs of implementing an EHR81 for an IDN that included a 

medical center with a 280-bed acute care hospital, 16 hospital-based and satellite outpatient 

clinics, a research institute, and a network of about 400 employed physicians. A vendordeveloped EHR was estimated to cost approximately $19 million (in 2001 dollars) for the sevenyear implementation period. This included costs for the various software products, server 

hardware, professional services related to installation and training, as well as desktop devices, 

monitors, biometric security devices, imaging hardware and software, additional technicalsupport staff, and other associated costs. 

(2) Cost of the Implementation Process. Only two studies provided an estimate of costs 

associated with the EHR implementation process. Both were for ambulatory settings.52, 80 One 

estimated an implementation cost of $3,400 per provider (in 2002 dollars) in the first year 

associated with workflow process redesign, training, and historical paper chart abstracting.52 It 

also estimated a revenue loss of $11,200 in the first year due to temporary loss of productivity. 

The total implementation process cost, $14,600 per provider, is 1.5 times the estimated EHR 

system cost. 

The Swedish study used a societal perspective and included costs of training and unexpected 

costs pertained to self-training during working hours, loss of normal activities in leisure hours, 

increase in administrative work load, extra service, and medical records summary.80 These costs 

were estimated at $75,000 (in 1995 U.S. dollars), approximately 30 percent of the EHR system 

cost. 

A third study projected the costs of implementing health care information exchange and 

interoperability where EHR is a requirement for Levels 3 and 4 implementation,82 which, 

according to the authors’ taxonomy, refers to the ability to handle machine-organizable data and 

machine-interpretable data, respectively. The authors projected costs for multiple stakeholders of 

the healthcare system for Levels 3 and 4 health care information exchange and interoperability. 

The national ten-year rollout cost of Levels 3 and 4 were estimated to be $320 billion and $276 

billion, respectively. Additionally, the national ten-year annual costs were estimated at $20.2 

billion and $16.5 billion, respectively. 
 47

Quantified Benefits from an EHR System. Benefits of an EHR system or health care 

information exchange and interoperability were also quantified in the six studies that assessed 

the cost of implementing such a system. 

One study was based on the Partners HealthCare ambulatory EHR, which not only provided 

health information and data storage capability but also possessed results management, order 

entry management, point-of-care decision support, and administrative information management 

functionalities. Therefore, many benefits were expected. The study divided the benefits into three 

categories: (1) payer-independent benefits, including savings from chart pulls and transcription; 

(2) benefits under capitated reimbursement, including averted costs from ADEs, drug utilization, 

laboratory utilization, and radiology utilization; and (3) benefits under fee-for-service 

reimbursement, including improvement in charge capture and decreased billing errors.52 The 

authors predicted that savings from chart pulls and transcription would be seen immediately after 

the EHR implementation, and costs associated with ADEs and drug utilization could be averted 

from second year on, but other potential savings would not be realized until the fourth year. Fiveyear total benefits of an EHR implementation were estimated to be $129,300 per provider (in 

2002 dollars), or a net savings of $86,400 per provider (in 2002 dollars). Sensitivity analyses 

showed that the estimates were sensitive to the assumption of the proportion of patients whose 

care was capitated. The net financial value could range from a $2,300 net cost to a $330,900 net 

benefit per provider. 

The Swedish study examined an EHR system with functionality limited only to health 

information and data storage. Therefore, the expected benefits were limited and included 

increase in knowledge capital for the primary health care team, easier and quicker 

communication for general practitioners during telephone consultations, clearer information to 

patients, and time saved in retrieving paper-based medical records.80 Only the value of the last 

item was estimated, at a total of approximately $10,500 (in U.S. 1995 dollars) for the first year of 

EHR implementation. 

Despite potential savings from ADE prevention and reduced resource utilization under 

capitated reimbursement, the study of the ambulatory EHR system in a HMO (13 outpatient 

clinics) did not quantify this aspect of benefits. Instead, it quantified only the averted costs 

associated with improved efficiency.53, 54 The study estimated an annual savings of $3,700,000 

(in 1996 dollars) from reduced medical record room and support staff, elimination of clinical 

forms, and automatic collection of billing data. 

The cancer center study projected the benefits of an EHR over ten years.55 This projection 

made several key assumptions, including no benefit until the third year after implementation, 

benefits to phase-in as the EHR system became functional, physician acceptance and use of the 

system, a link between business management benefits and managed care, and productivity 

changes. The authors divided the benefits into capture and access, decision support, optimization 

of clinical practice, business management, and streamlining of patient flow. The estimated total 

quantified benefits were $129.69 million over ten years. Adjusted by the total implementation 

and system costs, the authors’ assigned confidence factor, and 9.5-percent discount rate, the net 

value was predicted to be $24.9 million (in 1994 dollars). 

The other IDN expected even greater benefits from an EHR implementation. The authors 

predicted approximately $68.5 million in gross quantifiable benefits over a seven-year period; 
 48

subtracting the cost of the EHR system implementation, the net benefit would be $31.4 million 

(in 2001 dollars), using a 10-percent discount rate.81 The authors predicted and quantified 

benefits from savings in laboratory and radiology order entry, pharmacy order entry, 

documentation availability of information, and charge capture. 

Another study estimated that implementation of a standardized interoperable EHR system by 

all healthcare organizations in the United States would yield substantial financial benefits. The 

health care information exchange and interoperability study predicted that investment on a fully 

standardized, Level-4 nationwide system will have the most financial return, a net value of $77.8 

billion per year once fully implemented.82 Non-standardized health care information exchange 

and interoperability also can have positive financial returns, but the returns are smaller compared 

to the Level-4 implementation. 

In summary, despite the heterogeneity in the analytic methods used, all five cost-benefit 

analyses predicted substantial savings from EHR (and health care information exchange and 

interoperability) implementation.52-55, 81, 82 In other words, the quantifiable benefits are projected 

to outweigh the investment costs. However, the predicted time needed to break even varied from 

three52, 81 to six55 to perhaps as long as 13 years.54

Conclusion. Our evidence review found consistent predictions from five cost-benefit studies 

that implementation of an EHR system can be financially viable at the individual organization 

level or through a nationwide implementation with high levels of health care information 

exchange and interoperability.52-55, 81, 82 However, there are several caveats. 

1. All studies are predictive analyses that are based on many analytical assumptions and 

limited empirical data. The strength of evidence is considered weak. 

2. In all studies, the EHR system was assumed to have multiple functionalities that include, 

at minimum, health information and data storage, administrative processes, decision 

support, and results management, as well as information exchange capabilities. The 

functional capability of an EHR system is critical to the benefit accrued. 

3. The individual organizations that were the subjects of four studies were all large 

organizations. Large organizations involve many people, units, and subsystems and have 

complicated processes and interactions. They can benefit greatly from automated, 

transparent information processing through HIT, and substantial economies of scale. The 

literature review did not identify cost-benefit studies for EHR implementation in small 

organizations. 

4. The costs of implementing an EHR system may be underestimated. Only one of the five 

cost-benefit analyses included the cost of the implementation process,52 and it found that 

this cost was 1.5 times the cost of the EHR system. Implementing an EHR system 

requires extensive changes in the organizational processes, individual behaviors, and the 

interactions between the two. These resulting costs are often omitted or not reported from 

studies but can be substantial. 

5. The financial benefits depend on the financing system. As shown in the sensitivity 

analysis of one study,52 the benefit estimates are most sensitive to the assumption of the 

proportion of capitated patients. Realizing all quantifiable benefits of EHR 

implementation would require changes to the current health care financing system. 
 49

6. Both the cost and the benefit of attaining interoperability among EHR systems are 

directly proportional to the level of data exchange achieved. For example, the cost of 

achieving machine-organizable (Level 3) or machine-interpretable (Level 4) 

interoperability is greatest, but it offers the most potential for increased efficiency, 

improved healthcare utilization, and reduced costs. 

In conclusion, there is some empirical evidence to support the positive economic value of an 

EHR system and the component parts of EHRs. However, realizing the projected benefits will 

require proper alignment of the healthcare financing system, strong leadership, effective 

implementation strategies, and focused efforts to successfully adapt the EHR system. 

 51

Table 1. Summary of studies assessing the economic impact of an EHR system 

Reference HIT 

elements 

IOM categories 

Organization Known 

system 

Sources of data 

Cost variables Benefit variables 

Wang, 

200352

EHR, Electronic prescribing 

HIDS, RM, OEM, DS, AP 

Outpatient Part Primary data 

from EMR, literature review, expert group consensus 

Software license, implementation, support, hardware, productivity loss 

Quantified: Chart pull, transcription, prevention of ADE, drug, lab, radiology, charge capture, billing, & net value of AEMR during a five-year period 

AriasVimarlund, 

199680

EHR HIDS Outpatient Work place 

observations and key informant interviews 

Direct: Training, hardware & software, project manager system supplier, maintenance; Unexpected: selftraining at work, loss of leisure hours, increase in administrative workload, extra service, summarizing medical records; Indirect: unquantified 

Quantified: time saved from chart pulling; Unquantified: increase iknowledge capital for care team, easier and quicker communication for GPs during telephone consultationsclearer information to patients 

Agrawal, 

200283 EHR HIDS, AP Outpatient Part Hypothetical scenario, 

published studies 

Unquantified: hardware, software, network, maintenance, installation and training, opportunity cost 

Quantified: chart pull, transcription, time to document diagnostic codes; Unquantified: charge capture, cash flow, prescribing, malpractice premium, health resource waste, ADE and injuries, delivery of preventive and health maintenance procedures, provider and patient satisfaction 
 52

Reference HIT 

elements 

IOM categories 

Organization Known 

system 

Sources of data 

Cost variables Benefit variables 

Khaoury, 199853

EHR, Decision Support 

HIDS, RM, DS, AP 

Outpatient KP Not described Personnel, printing, 

network, hardware, license 

Quantified: Medical record room and supporstaff, automated clinical forms, automated billing data; Unquantified: support of a quality program 

Schmitt, 200281

EHR HIDS, 

OEM, AP 

IDN Opinion from 

clinical advisory team, vendor, site visit observation 

Annual cost over a seven-year period, with no detail breakdowns and assuming no cost associated with the temporary reduction in physician productivity 

ADE, capitated drug benefits, inpatient medication cost, laboratory, radiology, pharmacy, documentation, information at the point of care, charge capture, & net benefit over the 7 years period 

Valancy, 

200284 EHR N/A Outpatient Illustrative 5-year annual cost with detail on hardware, 

software, vendor support, and data entry cost 

Not reported 

Khoury, 199754

EHR HIDS, DS, 

AP 

Outpatient KP Not described Personnel, printing, 

network, hardware, license 

Quantified: Medical record room and supporstaff, automated clinical forms, automated billing data; Unquantified: patient communications, guideline compliance, disease management and prevention 
 53

Reference HIT 

elements 

IOM categories 

Organization Known 

system 

Sources of data 

Cost variables Benefit variables 

Kian, 199555 EHR HIDS, AP IDN Literature 

review, surveying 10 institutions, interviewing vendors, process mapping sessions 

10 years annual cost including hard and software, integration, network infrastructure, vertical integration, I/S infrastructure, institutional issues 

Quantified: Data capture and access, decision support, business management, streamlining patient flownet impact over 10 years period Unquantified: patient readmission, staff reduction 

Walker, 

200582

EHR and health care information exchange and interoperability

DS Organizations, 

nationwide 

 Literature 

review, published data, expert interviews, expert panel 

Interface development, electronic health record acquisition, maintenance 

Quantified: Lab, radiology, pharmacy, chart request and referral, reporting, provider-payer transaction, & net value of health care information exchange and interoperability 

NOTE: HIDS = health information and data storage; RM = results management; OEM = order entry management; DS = decision support; el

connectivity; AP = administrative processes. 
 54

Health Information Technology and Patient Centeredness 

Many advocates of HIT believe that one of its primary goals is to increase the extent to 

which the patient is at the center of his or her health care. For this report, studies of HIT and 

“patient centeredness” were defined as those that assessed HIT systems that included the element 

of patient decision support/consumer health informatics, telemedicine, or dataexchange/community health information networks or that reported patient satisfaction as an 

outcome. From the database of 256 articles there were 34 unique studies or systematic reviews 

meeting these criteria. 

Ten studies assessed computer-generated reminders. Of these, seven assessed the use of 

reminder programs to improve the delivery of preventive care such as mammography and 

immunizations.27, 85-90 All studies reported greater use of preventive services by patients—or the 

physicians of patients—who received computer-generated reminders. Two other studies assessed 

the effect of computer-supported or -generated reminder systems for refilling medications,91, 92

neither of which reported statistically significant improvements in compliance. A third study 

assessed the effect of a computer-generated reminder chart on patients’ compliance with drug 

regimens, which reported significant improvements in mean compliance score for patients 

receiving an automatically generated reminder chart.93 

Seven original studies evaluated various aspects of telemedicine, and a review article 

assessed the role of telemedicine in surgery.94 In the context of surgery, telemedicine included 

tele-mentoring, tele-proctoring, tele-conferencing, and tele-presence surgery, all of which are 

designed to allow physicians to communicate and improve the technical delivery of remote 

surgical procedures. Thus, this article was not considered relevant to patient centeredness. Five 

studies assessed telemedicine in particular contexts, including the intensive care unit,95 the 

control of essential hypertension,96 the evaluation of patients in an outpatient pulmonary clinic,97

the role of telemedicine as one component of HIT and its importance to child safety,98 and the 

use of a remote video system that allowed nurses and patients to interact in real time for patients 

with a variety of health conditions.99 All of these articles reported benefits from the use of 

telemedicine technologies. The last article in this group was an assessment of ComputerLink, 

which was conceived as an alternative to traditional caregiver support services such as support 

groups and health education programs. It was tested in a 12-month randomized trial in family 

caregivers of people with Alzheimer’s disease. Compared with the control group that did not 

have access to ComputerLink, caregivers in the experimental group reported more improvement 

in caregiver strain.100 

Three studies assessed the effect on patient trust and satisfaction of some aspects of HIT that 

are used during a patient consultation. In a pre-post study of general surgical patients, 96 percent 

of patients stated that their contact with a doctor was as easy and as personal after installation of 

a computer in the office as before the installation.101 In a controlled trial, the use of an automatic 

voice recognition system for transcribing progress notes was not associated with any significant 

negative effect on patient satisfaction, and was associated with some positive effects in terms of 

preventive maintenance and patient education.102 We also identified a study that assessed the 

effects of a nursing module used at the point of care. In this time-series study, an integrated, 

menu-driven electronic health record was associated with marked increases in nursing 
 55

documentation and no effect on patient satisfaction.103 One study assessed physician 

satisfaction, and was excluded.104

Two studies assessed the effect of computer-guided management of patients. The first study 

compared the effect of computer-controlled administration of analgesic for post-operative pain to 

that of patient-controlled administration. The computer-controlled infusion used custom-written 

software designed to rapidly attain and maintain a theoretical target plasma concentration of the 

analgesic. In a double-blind randomized trial, the study found that computer-controlled analgesia 

conferred a more rapid onset of pain relief and was as effective as patient-controlled 

administration in providing post-operative analgesia.105 Which of these interventions is more 

“patient-centric” may be debatable, however. In the other study, computer-guided behavioral 

therapy that allowed patients to progress through a self-paced workbook was compared with 

clinician-directed behavioral therapy and with relaxation therapy in a randomized trial of patients 

with obsessive-compulsive disorder. At ten weeks, the Yale-Brown obsessive-compulsive scale 

showed significantly greater improvement in the patients receiving clinician-guided behavior 

therapy than in the group receiving computer-guided behavior therapy, and both of these were 

significantly greater than the improvement attained with relaxation therapy, which was found to 

be essentially ineffective. This study concluded that computer-guided behavior therapy was 

effective and might be a helpful first step in treating patients with obsessive-compulsive disorder 

when clinical-guided behavior therapy was unavailable.106

Five studies assessed various aspects of consumer health informatics. The interventions 

included a clinical trial of an interactive computerized patient education system in family 

practice;107 and assessments of the effects of computer tailored smoking cessation in family 

practice,108 the effectiveness of a computer-generated patient health summary in changing 

patients’ knowledge, attitudes, and behavior concerning health promotion,109 and the use of selfadministered computerized assessments for psychiatric disorders in patients in primary care.110

All of these studies reported benefits of the computerized health informatics system. A review of 

37 studies of computer-generated health behavior interventions intended to motivate individuals 

to adopt various treatment regimens concluded that such systems are effective.111 

Two review articles assessed the effect of various HIT systems that are directly accessed by 

the consumer or patient. One of the reviews assessed ten comparative studies of consumers using 

the Internet to access health information and services. This review included controlled studies, 

before-and-after studies, and interrupted time-series analyses of Internet users versus nonusers, 

or of the use of the Internet versus other communication media. The authors concluded that 

rigorous research regarding the effects of consumer Internet use on health outcomes is lacking. 

In the ten studies they assessed, all showed some beneficial effects on health outcomes, although 

the authors note the methodological quality of many studies was poor.112 A second review article 

assessed comparative studies evaluating the health or social outcomes of virtual peer-to-peer 

communities, which they characterized as a type of electronic support group. Among 45 

publications describing 38 studies (of which 20 were randomized trials), only six evaluated 

“pure” peer-to-peer communities without other interventions. The other studies assessed 

complex interventions, of which a peer-to-peer community was only one component. The 

authors concluded that no good evidence exists on the effects of consumer-led peer-to-peer 

communities, partly because most such interventions have been evaluated only as part of a 

complex intervention or interventions involving health professionals.113
 56

One review article assessed what is known about email consultations in healthcare. The 

authors noted that a rapidly expanding proportion of the population has access to email and that, 

while email consultations have the potential to play an important role in the delivery of 

preventive health care and facilitation of self-management of chronic disorders, there is little 

evidence from controlled trials that this potential benefit can be translated to routine clinical 

care.114

Another review article assessed studies of “electronic communication with patients,” which 

was defined to include studies of computerized communication, telephone follow-up and 

counseling, telephone reminders, interactive telephone systems, telephone access, and telephone 

screening. This article concluded that distance medicine technology has benefits in the areas of 

preventive care, management of osteoarthritis, cardiac rehabilitation, and diabetes care and that 

“distance medicine technology enables greater continuity of care by improving access and 

supporting the coordination of activities by a clinician.”115

Last, two studies dealt with data exchange networks or community health information 

networks. One study described the experience of developers of an electronic laboratory reporting 

system.116 In the second study, researchers developed a cost-benefit model and used published 

evidence and expert opinion to assess the ten-year rollout and annual cost of healthcare 

information exchange and interoperability, a development that would allow providers to access 

patient health care information in any clinical setting. The researchers concluded that a fully 

standardized interoperable system could save $77.8 billion a year, once fully implemented.82

In summary, evidence for an effect of HIT on patient-centeredness in health care is sparse. 

The best evidence is the beneficial effect of using computerized patient reminders for preventive 

care. The evidence for benefits of telemedicine and consumer health informatics is also limited 

to specific contexts. Finally, the evidence is much more limited for effects of more general 

interactive HIT (such as the internet or email on health) or the effect of implementing HIT 

systems (such as the electronic health record) on patient trust and satisfaction. 

Barriers to HIT Implementation 

All studies initially reviewed were screened for data on barriers to adoption and 

implementation. For this analysis, qualitative studies that were primarily focused on barriers and 

studies that collected quantitative data on barriers were included. Studies in which barriers were 

briefly discussed, but were not a primary focus, were excluded. A primary focus on barriers was 

identified through reviewer consensus. 

We identified 20 publications that focused on the barriers to implementing HIT. Of these, 8 

reported the actual or potential barriers encountered with specific HIT implementations,55, 62, 116-

124 usually as part of an article discussing the implementation. Two articles were short opinion 

pieces about potential barriers from the physician perspective.125, 126 Two studies assessed the 

physician time for order entry using CPOE compared to paper methods;118, 127 both demonstrated 

that CPOE took more physician time, although the study by Overhage and colleagues found this 

additional time to be modest. A third study assessed the effect on primary care physicians’ time 

before and after implementation of an EHR system and reported that the time for a patient visit 

actually fell by half a minute with EHR use.128 Last, one study compared physician user 
 57

satisfaction with two HIT systems: the VA CPRS system and the Mt. Sinai hospital physician 

order entry system. This study demonstrated CPRS users to be much more satisfied than Mt. 

Sinai hospital users on many dimensions and also demonstrated that satisfaction was correlated 

most strongly with the ability of the HIT system to perform tasks in a “straightforward” 

manner.117 Finally, one article was a systematic review of physician use of electronic retrieval 

systems such as Medline.129

The other five articles focused more broadly on barriers to HIT implementation. One 

systematic review130 summarized barriers mentioned in the medical and pediatric literature that 

are significant for pediatric practices. These barriers were divided into four categories. 

Situational barriers included time and financial pressures, unproven return on investment, 

insufficient access to the internet or to computer technology in the office setting, the prohibitive 

cost of information technology for small practices, and software not being supportive of pediatric 

practice needs. Cognitive and or physical barriers include physical disabilities and insufficient 

computer skills. Liability barriers included confidentiality concerns. Finally, knowledge and 

attitudinal barriers included insufficient research about information technology in pediatrics, 

insufficient knowledge about benefits afforded by information technology, apprehension about 

change, and philosophical opposition to information technology. 

Two studies used surveys to identify barriers in the use of electronic medical records131 and 

barriers to implementing CPOE systems in U.S. hospitals.132 In the first of these studies, the 

authors conducted 90 interviews with electronic medical record managers and physician 

champions in 30 physicians’ organizations between 2000 and 2002. Key barriers to electronic 

medical record use were high initial financial costs, slow and uncertain financial payoffs, and 

high initial costs in terms of physician time. Additional barriers included difficulties with 

technology, complementary changes in support, electronic data exchange, financial incentives, 

and physician attitudes. The authors note that these barriers were most acute for physicians in 

solo/small group practice, which account for a large proportion of U.S. physicians. The second 

article132 reported the results of 52 interviews at 26 hospitals in various stages of implementation 

of CPOE—from not considering implementation to fully implemented. Most respondents were 

Chief Information Officers; the remainder consisted of Chief Financial Officers, Chief Medical 

Officers, and other management officials. Three main barriers to CPOE adoption were identified. 

The first was physician and organizational resistance due to the perceived negative impact on the 

physician’s workflow. The authors noted that resistance from physicians could escalate to the 

point of a “physician rebellion,” which could derail the entire implementation process. The 

second barrier identified was the high cost, with estimates from prior studies for the cost of 

CPOE ranging from $3 million to $10 million, depending on the hospital’s size and the level of 

existing information technology infrastructure. The third major barrier identified was 

product/vendor immaturity. Survey respondents reported that many current vendor products did 

not fit the needs of their hospital, and extensive software modifications were required to 

accommodate established workflow in the hospital. 

We also identified two recent prominent editorials about barriers to HIT implementation that 

summarized the issues succinctly.133, 134 The first of these133 identified several challenges for 

adoption of electronic health records. These included cost, technical issues, system 

interoperability, concerns about privacy and confidentiality, and a lack of a well trained clinical 

informatics work force to lead the process. This author identified financing as the biggest 
 58

impediment, which he attributed to a misalignment of costs and benefits. He noted that while 

some studies have suggested a substantially positive return on HIT investment for the health care 

system as a whole, those who are expected to pay for the systems (physicians and other practice 

organizations) see only about 11 percent of that return on investment. The rest of the savings go 

to those who typically do not pay directly for the electronic health record. Another major 

challenge he identified was system and data interoperability, noting that most health care data 

(whether on paper or electronic) are trapped in “silos.” A third concern was privacy and 

confidentiality: the author stated that physicians, other health care professionals, and healthcare 

organizations must be vigilant in protecting patient privacy. The last major barrier identified was 

the need for a workforce capable of leading the implementation of information technology. 

The second editorial134 stated that, despite predictions of a “bright and near future” for the 

use of HIT, this future never seems to be realized. The authors attributed the lack of progress in 

HIT implementation to a lack of attention to the social component, citing the need to view the 

clinical workplace as a complex system in which technologies, people, and organizational 

routines dynamically interact, which leads to the following observations: 

 “(1) Organizations are simultaneously social (e.g. consisting of people, 

values, norms and culture) and technical (i.e., without tools, equipment, 

procedures, technology and facilities the people could not work and the 

organization would not exist). (2) These social and technical elements are deeply 

inter-dependent and inter-related—hence the term socio-technical systems. Every 

change in one element affects the other. (3) Accordingly, good design and 

implementation is not a technical problem but rather one of jointly optimizing the 

combined socio-technical system.” 

The authors also note, “...an information technology in and of itself cannot do anything, and 

when the patterns of its use are not tailored to the workers and their environment to yield high 

quality care, the technological interventions will not be productive. This implies that any IT 

acquisitions or implementation trajectory should, first and foremost, be an organization change 

trajectory.” 

In summary, studies have identified a large number of barriers to the implementation of HIT. 

These barriers can be classified as situational barriers (including time and financial concerns), 

cognitive and or physical barriers (include physical disabilities and insufficient computer skills), 

liability barriers (including confidentiality concerns), and knowledge and attitudinal barriers. 

Cutting across all these categories, however, may be the need for clinical medicine as it is now 

practiced in the majority of settings to undergo a major structural and ideological reorganization, 

so it can be integrated with and enjoy the benefits of HIT. 
 59

Chapter 4. Discussion 

Limitations 

The primary limitation of this review is the quality and quantity of the available studies. As 

we have noted, understanding the benefits and costs of implementing a HIT system requires an 

understanding of the intervention in terms of its technical components as well as its human 

factors, project management, and organizational component; and understanding the organization 

implementing the HIT system requires understanding many things about it, including its past and 

current culture of change and the economic situation in which it operates. Most of this 

information is absent for most of the published studies of HIT. Past limits on word count in 

published articles may have prevented some authors from including such information in their 

published reports, but recognition of the information that is needed and the recent practice of 

allowing supplementary methodological information to be posted online should obviate the 

problem. 

A second limitation is that not all relevant published studies may have been identified. While 

our search efforts were comprehensive, it is inevitable that we did not find some relevant studies. 

An advantage of our interactive database of evidence (http://healthit.ahrq.gov/tools/rand) is that 

it can be updated easily, so we invite readers to send us the citations for relevant articles we may 

have missed. 

We selected only articles that were classified as systematic reviews, hypothesis testing, or 

predictive analysis for more detailed review and structured abstracts in our interactive database. 

These articles tend to have less description about how the HIT actually operated and its 

implementation processes than do qualitative, descriptive articles. Although in general we did 

not find good evidence of such critical information in the literature during our review processes, 

we provide citations of qualitative articles in our interactive database for interested readers. 

However, it should be noted that while these qualitative articles might contain more contextual 

information about the HIT systems, they are completely lacking in any generalizable knowledge 

about the benefits of HIT, such as reduction in errors or quality improvement. Any such studies 

that compared outcomes (such as error rates) with and without a HIT system would have been 

classified as hypothesis-testing studies and would have been included in our analyses. 

A third limitation is that we considered only published studies. As noted by our TEP, 

substantially more information regarding implementation may have been obtained by contacting 

leading HIT implementers and conducting structured interviews with them. An additional 

limitation of reliance on published studies is that less successful, or failed, HIT attempts may be 

less likely to be published than successful ones (a result of publication bias). 

A final limitation is that many of the costs and financial benefits of EHR will change over the 

years, because they depend on the changing price of such factors as hardware, software, labor 

costs, and medical prices. Consequently, it is not easy to translate costs as they were reported in 

the original article into today’s costs. 
 60

Conclusions 

General Conclusions 

1. Predictive analyses, based on statistical modeling techniques, suggest that HIT has the 

potential to enable a dramatic transformation in the delivery of health care, making it safer, 

more effective, and more efficient. The empirical research evidence base supporting HIT 

benefits is more limited. 

2. Organizations that have realized some of these major gains through the implementation of 

multi-functional, interoperable HIT systems built around an electronic health record include 

the VA, Partners, and Regenstrief Institute. 

3. The impact of HIT implementation on cost and quality will not be consistent across 

institutions, independent of context. The specific context within which HIT is implemented, 

including the setting, the clinical issues, and the patient populations, greatly influences its use 

and effects. 

4. More widespread implementation of HIT is limited by the lack of generalizable knowledge 

about what types of HIT and methods for its implementation will result in changes in benefits 

and costs that are specific for specific health organizations, especially for small practices and 

small hospitals. 

5. The reporting of HIT developments and implementations needs to be improved with greater 

attention to descriptions of both the intervention and the organizational/economic 

environment in which it is implemented. 

6. A high priority must be placed on establishing standards for the information that needs to be 

measured and reported in HIT implementation studies, similar to the CONSORT standards 

for clinical trials of therapeutics. 

7. Using existing published evidence, it is not possible to draw firm conclusions about which 

HIT functionalities are most likely to achieve certain health benefits. The assessment of costs 

is even more uncertain. 

8. Existing evidence is not sufficient to clearly define “who pays for” and “who benefits from” 

HIT implementation in any healthcare organization except those, such as Kaiser and the VA, 

that are responsible for paying for and delivering all the care for the defined population. 

9. Models can be built to estimate the costs and benefits of interoperable HIT systems within 

and across health care provider settings, payers/purchasers and cumulatively across the health 

care continuum, but these models are based on many assumptions. 

10. Implementation of HIT faces many barriers. 

Pediatrics 

1. Limited empiric evidence exists to support a benefit for HIT use in pediatrics in the areas of 

medication safety, clinical decision-support, process improvement, and cost reduction. 
 61

2. Only one scientific study weighed these benefits against the costs or cost-effectiveness of 

implementing HIT systems in pediatric healthcare settings. 

3. A majority of HIT systems for use in pediatric practices were tested and/or developed in 

academic settings, and the ability to generalize these findings to commercially available 

systems used in nonacademic settings is limited. 

EHRs and the Quality of Ambulatory Care 

1. A small set of high quality studies shows that implementation of a comprehensive 

ambulatory EHR improves quality of care. Available evidence focuses primarily on the 

impact of ambulatory EHRs on decreasing overused health services by enhancing access to 

data, providing capabilities for real-time analysis of clinical data, and acting as platforms for 

decision support. 

2. Ambulatory EHRs improve the structure of care delivery, improve clinical processes, and 

enhance outcomes. Most available evidence shows the effects of ambulatory EHRs on 

processes of care. 

3. Interpreting the precise causal effects of ambulatory EHRs on quality is difficult due to lack 

of systematic and detailed descriptions of system capabilities, limited data (either qualitative 

or quantitative) on the workflow redesign and organizational changes that accompanied 

implementation of an ambulatory EHR (or implementation of a new function in an existing 

EHR package), use of ad hoc measures to assess quality, and use of study designs that do not 

explicitly take into account sources of bias and confounding. Thus, while existing evidence 

may have high internal validity, the generalizability of findings is limited. 

4. Although substantial potential exists, evidence for the ability of ambulatory care EHRs to 

improve quality by making healthcare more consumer- and patient-centered is scant. 

Economic Value of an HIT and EHR System 

1. The main quantifiable benefits of an EHR system were savings from data capture and access; 

decision support to improve efficiency, quality, and safety of care; business management 

related to staffing, billing, and overheads; and streamlining patient flow. 

2. Few studies quantitatively assessed the costs to implement an EHR system and the financial 

benefits reaped from it. 

3. All the cost-benefit analyses of an EHR system predicted that the financial benefits would 

significantly outweigh the costs, in a timeframe that varied from three to thirteen years, but 

this evidence is limited to large organizations and multi-functional EHR systems. 

4. The positive economic estimates for EHR system implementation are encouraging but are 

based on limited evidence at this time. Only limited empirical evidence supports the 

assumptions made in the predictive analyses. Most studies omitted the costs of implementing 

an EHR system that were associated with the temporary loss of productivity and the cost of 
 62

process redesigns. Moreover, realization of the financial return is highly sensitive to the 

organization’s financial incentives. 

5. There is some evidence regarding the positive economic value of implementing component 

parts of an EHR system, with models suggesting that many of the benefits do not accrue 

unless a broadly functional system is implemented. 

Recommendations for Future Research 

This section is divided into two parts: (1) research recommendations and (2) 

recommendations regarding the appropriate types of researchers to carry out the recommended 

work. 

Recommendations 

1. High on the list of future research is the need for agreed-upon standards for reporting HIT 

implementation studies, similar in purpose to the CONSORT standards for the reporting of 

clinical therapeutics trials. 

2. The organizational change and workflow redesign required by and accompanying HIT 

implementation (or implementation of a new HIT function) need to be described and 

measured with greater validity, reliability, and precision in order to understand the impact of 

HIT on care delivery. Without such information, the true “intervention” remains unclear, and 

the generalizability of results will remain limited. This kind of reporting will require the 

development and dissemination of publishing standards. 

3. While HIT implementation does not easily lend itself to randomized trials, better use of 

quasi-experimental study designs and other study designs of high internal validity could 

greatly enhance the clinical relevance of results, reduce bias and confounding, and increase 

the generalizability of findings. Currently, the published literature is dominated by simple 

pre-post implementation designs. 

4. Creative, alternative research methodology should be considered to estimate costs and 

benefits of HIT as a supplement to traditional hypothesis-testing studies. Traditional 

experimental or quasi-experimental approaches may be impractical because they are 

expensive, time-consuming, and interfere with HIT implementation. Qualitative studies are 

often subjective, descriptive, and lack generalizability. Simulation modeling is a promising 

alternative to generate knowledge and evidence; it is different from analytical modeling 

where the result functionally depends on the input (a number of parameters). Simulation, or 

dynamic, modeling uses a set of rules that define how the system being modeled will change 

in the future, given its present state, existing knowledge, and foreseeable uncertainties. For 

complex problems like HIT implementation, where time dynamics is important and 

experimenting with the real system is expensive or impossible, simulation modeling can 

support estimates of cost, benefit, and net value of HIT systems. 
 63

5. The costs and benefits of HIT depend not only on the internal system (the practice 

environment) but also on the interactions with the external system, including consumers 

(patients and potential users of the healthcare system), medical service suppliers 

(laboratories, radiology centers, other healthcare organizations), technology suppliers, and 

the regulatory and financing systems an organization operates. Multi-perspective studies are 

needed to investigate the flow of costs and benefits in order to maximize the benefits of HIT 

in the larger healthcare delivery system. Again, simulation modeling may be the best 

methodology for this type of research. 

6. The conceptual foundation for the impact of EHRs on improving care is strong. More 

research concerning the efficacy and effectiveness of EHRs across health care settings, 

providers, and patient populations needs to be carried out. Such research will require 

focusing on how EHR tools are implemented and utilized in day-to-day practice, a 

broadening of environments to include nonacademic/nonintegrated network practices, the 

development of methods and instruments directed at evaluation of externally developed 

systems, and a broader understanding of the human factors issues relevant to healthcare. 

7. More research is needed on which specific components of an EHR are beneficial and also on 

evaluating new specific components—for example, clinical decision-support. Much of the 

existing decision support relies on simple rules, and it should be possible to provide 

substantially better assistance with the use of more-complex rules and models. 

8. More research is needed to evaluate the effects of EHRs on improving quality by making 

care more consumer-centered. 

9. Process and outcome benefits of HIT that are important and unique to pediatrics must be 

better quantified, given the unique workflow and information needs of pediatric 

organizations and practice settings. A growing body of epidemiologic studies has 

demonstrated the frequency of medication errors in the pediatric healthcare setting. Welldesigned studies are needed to demonstrate empirically the benefit of HIT in improving 

patient safety, not only in the hospital environment, but also in ambulatory and other settings. 

10. Well-designed studies measuring the costs of HIT implementation and resultant benefits in 

pediatrics and other vulnerable populations (e.g., chronically ill, disabled, etc.) are needed, 

especially in nonacademic settings and with commercially available HIT systems. 

Recommendations Regarding Public and Private Types of 

Organizations to Perform the Proposed Research and/or Analysis 

 The assessment of HIT implementations of greatest relevance to most U.S. healthcare 

institutions will occur in nonacademic settings. Most nonacademic settings have limited research 

expertise or infrastructure to design and support a research project on HIT. If extramural funds 

are desired for an evaluation of HIT implementation, the ability to secure funding coincident 

with the project plan is difficult, if not impossible, especially given the funding cycle of grants. 

Also, to use a pre-and post-implementation design, the researcher needs funding for an extended 

period of time to collect enough data to adequately power the study before the HIT system is in 
 64

place. For financial and pragmatic reasons, this pre-implementation data collection cannot delay 

the HIT implementation process. 

Therefore, we would suggest that for HIT research to be feasible in nonacademic settings 

with commercial systems, some important steps should be taken: 

1. Create incentives (e.g., matching funds) for nonacademic medical centers and provider 

organizations to perform high-quality evaluations of vendor-based HIT implementation. 

These projects should be funded by organizational dollars and support should be provided for 

academic investigators to partner with such organizations. These measures would help 

organizations that lack a built-in research infrastructure to conduct rigorous research. 

2. Provide a number of extramural funding mechanisms (government, state, foundation, or even 

vendor) to evaluate HIT with limited-funding cycles, allowing for adequate preimplementation measurements and/or rigorous study design. The investigators typically do 

not determine the timing of implementation, which is often delayed, and funders much be 

cognizant of this and not penalize the investigators, by disallowing no-cost extensions. 

3. Devise a standard means to adequately assess and describe the “socio-technical” milieu of an 

organization relevant to HIT implementation. 
 65

References 

 1. Kohn LT, Corrigan JM, Donaldson M. To 

Err is Human: Building a Safer Health 

System. Washington, DC: National 

Academy Press. 2000. 

 2. Office of the Coordination of National 

Health Information, Department of Health 

and Human Services. The decade of health 

information technology: delivering 

consumer-centric and information-rich 

health care: framework for strategic action. 

Federal Register 2004;Sec. 411.357 

(u):16142. 

 3. Ash JS, Gorman PN, Seshadri V, et al. 

Computerized physician order entry in U.S. 

hospitals: results of a 2002 survey. J Am 

Med Inform Assoc 2004;11(2):95-9. 

 4. Miller RH, Hillman JM, Given RS. 

Physician use of IT: results from the 

Deloitte Research Survey. J Healthc Inf 

Manag 2004;18(1):72-80. 

 5. Terry K. Doctors and EHRs. Med Econ 

2005;82(2):72-4, 77-8, 80-4. 

 6. Dexter PR, Perkins S, Overhage JM, et al. A 

computerized reminder system to increase 

the use of preventive care for hospitalized 

patients. N Engl J Med 2001;345(13):965-

70. 

 7. Sisk JE, Moskowitz AJ, Whang W, et al. 

Cost-effectiveness of vaccination against 

pneumococcal bacteremia among elderly 

people. JAMA 1997;278(16):1333-9. 

 8. Leape LL, Berwick DM, Bates DW. What 

practices will most improve safety? 

Evidence-based medicine meets patient 

safety. JAMA 2002;288(4):501-507. 

 9. Shojania KG, Duncan BW, McDonald KM, 

et al. Safe but sound. Patient safety meets 

evidence-based medicine. JAMA 

2002;288(4):508-513. 

 10. Kaushal R, Shojania KG, Bates DW. Effects 

of computerized physician order entry and 

clinical decision support systems on 

medication safety: a systematic review. Arch 

Intern Med 2003;163(12):1409-16. 

 11. Johnston ME, Langton KB, Haynes RB, et 

al. Effects of computer-based clinical 

decision support systems on clinician 

performance and patient outcome. A critical 

appraisal of research. Ann Intern Med 

1994;120(2):135-42. 

 12. Corley ST. Electronic prescribing: a review 

of costs and benefits. Top Health Inf 

Manage 2003;24(1):29-38. 

 13. Garg AX, Adhikari NK, McDonald H, et al. 

Effects of computerized clinical decision 

support systems on practitioner performance 

and patient outcomes: a systematic review. 

JAMA 2005;293(10):1223-38. 

 14. Shiffman RN, Liaw Y, Brandt CA, et al. 

Computer-based guideline implementation 

systems: a systematic review of 

functionality and effectiveness. J Am Med 

Inform Assoc 1999;6(2):104-14. 

 15. American Academy of Pediatrics: Task 

Force on Medical Informatics. Special 

requirements for electronic medical record 

systems in pediatrics. Pediatrics 

2001;108(2):513-5. 

 16. Kaushal R, Bates DW, Landrigan C, et al. 

Medication errors and adverse drug events 

in pediatric inpatients. JAMA 

2001;285(16):2114-20. 

 17. Folli HL, Poole RL, Benitz WE, et al. 

Medication error prevention by clinical 

pharmacists in two children's hospitals. 

Pediatrics 1987;79(5):718-22. 

 18. Jonville AP, Autret E, Bavoux F, et al. 

Characteristics of medication errors in 

pediatrics. DICP 1991;25(10):1113-8. 
 66

 19. Holdsworth MT, Fichtl RE, Behta M, et al. 

Incidence and impact of adverse drug events 

in pediatric inpatients. Arch Pediatr Adolesc 

Med 2003;157(1):60-5. 

 20. Koren G, Barzilay Z, Greenwald M. Tenfold 

errors in administration of drug doses: a 

neglected iatrogenic disease in pediatrics. 

Pediatrics 1986;77(6):848-9. 

 21. Diav-Citrin O, Ratnapalan S, Grouhi M, et 

al. Medication errors in paediatrics: a case 

report and systematic review of risk factors. 

Paediatr Drugs 2000;2(3):239-42. 

 22. Crossing The Quality Chasm : A New 

Health System for the 21st Century. 

Washington, D.C.: National Academy Press; 

2001. 

 23. Mullett CJ, Evans RS, Christenson JC, et al. 

Development and impact of a computerized 

pediatric antiinfective decision support 

program. Pediatrics 2001;108(4):E75. 

 24. Fortescue EB, Kaushal R, Landrigan CP, et 

al. Prioritizing strategies for preventing 

medication errors and adverse drug events in 

pediatric inpatients. Pediatrics 2003;111(4 

Pt 1):722-9. 

 25. Potts AL, Barr FE, Gregory DF, et al. 

Computerized Physician Order Entry and 

Medication Errors in a Pediatric Critical 

Care Unit. Pediatrics 2004;113(1):59-63. 

 26. Cordero L, Kuehn L, Kumar RR, et al. 

Impact of computerized physician order 

entry on clinical practice in a newborn 

intensive care unit. J Perinatol 2004;24(2 

):88-93. 

 27. Ornstein SM, Garr DR, Jenkins RG, et al. 

Implementation and evaluation of a 

computer-based preventive services system. 

Fam Med 1995;27(4):260-6. 

 28. Szilagyi PG, Rodewald LE, Savageau J, et 

al. Improving influenza vaccination rates in 

children with asthma: a test of a 

computerized reminder system and an 

analysis of factors predicting vaccination 

compliance. Pediatrics 1992;90(6):871-5. 

 29. Chiarelli F, Tumini S, Morgese G, et al. 

Controlled study in diabetic children 

comparing insulin-dosage adjustment by 

manual and computer algorithms . Diabetes 

Care 1990;13(10):1080-4. 

 30. Schriger DL, Baraff LJ, Buller K, et al. 

Implementation of clinical guidelines via a 

computer charting system: effect on the care 

of febrile children less than three years of 

age. J Am Med Inform Assoc 

2000;7(2):186-95. 

 31. Christakis DA, Zimmerman FJ, Wright JA, 

et al. A randomized controlled trial of pointof-care evidence to improve the antibiotic 

prescribing practices for otitis media in 

children. Pediatrics 2001;107(2):E15. 

 32. Margolis CZ, Warshawsky SS, Goldman L, 

et al. Computerized algorithms and 

pediatricians' management of common 

problems in a community clinic. Acad Med 

1992;67(4):282-4. 

 33. Carroll AE, Tarczy-Hornoch P, O'Reilly E, 

et al. The effect of point-of-care personal 

digital assistant use on resident 

documentation discrepancies. Pediatrics 

2004;113(3 Pt 1):450-4. 

 34. Quattlebaum TG, Darden PM, Sperry JB. 

Effectiveness of computer-generated 

appointment reminders. Pediatrics 

1991;88(4):801-5. 

 35. McGlynn EA, Asch SM, Adams J, et al. The 

quality of health care delivered to adults in 

the United States. N Engl J Med 

2003;348(26):2635-45. 

 36. Chassin MR, Galvin RW. The urgent need 

to improve health care quality. Institute of 

Medicine National Roundtable on Health 

Care Quality. JAMA 1998;280(11):1000-5. 

 37. Eisele CW, Slee VN, Hoffmann RG . Can 

the practice of internal medicine be 

evaluated? Ann Intern Med 1956;44(1):144-

61. 

 38. Brook RH, Appel FA. Quality-of-care 

assessment: choosing a method for peer 

review. N Engl J Med 1973;288(25):1323-9. 
 67

 39. Knaus WA, Draper EA, Wagner DP, et al. 

An evaluation of outcome from intensive 

care in major medical centers. Ann Intern 

Med 1986;104(3):410-8. 

 40. Dubois RW, Brook RH, Rogers WH. 

Adjusted hospital death rates: a potential 

screen for quality of medical care. Am J 

Public Health 1987;77(9):1162-6. 

 41. Hussey PS, Anderson GF, Osborn R, et al. 

How does the quality of care compare in 

five countries? Health Aff (Millwood) 

2004;23(3):89-99. 

 42. Hippisley-Cox J, O'Hanlon S, Coupland C. 

Association of deprivation, ethnicity, and 

sex with quality indicators for diabetes: 

population based survey of 53,000 patients 

in primary care. BMJ 2004;329(7477):1267-

9. 

 43. Barros FC, Vaughan JP, Victoria CG. Why 

so many caesarean sections? The need for a 

further policy change in Brazil. Health 

Policy Plan 1986;1(1):19-29. 

 44. Donabedian A. Explorations in Quality 

Assessment and Monitoring. Vol I. The 

Definition of Quality and Approaches to its 

Assessment. Ann Arbor, MI: Health 

Administration Press; 1980. 

 45. Tierney WM, Miller ME, McDonald CJ. 

The effect on test ordering of informing 

physicians of the charges for outpatient 

diagnostic tests. N Engl J Med 1990 

;322(21):1499-504. 

 46. Tierney WM, McDonald CJ, Hui SL, et al. 

Computer predictions of abnormal test 

results. Effects on outpatient testing. JAMA 

1988;259(8):1194-8. 

 47. Tierney WM, McDonald CJ, Martin DK, et 

al. Computerized display of past test results. 

Effect on outpatient testing. Ann Intern Med 

1987; 107(4):569-74. 

 48. Kucher N, Koo S, Quiroz R, et al. Electronic 

alerts to prevent venous thromboembolism 

among hospitalized patients. N Engl J Med 

2005;352(10):969-77. 

 49. Chin HL, Wallace P. Embedding guidelines 

into direct physician order entry: simple 

methods, powerful results. Proc AMIA 

Symp 1999;221-5. 

 50. van Wijk MA, van der Lei J, Mosseveld M, 

et al. Assessment of decision support for 

blood test ordering in primary care. a 

randomized trial. Ann Intern Med 

2001;134(4):274-81. 

 51. Safran C, Rind DM, Davis RB, et al. 

Guidelines for management of HIV 

infection with computer-based patient's 

record. Lancet 1995;346(8971):341-6. 

 52. Wang SJ, Middleton B, Prosser LA, et al. A 

cost-benefit analysis of electronic medical 

records in primary care. Am J Med 

2003;114:397-403. 

 53. Khoury AT. Support of quality and business 

goals by an ambulatory automated medical 

record system in Kaiser Permanente of 

Ohio. Eff Clin Pract 1998;1(2):73-82. 

 54. Khoury A. Finding value in EMRs 

(electronic medical records). Health Manag 

Technol 1997;18(8):34, 36. 

 55. Kian LA, Stewart MW, Bagby C, et al. 

Justifying the cost of a computer-based 

patient record. Healthc Financ Manage 

1995;49(7):58-60, 62, 64-7. 

 56. McDowell I, Newell C, Rosser W. A 

randomized trial of computerized reminders 

for blood pressure screening in primary care. 

Med Care 1989;27(3):297-305. 

 57. McDowell I, Newell C, Rosser W. 

Computerized reminders to encourage 

cervical screening in family practice. J Fam 

Pract 1989;28(4):420-4. 

 58. Rosser WW, Hutchison BG, McDowell I, et 

al. Use of reminders to increase compliance 

with tetanus booster vaccination. CMAJ 

1992;146(6):911-7. 

 59. McDowell I, Newell C, Rosser W. 

Comparison of three methods of recalling 

patients for influenza vaccination. CMAJ 

1986;135(9):991-7. 
 68

 60. Rosser WW, McDowell I, Newell C . Use of 

reminders for preventive procedures in 

family medicine. CMAJ 1991;145(7):807-

14. 

 61. Simon GE, VonKorff M, Rutter C, et al. 

Randomised trial of monitoring, feedback, 

and management of care by telephone to 

improve treatment of depression in primary 

care. BMJ 2000;320(7234):550-4. 

 62. Cameron S, Regalado M, Quitoles M, et al. 

Harnessing technology: the creation of an 

electronic care management record in a 

social health maintenance organization. 

Manag Care Q 1999;7(1):11-5. 

 63. Daly JM, Buckwalter K, Maas M. Written 

and computerized care plans. Organizational 

processes and effect on patient outcomes. J 

Gerontol Nurs 2002;28(9):14-23. 

 64. Krall MA. Acceptance and performance by 

clinicians using an ambulatory electronic 

medical record in an HMO. Proc Annu 

Symp Comput Appl Med Care 1995;708-11. 

 65. Bulpitt CJ, Beilin LJ, Coles EC, et al. 

Randomised controlled trial of computerheld medical records in hypertensive 

patients. Br Med J 1976;1(6011):677-9. 

 66. Burkle T, Michel A, Horch W, et al. 

Computer based nursing documentation 

means to achieve the goal. Int J Med Inf 

1998;52(1-3):71-80. 

 67. Kilgore ML, Flint D, Pearce R. The varying 

impact of two clinical information systems 

in a cardiovascular intensive care unit. J 

Cardiovasc Manag 1998;9(2):31-5. 

 68. Evans RS, Pestotnik SL, Classen DC, et al. 

Prevention of adverse drug events through 

computerized surveillance. Proc Annu Symp 

Comput Appl Med Care 1992;437-441. 

 69. Classen DC, Pestotnik SL, Evans RS, et al. 

Adverse drug events in hospitalized patients. 

Excess length of stay, extra costs, and 

attributable mortality. JAMA 

1997;277(4):301-306. 

 70. Evans RS, Classen DC, Stevens LE, et al. 

Using a hospital information system to 

assess the effects of adverse drug events. 

Proc Annu Symp Comput Appl Med Care 

1993;161-165. 

 71. Tierney WM, Murray MD, Gaskins DL, et 

al. Using computer-based medical records to 

predict mortality risk for inner-city patients 

with reactive airways disease. J Am Med 

Inform Assoc 1997;4(4):313-21. 

 72. Bates DW, Kuperman GJ, Rittenberg E, et 

al. A randomized trial of a computer-based 

intervention to reduce utilization of 

redundant laboratory tests. Am J Med 

1999;106(2):144-50. 

 73. Ornstein SM, MacFarlane LL, Jenkins RG, 

et al. Medication cost information in a 

computer-based patient record system. 

Impact on prescribing in a family medicine 

clinical practice. Arch Fam Med 

1999;8(2):118-21. 

 74. Wells BJ, Lobel KD, Dickerson LM. Using 

the electronic medical record to enhance the 

use of combination drugs. Am J Med Qual 

2003;18(4):147-9. 

 75. Mc Donald CJ. Use of a computer to detect 

and respond to clinical events: its effect on 

clinician behavior. Ann Intern Med 

1976;84(2):162-7. 

 76. Jones DL, Kroenke K, Landry FJ, et al. Cost 

savings using a stepped-care prescribing 

protocol for nonsteroidal anti-inflammatory 

drugs. JAMA 1996;275(12):926-30. 

 77. Wilson GA, McDonald CJ, McCabe GP= Jr. 

The effect of immediate access to a 

computerized medical record on physician 

test ordering: a controlled clinical trial in the 

emergency room. Am J Public Health 

1982;72(7):698-702. 

 78. Tierney WM, Miller ME, Overhage JM, et 

al. Physician inpatient order writing on 

microcomputer workstations. Effects on 

resource utilization. JAMA 1993;269(3 

):379-83. 
 69

 79. Evans RS, Pestotnik SL, Classen DC, et al. 

A computer-assisted management program 

for antibiotics and other antiinfective agents. 

N Engl J Med 1998;338(4):232-8. 

 80. Arias-Vimarlund V, Ljunggren M, Timpka 

T. Implementation of computer-based 

patient records in primary care: the societal 

health economic effects. Proc AMIA Annu 

Fall Symp 1996;503-7. 

 81. Schmitt KF, Wofford DA. Financial analysis 

projects clear returns from electronic 

medical records. Healthc Financ Manage 

2002;56(1):52-7. 

 82. Walker J, Pan E, Johnston D, et al. The 

value of health care information exchange 

and interoperability. Health Affairs 

2005;w5.10. 

 83. Agrawal A. Return on investment analysis 

for a computer-based patient record in the 

outpatient clinic setting. J Assoc Acad 

Minor Phys 2002;13(3):61-5. 

 84. Valancy J. How much will that EMR system 

really cost? Fam Pract Manag 2002;9(4):57-

8. 

 85. Chambers CV, Balaban DJ, Carlson BL, et 

al. Microcomputer-generated reminders. 

Improving the compliance of primary care 

physicians with mammography screening 

guidelines. J Fam Pract 1989;29(3):273-80. 

 86. Becker DM, Gomez EB, Kaiser DL, et al. 

Improving preventive care at a medical 

clinic: how can the patient help? Am J Prev 

Med 1989;5(6): 353-9. 

 87. Moran WP, Nelson K, Wofford JL, et al. 

Computer-generated mailed reminders for 

influenza immunization: a clinical trial. J 

Gen Intern Med 1992;7(5):535-7. 

 88. Frame PS, Zimmer JG, Werth PL, et al. 

Computer-based vs manual health 

maintenance tracking. A controlled trial. 

Arch Fam Med 1994;3(7):581-8. 

 89. Barton MB, Schoenbaum SC. Improving 

influenza vaccination performance in an 

HMO setting: the use of computer-generated 

reminders and peer comparison feedback. 

Am J Public Health 1990;80(5):534-6. 

 90. McDowell I, Newell C, Rosser W. A followup study of patients advised to obtain 

influenza immunizations. Fam Med 

1990;22(4):303-6. 

 91. Baird TK, Broekemeier RL, Anderson MW. 

Effectiveness of a computer-supported refill 

reminder system. Am J Hosp Pharm 

1984;41(11):2395-7. 

 92. Simkins CV, Wenzloff NJ. Evaluation of a 

computerized reminder system in the 

enhancement of patient medication refill 

compliance. Drug Intell Clin Pharm 

1986;20(10):799-802. 

 93. Raynor DK, Booth TG, Blenkinsopp A. 

Effects of computer generated reminder 

charts on patients' compliance with drug 

regimens. BMJ 1993;306(6886):1158-61. 

 94. Eadie LH, Seifalian AM, Davidson BR. 

Telemedicine in surgery. Br J Surg 

2003;90(6):647-58. 

 95. Rosenfeld BA, Dorman T, Breslow MJ, et 

al. Intensive care unit telemedicine: alternate 

paradigm for providing continuous 

intensivist care. Crit Care Med 

2000;28(12):3925-31. 

 96. Rogers MA, Small D, Buchan DA, et al. 

Home monitoring service improves mean 

arterial pressure in patients with essential 

hypertension. A randomized, controlled 

trial. Ann Intern Med 2001;134(11):1024-

32. 

 97. Pacht ER, Turner JW, Gailiun M, et al. 

Effectiveness of telemedicine in the 

outpatient pulmonary clinic. Telemed J 

1998;4(4):287-92. 

 98. Johnson KB, Davison CL. Information 

technology: its importance to child safety. 

Ambul Pediatr 2004;4( 1):64-72. 
 70

 99. Johnston B, Wheeler L, Deuser J, et al. 

Outcomes of the Kaiser Permanente TeleHome Health Research Project. Arch Fam 

Med 2000;9(1):40-5. 

 100. Bass DM, McClendon MJ, Brennan PF, et 

al. The buffering effect of a computer 

support network on caregiver strain. J Aging 

Health 1998;10(1):20-43. 

 101. Rethans JJ, Hoppener P, Wolfs G, et al. Do 

personal computers make doctors less 

personal? Br Med J (Clin Res Ed) 

1988;296(6634):1446-8. 

 102. Threet E, Fargues MP. Economic evaluation 

of voice recognition for the clinicians' 

desktop at the Naval Hospital Roosevelt 

Roads. Mil Med 1999;164(2):119-26. 

 103. Nahm R, Poston I. Measurement of the 

effects of an integrated, point-of-care 

computer system on quality of nursing 

documentation and patient satisfaction. 

Comput Nurs 2000;18(5):220-9. 

 104. O'Connell RT, Cho C, Shah N, et al. Take 

note(s): differential EHR satisfaction with 

two implementations under one roof. J Am 

Med Inform Assoc 2004;11(1):43-9. 

 105. van den Nieuwenhuyzen MC, Engbers FH, 

Burm AG, et al. Computer-controlled 

infusion of alfentanil versus patientcontrolled administration of morphine for 

postoperative analgesia: a double-blind 

randomized trial. Anesth Analg 

1995;81(4):671-9. 

 106. Greist JH, Marks IM, Baer L, et al. Behavior 

therapy for obsessive-compulsive disorder 

guided by a computer or by a clinician 

compared with relaxation as a control. J Clin 

Psychiatry 2002;63(2):138-45. 

 107. Krishna S, Balas EA, Spencer DC, et al. 

Clinical trials of interactive computerized 

patient education: implications for family 

practice. J Fam Pract 1997;45(1):25-33. 

 108. Strecher VJ, Kreuter M, Den Boer DJ, et al. 

The effects of computer-tailored smoking 

cessation messages in family practice 

settings. J Fam Pract 1994;39(3):262-70. 

 109. Liaw T, Lawrence M, Rendell J. The effect 

of a computer-generated patient-held 

medical record summary and/or a written 

personal health record on patients' attitudes, 

knowledge and behaviour concerning health 

promotion. Fam Pract 1996;13(3):289-93. 

 110. Lewis G, Sharp D, Bartholomew J, et al. 

Computerized assessment of common 

mental disorders in primary care: effect on 

clinical outcome. Fam Pract 

1996;13(2):120-6. 

 111. Revere D, Dunbar PJ. Review of computergenerated outpatient health behavior 

interventions: clinical encounters "in 

absentia". J Am Med Inform Assoc 

2001;8(1):62-79. 

 112. Bessell TL, McDonald S, Silagy CA, et al. 

Do Internet interventions for consumers 

cause more harm than good? A systematic 

review. Health Expect 2002;5(1):28-37. 

 113. Eysenbach G, Powell J, Englesakis M, et al. 

Health related virtual communities and 

electronic support groups: systematic review 

of the effects of online peer to peer 

interactions. BMJ 2004;328(7449):1166. 

 114. Car J, Sheikh A. Email consultations in 

health care: 1--scope and effectiveness. BMJ 

2004;329 (7463):435-8. 

 115. Balas EA, Jaffrey F, Kuperman GJ, et al. 

Electronic communication with patients. 

Evaluation of distance medicine technology. 

JAMA 1997;278(2):152-9. 

 116. Overhage JM, Suico J, McDonald CJ. 

Electronic laboratory reporting: barriers, 

solutions and findings. J Public Health 

Manag Pract 2001;7(6): 60-6. 

 117. Murff HJ, Kannry J. Physician satisfaction 

with two order entry systems. J Am Med 

Inform Assoc 2001;8(5):499-509. 

 118. Overhage JM, Perkins S, Tierney WM, et al. 

Controlled trial of direct physician order 

entry: effects on physicians' time utilization 

in ambulatory primary care internal 

medicine practices. J Am Med Inform Assoc 

2001; 8(4):361-71. 
 71

 119. Wong DH, Gallegos Y, Weinger MB, et al. 

Changes in intensive care unit nurse task 

activity after installation of a thirdgeneration intensive care unit information 

system. Crit Care Med 2003;31 (10):2488-

94. 

 120. Abookire SA, Teich JM, Sandige H, et al. 

Improving allergy alerting in a computerized 

physician order entry system. Proc AMIA 

Symp 2000;2-6. 

 121. Ammenwerth E, Mansmann U, Iller C, et al. 

Factors affecting and affected by user 

acceptance of computer-based nursing 

documentation: results of a two-year study. J 

Am Med Inform Assoc 2003;10 (1):69-84. 

 122. Gamm LD, Barsukiewicz CK, Dansky KH, 

et al. Investigating changes in end-user 

satisfaction with installation of an electronic 

medical record in ambulatory care settings. J 

Healthc Inf Manag 1998;12 (4):53-65. 

 123. Burkle T, Kuch R, Prokosch HU, et al. 

Stepwise evaluation of information systems 

in an university hospital. Methods Inf Med 

1999;38(1):9-15. 

 124. Elbourne D, Richardson M, Chalmers I, et 

al. The Newbury Maternity Care Study: a 

randomized controlled trial to assess a 

policy of women holding their own obstetric 

records. Br J Obstet Gynaecol 

1987;94(7):612-9. 

 125. Lazarus SS. Physicians' use of electronic 

medical records--identifying and crossing 

the barriers. Med Group Manage J 

1999;46(3):12-4. 

 126. Berkowitz LL. Breaking down the barriers. 

Improving physician buy-in of CPR 

systems. Healthc Inform 1997;14(10):73-6. 

 127. Bates DW, Boyle DL, Teich JM. Impact of 

computerized physician order entry on 

physician time. Proc Annu Symp Comput 

Appl Med Care 1994;996. 

 128. Pizziferri L, Kittler AF, Volk LA, et al. 

Primary care physician time utilization 

before and after implementation of an 

electronic health record: a time-motion 

study. J Biomed Inform 2005;38(3):176-88. 

 129. Hersh WR, Hickam DH. How well do 

physicians use electronic information 

retrieval systems? A framework for 

investigation and systematic review. JAMA 

1998;280(15):1347-52. 

 130. Johnson KB. Barriers that impede the 

adoption of pediatric information 

technology. Arch Pediatr Adolesc Med 

2001;155(12):1374-9. 

 131. Miller RH, Sim I. Physicians' use of 

electronic medical records: barriers and 

solutions. Health Aff (Millwood) 

2004;23(2):116-26. 

 132. Poon EG, Blumenthal D, Jaggi T, et al. 

Overcoming barriers to adopting and 

implementing computerized physician order 

entry systems in U.S. hospitals. Health Aff 

(Millwood) 2004;23(4):184-90. 

 133. Hersh W. Health care information 

technology: progress and barriers. JAMA 

2004;292(18):2273-4. 

 134. Wears RL, Berg M. Computer technology 

and clinical work: still waiting for Godot. 

JAMA 2005;293(10):1261-3. 
Appendix A. Technical Expert Panel 

A-1

David Bates, MD, MSc 

Chief, General Medicine 

Brigham and Women’s Hospital 

620 Tremont Street, Room BC3-002M 

Boston, MA 02120-1613 

617-732-5650 

617-732-7072 (fax) 

dbates@partners.org 

John Byrnes, MD

Chief Medical Officer 

Spectrum Health 

100 Michigan Ave. NE 

Grand Rapids, MI 49503 

616-391-2774 

john.byrnes@spectrum-health.org 

David Classen, MD

Vice President 

First Consulting Group 

561 East Northmont Way 

Salt Lake City, UT 84103 

801-532-3633 

801-596-3443 (fax) 

dclassen@fcg.com 

Simon Cohn, MD, MPH

National Director for Health 

Information Policy 

Kaiser Permanente Medical Care Program 

One Kaiser Plaza, 27L 

Oakland, CA 94612 

510-271-6461 

510-267-2194 (fax) 

simon.cohn@kp.org 

Mary Jo Deering, PhD

Director 

Health Communication and Telehealth 

Office of Disease Prevention and 

Health Promotion 

U.S. Department of Health and 

Human Services 

200 Independence Avenue SW 

Room 738-G 

Washington, DC 20201 

202-260-2652 

202-205-0463 (fax) 

mdeering@osophs.dhhs.gov

Mary Jo Deering, Ph.D.*

Director for Informatics Dissemination 

NCI Center for Bioinformatics 

National Cancer Institute 

National Institutes of Health, USDHHS 

6116 Executive Blvd. - #400 

Rockville, MD 20852 

Phone: 301-594-1273 

Fax: 301-480-3441 

E-mail: deeringm@mail.nih.gov 

Jennie Harvell

Senior Policy Analyst 

U.S. Department of Health and Human 

Services 

Humphrey Building 

200 Independence Avenue SW 

Washington, DC 20201 

202-690-6443 

202-401-7733 (fax) 

jennie.harvell@hhs.gov 

Robert Kolodner, MD

Acting Deputy CIO for Healthcare 

Department of Veterans Affairs 

810 Vermont Avenue NW 

Washington, DC 20420 

David Lansky, Ph.D.

Foundation for Accountability 

1200 NW Naito Parkway, Suite 470 

Portland, OR 97209 

503-223-2228 
Appendix A. Technical Expert Panel 

A-2

202-273-8663 

robkolodner@hq.med.va.gov 

503-223-4336 (fax) 

dlansky@facct.org 

Janet M. Marchibroda 

Chief Executive Officer 

eHealth Initiative 

1500 K Street, N.W., Suite 900 

Washington, D.C. 20005 

202-624-3270 

202-624-3263 (direct) 

Blackford Middleton, MD, MPH, MSc

Clinical Informatics R&D 

Partners Healthcare 

93 Worcester St., Box 81905 

Wellesley, MA 02481 

781-416-8528 

781-416-2286 (fax) 
A-3

Arnold Milstein, MD, MPH

Principal and Consultant 

William M. Mercer, Inc. 

3 Embarcadero Center 

Suite 1500 

San Francisco, CA 94111 

415-743-8803 

415-743-8950 (fax) 

arnold.milstein@mercer.com 

Elliott Sternberg, MD

Senior Vice President and 

Chief Medical Officer 

St. Joseph Health System 

500 S. Main Street, Suite 900 

Orange, CA 92868 

714-347-7841 

esternbe@corp.stjoe.org 

Paul C. Tang, MD

Chief Medical Officer 

Palo Alto Medical Foundation 

795 El Camino Real 

Palo Alto, CA 94301 

650-853-5775 

650-853-6050 (fax) 

tang@smi.stanford.edu 

Scott Weingarten, MD, MPH

Chief Medical Officer 

Zynx Health, Inc. 

9100 Wilshire Blvd., Suite 655-E 

Beverly Hills, CA 90210 

310-247-7700 

310-247-710 (fax) 

weingarten@zynx.com 

Scott Young, M.D. 

Director, Health Information Technology 

Programs and Research 

Agency for Healthcare Research and 

Quality 

540 Gaither Road 

Rockville, MD 20850 

301-427-1580 

301-427-1595 (fax) 

syoung@ahrq.gov 

David Cutler, PhD 

Economist, 

Department of Economics 

Littauer Center 

1875 Cambridge Street 

Harvard University 

Cambridge, MA 02138 
A-4

Marc Berg 

Prof. Dr. Marc Berg, MA, MD, PhD 

Chair of Dept. of Social Medical Sciences 

Institute of Health Policy and Management 

Erasmus University Rotterdam 

P.O. Box 1738 

3000 DR Rotterdam 

The Netherlands 

Phone: ++31-10-4088531 / 4088525 

Fax: ++31-10-4089094 

M.Berg@bmg.eur.nl 

Margaret Coopey RN, MGA, MPS 

Senior Health Policy Analyst 

Agency for Healthcare Research and 

Quality 

540 Gaither Road 

Rockville, Maryland 20850 

301-427-1618 

301-427-1520 (fax) 

mcoopey@ahrq.gov

David Atkins

Agency for Healthcare Research and 

Quality 

540 Gaither Road 

Rockville, Maryland 20850 

301-427-1618 

301-427-1520 (fax) 

datkins@ahrq.gov
B-1

Appendix B 

INFORMATION TECHNOLOGY IN HEALTH 

SEARCH STRATEGIES 

SEARCH METHODOLOGY – INFORMATION TECHNOLOGY + 

EFFECTS/OUTCOMES 

DATABASE SEARCHED: PUBMED 

YEARS OF COVERAGE: 1995-2003 

OTHER LIMITERS: 

 English 

SEARCH STRATEGY: 

automatic data processing[majr] OR medical informatics[majr] OR medical informatics 

applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR 

information technolog* OR information infrastructure* OR ehealth OR e-health 

AND 

adverse effects[sh] OR outcome and process assessment health care[mh] OR costs and 

cost analysis OR efficiency, organizational OR risk assessment OR outcome*[ti] OR 

outcome*[ab] OR cost[ti] OR cost[ab] OR costs[ti] OR costs[ab] OR efficien*[ti] OR 

efficien*[ab] OR risk*[ti] OR risk*[ab] OR adverse[ti] OR adverse[ab] 

AND 

systematic[sb] – NOTE: This is the pre-formulated search developed by Medline to 

include systematic reviews, meta analyses, etc. 

NUMBER OF ITEMS RETRIEVED: 516 

================================================================ 

SEARCH METHODOLOGY – INFORMATION TECHNOLOGY + IMPROVEMENT 

DATABASE SEARCHED: PUBMED 

YEARS OF COVERAGE: 2002-2004 

OTHER LIMITERS: 

 English 

SEARCH STRATEGY: 

automatic data processing[majr] OR medical informatics[majr] OR medical informatics 

applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR 

information technolog* OR information infrastructure* OR ehealth OR e-health 

AND 
B-2

outcome assessment health care OR process assessment health care OR workplace 

OR workflow* OR work flow* OR quality indicators, health care
B-3

AND 

improve*[tiab] OR chang*[tiab] 

NUMBER OF ITEMS RETRIEVED: 244 

================================================================== 

SEARCH METHODOLOGY – PRELIMINARY PUBMED SEARCH ON INFORMATION 

TECHNOLOGY BARRIERS 

Database Searched and Time Period Covered: 

 PubMed – 2000-2003 

Other Limiters: 

 English only 

Search Strategy: 

automatic data processing[majr] OR medical informatics[majr] OR medical informatics 

applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR 

attitude to computers OR information technolog* 

AND 

barrier* OR challeng* OR difficult* OR resist OR resisting OR resistance 

NUMBER OF ITEMS RETRIEVED: 1662 

=================================================================== 

INFORMATION TECHNOLOGY & HEALTH – POPULAR LITERATURE 

Database: Periodical Abstracts 

Query: ((de: information and de: technology) or (de: information and de: systems) or de: 

computer+ or de: digital or de: electronic or de: internet) and (de: health or de: 

healthcare or de: medical or de: medicine or de: physician+ or de: doctor+ or de: 

hospital+) and yr: 2002-2003 

NUMBER OF ITEMS RETRIEVED: 433 

=================================================================== 

SEARCH METHODOLOGY

Health Affairs Journal 

SEARCH #1 (PERFORMED 12/23/04) 

DATABASE SEARCHED AND TIME PERIOD COVERED: 

PUBMED 1966-2004 

SEARCH STRATEGY: 

information technology
B-4

AND 

information systems

NUMBER OF ITEMS RETRIEVED: 98

=================================================================== 

SEARCH METHODOLOGY 

COMPUTERIZED MEDICAL RECORDS – COST BENEFITS 

SEARCH #1 (PERFORMED 1/6/04) 

DATABASE SEARCHED AND TIME PERIOD COVERED: 

PUBMED 1995-2004 

SEARCH STRATEGY: 

medical records systems, computerized[majr] 

 AND 

 costs and cost analysis OR economics OR financ* OR economics(subheading) 

AND 

Review 

NUMBER OF ITEMS RETRIEVED: 26 

=================================================================== 

SEARCH #2 (PERFORMED 1/6/04) 

DATABASE SEARCHED AND TIME PERIOD COVERED: 

PUBMED 1995-2004 

SEARCH STRATEGY: 

medical records systems, computerized[majr] 

 AND 

 costs and cost analysis OR economics OR financ* OR economics(subheading) 

NOT 

Review 

NUMBER OF ITEMS RETRIEVED: 871 

NUMBER OF ITEMS SENT TO RESEARCHER (after deleting irrelevant items): 186 

(COVERING 2000-2003)
Appendix C 

Article ID: 0000 Health Information Technology Reviewers: Assigned on:

Reviewer: ______________ Round Two Screener

C-1

Check all that apply on each question. To change pre-screener data from “checked” to “unchecked” please write 

“uncheck” next to the appropriate box. 

1. What are the one or two main HIT elements being tested? 

Computerized Provider Order Entry...............................

Electronic Health Record ...............................................

Decision Support............................................................

Results Reporting/Viewing Systems ..............................

Electronic Prescribing (incl. Barcoding)........................

Mobile Computing..........................................................

Data Exchange Networks/ Community Health 

Information Network ..................................................

Patient Decision Support/ Consumer Health 

 Informatics .............................................................. 

Communication Systems .................................................

Administrative ..................................................................

Knowledge/ Information Retrieval Systems......................

Data Collection/ Data Summary Systems ........................

Telemedicine ..................................................................

HIT in general .................................................................

Other (specify: ________________________)................

Not HIT ............................................................................

N/A, N/R...........................................................................

2. Which IOM categories does the HIT address? 

Health information and data storage..............................

Results management.....................................................

 Order entry management ..........................................

Decision support ............................................................

Electronic communication and connectivity ...................

Patient support...............................................................

Administrative processes...............................................

Reporting and population health management ..............

N/A, N/R.........................................................................

3. What are the types of healthcare organization settings?

Hospital/inpatient ...........................................................

Outpatient/ambulatory....................................................

Integrated delivery Network (IDN)..................................

Emergency room............................................................

Nursing home ................................................................

Patient home..................................................................

Pediatrics .......................................................................

Pharmacy.......................................................................

Internet...........................................................................

Other setting (specify: __________________) ..............

N/A, N/R.........................................................................

4. What is the article’s purpose? [Circle one]

Descriptive 

1. Qualitative 2. Quantitative 3. Other descriptive 

Hypothesis testing: 

With Intervention, with concurrent comparison group: 

4. RCT 5. CCT 6. Cntrl. Before/After 

With intervention, without concurrent comparison group: 

7. Pre-Post 8. Time series 9. Historical control 

No intervention 

10. Cross-sectional 19. Case study with concurrent control 

Other hypothesis testing: 

11. Other hypothesis testing 

Predictive analysis 

12. Cost-effect. 13. Cost-benefit 14. Other pred.analysis 

Review 

15. Non-systematic 16.Systematic/MA 

Other Purpose 

18. Other (specify: _________________) 

5. Does this article report data from any of the 

following systems? 

Intermountain.................................................................

Partners.........................................................................

Regenstrief ....................................................................

VA..................................................................................

UK’s NHS ......................................................................

Kaiser ............................................................................

Vanderbilt University......................................................

N/A, N/R ........................................................................

 

6. Which outcomes are measured (numerically 

reported) in the article? 

Impact on patient safety.................................................

Impact on patient satisfaction ........................................

Impact on health care effectiveness and quality............

Impact on efficiency, utilization, and costs.....................

Time: Admin ; Nurs. ; Phys. ; Pt ; NOS 

Impact on healthcare access.........................................

Other (specify:_____________________) ................... 

N/A, N/R ....................................................................... 

7. Are barriers or facilitators the main focus of the 

paper, and/or are numerical results given? 

[CIRCLE ONE]

Yes ................................................................................ 1 

No.................................................................................. 2 

N/A, N/R ........................................................................ 8 

8. What years did the research take place? 

(Enter 4-digit years. N/A, N/R: enter 9999)

Year began Year ended 

NOTES:

 Check here if this article should be a Star Article 

Candidate 
Appendix D: List of TEP Members Who Provided Comments 

 

D-1

David Bates, MD, MSc 

Chief, General Medicine 

Brigham and Women’s Hospital 

620 Tremont Street, Room BC3-002M 

Boston, MA 02120-1613 

617-732-5650 

617-732-7072 (fax) 

dbates@partners.org 

David Atkins

Agency for Healthcare Research and Quality 

540 Gaither Road 

Rockville, Maryland 20850 

301-427-1618 

301-427-1520 (fax) 

datkins@ahrq.gov 

Marc Berg 

Prof. Dr. Marc Berg, MA, MD, PhD 

Chair of Dept. of Social Medical Sciences 

Institute of Health Policy and Management 

Erasmus University Rotterdam 

P.O. Box 1738 

3000 DR Rotterdam 

The Netherlands 

Phone: ++31-10-4088531 / 4088525 

Fax: ++31-10-4089094 

M.Berg@bmg.eur.nl 

Mary Jo Deering, Ph.D.*

Director for Informatics Dissemination 

NCI Center for Bioinformatics 

National Cancer Institute 

National Institutes of Health, USDHHS 

6116 Executive Blvd. - #400 

Rockville, MD 20852 

Phone: 301-594-1273 

Fax: 301-480-3441 

E-mail: deeringm@mail.nih.gov 

Jennie Harvell

Senior Policy Analyst 

U.S. Department of Health and Human 

Services 

Humphrey Building 

200 Independence Avenue SW 

Washington, DC 20201 

202-690-6443 

202-401-7733 (fax) 

jennie.harvell@hhs.gov 

Scott Weingarten, MD, MPH

Chief Medical Officer 

Zynx Health, Inc. 

9100 Wilshire Blvd., Suite 655-E 

Beverly Hills, CA 90210 

310-247-7700 

310-247-710 (fax) 

weingarten@zynx.com 

Arnold Milstein, MD, MPH

Principal and Consultant 

William M. Mercer, Inc. 

3 Embarcadero Center 

Suite 1500 

San Francisco, CA 94111 

415-743-8803 

415-743-8950 (fax) 

arnold.milstein@mercer.com 

Elliott Sternberg, MD

Senior Vice President and 

Chief Medical Officer 

St. Joseph Health System 

500 S. Main Street, Suite 900 

Orange, CA 92868 

714-347-7841 

esternbe@corp.stjoe.org 

Cynthia Baur 

Office of Public Health and Science, Office of 

Disease Prevention and Health Promotion 

HHS/OS/OPHS/ODPHP 

738G Humphrey Building 

telephone: (202) 205-2311 

telefax: (202) 205-0463 
Appendix D: List of TEP Members Who Provided Comments 

 

D-2

e-mail: cbaur@osophs.dhhs.gov
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

4. Conclusions

The overall cautious tone of the conclusions is appropriate given the limitations of the data. No response neede

4. Conclusions

Use IOM framework of HIT functions to outline conclusions about which are most likely to produce benefits.Report could do more to make its findings useful in the current context of decisions re. HIT - which systems should be implemented to support which functions.

This is a desirable gthink is achievable because the benefit context specific, defunctionalities of thcurrent state of the 

Overall What savings will derive from better quality, to whom do those savings accrue, and over what 

time frame?

We have indicated answer this questio

1. Introduction

1 Few data about stand-alones making a difference. Second paragraph, after "These systems," 

please remove "used as stand-alone clinical decision support tools, or" 

Done.

1. Introduction

3 Fourth par, after "Unless they have most of the business for that organization," add "or agree 

to collaborate" Also, remove definition of "free rider" problem.

Done.

1. Introduction

5 Not fair to impugn prior studies without specifying all? in advance. Moreover, most journals 

wouldn't publish the articles if you added more description of components. Please add caveats.

We are not meaninspecifying what we important compone"intervention." We 

1. Introduction

6 Regarding HIT articles, would need some standards about how to do what is expected. 

Standard is to provide bed size, academic/ not. I don't disagree that more info would be helpful.

No response needeWe all agree we nestatement for HIT i

3. Results 13 Second paragraph "..furthermore, it is not possible to calculate the cost of the development of 

an HIT system, since this process has occurred over many years." Too gross a generalization. Sometimes you add a module quickly. Replace "not possible" with "challenging." Change to "since this process often has occurred over.."

We added "as a whreferring to the maiaddition of a single 

Page 1
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

3. Results 13 paragraph 2, before last sentence, add "They are not commercially available from a vendor, 

and vendors supply most HIT systems in the U.S."

Done.

3. Results 14 "No study evaluated an HIT system with at least 4 of the 8 categories of functionality." You 

might list the categories here or explain.

We added this info

3. Results - EHR and ambulatory 

21 Do you just mean ambulatory EHRs? You have left out lots of inpatient studies. Yes, this section fo

EHRs.

3. Results - EHR and ambulatory care

24 "Evaluating the effects of EHR adoption was itself a form of structural change in this study." 

Please reword, unclear. What about the studies on preventive care? Should explain why omitted.

This section has beattempt to make thi

3. Results - Economic benefits of EHR

36 2nd par - can you repeat size to provide perspective? Done.

3. Results - Economic benefits of 

37 Should make separate point about importance of interoperability and that it needs to be 

machine level for high value.

We have included a importance of inter

3. Results -barriers

41 Add cites: Lee, F, Journal of American Medical Informatics Assoc Shu K, in MEDINFO 

proceedings, 2005 Pizziferri, et al 2005 These may be too recent for your entry criteria.

We were unable to American Medical (Shu K, in MEDINadded the Pizziferri barriers.

3. Results -barriers

42 Last paragraph - I don't think you should close with this. This editorial overall is about as 

evidence free as such things get. 

We deleted this tex

Page 2
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

4. Limitations 45 1st paragraph "Most of this information is absent for most published studies of HIT." You 

should say why journals have word limits, and standards for publication.

We added a caveat 

4. Future research

47 "This will require the development of high quality data collection instruments that will be 

feasible to use before, during and after the implementation process." I don't think this will help.

We changed this to such reports."

4. Future research

48 #6 "Well-designed studies are needed to empirically demonstrate the benefit of HIT in 

improving patient safety not only in the hospital environment, but also in ambulatory settings." These will be published soon.

We look forward to 

4. Future research

49 Add "support for academic investigators to partner with such organizations" (i.e. nonacademic centers, provider orgs) Done.

4. Future research

50 "Provide research support and expertise to grantee organizations lacking a built-in research 

infrastructure." Not realistic - won't happen. Suggest omitting.

We changed the wostrongly emphasize partnership.

4. Conclusions

vi "We identified no hypothesis testing study of HIT that assessed a system of broad 

functionality which included sufficient cost data and organizational context information to allow generalization to other health care settings." - What about the Regenstrief CPOE study? Suggest dropping this statement.

We meant cost data implementation cosa single componentinformation to this 

4. Conclusions

I think you are too critical of the published literature on HIT re. inclusions of administrative info, given lack of a standard and word limits on published papers, though I agree it is a good idea.

We added caveats t

Overall This is truly a stellar piece of work. No response neede

Page 3
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

1. Introduction

1 "The use of health information technology (HIT) holds tremendous promise in improving the 

efficiency, cost-effectiveness, quality and safety of medical care delivery in our nation’s healthcare system." see comment on conclusions. Unless tied to a source, such as an IOM report, this statement is more appropriate to a policy paper than an evidence report.

We changed this to having..."

1. Introduction

5 HIT interventions have a fifth component: communication (see in text comment) We have added com

existing componencomponent.

3. Results - Pediatrics

21 Tone and content of the 3 section summaries (within peds) are very different. Make more 

consistent.

In this revision, we to try and make the consistent.

3. Results - EHR and ambulatory

22 "studies that were accepted after screening, review, and reconciliation" unclear This was reworded

3. Results - EHR and ambulatory 

32 need to harmonize content and tone. Need to acknowledge extremely small number of 

studies, almost 50% are at same institution

Done.

3. Results - Economic 

benefits of 

33 use of "we" is not consistent across report We tried to make th

3. Results - Economic benefits of HER

34 health care information exchange and interoperability - HIEI - this seems like an unhelpful 

acronym

Done.

Page 4
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

4. Future research

50 This section seems unfinished in terms of development of ideas. This particular revi

unedited version of section has been mdraft.

Preface iii Add that ODPHR provided additional funding We have included a4. Conclusions

viii "HIT has the potential to enable a dramatic transformation in the delivery of health care, 

making it safer, more effective, and more efficient." I think this sentence should be reconsidered in light of the conclusion sections of the case studies. As an evidence report and not a policy paper, the sentence should more closely reflect the findings, which are that the evidence is very limited, from a very small number of studies and organizations, and is inconclusive. The findings are suggestive of future research more than anything else and of the need to systematize data collection about organizations and contexts in order to be able to make comparisons across studies and settings, per the final sentence of this section. Perhaps another point to make is that cost/benefit analysis may be a relatively recent way of looking at 

We revised this stapredictive analyses potential.

Overall 1 I read the report with pleasure. The database is excellent and useful. No response neede

Page 5
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

2. Methods 2 Most reviews and 'searches for evidence' on benefits (either financial or otherwise) of HIT in 

the health care literature fall into the trap of treating a HIT as just another 'new intervention' that has to be evaluated using RCTs or its somewhat lesser equals. The important contribution of this report is that it has picked up the message... that HIT cannot be treated that way.

No response neede

2. Methods 3 ...any 'IT innovation' is highly context-specific. There are so many key variables at stake, the 

authors argue, that no overall review is possible. Therefore, the authors revert to creating an 

No response neede

Page 6
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

Overall 4 This report candidly and powerfully lays out the dilemmas involved in the questions it sets 

out to answer. Simultaneously, however, I feel that it could have made a stronger step forward in arguing for a REPHRASING of the question for ' proof' for 'benefits' than it currently does. ..the important question would be whether the technology technically does what it promises to do (which is a question dependent on the specific application in question) and whether the organization is subsequently ' ready' to use this technology in such a way that it may reap these benefits. The latter question is about proper organizational change, about creating a ' fit' between the technology and the organization which will generate these potential benefits. Again, nobody in that world would consider asking for RCTs or other such designs. The potential benefit, after all, is plain common sense; the question is whether these specific instances (technology and organization) will be able to yield these benefits. That is a substantially different question from interventions such a novel drug where it is NOT plain common sense whether it will yield improved outcomes compared to a placebo or its competito

Whether a particular organization will realize the benefits that such ‘common sense' innovation

We have added thaand organizational important componeimplementation.

General I see absolutely no discussion in the main text to support the statement about lack of evidence 

for ambulatory EHRs and improving quality by making it more consumer and patient-centric. [I have] the desire to see more research on the costs and benefits to patients of HIT.The dearth of research on the value of HIT to the patient and the consumer, and the value of HIT to healthcare quality through more patient-centric focus, should be elevated in the discussion section of the report 

We have added a nInformation TechnoCenteredness"

1. Introduction

2 2nd para - isn't there some impact on costs, ie. increasing revenue by providing more 

vaccinations? Would this revenue be offset by reduced hospitalizations?

We have re-done thclear.

1. Introduction

3 4th para - add: The gains to HMOs of better care will be sure when capitation payments are 

risk adjusted. 

We have added the HMOs of better carcapitation paymentadjusted."

Page 7
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

1. Introduction

3 Remove comment on "free rider" problem. We have explained 

1. Introduction

3 After "insurers might want to subsidize it in part" add: i.e. based on the payer's share of the 

provider's case mix.

We have added, "ishare of the provide

1. Introduction

4 "There are financial aspects to many of these non-monetized items. For example, the 

intervention may reduce the cost of meeting a preexisting reporting requirement." Not sure of the point, given that even nonprofits ..must stay in business.

Agreed, but many tcare that have non-simply making that 

2. Methods 9 Report "Advanced Technologies to Low Health Care Costs and Improve Quality" - what 

settings were discussed?

Settings of care havsentence, (inpatient 

2. Methods 9 JAMIA meta-analysis on computer based clinical reminder systems - what settings were 

discussed?

Settings of care havsentence, (ambulato

3. Results - EHR and ambulatory 

20 Structure-process-outcome framework did not fit with types of outcomes discussed. 

Framework also interrupted the flow of discussion. Please rewrite

We have restructurand structure-proceallow for more flui

Page 8
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

3. Results - EHR and ambulatory 

20 3rd par - "Therefore, the roles of EHRs in improving quality is unclear at this time." 

However, later on the authors state that there is evidence of improvements in quality."5 of the 7 assessed quality through an analysis of some type of outcome." This conflicts with previous paragraph "6 of the 7 assessed quality with respect to some type of outcome."Par 3 - What is meant by "patient charts became available"? Electronic charts?

We have entirely resatisfy these comm

3. Results - Economic 

benefits of 

31 Somewhere early you need to define what constitutes an EHR, minimum functionality, etc. This is now done in 

paragraph.

3. Results - Economic 

benefits of 

31 Do you mean to imply that decision support is not part of EHR? This is better expla

paragraph.

3. Results - Economic 

benefits of 

31 Strengthen the discussion of evidence that emerges from each of the following studies that 

support an "economic appraisal" of the EHR component.

We have expanded 

3. Results - Economic 

benefits of 

33 1st Par - Provide an example or two about how these studies quantified / monetized costs and 

benefits.

How these variableindicated in the tex

3. Results - Economic 

benefits of 

33 Be consistent - either refer to the study location or don't. You mention Kaiser and Partners but 

not the other study sites.

We have changed t

3. Results - Economic benefits of HER

34 First study - ($9700 per provider) - what were the component parts of the EHR system? We now provide th

EHR system.

3. Results - Economic benefits of HER

37 Conclusion 1 - What about information exchange capabilities? We added this as p

Page 9
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

3. Results - Economic benefits of HER

37 Conclusion 5 - Revise last sentence to: Changes to the current health care financing system 

would be required to realize all quantifiable benefits of EHR implementation.

We have changed t"Realizing all quanimplementation wocurrent health care 

4. Conclusions

47 Change "Economic value" heading to "Economic value of HIT and EHR Implementation" This has been chan

an EHR System"

4. Conclusions

47 Conclusion 4. Under economic value - change 1st sentence to "The positive economic 

estimates for EHR system implementation are encouraging but at this time are based on limited evidence."

Done.

4. Conclusions

47 Add conclusion "There is some evidence regarding the positive economic value of 

implementing component parts of EHRs."

We have added this 

4. Future research

47 Change 1 to "Available evidence highlights the potential for EHRs to impact quality of care. 

However, a more systematic, evidence based understanding of the role of EHRs in quality improvement in ambulatory care and other settings is needed.

We have changed tevidence highlights ambulatory EHRs tHowever, a more syunderstanding of thin quality improveminformation is criticadoption, but also tadopters realize fro

4. Future research

48 Remove "ambulatory" before EHRs in all the sentences. Done.

Preface iv Health Communication and Telehealth provided funding - who is this? This was an incorre

Page 10
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

assorted typos, language usage We have made the Overall This is a wonderful review and a very intuitive organization of key findings. No response needeOverall Given our national predilections for leaping before we look, what you have compiled and 

made accessible will nonetheless provide a more valuable guide for pioneers than pure intuition. 

No response neede

1. Introduction

Intro focuses the report on costs, benefits, and barriers to HIT. Organization of the report and conclusions should specifically focus on those areas, or intro should be changed.

We have restructurreport.

4. Conclusions

Are there any recommendations for effective metrics to be tracked for organizations implementing HIT?

We have added a cneeds standards in 

Overall Add more on potential disconnect between who is funding and who is benefiting, i.e. 

physician adoption at a community based medical center is distinctly different than at an academic medical center based on financial relationships.

We have added to tthat "who pays" anquestion and that, eKaiser and the VA, published data to reother than to state texists. 

Overall There is a problem knowing when to "jump in" i.e. when will interfaces be optimized and 

products ready.

We agree this is a pscant evidence rega

Overall High quality is less of an issue when patients do not have the ability to choose / access all 

providers.

No response neede

Overall A simple projected case study might be illustrative, i.e.. adding Fine Pneumonia Protocols for 

pneumonia admission into CPOE or EHR.

We discussed this iinclude this, there astudies published, wdatabase. Interested there. 

Page 11
Appendix E. REVIEWER COMMENTS

"Health Information Technology - Costs and Benefits"

Section Page Comment 

Overall Report is terrific. No response needeOverall Comment on how this report extends or compliments recent JAMA review. It extends the recen

that the HIT literatuthe decision suppora few centers evaluThe generalizability evaluations is questgiven.

Page 12
F-1

Appendix F. Articles Included in HIT Interactive Database 

 1. Abookire SA, Teich JM, Sandige H, et al. 

Improving allergy alerting in a 

computerized physician order entry 

system. Proc AMIA Symp 2000;2-6. 

[Rec#: 54] 

 2. Agrawal A. Return on investment analysis for 

a computer-based patient record in 

the outpatient clinic setting. J Assoc 

Acad Minor Phys 2002;13(3):61-5. 

[Rec#: 431] 

 3. Al-Aynati MM, Chorneyko KA. Comparison 

of voice-automated transcription and 

human transcription in generating 

pathology reports. Arch Pathol Lab 

Med 2003;127(6):721-5. 

[Rec#: 432] 

 4. Alvis JM, Reves JG, Govier AV, et al. 

Computer-assisted continuous 

infusions of fentanyl during cardiac 

anesthesia: comparison with a manual 

method. Anesthesiology 

1985;63(1):41-9. 

[Rec#: 636] 

 5. Ammenwerth E, Eichstadter R, Haux R, et al. 

A randomized evaluation of a 

computer-based nursing 

documentation system. Methods Inf 

Med 2001;40(2):61-8. 

[Rec#: 637] 

 6. Ammenwerth E, Mansmann U, Iller C, et al. 

Factors affecting and affected by user 

acceptance of computer-based 

nursing documentation: results of a 

two-year study. J Am Med Inform 

Assoc 2003;10 (1):69-84. 

[Rec#: 181] 

 7. Anderson JG, Jay SJ, Anderson M, et al. 

Evaluating the capability of 

information technology to prevent 

adverse drug events: A computer 

simulation approach. J Am Med 

Inform Assoc 2002;9:479-490. 

[Rec#: 172] 

 8. Arias-Vimarlund V, Ljunggren M, Timpka T. 

Implementation of computer-based 

patient records in primary care: the 

societal health economic effects. Proc 

AMIA Annu Fall Symp 1996;503-7. 

[Rec#: 182] 

 9. Baird TK, Broekemeier RL, Anderson MW. 

Effectiveness of a computersupported refill reminder system. Am 

J Hosp Pharm 1984;41(11):2395-7. 

[Rec#: 740] 

 10. Balas EA, Austin SM, Mitchell JA, et al. The 

clinical value of computerized 

information services. A review of 98 

randomized clinical trials. Arch Fam 

Med 1996;5(5):271-8. 

[Rec#: 56] 

 11. Balas EA, Boren SA, Griffing G . 

Computerized management of 

diabetes: a synthesis of controlled 

trials. Proc AMIA Symp 1998;295-9. 

[Rec#: 185] 

 12. Balas EA, Jaffrey F, Kuperman GJ, et al. 

Electronic communication with 

patients. Evaluation of distance 

medicine technology. JAMA 

1997;278(2):152-9. 

[Rec#: 57] 

 13. Balas EA, Weingarten S, Garb CT, et al. 

Improving preventive care by 

prompting physicians. Arch Intern 

Med 2000;160(3):301-8. 

[Rec#: 58] 

 14. Baldwin DR, Beech CA, Evans AH, et al. 

Principals of design and evaluation of 

an information system for a 

department of respiratory medicine. 

Health Care Anal 1997;5(1):78-84. 

[Rec#: 187] 

 15. Ball A, Goodman M, Evans L, et al. Computer 

generated discharge summaries and 

their use as a case mix sensitive audit 

engine. A tale of two cities. J R Coll 

Physicians Lond 1999;33(3):260-3. 

[Rec#: 188] 
F-2

 16. Barton MB, Schoenbaum SC. Improving 

influenza vaccination performance in 

an HMO setting: the use of computergenerated reminders and peer 

comparison feedback. Am J Public 

Health 1990;80(5):534-6. 

[Rec#: 875] 

 17. Bass DM, McClendon MJ, Brennan PF, et al. 

The buffering effect of a computer 

support network on caregiver strain. J 

Aging Health 1998;10(1):20-43. 

[Rec#: 59] 

 18. Bates DW, Boyle DL, Teich JM. Impact of 

computerized physician order entry 

on physician time. Proc Annu Symp 

Comput Appl Med Care 1994;996. 

[Rec#: 2] 

 19. Bates DW, Kuperman GJ, Rittenberg E, et al. 

A randomized trial of a computerbased intervention to reduce 

utilization of redundant laboratory 

tests. Am J Med 1999;106(2):144-50. 

[Rec#: 63] 

 20. Bates DW, Leape LL, Cullen DJ, et al. Effect 

of computerized physician order entry 

and a team intervention on prevention 

of serious medication errors. JAMA 

1998;280(15):1311-1316. 

[Rec#: 4] 

 21. Bates DW, Teich JM, Lee J, et al. The impact 

of computerized physician order entry 

on medication error prevention. J Am 

Med Inform Assoc 1999;6(4):313-21. 

[Rec#: 64] 

 22. Becker DM, Gomez EB, Kaiser DL, et al. 

Improving preventive care at a 

medical clinic: how can the patient 

help? Am J Prev Med 1989;5(6): 353-

9. 

[Rec#: 150] 

 23. Becker SH, Arenson RL. Costs and benefits of 

picture archiving and communication 

systems. J Am Med Inform Assoc 

1994;1(5):361-71. 

[Rec#: 7] 

 24. Bennett JW, Glasziou PP. Computerised 

reminders and feedback in medication 

management: a systematic review of 

randomised controlled trials. Med J 

Aust 2003;178(5):217-22. 

[Rec#: 192] 

 25. Bessell TL, McDonald S, Silagy CA, et al. Do 

Internet interventions for consumers 

cause more harm than good? A 

systematic review. Health Expect 

2002;5(1):28-37. 

[Rec#: 193] 

 26. Boldreghini S, Larrabee JH. Difference in 

nursing documentation before and 

after computerization: A pilot study. J 

Nurs Inform [serial online] 1998;2(1). 

[Rec#: 988] 

 27. Bouhaddou O, Lambert JG, Morgan GE. Iliad 

and Medical HouseCall: evaluating 

the impact of common sense 

knowledge on the diagnostic accuracy 

of a medical expert system. Proc 

Annu Symp Comput Appl Med Care 

1995;742-6. 

[Rec#: 725] 

 28. Brettle A. Information skills training: a 

systematic review of the literature. 

Health Info Libr J 2003;20 Suppl 1:3-

9. 

[Rec#: 197] 

 29. Brownbridge G, Evans A, Fitter M, et al. An 

interactive computerized protocol for 

the management of hypertension: 

effects on the general practitioner's 

clinical behaviour. J R Coll Gen Pract 

1986;36(286):198-202. 

[Rec#: 638] 

 30. Brownbridge G, Evans A, Wall T . Effect of 

computer use in the consultation on 

the delivery of care. Br Med J (Clin 

Res Ed) 1985;291(6496):639-42. 

[Rec#: 639] 

 31. Bulpitt CJ, Beilin LJ, Coles EC, et al. 

Randomised controlled trial of 

computer-held medical records in 

hypertensive patients. Br Med J 

1976;1(6011):677-9. 

[Rec#: 745] 
F-3

 32. Burack RC, Gimotty PA. Promoting screening 

mammography in inner-city settings. 

The sustained effectiveness of 

computerized reminders in a 

randomized controlled trial. Med 

Care 1997;35(9):921-31. 

[Rec#: 707] 

 33. Burack RC, Gimotty PA, George J, et al. 

Promoting screening mammography 

in inner-city settings: a randomized 

controlled trial of computerized 

reminders as a component of a 

program to facilitate mammography. 

Med Care 1994;32(6):609-24. 

[Rec#: 159] 

 34. Burkle T, Kuch R, Prokosch HU, et al. 

Stepwise evaluation of information 

systems in an university hospital. 

Methods Inf Med 1999;38(1):9-15. 

[Rec#: 641] 

 35. Burkle T, Michel A, Horch W, et al. Computer 

based nursing documentation means 

to achieve the goal. Int J Med Inf 

1998;52(1-3):71-80. 

[Rec#: 642] 

 36. Burton ME, Ash CL, Hill DP= Jr, et al. A 

controlled trial of the cost benefit of 

computerized bayesian 

aminoglycoside administration. Clin 

Pharmacol Ther 1991;49(6):685-94. 

[Rec#: 643] 

 37. Bushko RG, Havlicek PL, Deppert E, et al. 

Impact of voice- and knowledgeenabled clinical reporting--US 

example. Stud Health Technol Inform 

2002;80:265-74. 

[Rec#: 460] 

 38. Cameron S, Regalado M, Quitoles M, et al. 

Harnessing technology: the creation 

of an electronic care management 

record in a social health maintenance 

organization. Manag Care Q 

1999;7(1):11-5. 

[Rec#: 461] 

 39. Campbell SM, Roland MO, Gormanly B. 

Evaluation of a computerized 

appointment system in general 

practice. Br J Gen Pract 

1996;46(409):477-8. 

[Rec#: 880] 

 40. Cannon DS, Allen SN. A comparison of the 

effects of computer and manual 

reminders on compliance with a 

mental health clinical practice 

guideline. J Am Med Inform Assoc 

2000;7(2):196-203. 

[Rec#: 167] 

 41. Car J, Sheikh A. Email consultations in health 

care: 1--scope and effectiveness. BMJ 

2004;329( 7463):435-8. 

[Rec#: 950] 

 42. Carmenates J, Keith MR. Impact of 

automation on pharmacist 

interventions and medication errors in 

a correctional health care system. Am 

J Health Syst Pharm 2001;58(9):779-

83. 

[Rec#: 201] 

 43. Carroll AE, Tarczy-Hornoch P, O'Reilly E, et 

al. The effect of point-of-care 

personal digital assistant use on 

resident documentation discrepancies. 

Pediatrics 2004;113(3 Pt 1):450-4. 

[Rec#: 919] 

 44. Carroll AE, Tarczy-Hornoch P, O'Reilly E, et 

al. The effect of point-of-care 

personal digital assistant use on 

resident documentation discrepancies. 

Pediatrics 2004;113(3 Pt 1):450-4. 

[Rec#: 989] 

 45. Casner PR, Reilly R, Ho H. A randomized 

controlled trial of computerized 

pharmacokinetic theophylline dosing 

versus empiric physician dosing. Clin 

Pharmacol Ther 1993;53(6):684-90. 

[Rec#: 644] 

 46. Chambers CV, Balaban DJ, Carlson BL, et al. 

The effect of microcomputergenerated reminders on influenza 

vaccination rates in a universitybased family practice center. J Am 

Board Fam Pract 1991;4 (1):19-26. 

[Rec#: 882] 
F-4

 47. Chambers CV, Balaban DJ, Carlson BL, et al. 

Microcomputer-generated reminders. 

Improving the compliance of primary 

care physicians with mammography 

screening guidelines. J Fam Pract 

1989;29(3):273-80. 

[Rec#: 149] 

 48. Chatellier G, Colombet I, Degoulet P. 

Computer-adjusted dosage of 

anticoagulant therapy improves the 

quality of anticoagulation. Medinfo 

1998;9 Pt 2:819-23. 

[Rec#: 203] 

 49. Chatellier G, Colombet I, Degoulet P. An 

overview of the effect of computerassisted management of anticoagulant 

therapy on the quality of 

anticoagulation. Int J Med Inf 

1998;49(3):311-20. 

[Rec#: 204] 

 50. Chen P, Tanasijevic MJ, Schoenenberger RA, 

et al. A computer-based intervention 

for improving the appropriateness of 

antiepileptic drug level monitoring . 

Am J Clin Pathol 2003;119( 3):432-8. 

[Rec#: 205] 

 51. Chertow GM, Lee J, Kuperman GJ, et al. 

Guided medication dosing for 

inpatients with renal insufficiency. 

JAMA 2001;286(22):2839-44. 

[Rec#: 162] 

 52. Chiarelli F, Tumini S, Morgese G, et al. 

Controlled study in diabetic children 

comparing insulin-dosage adjustment 

by manual and computer algorithms . 

Diabetes Care 1990;13(10):1080-4. 

[Rec#: 645] 

 53. Chin HL, Wallace P. Embedding guidelines 

into direct physician order entry: 

simple methods, powerful results. 

Proc AMIA Symp 1999;221-5. 

[Rec#: 465] 

 54. Chin JM, Muller RJ, Lucarelli CD. A 

pharmacy intervention program: 

recognizing pharmacy's contribution 

to improving patient care. Hosp 

Pharm 1995;30(2):120, 123-6, 129-

30. 

[Rec#: 206] 

 55. Christakis DA, Zimmerman FJ, Wright JA, et 

al. A randomized controlled trial of 

point-of-care evidence to improve the 

antibiotic prescribing practices for 

otitis media in children. Pediatrics 

2001;107(2):E15. 

[Rec#: 732] 

 56. Christensen-Szalanski JJ, Diehr PH, Wood 

RW, et al. Phased trial of a proven 

algorithm at a new primary care 

clinic. Am J Public Health 

1982;72(1):16-21. 

[Rec#: 883] 

 57. Classen DC, Pestotnik SL, Evans RS, et al. 

Adverse drug events in hospitalized 

patients. Excess length of stay, extra 

costs, and attributable mortality. 

JAMA 1997;277(4):301-306. 

[Rec#: 11] 

 58. Coe FL, Norton E, Oparil S, et al. Treatment 

of hypertension by computer and 

physician-a prospective controlled 

study. J Chronic Dis 1977;30(2):81-

92. 

[Rec#: 746] 

 59. Cordero L, Kuehn L, Kumar RR, et al. Impact 

of computerized physician order entry 

on clinical practice in a newborn 

intensive care unit. J Perinatol 

2004;24(2 ):88-93. 

[Rec#: 920] 

 60. Daly JM, Buckwalter K, Maas M. Written and 

computerized care plans. 

Organizational processes and effect 

on patient outcomes. J Gerontol Nurs 

2002;28(9):14-23. 

[Rec#: 646] 

 61. Davidson KW, Kahn A, Price RD. Reduction 

of adverse drug reactions by 

computerized drug interaction 

screening. J Fam Pract 

1987;25(4):371-5. 

[Rec#: 885] 

 62. Day F, Hoang LP, Ouk S, et al. The impact of 

a guideline-driven computer charting 

system on the emergency care of 

patients with acute low back pain. 

Proc Annu Symp Comput Appl Med 

Care 1995;576-80. 
F-5

[Rec#: 724] 

 63. Demakis JG, Beauchamp C, Cull WL, et al. 

Improving residents' compliance with 

standards of ambulatory care: results 

from the VA Cooperative Study on 

Computerized Reminders. JAMA 

2000;284(11):1411-6. 

[Rec#: 729] 

 64. Dexter PR, Perkins SM, Maharry KS, et al. 

Inpatient Computer-based Standing 

Orders vs physician reminders to 

increase influenza and pneumococcal 

vaccination rates. JAMA 

2004;292(19):2366-2371. 

[Rec#: 948] 

 65. Dexter PR, Wolinsky FD, Gramelspacher GP, 

et al. Effectiveness of computergenerated reminders for increasing 

discussions about advance directives 

and completion of advance directive 

forms. A randomized, controlled trial. 

Ann Intern Med 1998;128(2):102-10. 

[Rec#: 649] 

 66. Dickinson JC, Warshaw GA, Gehlbach SH, et 

al. Improving hypertension control: 

impact of computer feedback and 

physician education. Med Care 

1981;19(8):843-54. 

[Rec#: 743] 

 67. Donald JB. Prescribing costs when computers 

are used to issue all prescriptions. 

BMJ 1989;299( 6690):28-30. 

[Rec#: 651] 

 68. Durieux P, Nizard R, Ravaud P, et al. A 

clinical decision support system for 

prevention of venous 

thromboembolism: effect on 

physician behavior. JAMA 

2000;283(21):2816-21. 

[Rec#: 71] 

 69. Eadie LH, Seifalian AM, Davidson BR. 

Telemedicine in surgery. Br J Surg 

2003;90(6):647-58. 

[Rec#: 224] 

 70. Eccles M, McColl E, Steen N, et al. Effect of 

computerised evidence based 

guidelines on management of asthma 

and angina in adults in primary care: 

cluster randomised controlled trial. 

BMJ 2002;325(7370):941. 

[Rec#: 72] 

 71. Ehler D, Vacek JL, Bansal S, et al. Transition 

to an all-digital echocardiography 

laboratory: a large, multi-site private 

cardiology practice experience. J Am 

Soc Echocardiogr 2000;13(12):1109-

16. 

[Rec#: 481] 

 72. Evans RS, Classen DC, Pestotnik SL, et al. 

Improving empiric antibiotic 

selection using computer decision 

support. Arch Intern Med 1994; 

154(8):878-84. 

[Rec#: 866] 

 73. Evans RS, Classen DC, Stevens LE, et al. 

Using a hospital information system 

to assess the effects of adverse drug 

events. Proc Annu Symp Comput 

Appl Med Care 1993;161-165. 

[Rec#: 17] 

 74. Evans RS, Larsen RA, Burke JP, et al. 

Computer surveillance of hospitalacquired infections and antibiotic use. 

JAMA 1986;256(8):1007-11. 

[Rec#: 74] 

 75. Evans RS, Pestotnik SL, Classen DC, et al. 

Prevention of adverse drug events 

through computerized surveillance. 

Proc Annu Symp Comput Appl Med 

Care 1992;437-441. 

[Rec#: 18] 

 76. Evans RS, Pestotnik SL, Classen DC, et al. 

Evaluation of a computer-assisted 

antibiotic-dose monitor. Ann 

Pharmacother 1999;33(10):1026-31. 

[Rec#: 870] 

 77. Evans RS, Pestotnik SL, Classen DC, et al. A 

computer-assisted management 

program for antibiotics and other 

antiinfective agents. N Engl J Med 

1998;338(4):232-8. 

[Rec#: 75] 
F-6

 78. Eysenbach G, Powell J, Englesakis M, et al. 

Health related virtual communities 

and electronic support groups: 

systematic review of the effects of 

online peer to peer interactions. BMJ 

2004;328(7449):1166. 

[Rec#: 923] 

 79. Fihn SD, McDonell MB, Vermes D, et al. A 

computerized intervention to improve 

timing of outpatient follow-up: a 

multicenter randomized trial in 

patients treated with warfarin. 

National Consortium of 

Anticoagulation Clinics. J Gen Intern 

Med 1994;9(3):131-9. 

[Rec#: 654] 

 80. Fitzmaurice DA, Hobbs FD, Delaney BC, et 

al. Review of computerized decision 

support systems for oral 

anticoagulation management. Br J 

Haematol 1998;102(4):907-9. 

[Rec#: 234] 

 81. Fitzmaurice DA, Hobbs FD, Murray ET, et al. 

Evaluation of computerized decision 

support for oral anticoagulation 

management based in primary care. 

Br J Gen Pract 1996;46(410):533-5. 

[Rec#: 751] 

 82. Fortescue EB , Kaushal R, Landrigan CP, et al. 

Prioritizing strategies for preventing 

medication errors and adverse drug 

events in pediatric inpatients. 

Pediatrics 2003;111(4 Pt 1): 722-9. 

[Rec#: 990] 

 83. Frame PS, Zimmer JG, Werth PL, et al. 

Computer-based vs manual health 

maintenance tracking. A controlled 

trial. Arch Fam Med 1994;3(7): 581-

8. 

[Rec#: 160] 

 84. Frances CD, Alperin P, Adler JS, et al. Does a 

fixed physician reminder system 

improve the care of patients with 

coronary artery disease? A 

randomized controlled trial. West J 

Med 2001;175(3):165-6. 

[Rec#: 733] 

 85. Friedman CP, Elstein AS, Wolf FM, et al. 

Enhancement of clinicians' diagnostic 

reasoning by computer-based 

consultation: a multisite study of 2 

systems. JAMA 1999;282(19):1851-

6. 

[Rec#: 78] 

 86. Gajewski LK, Vanscoy GJ, Dean JC, et al. The 

development of a computerized LANbased investigational drug 

information database. Hosp Pharm 

1995;30(12):1113, 1116-8, 1120. 

[Rec#: 240] 

 87. Gamm LD, Barsukiewicz CK, Dansky KH, et 

al. Investigating changes in end-user 

satisfaction with installation of an 

electronic medical record in 

ambulatory care settings. J Healthc 

Inf Manag 1998;12 (4):53-65. 

[Rec#: 488] 

 88. Garg AX, Adhikari NK, McDonald H, et al. 

Effects of computerized clinical 

decision support systems on 

practitioner performance and patient 

outcomes: a systematic review. 

JAMA 2005;293(10):1223-38. 

[Rec#: 994] 

 89. Garr DR, Ornstein SM, Jenkins RG, et al. The 

effect of routine use of computergenerated preventive reminders in a 

clinical practice. Am J Prev Med 

1993;9(1):55-61. 

[Rec#: 655] 

 90. Garrido T, Jamieson L, Zhou Y, et al. Effect of 

electronic health records in 

ambulatory care: retrospective, serial, 

cross sectional study. BMJ 

2005;330(7491):581. 

[Rec#: 996] 

 91. Gehlbach SH, Wilkinson WE, Hammond WE, 

et al. Improving drug prescribing in a 

primary care practice. Med Care 

1984;22(3):193-201. 

[Rec#: 656] 

 92. Gilliland A, Mills KA, Steele K. General 

practitioner records on computer--

handle with care. Fam Pract 

1992;9(4):441-50. 

[Rec#: 657] 
F-7

 93. Gioia PC. Quality improvement in pediatric 

well care with an electronic record. 

Proc AMIA Symp 2001;209-13. 

[Rec#: 493] 

 94. Goethe JW, Schwartz HI, Szarek BL. 

Physician compliance with practice 

guidelines. Conn Med 

1997;61(9):553-8. 

[Rec#: 708] 

 95. Goldman L, Cook EF, Brand DA, et al. A 

computer protocol to predict 

myocardial infarction in emergency 

department patients with chest pain. 

N Engl J Med 1988;318(13):797-803. 

[Rec#: 658] 

 96. Gonzalez ER, Vanderheyden BA, Ornato JP, 

et al. Computer-assisted optimization 

of aminophylline therapy in the 

emergency department. Am J Emerg 

Med 1989;7(4):395-401. 

[Rec#: 659] 

 97. Gray J, Majeed A, Kerry S, et al. Identifying 

patients with ischaemic heart disease 

in general practice: cross sectional 

study of paper and computerised 

medical records. Bmj 

2000;321(7260):548-50. 

[Rec#: 246] 

 98. Greist JH, Marks IM, Baer L, et al. Behavior 

therapy for obsessive-compulsive 

disorder guided by a computer or by a 

clinician compared with relaxation as 

a control. J Clin Psychiatry 

2002;63(2):138-45. 

[Rec#: 249] 

 99. Harris RP, O'Malley MS, Fletcher SW, et al. 

Prompting physicians for preventive 

procedures: a five-year study of 

manual and computer reminders. Am 

J Prev Med 1990;6(3):145-52. 

[Rec#: 888] 

 100. Heden B, Ohlin H, Rittner R, et al. Acute 

myocardial infarction detected in the 

12-lead ECG by artificial neural 

networks. Circulation 

1997;96(6):1798-802. 

[Rec#: 81] 

 101. Hersh WR, Hickam DH. How well do 

physicians use electronic information 

retrieval systems? A framework for 

investigation and systematic review. 

Jama 1998;280(15):1347-52. 

[Rec#: 257] 

 102. Hershey CO, Porter DK, Breslau D, et al. 

Influence of simple computerized 

feedback on prescription charges in 

an ambulatory clinic. A randomized 

clinical trial. Med Care 

1986;24(6):472-81. 

[Rec#: 734] 

 103. Herzmark G, Brownbridge G, Fitter M, et al. 

Consultation use of a computer by 

general practitioners. J R Coll Gen 

Pract 1984;34(269):649-54. 

[Rec#: 662] 

 104. Hetlevik I, Holmen J, Kruger O. Implementing 

clinical guidelines in the treatment of 

hypertension in general practice. 

Evaluation of patient outcome related 

to implementation of a computerbased clinical decision support 

system . Scand J Prim Health Care 

1999;17(1):35-40. 

[Rec#: 728] 

 105. Hickling K, Begg E, Moore ML. A 

prospective randomised trial 

comparing individualised 

pharmacokinetic dosage prediction 

for aminoglycosides with prediction 

based on estimated creatinine 

clearance in critically ill patients. 

Intensive Care Med 1989;15(4):233-

7. 

[Rec#: 663] 

 106. Hobbs FD, Delaney BC, Carson A, et al. A 

prospective controlled trial of 

computerized decision support for 

lipid management in primary care. 

Fam Pract 1996;13(2):133-7. 

[Rec#: 664] 

 107. Honigman B, Lee J, Rothschild J, et al. Using 

computerized data to identify adverse 

drug events in outpatients. J Am Med 

Inform Assoc 2001;8(3):254-66. 

[Rec#: 83] 
F-8

 108. Hooker RC, Cowap N, Newson R, et al. Better 

by half: hypertension in the elderly 

and the 'rule of halves': a primary care 

audit of the clinical computer record 

as a springboard to improving care. 

Fam Pract 1999;16(2):123-8. 

[Rec#: 262] 

 109. Hutchison BG. Effect of computer-generated 

nurse/physician reminders on 

influenza immunization among 

seniors. Fam Med 1989;21(6):433-7. 

[Rec#: 889] 

 110. Jha AK, Kuperman GJ, Teich JM, et al. 

Identifying adverse drug events: 

development of a computer-based 

monitor and comparison with chart 

review and stimulated voluntary 

report. J Am Med Inform Assoc 

1998;5(3):305-14. 

[Rec#: 861] 

 111. Johnson KB, Davison CL. Information 

technology: its importance to child 

safety. Ambul Pediatr 2004;4(1):64-

72. 

[Rec#: 939] 

 112. Johnston B, Wheeler L, Deuser J, et al. 

Outcomes of the Kaiser Permanente 

Tele-Home Health Research Project. 

Arch Fam Med 2000;9(1):40-5. 

[Rec#: 85] 

 113. Jones DL, Kroenke K, Landry FJ, et al. Cost 

savings using a stepped-care 

prescribing protocol for nonsteroidal 

anti-inflammatory drugs. JAMA 

1996;275(12):926-30. 

[Rec#: 890] 

 114. Kaltenthaler E, Shackley P, Stevens K, et al. A 

systematic review and economic 

evaluation of computerised cognitive 

behaviour therapy for depression and 

anxiety. Health Technol Assess 

2002;6(22):1-89. 

[Rec#: 272] 

 115. Kaushal R, Shojania KG, Bates DW. Effects 

of computerized physician order entry 

and clinical decision support systems 

on medication safety: a systematic 

review. Arch Intern Med 

2003;163(12):1409-16. 

[Rec#: 22] 

 116. Kenkre J, Drury VW, Lancashire RJ. Nurse 

management of hypertension clinics 

in general practice assisted by a 

computer. Fam Pract 1985;2(1): 17-

22. 

[Rec#: 747] 

 117. Kerr EA, Smith DM, Hogan MM, et al. 

Comparing clinical automated, 

medical record, and hybrid data 

sources for diabetes quality measures. 

Jt Comm J Qual Improv 

2002;28(10):555-65. 

[Rec#: 165] 

 118. Khoury A. Finding value in EMRs (electronic 

medical records). Health Manag 

Technol 1997;18(8):34, 36. 

[Rec#: 801] 

 119. Kian LA, Stewart MW, Bagby C, et al. 

Justifying the cost of a computerbased patient record. Healthc Financ 

Manage 1995;49(7):58-60, 62, 64-7. 

[Rec#: 802] 

 120. Kilgore ML, Flint D, Pearce R . The varying 

impact of two clinical information 

systems in a cardiovascular intensive 

care unit. J Cardiovasc Manag 

1998;9(2):31-5. 

[Rec#: 668] 

 121. King WJ, Paice N, Rangrej J, et al. The effect 

of computerized physician order entry 

on medication errors and adverse 

drug events in pediatric inpatients. 

Pediatrics 2003;112(3 Pt 1):506-9. 

[Rec#: 991] 

 122. Kovner C, Schuchman L, Mallard C. The 

application of pen-based computer 

technology to home health care. 

Comput Nurs 1997;15(5):237-44. 

[Rec#: 803] 

 123. Krall MA. Acceptance and performance by 

clinicians using an ambulatory 

electronic medical record in an HMO. 

Proc Annu Symp Comput Appl Med 

Care 1995;708-11. 

[Rec#: 804] 
F-9

 124. Kramer TL, Owen RR, Cannon D, et al. How 

well do automated performance 

measures assess guideline 

implementation for new-onset 

depression in the Veterans Health 

Administration? Jt Comm J Qual Saf 

2003;29(9):479-89. 

[Rec#: 168] 

 125. Krishna S, Balas EA, Spencer DC, et al. 

Clinical trials of interactive 

computerized patient education: 

implications for family practice. J 

Fam Pract 1997;45(1):25-33. 

[Rec#: 89] 

 126. Kucher N, Koo S, Quiroz R, et al. Electronic 

alerts to prevent venous 

thromboembolism among 

hospitalized patients. N Engl J Med 

2005;352(10):969-77. 

[Rec#: 995] 

 127. Kuperman GJ , Teich JM, Bates DW, et al. 

Detecting alerts, notifying the 

physician, and offering action items: 

a comprehensive alerting system. 

Proc AMIA Annu Fall Symp 

1996;704-708. 

[Rec#: 25] 

 128. Kuperman GJ, Teich JM, Tanasijevic MJ, et 

al. Improving response to critical 

laboratory results with automation: 

results of a randomized controlled 

trial. J Am Med Inform Assoc 

1999;6(6):512-22. 

[Rec#: 92] 

 129. Larrabee JH, Boldreghini S, Elder-Sorrells K, 

et al. Evaluation of documentation 

before and after implementation of a 

nursing information system in an 

acute care hospital. Comput Nurs 

2001;19(2):56-65; quiz 66-8. 

[Rec#: 670] 

 130. Larsen RA, Evans RS, Burke JP, et al. 

Improved perioperative antibiotic use 

and reduced surgical wound 

infections through use of computer 

decision analysis. Infect Control 

Hosp Epidemiol 1989;10(7):316-20. 

[Rec#: 671] 

 131. Lewis G, Sharp D, Bartholomew J, et al. 

Computerized assessment of common 

mental disorders in primary care: 

effect on clinical outcome. Fam Pract 

1996;13(2):120-6. 

[Rec#: 891] 

 132. Liaw T, Lawrence M, Rendell J . The effect of 

a computer-generated patient-held 

medical record summary and/or a 

written personal health record on 

patients' attitudes, knowledge and 

behaviour concerning health 

promotion. Fam Pract 

1996;13(3):289-93. 

[Rec#: 893] 

 133. Lisboa PJ. A review of evidence of health 

benefit from artificial neural networks 

in medical intervention. Neural Netw 

2002;15(1):11-39. 

[Rec#: 287] 

 134. Litzelman DK, Dittus RS, Miller ME, et al. 

Requiring physicians to respond to 

computerized reminders improves 

their compliance with preventive care 

protocols. J Gen Intern Med 

1993;8(6):311-7. 

[Rec#: 152] 

 135. Lobach DF. Electronically distributed, 

computer-generated, individualized 

feedback enhances the use of a 

computerized practice guideline. Proc 

AMIA Annu Fall Symp 1996;493-7. 

[Rec#: 894] 

 136. Lobach DF, Hammond WE. Computerized 

decision support based on a clinical 

practice guideline improves 

compliance with care standards. Am J 

Med 1997;102(1):89-98. 

[Rec#: 709] 

 137. Lobach DF, Hammond WE. Development and 

evaluation of a Computer-Assisted 

Management Protocol (CAMP): 

improved compliance with care 

guidelines for diabetes mellitus. Proc 

Annu Symp Comput Appl Med Care 

1994;787-91. 

[Rec#: 895] 
F-10

 138. Margolis CZ, Warshawsky SS, Goldman L, et 

al. Computerized algorithms and 

pediatricians' management of 

common problems in a community 

clinic. Acad Med 1992;67(4):282-4. 

[Rec#: 710] 

 139. Marr PB, Duthie E, Glassman KS, et al. 

Bedside terminals and quality of 

nursing documentation. Comput Nurs 

1993;11(4):176-82. 

[Rec#: 673] 

 140. Mc Donald CJ. Use of a computer to detect 

and respond to clinical events: its 

effect on clinician behavior. Ann 

Intern Med 1976;84(2):162-7. 

[Rec#: 726] 

 141. McAlister NH, Covvey HD, Tong C, et al. 

Randomised controlled trial of 

computer assisted management of 

hypertension in primary care. Br Med 

J (Clin Res Ed) 1986;293(6548):670-

4. 

[Rec#: 674] 

 142. McCartney P, Macdowall W, Thorogood M. A 

randomised controlled trial of 

feedback to general practitioners of 

their prophylactic aspirin prescribing. 

BMJ 1997;315(7099):35-6. 

[Rec#: 737] 

 143. McCowan C, Neville RG, Ricketts IW, et al. 

Lessons from a randomized 

controlled trial designed to evaluate 

computer decision support software 

to improve the management of 

asthma. Med Inform Internet Med 

2001;26(3):191-201. 

[Rec#: 731] 

 144. McDonald CJ. Protocol-based computer 

reminders, the quality of care and the 

non-perfectability of man. N Engl J 

Med 1976;295(24):1351-5. 

[Rec#: 96] 

 145. McDonald CJ, Hui SL, Smith DM, et al. 

Reminders to physicians from an 

introspective computer medical 

record. A two-year randomized trial. 

Ann Intern Med 1984;100(1):130-8. 

[Rec#: 141] 

 146. McDonald CJ, Hui SL, Tierney WM. Effects 

of computer reminders for influenza 

vaccination on morbidity during 

influenza epidemics. MD Comput 

1992;9(5):304-12. 

[Rec#: 142] 

 147. McDonald CJ, Wilson GA, McCabe GP= Jr. 

Physician response to computer 

reminders. JAMA 

1980;244(14):1579-81. 

[Rec#: 727] 

 148. McDowell I, Newell C, Rosser W. 

Comparison of three methods of 

recalling patients for influenza 

vaccination. CMAJ 1986;135(9):991-

7. 

[Rec#: 147] 

 149. McDowell I, Newell C, Rosser W. 

Computerized reminders to encourage 

cervical screening in family practice. 

J Fam Pract 1989;28(4):420-4. 

[Rec#: 145] 

 150. McDowell I, Newell C, Rosser W. A followup study of patients advised to obtain 

influenza immunizations. Fam Med 

1990;22(4):303-6. 

[Rec#: 897] 

 151. McDowell I, Newell C, Rosser W. A 

randomized trial of computerized 

reminders for blood pressure 

screening in primary care. Med Care 

1989;27(3):297-305. 

[Rec#: 144] 

 152. McMichael J, Lieberman R, Doyle H, et al. An 

intelligent and cost-effective 

computer dosing system for 

individualizing FK506 therapy in 

transplantation and autoimmune 

disorders. J Clin Pharmacol 

1993;33(7):599-605. 

[Rec#: 677] 

 153. McNeely MD. The use of expert systems for 

improving test use and enhancing the 

accuracy of diagnosis. Clin Lab Med 

2002;22(2):515-28. 

[Rec#: 304] 
F-11

 154. McPhee SJ, Bird JA, Fordham D, et al. 

Promoting cancer prevention 

activities by primary care physicians. 

Results of a randomized, controlled 

trial. JAMA 1991;266(4):538-44. 

[Rec#: 158] 

 155. Mekhjian HS, Kumar RR, Kuehn L, et al. 

Immediate benefits realized following 

implementation of physician order 

entry at an academic medical center . 

J Am Med Inform Assoc 

2002;9(5):529-39. 

[Rec#: 26] 

 156. Mitchell E, Sullivan F. A descriptive feast but 

an evaluative famine: systematic 

review of published articles on 

primary care computing during 1980-

97. Bmj 2001;322(7281):279-82. 

[Rec#: 312] 

 157. Montani S, Bellazzi R, Quaglini S, et al. Metaanalysis of the effect of the use of 

computer-based systems on the 

metabolic control of patients with 

diabetes mellitus. Diabetes Technol 

Ther 2001;3(3):347-56. 

[Rec#: 315] 

 158. Montgomery AA, Fahey T. A systematic 

review of the use of computers in the 

management of hypertension. J 

Epidemiol Community Health 

1998;52(8):520-5. 

[Rec#: 316] 

 159. Montgomery AA, Fahey T, Peters TJ, et al. 

Evaluation of computer based clinical 

decision support system and risk 

chart for management of hypertension 

in primary care: randomised 

controlled trial. BMJ 

2000;320(7236):686-90. 

[Rec#: 730] 

 160. Moran WP, Nelson K, Wofford JL, et al. 

Computer-generated mailed 

reminders for influenza 

immunization: a clinical trial. J Gen 

Intern Med 1992;7(5):535-7. 

[Rec#: 157] 

 161. Mullett CJ, Evans RS, Christenson JC, et al. 

Development and impact of a 

computerized pediatric antiinfective 

decision support program. Pediatrics 

2001;108(4):E75. 

[Rec#: 27] 

 162. Mungall DR, Anbe D, Forrester PL, et al. A 

prospective randomized comparison 

of the accuracy of computer-assisted 

versus GUSTO nomogram--directed 

heparin therapy. Clin Pharmacol Ther 

1994;55(5):591-6. 

[Rec#: 678] 

 163. Murchie CJ, Kenny GN. Comparison among 

manual, computer-assisted, and 

closed-loop control of blood pressure 

after cardiac surgery. J Cardiothorac 

Anesth 1989;3(1):16-9. 

[Rec#: 679] 

 164. Murff HJ, Kannry J. Physician satisfaction 

with two order entry systems. J Am 

Med Inform Assoc 2001;8(5):499-

509. 

[Rec#: 28] 

 165. Nahm R, Poston I. Measurement of the effects 

of an integrated, point-of-care 

computer system on quality of 

nursing documentation and patient 

satisfaction. Comput Nurs 

2000;18(5):220-9. 

[Rec#: 680] 

 166. Nightingale PG, Adu D, Richards NT, et al. 

Implementation of rules based 

computerised bedside prescribing and 

administration: intervention study. 

BMJ 2000;320(7237):750-3. 

[Rec#: 324] 

 167. Nilasena DS, Lincoln MJ. A computergenerated reminder system improves 

physician compliance with diabetes 

preventive care guidelines. Proc 

Annu Symp Comput Appl Med Care 

1995;640-5. 

[Rec#: 712] 
F-12

 168. Nilasena DS, Lincoln MJ, Turner CW, et al. 

Development and implementation of 

a computer-generated reminder 

system for diabetes preventive care. 

Proc Annu Symp Comput Appl Med 

Care 1994;831-5. 

[Rec#: 711] 

 169. Oren E, Shaffer ER, Guglielmo BJ. Impact of 

emerging technologies on medication 

errors and adverse drug events. Am J 

Health Syst Pharm 

2003;60(14):1447-58. 

[Rec#: 329] 

 170. Ornstein SM, Garr DR, Jenkins RG, et al. 

Implementation and evaluation of a 

computer-based preventive services 

system. Fam Med 1995;27(4):260-6. 

[Rec#: 714] 

 171. Ornstein SM, Garr DR, Jenkins RG, et al. 

Computer-generated physician and 

patient reminders. Tools to improve 

population adherence to selected 

preventive services. J Fam Pract 

1991;32(1):82-90. 

[Rec#: 151] 

 172. Ornstein SM, MacFarlane LL, Jenkins RG, et 

al. Medication cost information in a 

computer-based patient record 

system. Impact on prescribing in a 

family medicine clinical practice. 

Arch Fam Med 1999;8(2):118-21. 

[Rec#: 569] 

 173. Overhage JM, Perkins S, Tierney WM, et al. 

Controlled trial of direct physician 

order entry: effects on physicians' 

time utilization in ambulatory 

primary care internal medicine 

practices. J Am Med Inform Assoc 

2001; 8(4):361-71. 

[Rec#: 30] 

 174. Overhage JM, Suico J, McDonald CJ. 

Electronic laboratory reporting: 

barriers, solutions and findings. J 

Public Health Manag Pract 2001;7(6): 

60-6. 

[Rec#: 103] 

 175. Overhage JM, Tierney WM, McDonald CJ. 

Computer reminders to implement 

preventive care guidelines for 

hospitalized patients. Arch Intern 

Med 1996;156(14):1551-6. 

[Rec#: 31] 

 176. Overhage JM, Tierney WM, Zhou XH, et al. A 

randomized trial of "corollary orders" 

to prevent errors of omission. J Am 

Med Inform Assoc 1997;4(5):364-75. 

[Rec#: 32] 

 177. Pacht ER, Turner JW, Gailiun M, et al. 

Effectiveness of telemedicine in the 

outpatient pulmonary clinic. Telemed 

J 1998;4(4):287-92. 

[Rec#: 104] 

 178. Perlini S, Piepoli M, Marti G, et al. Treatment 

of chronic heart failure: an expert 

system advisor for general 

practitioners. Acta Cardiol 

1990;45(5):365-78. 

[Rec#: 752] 

 179. Persson M, Mjorndal T, Carlberg B, et al. 

Evaluation of a computer-based 

decision support system for treatment 

of hypertension with drugs: 

retrospective, nonintervention testing 

of cost and guideline adherence. J 

Intern Med 2000;247(1):87-93. 

[Rec#: 337] 

 180. Peterson CM, Jovanovic L, Chanoch LH. 

Randomized trial of computerassisted insulin delivery in patients 

with type I diabetes beginning pump 

therapy. Am J Med 1986;81(1):69-72. 

[Rec#: 683] 

 181. Pierpont GL, Thilgen D. Effect of 

computerized charting on nursing 

activity in intensive care. Crit Care 

Med 1995;23(6):1067-73. 

[Rec#: 832] 

 182. Pizziferri L, Kittler AF, Volk LA, et al. 

Primary care physician time 

utilization before and after 

implementation of an electronic 

health record: a time-motion study. J 

Biomed Inform 2005;38(3):176-88. 

[Rec#: 2200] 
F-13

 183. Poller L, Wright D, Rowlands M. Prospective 

comparative study of computer 

programs used for management of 

warfarin. J Clin Pathol 

1993;46(4):299-303. 

[Rec#: 684] 

 184. Potts AL, Barr FE, Gregory DF, et al. 

Computerized Physician Order Entry 

and Medication Errors in a Pediatric 

Critical Care Unit. Pediatrics 

2004;113(1 ):59-63. 

[Rec#: 163] 

 185. Pringle M, Robins S, Brown G. Computer 

assisted screening: effect on the 

patient and his consultation. Br Med J 

(Clin Res Ed) 1985;290(6483):1709-

12. 

[Rec#: 685] 

 186. Pringle M, Robins S, Brown G. Timer: a new 

objective measure of consultation 

content and its application to 

computer assisted consultations. Br 

Med J (Clin Res Ed) 

1986;293(6538):20-2. 

[Rec#: 686] 

 187. Quattlebaum TG, Darden PM, Sperry JB. 

Effectiveness of computer-generated 

appointment reminders. Pediatrics 

1991;88(4):801-5. 

[Rec#: 903] 

 188. Ramaswamy MR, Chaljub G, Esch O, et al. 

Continuous speech recognition in MR 

imaging reporting: advantages, 

disadvantages, and impact. AJR Am J 

Roentgenol 2000;174(3):617-22. 

[Rec#: 576] 

 189. Raschke RA, Gollihare B, Wunderlich TA, et 

al. A computer alert system to 

prevent injury from adverse drug 

events: development and evaluation 

in a community teaching hospital. 

JAMA 1998;280(15):1317-20. 

[Rec#: 864] 

 190. Raynor DK, Booth TG, Blenkinsopp A. 

Effects of computer generated 

reminder charts on patients' 

compliance with drug regimens. BMJ 

1993;306(6886):1158-61. 

[Rec#: 742] 

 191. Rethans JJ, Hoppener P, Wolfs G, et al. Do 

personal computers make doctors less 

personal? Br Med J (Clin Res Ed) 

1988;296(6634):1446-8. 

[Rec#: 688] 

 192. Revere D, Dunbar PJ. Review of computergenerated outpatient health behavior 

interventions: clinical encounters "in 

absentia". J Am Med Inform Assoc 

2001;8(1):62-79. 

[Rec#: 346] 

 193. Rind DM, Safran C, Phillips RS, et al. Effect 

of computer-based alerts on the 

treatment and outcomes of 

hospitalized patients. Arch Intern 

Med 1994;154(13 ):1511-7. 

[Rec#: 865] 

 194. Robbins JA, Dickinson WA, Bartel AG, et al. 

Lipid management program: results 

of applying national guidelines in a 

private practice. South Med J 

1993;86(3):289-92. 

[Rec#: 715] 

 195. Robson J, Boomla K, Fitzpatrick S, et al. 

Using nurses for preventive activities 

with computer assisted follow up: a 

randomised controlled trial. BMJ 

1989;298(6671):433-6. 

[Rec#: 905] 

 196. Rodman JH, Jelliffe RW, Kolb E, et al. 

Clinical studies with computerassisted initial lidocaine therapy. 

Arch Intern Med 1984;144(4):703-9. 

[Rec#: 689] 

 197. Rogers MA, Small D, Buchan DA, et al. Home 

monitoring service improves mean 

arterial pressure in patients with 

essential hypertension. A 

randomized, controlled trial. Ann 

Intern Med 2001;134(11):1024-32. 

[Rec#: 109] 

 198. Rollman BL, Hanusa BH, Gilbert T, et al. The 

electronic medical record. A 

randomized trial of its impact on 

primary care physicians' initial 

management of major depression 

[corrected]. Arch Intern Med 

2001;161(2):189-97. 

[Rec#: 352] 
F-14

 199. Rollman BL, Hanusa BH, Lowe HJ, et al. A 

randomized trial using computerized 

decision support to improve treatment 

of major depression in primary care. J 

Gen Intern Med 2002;17(7):493-503. 

[Rec#: 110] 

 200. Rosado B, Menzies S, Harbauer A, et al. 

Accuracy of computer diagnosis of 

melanoma: a quantitative metaanalysis. Arch Dermatol 2003; 

139(3):361-7; discussion 366. 

[Rec#: 355] 

 201. Rosenfeld BA, Dorman T, Breslow MJ, et al. 

Intensive care unit telemedicine: 

alternate paradigm for providing 

continuous intensivist care. Crit Care 

Med 2000;28(12):3925-31. 

[Rec#: 111] 

 202. Rosenstein AH. Measuring the benefit of 

performance improvement and 

decision support. Am J Med Qual 

1999;14(6):262-9. 

[Rec#: 357] 

 203. Rosenstein AH. Measuring the benefits of 

clinical decision support: return on 

investment. Pharm Pract Manag Q 

2000;20(1):28-43. 

[Rec#: 358] 

 204. Rosser WW, Hutchison BG, McDowell I, et 

al. Use of reminders to increase 

compliance with tetanus booster 

vaccination. CMAJ 1992;146(6):911-

7. 

[Rec#: 146] 

 205. Rossi RA, Every NR. A computerized 

intervention to decrease the use of 

calcium channel blockers in 

hypertension. J Gen Intern Med 1997; 

12(11):672-8. 

[Rec#: 716] 

 206. Ruiz R, Borches D, Gonzalez A, et al. A new 

sodium-nitroprusside-infusion 

controller for the regulation of arterial 

blood pressure. Biomed Instrum 

Technol 1993;27(3):244-51. 

[Rec#: 691] 

 207. Ryff-de Leche A, Engler H, Nutzi E, et al. 

Clinical application of two 

computerized diabetes management 

systems: comparison with the logbook method. Diabetes Res 

1992;19(3):97-105. 

[Rec#: 692] 

 208. Safran C, Rind DM, Davis RB, et al. 

Guidelines for management of HIV 

infection with computer-based 

patient's record. Lancet 

1995;346(8971):341-6. 

[Rec#: 717] 

 209. Sanders DL, Miller RA. The effects on 

clinician ordering patterns of a 

computerized decision support 

system for neuroradiology imaging 

studies. Proc AMIA Symp 2001;583-

7. 

[Rec#: 112] 

 210. Schriger DL, Baraff LJ, Buller K, et al. 

Implementation of clinical guidelines 

via a computer charting system: effect 

on the care of febrile children less 

than three years of age. J Am Med 

Inform Assoc 2000;7(2):186-95. 

[Rec#: 596] 

 211. Schriger DL, Baraff LJ, Rogers WH, et al. 

Implementation of clinical guidelines 

using a computer charting system. 

Effect on the initial care of health 

care workers exposed to body fluids. 

Jama 1997;278(19):1585-90. 

[Rec#: 368] 

 212. Selmi PM, Klein MH, Greist JH, et al. 

Computer-administered cognitivebehavioral therapy for depression. 

Am J Psychiatry 1990;147(1):51-6. 

[Rec#: 753] 

 213. Shea S, DuMouchel W, Bahamonde L. A 

meta-analysis of 16 randomized 

controlled trials to evaluate computerbased clinical reminder systems for 

preventive care in the ambulatory 

setting. J Am Med Inform Assoc 

1996;3(6): 399-409. 

[Rec#: 118] 
F-15

 214. Shiffman RN, Liaw Y, Brandt CA, et al. 

Computer-based guideline 

implementation systems: a systematic 

review of functionality and 

effectiveness. J Am Med Inform 

Assoc 1999;6(2):104-14. 

[Rec#: 375] 

 215. Shojania KG, Yokoe D, Platt R, et al. 

Reducing vancomycin use utilizing a 

computer guideline: results of a 

randomized controlled trial. J Am 

Med Inform Assoc 1998;5(6):554-62. 

[Rec#: 37] 

 216. Simkins CV, Wenzloff NJ. Evaluation of a 

computerized reminder system in the 

enhancement of patient medication 

refill compliance. Drug Intell Clin 

Pharm 1986;20(10):799-802. 

[Rec#: 741] 

 217. Simon GE, VonKorff M, Rutter C, et al. 

Randomised trial of monitoring, 

feedback, and management of care by 

telephone to improve treatment of 

depression in primary care. BMJ 

2000;320(7234):550-4. 

[Rec#: 738] 

 218. Singh AK, Moidu K, Trell E, et al. Impact on 

the management and delivery of 

primary health care by a computerbased information system. Comput 

Methods Programs Biomed 

1992;37(1):55-64. 

[Rec#: 909] 

 219. Steele MA, Bess DT, Franse VL, et al. Cost 

effectiveness of two interventions for 

reducing outpatient prescribing costs. 

DICP 1989; 23(6):497-500. 

[Rec#: 735] 

 220. Strecher VJ, Kreuter M, Den Boer DJ, et al. 

The effects of computer-tailored 

smoking cessation messages in family 

practice settings. J Fam Pract 

1994;39(3):262-70. 

[Rec#: 911] 

 221. Sullivan F, Mitchell E. Has general 

practitioner computing made a 

difference to patient care? A 

systematic review of published 

reports. Bmj 1995;311(7009):848-52. 

[Rec#: 384] 

 222. Sun J, Chang MW. Initialization of warfarin 

dosages using computer modeling. 

Arch Phys Med Rehabil 1995;76 

(5):453-6. 

[Rec#: 385] 

 223. Szilagyi PG, Rodewald LE, Savageau J, et al. 

Improving influenza vaccination rates 

in children with asthma: a test of a 

computerized reminder system and an 

analysis of factors predicting 

vaccination compliance. Pediatrics 

1992;90(6):871-5. 

[Rec#: 155] 

 224. Tape TG, Campbell JR. Computerized medical 

records and preventive health care: 

success depends on many factors. Am 

J Med 1993;94(6):619-25. 

[Rec#: 718] 

 225. Teich JM, Merchia PR, Schmiz JL, et al. 

Effects of computerized physician 

order entry on prescribing practices. 

Arch Intern Med 2000;160(18):2741-

2747. 

[Rec#: 39] 

 226. Theil DR, Stanley TE= 3rd, White WD, et al. 

Midazolam and fentanyl continuous 

infusion anesthesia for cardiac 

surgery: a comparison of computerassisted versus manual infusion 

systems. J Cardiothorac Vasc Anesth 

1993; 7(3):300-6. 

[Rec#: 696] 

 227. Thiru K, Hassey A, Sullivan F . Systematic 

review of scope and quality of 

electronic patient record data in 

primary care. Bmj 

2003;326(7398):1070. 

[Rec#: 392] 

 228. Threet E, Fargues MP. Economic evaluation of 

voice recognition for the clinicians' 

desktop at the Naval Hospital 

Roosevelt Roads. Mil Med 

1999;164(2):119-26. 

[Rec#: 615] 
F-16

 229. Tierney WM, Hui SL, McDonald CJ. Delayed 

feedback of physician performance 

versus immediate reminders to 

perform preventive care. Effects on 

physician compliance. Med Care 

1986;24(8):659-66. 

[Rec#: 43] 

 230. Tierney WM, McDonald CJ, Hui SL, et al. 

Computer predictions of abnormal 

test results. Effects on outpatient 

testing. JAMA 1988;259(8):1194-8. 

[Rec#: 44] 

 231. Tierney WM, McDonald CJ, Martin DK, et al. 

Computerized display of past test 

results. Effect on outpatient testing. 

Ann Intern Med 1987; 107(4):569-74. 

[Rec#: 45] 

 232. Tierney WM, Miller ME, McDonald CJ. The 

effect on test ordering of informing 

physicians of the charges for 

outpatient diagnostic tests. N Engl J 

Med 1990 ;322(21):1499-504. 

[Rec#: 46] 

 233. Tierney WM, Miller ME, Overhage JM, et al. 

Physician inpatient order writing on 

microcomputer workstations. Effects 

on resource utilization. JAMA 

1993;269(3 ):379-83. 

[Rec#: 47] 

 234. Timmermans D, van Bockel H, Kievit J. 

Improving the quality of surgeons' 

treatment decisions: a comparison of 

clinical decision making with a 

computerised evidence based decision 

analytical model. Qual Health Care 

2001;10(1):4-9. 

[Rec#: 395] 

 235. Turner BJ, Day SC, Borenstein B. A 

controlled trial to improve delivery of 

preventive care: physician or patient 

reminders? J Gen Intern Med 

1989;4(5):403-9. 

[Rec#: 143] 

 236. Turner RC, Peden JG= Jr, O'Brien K. Patientcarried card prompts vs computergenerated prompts to remind private 

practice physicians to perform health 

maintenance measures. Arch Intern 

Med 1994;154(17):1957-60. 

[Rec#: 719] 

 237. Valancy J. How much will that EMR system 

really cost? Fam Pract Manag 

2002;9(4):57-8. 

[Rec#: 619] 

 238. van den Hoogen JP, van Ree JW . Preventive 

cardiology in general practice: 

computer-assisted hypertension care. 

J Hum Hypertens 1990;4(4):365-7. 

[Rec#: 744] 

 239. van den Nieuwenhuyzen MC, Engbers FH, 

Burm AG, et al. Computer-controlled 

infusion of alfentanil versus patientcontrolled administration of morphine 

for postoperative analgesia: a doubleblind randomized trial. Anesth Analg 

1995;81(4):671-9. 

[Rec#: 698] 

 240. van Wijk MA, van der Lei J, Mosseveld M, et 

al. Assessment of decision support for 

blood test ordering in primary care. a 

randomized trial. Ann Intern Med 

2001;134(4):274-81. 

[Rec#: 997] 

 241. Verner D, Seligmann H, Platt S, et al. 

Computer assisted design of a 

theophylline dosing regimen in acute 

bronchospasm: serum concentrations 

and clinical outcome. Eur J Clin 

Pharmacol 1992;43(1):29-33. 

[Rec#: 699] 

 242. Vincent EC, Hardin PA, Norman LA, et al. 

The effects of a computer-assisted 

reminder system on patient 

compliance with recommended health 

maintenance procedures. Proc Annu 

Symp Comput Appl Med Care 

1995;656-60. 

[Rec#: 720] 

 243. Walker J, Pan E, Johnston D, et al. The value 

of health care information exchange 

and interoperability. Health Affairs 

2005;w5.10. 

[Rec#: 987] 
F-17

 244. Walton R, Dovey S, Harvey E, et al. Computer 

support for determining drug dose: 

systematic review and meta-analysis. 

Bmj 1999;318(7189):984-90. 

[Rec#: 401] 

 245. Wang SJ, Middleton B, Prosser LA, et al. A 

cost-benefit analysis of electronic 

medical records in primary care. Am 

J Med 2003;114:397-403. 

[Rec#: 171] 

 246. Warshawsky SS, Pliskin JS, Urkin J, et al. 

Physician use of a computerized 

medical record system during the 

patient encounter: a descriptive study. 

Comput Methods Programs Biomed 

1994;43(3-4):269-73. 

[Rec#: 915] 

 247. Weingarten MA, Bazel D, Shannon HS. 

Computerized protocol for preventive 

medicine: a controlled self-audit in 

family practice. Fam Pract 

1989;6(2):120-4. 

[Rec#: 701] 

 248. Wells BJ, Lobel KD, Dickerson LM. Using the 

electronic medical record to enhance 

the use of combination drugs. Am J 

Med Qual 2003;18(4):147-9. 

[Rec#: 628] 

 249. White KS, Lindsay A, Pryor TA, et al. 

Application of a computerized 

medical decision-making process to 

the problem of digoxin intoxication. J 

Am Coll Cardiol 1984;4(3):571-6. 

[Rec#: 702] 

 250. White RH, Hong R, Venook AP, et al. 

Initiation of warfarin therapy: 

comparison of physician dosing with 

computer-assisted dosing. J Gen 

Intern Med 1987;2(3):141-8. 

[Rec#: 703] 

 251. White RH, Mungall D. Outpatient 

management of warfarin therapy: 

comparison of computer-predicted 

dosage adjustment to skilled 

professional care. Ther Drug Monit 

1991;13(1):46-50. 

[Rec#: 704] 

 252. Willcourt RJ, Pager D, Wendel J, et al. 

Induction of labor with pulsatile 

oxytocin by a computer-controlled 

pump. Am J Obstet Gynecol 

1994;170(2):603-8. 

[Rec#: 705] 

 253. Willson D, Ashton C, Wingate N, et al. 

Computerized support of pressure 

ulcer prevention and treatment 

protocols. Proc Annu Symp Comput 

Appl Med Care 1995;646-50. 

[Rec#: 721] 

 254. Wilson AL, Hill JJ, Wilson RG, et al. 

Computerized medication 

administration records decrease 

medication occurrences. Pharm Pract 

Manag Q 1997;17(1):17-29. 

[Rec#: 860] 

 255. Wilson GA, McDonald CJ, McCabe GP= Jr. 

The effect of immediate access to a 

computerized medical record on 

physician test ordering: a controlled 

clinical trial in the emergency room. 

Am J Public Health 1982;72(7):698-

702. 

[Rec#: 706] 

 256. Wong DH, Gallegos Y, Weinger MB, et al. 

Changes in intensive care unit nurse 

task activity after installation of a 

third-generation intensive care unit 

information system. Crit Care Med 

2003;31 (10):2488-94. 

[Rec#: 53] 
G-1

Appendix G. List of Excluded Articles 

 1. Beyond guidelines: tool arms physicians with 

critical knowledge at the point of 

care. Dis Manag Advis 2002;8(1):9-

12, 1. 

[Rec#: 173] 

 2. Capitated groups reaping benefits of EMR 

systems. Capitation Manag Rep 

2003;10(6):81-4. 

[Rec#: 426] 

 3. CareGroup site takes some of the 

administrative hassle out of 

healthcare. Internet Healthc Strateg 

2001;3(7):4-6. 

[Rec#: 421] 

 4. Computerization saves money, time, and trees. 

Healthc Benchmarks 2000;7(1):1-6. 

[Rec#: 418] 

 5. Computerized system alerts docs to costs. ED 

Manag 1999;11(9):100-2. 

[Rec#: 417] 

 6. Conversion to computerized patient records 

streamlines ER process, saves money. 

Health Care Cost Reengineering Rep 

1997;2(3):37-40. 

[Rec#: 754] 

 7. Electronic system places patients in post-acute 

care. Hosp Case Manag 2003;11 

(6):85-6. 

[Rec#: 427] 

 8. Evidence-based medicine system helps 

physicians order the right tests. 

Perform Improv Advis 2003; 7(3):40-

4. 

[Rec#: 174] 

 9. Innovative CareEngine system roots out suboptimal care. Dis Manag Advis 

2003;9(1):9-13, 1. 

[Rec#: 175] 

 10. Innovative charting can slash costs, bolster 

efficiency, enhance quality. ED 

Manag 1995;7(11):121-3. 

[Rec#: 756] 

 11. Medication safety issue brief. Using 

automation to reduce errors. Part 2. 

Hosp Health Netw 2001; 75(2):33-4. 

[Rec#: 176] 

 12. New data storage system saves $20,000 per 

year. Healthc Benchmarks 

2001;8(12):139-40, 142. 

[Rec#: 423] 

 13. Paperless records will improve quality. Hosp 

Peer Rev 2002;27(7):99-100. 

[Rec#: 424] 

 14. Physicians go online for electronic medical 

records. Data Strateg Benchmarks 

2000;4(12):183-5, 177. 

[Rec#: 419] 

 15. Practice brief. Health informatics standards 

and information transfer: exploring 

the HIM role. American Health 

Information Management 

Association. J AHIMA 

2001;72(1):68A-68D. 

[Rec#: 420] 

 16. What's happening in information systems 

technology that is reshaping health 

care delivery, and how might it assist 

in infection prevention and control? 

Am J Infect Control 1996;24(6):476-

80. 

[Rec#: 178] 

 17. Adderley D, Hyde C, Mauseth P. The 

computer age impacts nurses. 

Comput Nurs 1997;15(1):43-6. 

[Rec#: 758] 

 18. Ahearn MD, Kerr SJ. General practitioners' 

perceptions of the pharmaceutical 

decision-support tools in their 

prescribing software. Med J Aust 

2003;179(1):34-7. 

[Rec#: 179] 

 19. Al-Ubaydli M. Handheld computers. BMJ 

2004;328(7449):1181-4. 

[Rec#: 930] 
G-2

 20. Allaert FA. The use of personal health 

information for controlling the costs 

of delivering health care: does the end 

justify the means? Int J Biomed 

Comput 1996;43(1-2):79-82. 

[Rec#: 759] 

 21. Alper BS, Stevermer JJ, White DS, et al. 

Answering family physicians' clinical 

questions using electronic medical 

databases. J Fam Pract 

2001;50(11):960-5. 

[Rec#: 180] 

 22. Amatayakul M. Making the case for electronic 

records. Health Data Manag 

1997;5(5):56-7, 59, 61-3. 

[Rec#: 760] 

 23. Ammon D. What works: Ready, set, grow. A 

local practice implements electronic 

medical record (EMR) technology, 

triples patient volume and reduces per 

patient costs by more than 10 percent. 

Health Manag Technol 

2002;23(12):42-3. 

[Rec#: 434] 

 24. Anderson B. Document imaging and 

workflow technology saves $1.2 

million annually. Health Manag 

Technol 1999;20 (11):18-20, 22. 

[Rec#: 435] 

 25. Anderson G, Knickman JR. Changing the 

chronic care system to meet people's 

needs. Health Aff (Millwood) 

2001;20 (6):146-60. 

[Rec#: 965] 

 26. Anderson HJ, Bunschoten B. Creating 

electronic records. A progress report. 

Health Data Manag 1996;4(9):36-8, 

41-2, 44. 

[Rec#: 761] 

 27. Anderson JG. Security of the distributed 

electronic patient record: a case-based 

approach to identifying policy issues. 

Int J Med Inf 2000;60(2):111-8. 

[Rec#: 436] 

 28. Antoine W. Electronic miracle. Memorial 

Hermann put its patient files online 

and watch its revenue grow. Health 

Forum J 2002;45(4):34-5. 

[Rec#: 437] 

 29. Arikian SR, Mario GG, Doyle JJ . Building a 

computerized disease management 

operating system. Oncology 

(Huntingt) 1996;10(11 Suppl):223-

32. 

[Rec#: 183] 

 30. Asher JR. Time and money. A look at message 

automation via a computerized 

patient record. MGMA Connex 

2003;3(9):28-9. 

[Rec#: 438] 

 31. Atkinson JC, Zeller GG, Shah C . Electronic 

patient records for dental school 

clinics: more than paperless systems. 

J Dent Educ 2002;66(5):634-42. 

[Rec#: 439] 

 32. Aubert BA, Hamel G. Adoption of smart cards 

in the medical sector: the Canadian 

experience. Soc Sci Med 

2001;53(7):879-94. 

[Rec#: 55] 

 33. Austin T, Iliffe S, Leaning M, et al. A 

prototype computer decision support 

system for the management of 

asthma. J Med Syst 1996;20(1):45-

55. 

[Rec#: 750] 

 34. Bakken S. An informatics infrastructure is 

essential for evidence-based practice. 

J Am Med Inform Assoc 

2001;8(3):199-201. 

[Rec#: 184] 

 35. Bakker A, Hammond E, Ball M. Summary 

report of observations, conclusions 

and recommendations. Int J Biomed 

Comput 1995;39(1):11-5. 

[Rec#: 763] 

 36. Balas EA, Stockham MG, Mitchell MA, et al. 

The Columbia Registry of 

Information and Utilization 

Management Trials. J Am Med 

Inform Assoc 1995;2(5):307-15. 

[Rec#: 186] 
G-3

 37. Baldwin G. CIO secrets to calculating return 

on investment. Health Data Manag 

2000;8(7):54-9, 62-4, 66-7. 

[Rec#: 440] 

 38. Ball MJ, Weaver C, Abbott PA. Enabling 

technologies promise to revitalize the 

role of nursing in an era of patient 

safety. Int J Med Inf 2003;69(1):29-

38. 

[Rec#: 441] 

 39. Baltzer DJ. A realistic take on tech. Hosp 

Health Netw 2002;76(3):12. 

[Rec#: 442] 

 40. Barnett GO, Winickoff RN, Morgan MM, et 

al. A computer-based monitoring 

system for follow-up of elevated 

blood pressure. Med Care 

1983;21(4):400-9. 

[Rec#: 140] 

 41. Barrett MJ. The evolving computerized 

medical record, Part II. Healthc 

Inform 2000;17(9):83-4, 86, 88. 

[Rec#: 443] 

 42. Barthell EN, Pemble KR. The National 

Emergency Medical Extranet project. 

J Emerg Med 2003;24(1):95-100. 

[Rec#: 189] 

 43. Basch P. Data control. Primary care practice 

improves productivity and 

profitability with electronic medical 

system. Healthc Inform 

1999;16(6):159-62. 

[Rec#: 444] 

 44. Bates DW. Using information systems to 

improve practice. Schweiz Med 

Wochenschr 1999;129(49):1913-9. 

[Rec#: 190] 

 45. Bates DW, Cohen M, Leape LL, et al. 

Reducing the frequency of errors in 

medicine using information 

technology. J Am Med Inform Assoc 

2001;8(4):299-308. 

[Rec#: 60] 

 46. Bates DW, Ebell M, Gotlieb E, et al. A 

proposal for electronic medical 

records in U.S. primary care. J Am 

Med Inform Assoc 2003;10(1):1-10. 

[Rec#: 61] 

 47. Bates DW, Evans RS, Murff H, et al. 

Detecting adverse events using 

information technology. J Am Med 

Inform Assoc 2003;10(2):115-28. 

[Rec#: 191] 

 48. Bates DW, Gawande AA. Improving safety 

with information technology. N Engl 

J Med 2003;348(25):2526-34. 

[Rec#: 62] 

 49. Bates DW, Kuperman GJ, Jha A, et al. Does 

the computerized display of charges 

affect inpatient ancillary test 

utilization? Arch Intern Med 

1997;157(21):2501-2508. 

[Rec#: 3] 

 50. Bates DW, Makary MA, Teich JM, et al. 

Asking residents about adverse events 

in a computer dialogue: how accurate 

are they? Jt Comm J Qual Improv 

1998 ;24(4):197-202. 

[Rec#: 445] 

 51. Bates DW, Spell N, Cullen DJ, et al. The costs 

of adverse drug events in hospitalized 

patients. Adverse Drug Events 

Prevention Study Group. JAMA 

1997;277(4):307-311. 

[Rec#: 5] 

 52. Beard N, Thomas D. Realising your potential. 

Health Serv J 1990;100(5203):816-

817. 

[Rec#: 6] 

 53. Bellazzi R, Riva A, Montani S, et al. A webbased system for diabetes 

management: the technical and 

clinical infrastructure. Proc AMIA 

Symp 1998;972. 

[Rec#: 65] 

 54. Berg M. Patient care information systems and 

health care work: a sociotechnical 

approach. Int J Med Inf 

1999;55(2):87-101. 

[Rec#: 868] 
G-4

 55. Bergeron BP. What price for your patient 

data? Medical record freeware may 

also become shareware. Postgrad 

Med 2000;108(2):33-5. 

[Rec#: 447] 

 56. Berkowitz LL. Breaking down the barriers. 

Improving physician buy-in of CPR 

systems. Healthc Inform 

1997;14(10):73-6. 

[Rec#: 768] 

 57. Bingham A. Computerized patient records 

benefit physician offices. Healthc 

Financ Manage 1997;51(9):68-70. 

[Rec#: 769] 

 58. Birkmeyer CM, Lee J, Bates DW, et al. Will 

electronic order entry reduce health 

care costs? Eff Clin Pract 

2002;5(2):67-74. 

[Rec#: 8] 

 59. Birkmeyer JD, Welch HG. A reader's guide to 

surgical decision analysis. J Am Coll 

Surg 1997;184(6):589-95. 

[Rec#: 194] 

 60. Biskup J, Bleumer G. Cryptographic 

protection of health information: cost 

and benefit. Int J Biomed Comput 

1996;43 (1-2):61-7. 

[Rec#: 770] 

 61. Boger E. Electronic tracking board reduces ED 

patient length of stay at Indiana 

Hospital. J Emerg Nurs 

2003;29(1):39-43. 

[Rec#: 449] 

 62. Brailer DJ. Translating ideals for health 

information technology into practice. 

Health Aff (Millwood) 2004;Suppl 

Web Exclusives:W4-318-20. 

[Rec#: 954] 

 63. Brassey J, Elwyn G, Price C, et al. Just in time 

information for clinicians: a 

questionnaire evaluation of the 

ATTRACT project. Bmj 2001; 

322(7285):529-30. 

[Rec#: 195] 

 64. Breitfeld PP, Weisburd M, Overhage JM, et al. 

Pilot study of a point-of-use decision 

support tool for cancer clinical trials 

eligibility. J Am Med Inform Assoc 

1999;6(6):466-77. 

[Rec#: 66] 

 65. Brender J, Nohr C, McNair P. Research needs 

and priorities in health informatics. 

Int J Med Inf 2000;58-59:257-89. 

[Rec#: 196] 

 66. Briggs B. Doctors sound off on I.T. concerns. 

Health Data Manag 2001;9(1):38-40, 

42-4, 46-8. 

[Rec#: 451] 

 67. Briggs B. Electronic stepping stones. Health 

Data Manag 2002;10(2):36-8. 

[Rec#: 453] 

 68. Briggs B. Gathering I.T. intelligence. Health 

Data Manag 2001;9(4):28-32, 34, 36 

passim. 

[Rec#: 452] 

 69. Briscoe M. Obstacles to the use of computers 

in British mental health services. 

Psychiatr Serv 1997;48(3):329-30. 

[Rec#: 772] 

 70. Broude AL. Build quality by investing in 

technology. An interview with Alan 

L. Broude. Healthc Financ Manage 

2003;57 (2):36-8. 

[Rec#: 454] 

 71. Brown EG. Methods and pitfalls in searching 

drug safety databases utilising the 

Medical Dictionary for Regulatory 

Activities (MedDRA). Drug Saf 

2003;26(3):145-58. 

[Rec#: 198] 

 72. Brown SH. No free lunch: institutional 

preparations for computer-based 

patient records. Proc AMIA Symp 

1999;486-90. 

[Rec#: 456] 

 73. Brundage MD, Groome PA, Feldman-Stewart 

D, et al. Decision analysis in locally 

advanced non-small-cell lung cancer: 

is it useful? J Clin Oncol 

1997;15(3):873-83. 
G-5

[Rec#: 199] 

 74. Brushwood DB. Maximizing the value of 

electronic prescription monitoring 

programs. J Law Med Ethics 

2003;31(1):41-54. 

[Rec#: 457] 

 75. Bundt TS. Transcription cost analysis reveals 

opportunities for savings, 

efficiencies. J AHIMA 

2002;73(8):55-60. 

[Rec#: 458] 

 76. Burkle T, Kuch R, Passian A, et al. The impact 

of computer implementation on 

nursing work patterns: study design 

and preliminary results. Medinfo 

1995;8 Pt 2:1321-5. 

[Rec#: 640] 

 77. Burstin H. Traversing the digital divide. 

Health Aff (Millwood) 

2000;19(6):245-9. 

[Rec#: 967] 

 78. Burstin H, Clancy C. Broader adoption of 

information technology needed . 

Health Aff (Millwood) 

2002;21(6):266-7; author reply 267-8. 

[Rec#: 959] 

 79. Bush J. Open-source software: just what the 

doctor ordered? Fam Pract Manag 

2003;10(6):65, 69. 

[Rec#: 459] 

 80. Butler MA, Bender AD. Intensive care unit 

bedside documentation systems. 

Realizing cost savings and quality 

improvements. Comput Nurs 

1999;17(1):32-8; quiz 39-40. 

[Rec#: 67] 

 81. Byrnes JJ. A revolutionary advance in disease 

management. Healthc Leadersh 

Manag Rep 2001;9(1):14-20. 

[Rec#: 200] 

 82. Cantlon G. A new proposal for cooperation in 

HIT development. Health Aff 

(Millwood) 1999;18(2):257-9. 

[Rec#: 969] 

 83. Carlson RW, Tu SW, Lane NM, et al. 

Computer-based screening of patients 

with HIV/AIDS for clinical-trial 

eligibility. Online J Curr Clin Trials 

1995;Doc No 179:[3347 words; 32 

paragraphs]. 

[Rec#: 68] 

 84. Carroll AE, Saluja S, Tarczy-Hornoch P. 

Development of a Personal Digital 

Assistant (PDA) based client/server 

NICU patient data and charting 

system . Proc AMIA Symp 2001;100-

4. 

[Rec#: 462] 

 85. Cartwright J, de Sylva S, Glasgow M, et al. 

Inaccessible information is useless 

information: addressing the 

knowledge gap. J Med Pract Manage 

2002;18(1):36-41. 

[Rec#: 202] 

 86. Chambliss ML, Rasco T, Clark RD, et al. The 

mini electronic medical record: a lowcost, low-risk partial solution. J Fam 

Pract 2001;50(12):1063-5. 

[Rec#: 463] 

 87. Chin HL, Krall M. Implementation of a 

comprehensive computer-based 

patient record system in Kaiser 

Permanente's Northwest Region. MD 

Comput 1997;14(1):41-5. 

[Rec#: 773] 

 88. Chin HL, Krall MA. Successful 

implementation of a comprehensive 

computer-based patient record system 

in Kaiser Permanente Northwest: 

strategy and experience. Eff Clin 

Pract 1998;1(2):51-60. 

[Rec#: 464] 

 89. Chin TL. Using automation to reduce 

medication errors. Health Data 

Manag 1997;5(7):74, 76, 78-83. 

[Rec#: 774] 

 90. Chrischilles EA, Fulda TR, Byrns PJ, et al. 

The role of pharmacy computer 

systems in preventing medication 

errors. J Am Pharm Assoc (Wash) 

2002;42(3):439-48. 

[Rec#: 207] 
G-6

 91. Churgin P, Strawn K. Population health 

management with computerized 

patient records. Eff Clin Pract 

1998;1(2):61-5. 

[Rec#: 466] 

 92. Churgin PG. Computerized patient records: 

the patients' response. HMO Pract 

1995;9(4):182-5. 

[Rec#: 775] 

 93. Classen DC, Burke JP, Pestotnik SL, et al. 

Clinical and financial impact of 

intravenous erythromycin therapy in 

hospitalized patients. Ann 

Pharmacother 1999;33(6):669-673. 

[Rec#: 9] 

 94. Classen DC, Pestotnik SL, Evans RS, et al. 

Computerized surveillance of adverse 

drug events in hospital patients. 

JAMA 1991;266(20):2847-2851. 

[Rec#: 10] 

 95. Coffey RM, Ball JK, Johantgen M, et al. The 

case for national health data 

standards. Health Aff (Millwood) 

1997;16(5):58-72. 

[Rec#: 980] 

 96. Coiera E. Four rules for the reinvention of 

health care. BMJ 

2004;328(7449):1197-9. 

[Rec#: 935] 

 97. Cole L, Houston S. Linking outcomes 

management and practice 

improvement. Structured care 

methodologies: evolution and use in 

patient care delivery. Outcomes 

Manag Nurs Pract 1999;3(2):53-9; 

quiz 59-60. 

[Rec#: 208] 

 98. Connelly DP, Aller RD. Outcomes and 

informatics. Arch Pathol Lab Med 

1997;121(11 ):1176-82. 

[Rec#: 209] 

 99. Cooper CP, Williams KN, Carey KA, et al. 

Advertising campaign on a major 

internet search engine to promote 

colorectal cancer screening. BMJ 

2004;328(7449):1179-80. 

[Rec#: 928] 

 100. Corley ST. Electronic prescribing: a review of 

costs and benefits. Top Health Inf 

Manage 2003;24(1):29-38. 

[Rec#: 12] 

 101. Covell DG, Uman GC, Manning PR. 

Information needs in office practice: 

are they being met? Ann Intern Med 

1985;103(4):596-9. 

[Rec#: 69] 

 102. Cox PMJr, D'Amato S, Tillotson DJ. Reducing 

medication errors. Am J Med Qual 

2001;16(3):81-6. 

[Rec#: 211] 

 103. Coye MJ, Bernstein WS. Improving America's 

health care system by investing in 

information technology. Health Aff 

(Millwood) 2003;22(4):56-8. 

[Rec#: 468] 

 104. Coyle GA, Heinen M. Scan your way to a 

comprehensive electronic medical 

record. Augment medication 

administration accuracy and increase 

documentation efficiency with bar 

coding technology. Nurs Manage 

2002;33(12):56, 58-9. 

[Rec#: 469] 

 105. Coyle-Toerner P, Collins L. How the clinical 

customization of an EMR means 

good business: a case study of Queen 

City Physicians. J Med Pract Manage 

2003;19(1):27-31. 

[Rec#: 470] 

 106. Cramer H, Katz PA. Electronic medical 

records: honing the competitive edge. 

Adm Radiol J 1996;15(2):16, 19-22. 

[Rec#: 776] 

 107. Cross M. Malpractice insurers endorse 

electronic records as a way to reduce 

risk. Health Data Manag 

1996;4(6):67-8, 70, 72. 

[Rec#: 778] 

 108. Currell R, Urquhart C. Nursing record 

systems: effects on nursing practice 

and health care outcomes. Cochrane 

Database Syst Rev 2003;( 

3):CD002099. 

[Rec#: 212] 
G-7

 109. Dakins DR. Is the future in the cards? Health 

Data Manag 2001;9(7):44-8. 

[Rec#: 471] 

 110. Dambro MR, Weiss BD, McClure CL, et al. 

An unsuccessful experience with 

computerized medical records in an 

academic medical center. J Med Educ 

1988;63(8):617-23. 

[Rec#: 647] 

 111. Darbro DA, Eichenauer J, Darbro MJ. 

Computerized medical record war 

story: practical experiences of 

computer implementation at the front. 

J Ambul Care Manage 

2000;23(2):38-42. 

[Rec#: 472] 

 112. Dassenko D, Slowinski T. Using the CPR to 

benefit a business office. Healthc 

Financ Manage 1995;49(7):68-70, 

72-3. 

[Rec#: 779] 

 113. Davidson EJ , Chismar WG. Planning and 

managing computerized order entry: a 

case study of IT- enabled 

organizational transformation. Top 

Health Inf Manage 1999;19(4):47-

61. 

[Rec#: 13] 

 114. Davis D, LeMaistre A. Has your organization 

leveraged the benefits of a 

computerized patient record? Nurs 

Case Manag 1997;2(6):240-5. 

[Rec#: 780] 

 115. Davis W. Nine years of computerized patient 

records in a small family practice. 

Fam Pract Manag 1997;4(3):69-70, 

73-6. 

[Rec#: 781] 

 116. de Boer HC, Heijmen BJ. A protocol for the 

reduction of systematic patient setup 

errors with minimal portal imaging 

workload. Int J Radiat Oncol Biol 

Phys 2001;50(5):1350-65. 

[Rec#: 214] 

 117. de Boer JC, Heijmen BJ. A new approach to 

off-line setup corrections: combining 

safety with minimum workload. Med 

Phys 2002;29(9):1998-2012. 

[Rec#: 215] 

 118. Dean JM, Vernon DD, Cook L, et al. 

Probabilistic linkage of computerized 

ambulance and inpatient hospital 

discharge records: a potential tool for 

evaluation of emergency medical 

services. Ann Emerg Med 

2001;37(6):616-26. 

[Rec#: 473] 

 119. Destache CJ, Meyer SK, Bittner MJ, et al. 

Impact of a clinical pharmacokinetic 

service on patients treated with 

aminoglycosides: a cost-benefit 

analysis. Ther Drug Monit 

1990;12(5):419-26. 

[Rec#: 648] 

 120. Destache CJ, Meyer SK, Rowley KM. Does 

accepting pharmacokinetic 

recommendations impact 

hospitalization? A cost-benefit 

analysis. Ther Drug Monit 

1990;12(5):427-33. 

[Rec#: 918] 

 121. Detmer DE, Deering MJ. Health information: 

a broader view. Health Aff 

(Millwood) 1999;18(2):256-9. 

[Rec#: 970] 

 122. Detmer DE. Building the national health 

information infrastructure for 

personal health, health care services, 

public health, and research. BMC 

Med Inform Decis Mak 2003;3(1):1. 

[Rec#: 70] 

 123. DeVore PA. Computer-assisted 

comprehensive geriatric assessment 

in a family physician's office. South 

Med J 1991;84 (8):953-5. 

[Rec#: 886] 

 124. DeWitt AL, Gunn SR, Hopkins P, et al. 

Critical care medicine mailing list: 

growth of an online forum. BMJ 

2004;328(7449):1180. 

[Rec#: 929] 

 125. Diehl M. Health system re-engineering: a 

CPRS economic decision model. Proc 

Annu Symp Comput Appl Med Care 

1995;688-92. 

[Rec#: 783] 
G-8

 126. Dilts D, Khamalah J, Plotkin A. Using cluster 

analysis for medical resource decision 

making. Med Decis Making 

1995;15(4):333-47. 

[Rec#: 220] 

 127. Donald JB. On line prescribing by computer. 

Br Med J (Clin Res Ed) 

1986;292(6525):937-9. 

[Rec#: 650] 

 128. Doolan DF, Bates DW. Computerized 

physician order entry systems in 

hospitals: mandates and incentives. 

Health Aff (Millwood) 

2002;21(4):180-8. 

[Rec#: 963] 

 129. Dorenfest S. The decade of the '90s. Poor use 

of IT investment contributes to the 

growing healthcare crisis. Healthc 

Inform 2000;17(8):64-7. 

[Rec#: 476] 

 130. Doupnik AM. An overview of Electronic 

Document Management System 

product offerings. Top Health Inf 

Manage 2002;23(1):62-73. 

[Rec#: 477] 

 131. Dowie J. What decision analysis can offer the 

clinical decision maker. Why 

outcome databases such as KIGS and 

KIMS are vital sources for decision 

analysis. Horm Res 1999;51 Suppl 

1:73-82. 

[Rec#: 222] 

 132. Dowie R, Gregory RP, Rowsell KV, et al. A 

decision analytic approach to 

commissioning ambulance cardiac 

services. J Manag Med 1998;12(2-

3):81-91, 79. 

[Rec#: 223] 

 133. Doyle O. Make benefits realization part of 

software purchase. Mod Healthc 

1989;19(37):48. 

[Rec#: 14] 

 134. Drazen E. Why don't we have computer-based 

patient records? J AHIMA 

1996;67(6):56-8; quiz 59-60. 

[Rec#: 784] 

 135. In: Drazen E, Fortin J, First Consulting Group. 

Digital hospitals move off the 

drawing board. 2003. [Rec#: 916] 

 136. Dujat C, Haux R, Schmucker P, et al. Digital 

optical archiving of medical records 

in hospital information systems--a 

practical approach towards the 

computer-based patient record? 

Methods Inf Med 1995;34(5):489-97. 

[Rec#: 785] 

 137. Duke JR, Mandell SF. The benefits and perils 

of system integration. J Med Syst 

1990;14(1-2):43-48. 

[Rec#: 16] 

 138. Durlach P. Focus on voice recognition. 

Talking up an emerging technology. 

Health Manag Technol 

2000;21(4):16-8. 

[Rec#: 479] 

 139. Eckman MH. Patient-centered decision 

making: a view of the past and a look 

toward the future. Med Decis Making 

2001;21(3):241-7. 

[Rec#: 225] 

 140. Edsall RL. Toward electronic medical records 

for the small practice. Fam Pract 

Manag 2000;7(6):10. 

[Rec#: 480] 

 141. Eichhorst B. Patient-centric HIS. A healthcare 

information system based on a 

longitudinal patient record provides 

benefits to patients--and clinicians, 

administrators and IT staff as well. 

Health Manag Technol 

2002;23(4):40-2. 

[Rec#: 482] 

 142. Elbourne D, Richardson M, Chalmers I, et al. 

The Newbury Maternity Care Study: 

a randomized controlled trial to 

assess a policy of women holding 

their own obstetric records. Br J 

Obstet Gynaecol 1987;94(7):612-9. 

[Rec#: 652] 

 143. Elson RB, Connelly DP. Computerized 

decision support systems in primary 

care. Prim Care 1995;22(2):365-84. 

[Rec#: 226] 
G-9

 144. Essin DJ, Lincoln TL. Implementing a lowcost computer-based patient record: a 

controlled vocabulary reduces data 

base design complexity. Proc Annu 

Symp Comput Appl Med Care 

1995;431-5. 

[Rec#: 787] 

 145. Estabrooks C, Goel V, Thiel E, et al. Decision 

aids: are they worth it? A systematic 

review. J Health Serv Res Policy 

2001;6(3):170-82. 

[Rec#: 227] 

 146. Everett JE. A decision support simulation 

model for the management of an 

elective surgery waiting system. 

Health Care Manag Sci 2002; 

5(2):89-95. 

[Rec#: 76] 

 147. Eytan TA, Goldberg HI. How effective is the 

computer-based clinical practice 

guideline? Eff Clin Pract 

2001;4(1):24-33. 

[Rec#: 228] 

 148. Fabrey RH= 2nd. Physician computer order 

entry in a hospital setting. Physician 

Exec 1996;22(4):31-3. 

[Rec#: 788] 

 149. Feldman P, Gold M, Chu K. Enhancing 

information for state health policy . 

Health Aff (Millwood) 

1994;13(3):236-50. 

[Rec#: 984] 

 150. Feldman PH, Gold M, Chu K. State health 

policy information: what worked? 

Health Aff (Millwood) 

1997;16(1):207-10. 

[Rec#: 983] 

 151. Ferguson GH, Hildman T, Nichols B. The 

effect of nursing care planning 

systems on patient outcomes. J Nurs 

Adm 1987;17(9):30-6. 

[Rec#: 653] 

 152. Fermann GJ, Suyama J. Point of care testing in 

the emergency department . J Emerg 

Med 2002;22(4):393-404. 

[Rec#: 229] 

 153. Fernando B, Savelyich BS, Avery AJ, et al. 

Prescribing safety features of general 

practice computer systems: 

evaluation using simulated test cases. 

BMJ 2004;328(7449):1171-2. 

[Rec#: 924] 

 154. Fernandopulle R, Ferris T, Epstein A, et al. A 

research agenda for bridging the 

'quality chasm.'. Health Aff 

(Millwood) 2003;22(2):178-90. 

[Rec#: 958] 

 155. Ferner RE. Computer aided prescribing leaves 

holes in the safety net. BMJ 

2004;328(7449):1172-3. 

[Rec#: 925] 

 156. Fieschi M, Dufour JC, Staccini P, et al. 

Medical decision support systems: 

old dilemmas and new paradigms? 

Methods Inf Med 2003;42(3):190-8. 

[Rec#: 230] 

 157. Finfgeld DL. Computer-assisted therapy: 

harbinger of the 21st century? Arch 

Psychiatr Nurs 1999;13(6):303-10. 

[Rec#: 231] 

 158. Finkelstein J, Cabrera MR, Hripcsak G. 

Internet-based home asthma 

telemonitoring: can patients handle 

the technology? Chest 

2000;117(1):148-55. 

[Rec#: 77] 

 159. Finnerty M, Altmansberger R, Bopp J, et al. 

Using state administrative and 

pharmacy data bases to develop a 

clinical decision support tool for 

schizophrenia guidelines. Schizophr 

Bull 2002;28(1):85-94. 

[Rec#: 232] 

 160. Firouzan PA, Coburn K. Research review: 

Survey of automation in health 

information management 

departments. Top Health Inf Manage 

1996; 17(1):52-60. 

[Rec#: 789] 

 161. Fischer S, Stewart TE, Mehta S, et al. 

Handheld computing in medicine. J 

Am Med Inform Assoc 

2003;10(2):139-49. 

[Rec#: 233] 
G-10

 162. Fleisher GR. What works. Children's Hospital 

ER automates records to save lives 

and money. Health Manag Technol 

1997;18(4):37. 

[Rec#: 790] 

 163. Fletcher J, Hicks NR, Kay JD, et al. Using 

decision analysis to compare policies 

for antenatal screening for Down's 

syndrome. Bmj 1995;311(7001):351-

6. 

[Rec#: 235] 

 164. Fordham D, McPhee SJ, Bird JA, et al. The 

Cancer Prevention Reminder System. 

MD Comput 1990;7(5):289-95. 

[Rec#: 154] 

 165. Fors UG, Sandberg HC. Computer-aided risk 

management--a software tool for the 

Hidep model. Quintessence Int 

2001;32 (4):309-20. 

[Rec#: 236] 

 166. Foster RA, Antonelli PJ. Computerized 

physician-order entry: are we there 

yet? Otolaryngol Clin North Am 

2002;35(6):1237-43, vii. 

[Rec#: 485] 

 167. Frame PS. Computerized health maintenance 

tracking systems: a clinician's guide 

to necessary and optional features. A 

report from the American Cancer 

Society Advisory Group on 

Preventive Health Care Reminder 

Systems. J Am Board Fam Pract 

1995;8(3):221-9. 

[Rec#: 237] 

 168. Fretschner R, Bleicher W, Heininger A, et al. 

Patient data management systems in 

critical care. J Am Soc Nephrol 

2001;12 Suppl 17:S83-6. 

[Rec#: 486] 

 169. Frijling BD, Lobo CM, Hulscher ME, et al. 

Multifaceted support to improve 

clinical decision making in diabetes 

care: a randomized controlled trial in 

general practice. Diabet Med 

2002;19(10):836-42. 

[Rec#: 238] 

 170. Frisse ME. Computers and productivity: is it 

time for a reality check? Acad Med 

1998;73(1):59-64. 

[Rec#: 487] 

 171. Gadd CS, Baskaran P, Lobach DF. 

Identification of design features to 

enhance utilization and acceptance of 

systems for Internet-based decision 

support at the point of care. Proc 

AMIA Symp 1998;91-5. 

[Rec#: 239] 

 172. Gardner E. Quantifying system benefits an 

elusive task. Mod Healthc 

1990;20(8):26-27. 

[Rec#: 19] 

 173. Gawande AA, Bates DW. The use of 

information technology in improving 

medical performance. Part II. 

Physician-support tools. MedGenMed 

2000;2(1):E13. 

[Rec#: 940] 

 174. Gawande AA, Bates DW. The use of 

information technology in improving 

medical performance. Part III. 

Patient-support tools. MedGenMed 

2000;2(1):E12. 

[Rec#: 241] 

 175. Gentile D. Toward growth in transcription 

solutions. Healthcare providers can 

gain minutes and even hours of 

efficiency with transcription 

technology. Health Manag Technol 

2003;24(7):38-40. 

[Rec#: 489] 

 176. Gillespie G. CIOs making soft sell when 

calculating ROI. Health Data Manag 

2002;10(8):44-8, 50. 

[Rec#: 491] 

 177. Gillespie G. In paper war, is OCR a good 

soldier? Health Data Manag 2003 

;11(10):58-60, 62 64, passim. 

[Rec#: 492] 

 178. Gillespie G. Online clinical guidelines help 

trim costs. Health Data Manag 

2000;8(1):38-45. 

[Rec#: 490] 
G-11

 179. Glandon GL, Shapiro RJ. Benefit-cost analysis 

of hospital information systems: the 

state of the (non) art. J Health Hum 

Resour Adm 1988;11(1):30-92. 

[Rec#: 21] 

 180. Glowinski A. Integrating guidelines and the 

clinical record: the role of 

semantically constrained 

terminologies. Stud Health Technol 

Inform 1995;16:207-18. 

[Rec#: 792] 

 181. Goddard BL. Termination of a contract to 

implement an enterprise electronic 

medical record system. J Am Med 

Inform Assoc 2000;7(6):564-8. 

[Rec#: 494] 

 182. Goedert J. Practice talks up substantial ROI. 

Health Data Manag 2003;11(4):54-6. 

[Rec#: 495] 

 183. Goldsmith J, Blumenthal D, Rishel W. Federal 

health information policy: a case of 

arrested development. Health Aff 

(Millwood) 2003;22(4):44-55. 

[Rec#: 496] 

 184. Goldstein MK, Coleman RW, Tu SW, et al. 

Translating research into practice: 

organizational issues in implementing 

automated decision support for 

hypertension in three medical centers. 

J Am Med Inform Assoc 

2004;11(5):368-76. 

[Rec#: 944] 

 185. Goodman KW. Critical care computing: 

outcomes, confidentiality, and 

appropriate use. Crit Care Clin 

1996;12(1):109-22. 

[Rec#: 497] 

 186. Gordon C. Practice guidelines and healthcare 

telematics: towards an alliance. Stud 

Health Technol Inform 1995;16:3-16. 

[Rec#: 243] 

 187. Gordon C, Gray JA, Toth B, et al. Systems of 

evidence-based healthcare and 

personalised health information: 

some international and national 

trends. Stud Health Technol Inform 

2000;77:23-8. 

[Rec#: 244] 

 188. Gorman C, Looker J, Fisk T, et al. A clinically 

useful diabetes electronic medical 

record: lessons from the past; pointers 

toward the future. Eur J Endocrinol 

1996;134(1):31-42. 

[Rec#: 498] 

 189. Grams RR, Morgan G. Medical record 

innovations that can improve 

physician productivity. J Med Syst 

1999;23(2):133-44. 

[Rec#: 499] 

 190. Grams RR, Moyer EH. The search for the 

elusive electronic medical record 

system--medical liability, the missing 

factor. J Med Syst 1997;21(1):1-10. 

[Rec#: 500] 

 191. Granados A. Health technology assessment 

and clinical decision making: which 

is the best evidence? Int J Technol 

Assess Health Care 1999;15(3):585-

92. 

[Rec#: 245] 

 192. Grandinetti D. The good news--and bad--about 

Web-based EMRs. Med Econ 

2000;77(17):73-4, 77, 80 passim. 

[Rec#: 501] 

 193. Greatbatch D, Heath C, Campion P, et al. How 

do desk-top computers affect the 

doctor-patient interaction? Fam Pract 

1995;12(1):32-6. 

[Rec#: 887] 

 194. Green J, Wintfeld N. How accurate are 

hospital discharge data for evaluating 

effectiveness of care? Med Care 

1993;31 (8):719-31. 

[Rec#: 79] 

 195. Greene J. The portal advantage. Hospitals lure 

docs with easy access to medical 

records. Hosp Health Netw 2001;75 

(10):56-8, 2. 

[Rec#: 502] 

 196. Greene ZB. Creating and managing a 

paperless health information 

management department. Top Health 

Inf Manage 2002;23(1):26-36. 

[Rec#: 503] 
G-12

 197. Greengold NL. A Web-based program for 

implementing evidence-based patient 

safety recommendations. Jt Comm J 

Qual Improv 2002;28(6):340-8. 

[Rec#: 247] 

 198. Greenhalgh T, Hughes J, Humphrey C, et al. A 

comparative case study of two 

models of a clinical informaticist 

service. Bmj 2002;324(7336):524-9. 

[Rec#: 248] 

 199. Grimshaw JM, Russell IT. Achieving health 

gain through clinical guidelines II: 

Ensuring guidelines change medical 

practice. Qual Health Care 

1994;3(1):45-52. 

[Rec#: 660] 

 200. Hagland M. Finding the e in healthcare. 

Healthc Inform 2001;18(11):21-4, 26. 

[Rec#: 506] 

 201. Hagland M. Moving gradually toward a 

paperless world. J AHIMA 

2000;71(8):26-31; quiz 33-4. 

[Rec#: 504] 

 202. Hagland M. Outpatient clinics without the 

paperwork. Health Manag Technol 

1997;18(6):14-9. 

[Rec#: 794] 

 203. Halley EC, Kambic PM. Concurrent process 

redesign and clinical documentation 

system implementation: a 6-month 

success story. Top Health Inf Manage 

1996;17(1):12-7. 

[Rec#: 795] 

 204. Halpern MT, Palmer CS, Simpson KN, et al. 

The economic and clinical efficiency 

of point-of-care testing for critically 

ill patients: a decision-analysis 

model. Am J Med Qual 1998;13(1):3-

12. 

[Rec#: 250] 

 205. Hancock WM, Walter PF. Reduce hospital 

costs with admissions and operating 

room scheduling systems. Softw 

Healthc 1986;4(1):42-6. 

[Rec#: 80] 

 206. Hannan TJ. Detecting adverse drug reactions 

to improve patient outcomes . Int J 

Med Inf 1999;55(1):61-4. 

[Rec#: 508] 

 207. Hannan TJ. Variation in health care--the roles 

of the electronic medical record. Int J 

Med Inf 1999;54(2):127-36. 

[Rec#: 507] 

 208. Hansen B. Handheld computers in diabetes 

management. Diabetes Self Manag 

2002;19(6):43-4, 46, 48-9. 

[Rec#: 509] 

 209. Hanson CW3rd, Marshall BE. Artificial 

intelligence applications in the 

intensive care unit. Crit Care Med 

2001;29(2):427-35. 

[Rec#: 251] 

 210. Harman J. Topics for our times: new health 

care data--new horizons for public 

health. Am J Public Health 

1998;88(7):1019-21. 

[Rec#: 511] 

 211. Hart JT, Thomas C, Gibbons B, et al. Twenty 

five years of case finding and audit in 

a socially deprived community. BMJ 

1991;302(6791):1509-13. 

[Rec#: 661] 

 212. Hasenfeld R , Shekelle PG. Is the 

methodological quality of guidelines 

declining in the US? Comparison of 

the quality of US Agency for Health 

Care Policy and Research (AHCPR) 

guidelines with those published 

subsequently. Qual Saf Health Care 

2003;12(6):428-34. 

[Rec#: 937] 

 213. Hassey A, Gerrett D, Wilson A . A survey of 

validity and utility of electronic 

patient records in a general practice. 

Bmj 2001;322(7299):1401-5. 

[Rec#: 252] 

 214. Heinrichs W, Monk S, Eberle B . [Automated 

anesthesia record systems]. 

Anaesthesist 1997;46(7):574-82. 

[Rec#: 512] 
G-13

 215. Hellbruck RP. [Electronic data processing 

assisted quality management in 

general practice networks]. 

Gesundheitswesen 2000;62(10):511-

5. 

[Rec#: 513] 

 216. Henry SB. Nursing informatics: state of the 

science. J Adv Nurs 

1995;22(6):1182-92. 

[Rec#: 254] 

 217. Henry SB, Douglas K, Galzagorry G, et al. A 

template-based approach to support 

utilization of clinical practice 

guidelines within an electronic health 

record. J Am Med Inform Assoc 

1998;5(3):237-44. 

[Rec#: 255] 

 218. Henry SB, Mead CN. Nursing classification 

systems: necessary but not sufficient 

for representing "what nurses do" for 

inclusion in computer-based patient 

record systems. J Am Med Inform 

Assoc 1997;4(3):222-32. 

[Rec#: 256] 

 219. Hicks J, Waldrum M, Wilson V, et al. 

Transcription system overhaul reaps 

savings, efficiency. J AHIMA 

2000;71 (10):58-63. 

[Rec#: 514] 

 220. Hodgkins ML. Are you ready for the 

computer-based patient record? J 

Ambul Care Manage 1995;18(3):1-8. 

[Rec#: 797] 

 221. Hoffer A. BPM: antidote to inefficiency. 

Business process management serves 

as a foundation for effective 

workflow automation in healthcare. 

Health Manag Technol 

2003;24(1):40-3. 

[Rec#: 515] 

 222. Hoffman C, Rice D, Sung HY. Persons with 

chronic conditions. Their prevalence 

and costs. JAMA 

1996;276(18):1473-9. 

[Rec#: 82] 

 223. Hofmans-Okkes IM, Lamberts H. The 

International Classification of 

Primary Care (ICPC): new 

applications in research and 

computer-based patient records in 

family practice. Fam Pract 

1996;13(3):294-302. 

[Rec#: 516] 

 224. Holmes-Rovner M, Kroll J, Rovner DR, et al. 

Patient decision support intervention: 

increased consistency with decision 

analytic models. Med Care 

1999;37(3):270-84. 

[Rec#: 260] 

 225. Holzman TG, Griffith A, Hunter WG, et al. 

Computer-assisted trauma care 

prototype. Medinfo 1995;8 Pt 2:1685. 

[Rec#: 261] 

 226. Homer CS, Davis GK, Everitt LS. The 

introduction of a woman-held record 

into a hospital antenatal clinic: the 

bring your own records study. Aust N 

Z J Obstet Gynaecol 1999;39( 1):54-

7. 

[Rec#: 665] 

 227. Hovenga EJ. Nursing information and the use 

of electronic health records. Aust 

Nurs J 2001;8(11):39-40. 

[Rec#: 263] 

 228. Howard WR. Development of an affordable 

data collection, reporting, and 

analysis system. Respir Care 

2003;48(2):131-7. 

[Rec#: 517] 

 229. Hozo I, Djulbegovic B. Using the Internet to 

calculate clinical action thresholds. 

Comput Biomed Res 1999;32(2):168-

85. 

[Rec#: 264] 

 230. Hughes KK, Dvorak EM. The use of decision 

analysis to examine ethical decision 

making by critical care nurses. Heart 

Lung 1997;26(3):238-48. 

[Rec#: 266] 
G-14

 231. Hurley SF, Dziukas LJ, McNeil JJ, et al. A 

randomized controlled clinical trial of 

pharmacokinetic theophylline dosing. 

Am Rev Respir Dis 

1986;134(6):1219-24. 

[Rec#: 666] 

 232. Hynes DM, Perrin RA, Rappaport S, et al. 

Informatics resources to support 

health care quality improvement in 

the veterans health administration. J 

Am Med Inform Assoc 

2004;11(5):344-50. 

[Rec#: 943] 

 233. Jacobs I. What works. Never going back. New 

Jersey ophthalmology practice moves 

from paper charts to an EMR system. 

Health Manag Technol 

2003;24(8):40-1. 

[Rec#: 518] 

 234. Jadad AR. Evidence-based decision making 

and asthma in the internet age: the 

tools of the trade. Allergy 2002;57 

Suppl 74:15-22. 

[Rec#: 268] 

 235. Jadad AR, Haynes RB, Hunt D, et al. The 

Internet and evidence-based decisionmaking: a needed synergy for 

efficient knowledge management in 

health care . CMAJ 2000;162(3):362-

5. 

[Rec#: 269] 

 236. Jantos LD, May RR, Sheppard RL. 

Streamlining medical group 

operations through an integrated 

computer-based patient record. J 

Healthc Inf Manag 1998;12(3):89-

101. 

[Rec#: 519] 

 237. Javitt JC. How to succeed in health 

information technology. Health Aff 

(Millwood) 2004;Suppl Web 

Exclusives:W4-321-4. 

[Rec#: 955] 

 238. Joch A. EMR the web way. Hosp Pract (Off 

Ed) 2000;35(5):suppl 12-6. 

[Rec#: 521] 

 239. Johnson KB. Barriers that impede the adoption 

of pediatric information technology. 

Arch Pediatr Adolesc Med 

2001;155(12):1374-9. 

[Rec#: 270] 

 240. Johnson KB, Cowan J. Clictate: a computerbased documentation tool for 

guideline-based care. J Med Syst 

2002;26(1):47-60. 

[Rec#: 271] 

 241. Joosting AC. Practice administration--a costeffective option. S Afr Med J 

2001;91(6):444-5. 

[Rec#: 522] 

 242. Jue J, Jerant AF. Electronic paper & pencil: 6 

easy steps to a low-cost electronic 

medical record. Fam Pract Manag 

2001;8(5):33-8. 

[Rec#: 523] 

 243. Kadas RM. The computer-based patient record 

is on its way. HMOs, the economy 

and HIPAA will drive adoption. 

Healthc Inform 2002;19(2):57-8. 

[Rec#: 524] 

 244. Kahn RA. Beyond HIPAA: the complexities 

of electronic record management . J 

AHIMA 2003;74(4):31-6; quiz 37-8. 

[Rec#: 525] 

 245. Kantor GS, Wilson WD, Midgley A. Opensource software and the primary care 

EMR. J Am Med Inform Assoc 

2003;10 (6):616; author reply 617. 

[Rec#: 526] 

 246. Kaplan B. An evaluation model for clinical 

information systems: clinical imaging 

systems. Medinfo 1995;8 Pt 2:1087. 

[Rec#: 273] 

 247. Kaplan B, Lundsgaarde HP. Toward an 

evaluation of an integrated clinical 

imaging system: identifying clinical 

benefits. Methods Inf Med 

1996;35(3):221-9. 

[Rec#: 527] 

 248. Katikireddi SV. HINARI: bridging the global 

information divide. BMJ 

2004;328(7449):1190-3. 
G-15

[Rec#: 933] 

 249. Kaufman DR, Patel VL, Hilliman C, et al. 

Usability in the real world: assessing 

medical information technologies in 

patients' homes. J Biomed Inform 

2003;36(1-2):45-60. 

[Rec#: 949] 

 250. Kazanjian A, Green CJ. Beyond effectiveness: 

the evaluation of information systems 

using A Comprehensive Health 

Technology Assessment Framework. 

Comput Biol Med 2002;32(3):165-

77. 

[Rec#: 274] 

 251. Keller LS, McDermott S, Alt-White A. Effects 

of computerized nurse careplanning 

on selected health care effectiveness 

measures. Proc Annu Symp Comput 

Appl Med Care 1991;38-42. 

[Rec#: 667] 

 252. Kelly B. The future of card technology in 

health care. Health Data Manag 

2002;10(5):64-8. 

[Rec#: 528] 

 253. Kelly G. Electronics, clinicians, and the NHS. 

BMJ 2000;321(7265):846-7. 

[Rec#: 529] 

 254. Kelly JT. 'After the chaos': expected benefits 

of health information management. 

Health Aff (Millwood) 1998;17 

(6):39-40. 

[Rec#: 975] 

 255. Kemper DW, Mettler M. Information therapy: 

a tale. Health Forum J 2002;45(1):16-

20. 

[Rec#: 275] 

 256. Kendall DB, Levine SR. Pursuing the promise 

of an information-age health care 

system. Health Aff (Millwood) 

1998;17(6):41-3. 

[Rec#: 974] 

 257. Kerkoulas PS. How to produce EDMS 

requirements and cost-benefit data. 

Top Health Inf Manage 2002;23(1):7-

15. 

[Rec#: 530] 

 258. Kerr EA, Krein SL, Vijan S, et al. Avoiding 

pitfalls in chronic disease quality 

measurement: a case for the next 

generation of technical quality 

measures. Am J Manag Care 

2001;7(11):1033-43. 

[Rec#: 164] 

 259. Kerr EA, Smith DM, Hogan MM, et al. 

Building a better quality measure: are 

some patients with 'poor quality' 

actually getting good care? Med Care 

2003;41(10):1173-82. 

[Rec#: 166] 

 260. Khabir J. What works. Handheld PCs, CPR 

software help clinic cuts costs, 

enhance service. Health Manag 

Technol 1997;18(8):42. 

[Rec#: 800] 

 261. Khoury AT. Support of quality and business 

goals by an ambulatory automated 

medical record system in Kaiser 

Permanente of Ohio. Eff Clin Pract 

1998;1(2):73-82. 

[Rec#: 531] 

 262. Kim MI, Johnson KB. Personal health records: 

evaluation of functionality and utility. 

J Am Med Inform Assoc 

2002;9(2):171-80. 

[Rec#: 87] 

 263. Kinyon CC. One CFO's success with 

transitioning to an automated patient 

record. Healthc Financ Manage 

2003;57 (2):52-6. 

[Rec#: 532] 

 264. Kirkby KC, Lambert TJ. Computer aids to 

treatment in psychiatry. Aust N Z J 

Psychiatry 1996;30(1):142-5. 

[Rec#: 278] 

 265. Kleinke JD. Vaporware.com: the failed 

promise of the health care Internet. 

Health Aff (Millwood) 2000;19 

(6):57-71. 

[Rec#: 966] 

 266. Kleinke JD. Release 0.0: clinical information 

technology in the real world. Health 

Aff (Millwood) 1998;17 (6):23-38. 

[Rec#: 533] 
G-16

 267. Kloss L. An electronic emergency. We need to 

push the industry toward standardized 

digital medical records. Mod Healthc 

2003;33(19):21. 

[Rec#: 534] 

 268. Kluge EH. Professional codes for electronic 

HC record protection: ethical, legal, 

economic and structural issues. Int J 

Med Inf 2000;60(2):85-96. 

[Rec#: 535] 

 269. Kok MR, Boon ME. Consequences of neural 

network technology for cervical 

screening: increase in diagnostic 

consistency and positive scores. 

Cancer 1996 ;78(1):112-7. 

[Rec#: 88] 

 270. Koppel R, Metlay JP, Cohen A, et al. Role of 

computerized physician order entry 

systems in facilitating medication 

errors. JAMA 2005;293(10):1197-

203. 

[Rec#: 993] 

 271. Krall MA, Chin H, Dworkin L, et al. 

Improving clinician acceptance and 

use of computerized documentation 

of coded diagnosis. Am J Manag 

Care 1997;3(4):597-601. 

[Rec#: 805] 

 272. Krobock JR. A taxonomy: hospital 

information systems evaluation 

methodologies. J Med Syst 

1984;8(5):419-429. 

[Rec#: 23] 

 273. Krohn R. Picture archiving and computing 

systems: the key to enterprise digital 

imaging. Manag Care Interface 

2002;15(9):36-9. 

[Rec#: 536] 

 274. Kuebler KK, Bruera E. Interactive 

collaborative consultation model in 

end-of-life care. J Pain Symptom 

Manage 2000;20(3):202-9. 

[Rec#: 90] 

 275. Kuntz KM, Tsevat J, Weinstein MC, et al. 

Expert panel vs decision-analysis 

recommendations for postdischarge 

coronary angiography after 

myocardial infarction. Jama 

1999;282(23):2246-51. 

[Rec#: 279] 

 276. Kuperman GJ, Gibson RF. Computer 

physician order entry: benefits, costs, 

and issues. Ann Intern Med 

2003;139(1):31-9. 

[Rec#: 24] 

 277. Kuperman GJ, Teich JM, Gandhi TK, et al. 

Patient safety and computerized 

medication ordering at Brigham and 

Women's Hospital. Jt Comm J Qual 

Improv 2001 ;27(10):509-21. 

[Rec#: 91] 

 278. Kuruvilla S , Dzenowagis J, Pleasant A, et al. 

Digital bridges need concrete 

foundations: lessons from the Health 

InterNetwork India. BMJ 

2004;328(7449):1193-6. 

[Rec#: 934] 

 279. Laerum H, Ellingsen G, Faxvaag A. Doctors' 

use of electronic medical records 

systems in hospitals: cross sectional 

survey. Bmj 2001;323(7325):1344-8. 

[Rec#: 280] 

 280. Laing K. The benefits and challenges of the 

computerized electronic medical 

record. Gastroenterol Nurs 

2002;25(2):41-5. 

[Rec#: 538] 

 281. Lakhani AD, Avery A, Gordon A, et al. 

Evaluation of a home based health 

record booklet. Arch Dis Child 

1984;59 (11):1076-81. 

[Rec#: 669] 

 282. Lambrinoudakis C, Gritzalis S . Managing 

medical and insurance information 

through a smart-card-based 

information system. J Med Syst 2000; 

24(4):213-34. 

[Rec#: 539] 
G-17

 283. Landis SE, Hulkower SD, Pierson S. 

Enhancing adherence with 

mammography through patient letters 

and physician prompts. A pilot study. 

N C Med J 1992;53(11):575-8. 

[Rec#: 156] 

 284. Langton KB, Horsman J, Hayward RS, et al. A 

Clinical Informatics Network 

(CLINT) to support the practice of 

evidence-based health care. Proc 

AMIA Annu Fall Symp 1996;428-32. 

[Rec#: 281] 

 285. Latimer EW. Assessing the impact of 

ambulatory computer-based medical 

record systems. MD Comput 

1999;16(2):44-6. 

[Rec#: 541] 

 286. Lawler F, Cacy JR, Viviani N, et al. 

Implementation and termination of a 

computerized medical information 

system. J Fam Pract 1996;42(3):233-

6. 

[Rec#: 806] 

 287. Lazarus R, Kleinman KP, Dashevsky I, et al. 

Using automated medical records for 

rapid identification of illness 

syndromes (syndromic surveillance): 

the example of lower respiratory 

infection. BMC Public Health 

2001;1(1): 9. 

[Rec#: 542] 

 288. Lazarus SS. Physicians' use of electronic 

medical records--identifying and 

crossing the barriers. Med Group 

Manage J 1999;46(3):12-4. 

[Rec#: 543] 

 289. Leape LL, Bates DW, Cullen DJ, et al. 

Systems analysis of adverse drug 

events. ADE Prevention Study 

Group. JAMA 1995;274(1):35-43. 

[Rec#: 862] 

 290. Leape LL, Berwick DM, Bates DW. What 

practices will most improve safety? 

Evidence-based medicine meets 

patient safety. JAMA 

2002;288(4):501-507. 

[Rec#: 170] 

 291. Lee FW. Can computer-aided systems 

engineering tools enhance the 

development of health care 

information systems? A critical 

analysis. Top Health Inf Manage 

1996;17(1):1-11. 

[Rec#: 282] 

 292. Lee IN, Lee SC, Embrechts M. Important 

variable selection techniques with 

multiple solutions for medical 

information applications. Med Inform 

Internet Med 2002;27(4):253-66. 

[Rec#: 283] 

 293. Lee ML, Warner HR. Performance of a 

diagnostic system (Iliad) as tool for 

quality assurance. Comput Biomed 

Res 1992;25(4):314-23. 

[Rec#: 93] 

 294. Lehmann ED, Deutsch T. Application of 

computers in diabetes care--a review. 

I. Computers for data collection and 

interpretation. Med Inform (Lond) 

1995;20(4 ):281-302. 

[Rec#: 749] 

 295. Lehmann ED, Deutsch T. Application of 

computers in diabetes care--a review. 

II. Computers for decision support 

and education. Med Inform (Lond) 

1995;20(4):303-29. 

[Rec#: 748] 

 296. Leipzig M. Implementing an electronic 

medical record system in ambulatory 

care. Hosp Technol Ser 

1996;15(12):2-4. 

[Rec#: 809] 

 297. Lenert LA, Cher DJ. Use of meta-analytic 

results to facilitate shared decision 

making. J Am Med Inform Assoc 

1999;6(5):412-9. 

[Rec#: 284] 

 298. Lepage EF, Gardner RM, Laub RM, et al. 

Improving blood transfusion practice: 

role of a computerized hospital 

information system. Transfusion 

1992;32(3 ):253-9. 

[Rec#: 94] 
G-18

 299. Lester J, Prady S, Finegan Y, et al. Learning 

from e-patients at Massachusetts 

General Hospital. BMJ 

2004;328(7449):1188-90. 

[Rec#: 932] 

 300. Liang MH, Shadick N. Feasibility and utility 

of adding disease-specific outcome 

measures to administrative databases 

to improve disease management. Ann 

Intern Med 1997;127(8 Pt 2):739-42. 

[Rec#: 285] 

 301. Liederman EM, Morefield CS. Web 

messaging: A new tool for patientphysician communication. J Am Med 

Inform Assoc 2003;10(3):260-70. 

[Rec#: 95] 

 302. Lillehaug SI, Lajoie SP. AI in medical 

education--another grand challenge 

for medical informatics. Artif Intell 

Med 1998;12(3):197-225. 

[Rec#: 286] 

 303. Linney B. Cardiology practice proves that 

electronic medical records do raise 

revenue. Physician Exec 

2003;29(3):34-6. 

[Rec#: 545] 

 304. Lipton HL, Miller RH, Wimbush JJ. 

Electronic prescribing: ready for 

prime time? J Healthc Inf Manag 

2003;17(4):72-9. 

[Rec#: 546] 

 305. Liu BJ, Cao F, Zhou MZ, et al . Trends in 

PACS image storage and archive . 

Comput Med Imaging Graph 

2003;27(2-3):165-74. 

[Rec#: 288] 

 306. Long PW, Larkins RG, Patterson CG, et al. 

Facilitating best practice: transferring 

the lessons of the Clinical Support 

Systems Program. J Qual Clin Pract 

2001;21(4):157-9. 

[Rec#: 289] 

 307. Lowenstein E, Greenberg D. Next-generation 

informatics. A new picture of quality 

medicine. Physician Exec 

1996;22(11):14-6. 

[Rec#: 811] 

 308. Lowes B. Firewalls: cheap and easy 

protection. Med Econ 

2003;80(17):20-1. 

[Rec#: 547] 

 309. Lowes R. Build an EMR for next to nothing. 

Med Econ 2003; 80(11):29-30. 

[Rec#: 551] 

 310. Lowes R. EMR? Not exactly. Good enough? 

For now. Med Econ 2002;79(15):32-

4. 

[Rec#: 550] 

 311. Lowes R. They're still working on the 

electronic medical record. Med Econ 

2000;77(13):29-32, 38. 

[Rec#: 548] 

 312. Lowes R. Toward a handheld EMR. Med Econ 

2002;79(1):26-8. 

[Rec#: 549] 

 313. Lubarsky DA, Sanderson IC, Gilbert WC, et 

al. Using an anesthesia information 

management system as a cost 

containment tool. Description and 

validation. Anesthesiology 

1997;86(5):1161-9. 

[Rec#: 812] 

 314. Lumpkin JR, Richards MS. Transforming the 

public health information 

infrastructure. Health Aff (Millwood) 

2002;21 (6):45-56. 

[Rec#: 961] 

 315. Luxenberg SN, DuBois DD, Fraley CG, et al. 

Electronic forms: benefits drawbacks 

of a World Wide Web-based 

approach to data entry. Proc AMIA 

Annu Fall Symp 1997;804-8. 

[Rec#: 813] 

 316. Maglaveras N, Koutkias V, Chouvarda I, et al. 

Home care delivery through the 

mobile telecommunications platform: 

the Citizen Health System (CHS) 

perspective. Int J Med Inf 2002;68(1-

3):99-111. 

[Rec#: 291] 
G-19

 317. Mahoney ME. Transforming health 

information management through 

technology. Top Health Inf Manage 

2002;23 (1):52-61. 

[Rec#: 292] 

 318. Major LF, Turner MG. Assessing the 

information management 

requirements for behavioral health 

providers. J Healthc Manag 

2003;48(5):323-33; discussion 334-5. 

[Rec#: 552] 

 319. Makela M, Kunnamo L. Implementing 

evidence in Finnish primary care. Use 

of electronic guidelines in daily 

practice. Scand J Prim Health Care 

2001;19(4):214-7. 

[Rec#: 293] 

 320. Mandl KD, Kohane IS, Brandt AM. Electronic 

patient-physician communication: 

problems and promise. Ann Intern 

Med 1998;129(6):495-500. 

[Rec#: 294] 

 321. Marill KA, Gauharou ES, Nelson BK, et al. 

Prospective, randomized trial of 

template-assisted versus undirected 

written recording of physician records 

in the emergency department. Ann 

Emerg Med 1999;33(5):500-9. 

[Rec#: 553] 

 322. Markle Foundation, The Robert Wood 

Johnson Foundation. Financial, legal 

and organizational approaches to 

achieving electronic connectivity in 

healthcare: working group on 

financial, organizational, and legal 

sustainability of health information 

exchange. 2004. 

[Rec#: 951] 

 323. Marks I. A computerized system to measure 

treatment outcomes and cost. 

Psychiatr Serv 1996;47(8):811-2. 

[Rec#: 814] 

 324. Martin-Baranera M, Planas I, Palau J, et al. 

IMASIS computer-based medical 

record project: dealing with the 

human factor. Medinfo 1995;8 Pt 

1:333. 

[Rec#: 295] 

 325. Masys DR. Effects of current and future 

information technologies on the 

health care workforce. Health Aff 

(Millwood) 2002;21(5):33-41. 

[Rec#: 962] 

 326. Matchar DB, Samsa GP. Using outcomes data 

to identify best medical practice: the 

role of policy models. Hepatology 

1999;29(6 Suppl):36S-39S. 

[Rec#: 296] 

 327. Mathieson S. For a few dollars more. Health 

Serv J 2003;113(5872):suppl 8-9. 

[Rec#: 554] 

 328. Mathis R, Doyle S. A quality mix: using 

evidence and experience to evaluate 

new technologies. J Healthc Qual 

2003;25(5):4-6; quiz 6-7. 

[Rec#: 297] 

 329. Matthews P. Planning for successful outcomes 

in the new millennium. Top Health 

Inf Manage 2000;20(3):55-64. 

[Rec#: 298] 

 330. May S. A CPR system can grow a practice. 

Improved efficiencies in charting and 

workflow allow better and faster 

healthcare delivery. Healthc Inform 

2002;19(1):93-4. 

[Rec#: 555] 

 331. McAlearney AS, Schweikhart SB, Medow 

MA. Doctors' experience with 

handheld computers in clinical 

practice: qualitative study. BMJ 

2004;328(7449):1162. 

[Rec#: 922] 

 332. McArthur J, Dickinson A. Decision making 

the explicit evidence-based way: 

comparing benefits, harms and costs. 

Nurs Prax N Z 1999;14(1):33-42. 

[Rec#: 301] 

 333. McCormick K, Renner AL, Mayes R, et al. 

The federal and private sector roles in 

the development of minimum data 

sets and core health data elements. 

Comput Nurs 1997;15(2 Suppl):S23-

32. 

[Rec#: 815] 
G-20

 334. McCormick KA. Including oncology 

outcomes of care in the computerbased patient record. Oncology 

(Huntingt) 1995;9(11 Suppl):161-7. 

[Rec#: 556] 

 335. McDonald CJ. Need for standards in health 

information. Health Aff (Millwood) 

1998;17(6):44-6. 

[Rec#: 973] 

 336. McDonald CJ, Overhage JM, Mamlin BW, et 

al. Physicians, information 

technology, and health care systems: 

a journey, not a destination. J Am 

Med Inform Assoc 2004;11(2):121-4. 

[Rec#: 938] 

 337. McDonald CJ, Overhage JM, Dexter P, et al. 

What is done, what is needed and 

what is realistic to expect from 

medical informatics standards. Int J 

Med Inf 1998;48(1-3):5-12. 

[Rec#: 675] 

 338. McDonald CJ, Overhage JM, Tierney WM, et 

al. The Regenstrief Medical Record 

System: a quarter century experience. 

Int J Med Inf 1999;54(3):225-53. 

[Rec#: 676] 

 339. McGlynn EA, Asch SM, Adams J, et al. The 

quality of health care delivered to 

adults in the United States. N Engl J 

Med 2003;348(26):2635-45. 

[Rec#: 97] 

 340. McGowan JJ, Winstead-Fry P. Problem 

Knowledge Couplers: reengineering 

evidence-based medicine through 

interdisciplinary development, 

decision support, and research. Bull 

Med Libr Assoc 1999;87(4):462-70. 

[Rec#: 303] 

 341. McKenzie NC, Marks IM. Overcoming 

interface problems in computerized 

monitoring of clinical outcomes. MD 

Comput 1997;14(5):377-81. 

[Rec#: 816] 

 342. McPhee SJ, Bird JA, Jenkins CN, et al. 

Promoting cancer screening. A 

randomized, controlled trial of three 

interventions. Arch Intern Med 

1989;149(8):1866-72. 

[Rec#: 153] 

 343. McQueen L, Mittman BS, Demakis JG. 

Overview of the Veterans Health 

Administration (VHA) Quality 

Enhancement Research Initiative 

(QUERI). J Am Med Inform Assoc 

2004;11(5):339-43. 

[Rec#: 942] 

 344. Mehta V, Kushniruk A, Gauthier S, et al. Use 

of evidence in the process of practice 

change in a clinical team: a studyforming part of the Autocontrol 

Project. Int J Med Inf 1998;51(2-

3):169-80. 

[Rec#: 305] 

 345. Memel DS, Scott JP, McMillan DR, et al. 

Development and implementation of 

an information management and 

information technology strategy for 

improving healthcare services: a case 

study. J Healthc Inf Manag 

2001;15(3):261-85. 

[Rec#: 558] 

 346. Mendelson DN, Goodman CS, Ahn R, et al. 

Outcomes and effectiveness research 

in the private sector. Health Aff 

(Millwood) 1998;17(5):75-90. 

[Rec#: 977] 

 347. Mendelson DN, Salinsky EM. Health 

information systems and the role of 

state government. Health Aff 

(Millwood) 1997;16(3):106-19. 

[Rec#: 981] 

 348. Merkouris AV. Computer-based 

documentation and bedside terminals. 

J Nurs Manag 1995;3(2):81-5. 

[Rec#: 306] 

 349. Messina E. Eliminating serious risks in the 

outpatient clinic with an electronic 

medical record. J Healthc Risk 

Manag 1995;15(4):20-4. 

[Rec#: 817] 

 350. Metfessel BA. An automated tool for an 

analysis of compliance to evidencebased clinical guidelines. Medinfo 

2001;10(Pt 1):226-30. 

[Rec#: 307] 
G-21

 351. Meyer K, Dimond M, Shaver J, et al. The 

'Learnmore' project: information 

acquisition solutions for the longterm care information system. 

Medinfo 1995;8 Pt 1:276-9. 

[Rec#: 308] 

 352. Meyer TJ, Van Kooten D, Marsh S, et al. 

Reduction of polypharmacy by 

feedback to clinicians. J Gen Intern 

Med 1991;6(2):133-6. 

[Rec#: 736] 

 353. Michael P, Pritchett E. The impact of HIPAA 

electronic transmissions and health 

information privacy standards. J Am 

Diet Assoc 2001;101(5):524-8. 

[Rec#: 559] 

 354. Middleton G. The electronic revolution. 

Report writing in the 21st century. 

Emerg Med Serv 2002;31(8):65-8. 

[Rec#: 560] 

 355. Mildon J, Cohen T. Drivers in the electronic 

medical records market. Health 

Manag Technol 2001;22(5):14-6, 18. 

[Rec#: 561] 

 356. Miller RH, Sim I. Physicians' use of electronic 

medical records: barriers and 

solutions. Health Aff (Millwood) 

2004;23(2):116-26. 

[Rec#: 953] 

 357. Miller RA, Gardner RM. Recommendations 

for responsible monitoring and 

regulation of clinical software 

systems. American Medical 

Informatics Association, Computerbased Patient Record Institute, 

Medical Library Association, 

Association of Academic Health 

Science Libraries, American Health 

Information Management 

Association, American Nurses 

Association. J Am Med Inform Assoc 

1997;4(6):442-57. 

[Rec#: 309] 

 358. Miller RA, Gardner RM. Summary 

recommendations for responsible 

monitoring and regulation of clinical 

software systems. American Medical 

Informatics Association, The 

Computer-based Patient Record 

Institute, The Medical Library 

Association, The Association of 

Academic Health Science Libraries, 

The American Health Information 

Management Association, and The 

American Nurses Association. Ann 

Intern Med 1997;127(9):842-5. 

[Rec#: 310] 

 359. Miller SR. I think I scan. Case study: 

implementing the computerized 

patient record--not an EMR. MGMA 

Connex 2003;3(2):46-9, 1. 

[Rec#: 562] 

 360. Miller TE, Derse AR. Between strangers: the 

practice of medicine online. Health 

Aff (Millwood) 2002;21(4):168-79. 

[Rec#: 964] 

 361. Millonig MK, Jackson TL, Ellis WM. 

Improving medication use through 

pharmacists' access to patient-specific 

health care information. J Am Pharm 

Assoc (Wash) 2002;42(4):638-45. 

[Rec#: 311] 

 362. Milstein A, Galvin RS, Delbanco SF, et al. 

Improving the safety of health care: 

the leapfrog initiative. Eff Clin Pract 

2000;3(6):313-6. 

[Rec#: 872] 

 363. Mitchell RD, Jessup S. Staying one step 

ahead--development of an electronic 

record. Caring 1995;14(8):34, 37, 39-

41. 

[Rec#: 818] 

 364. Moehr JR. Informatics in the service of health, 

a look to the future. Methods Inf Med 

1998;37(2):165-70. 

[Rec#: 313] 

 365. Mohr DN. Benefits of an electronic clinical 

information system. Healthc Inf 

Manage 1997;11(4):49-57. 

[Rec#: 819] 
G-22

 366. Moidu K, Wigertz O, Trell E. Multi centre 

systems analysis study of primary 

health care: a study of socioorganizational and human factors. Int 

J Biomed Comput 1992;30(1):27-42. 

[Rec#: 899] 

 367. Monane M, Matthias DM, Nagle BA, et al. 

Improving prescribing patterns for the 

elderly through an online drug 

utilization review intervention: a 

system linking the physician, 

pharmacist, and computer. JAMA 

1998;280(14):1249-52. 

[Rec#: 314] 

 368. Montoya ID, Carlson JW. Point-of-care 

systems, informatics, and health care 

delivery. Health Care Superv 1996;15 

(2):17-26. 

[Rec#: 822] 

 369. Moran DW. Health information policy: on 

preparing for the next war. Health Aff 

(Millwood) 1998;17(6):9-22. 

[Rec#: 976] 

 370. Morris AH. Decision support and safety of 

clinical environments. Qual Saf 

Health Care 2002;11(1):69-75. 

[Rec#: 317] 

 371. Morris AH. Rational use of computerized 

protocols in the intensive care unit. 

Crit Care 2001;5(5):249-54. 

[Rec#: 318] 

 372. Morrissey J. Out to set the record. With a push 

from HHS, the effort to create 

electronic medical records continues 

to build momentum. Mod Healthc 

2003;33(42):28-32, 35. 

[Rec#: 563] 

 373. Muller CR. Computer applications in the 

search for unrelated stem cell donors. 

Transpl Immunol 2002;10(2-3):227-

40. 

[Rec#: 319] 

 374. Mullooly JP. Misclassification model for 

person-time analysis of automated 

medical care databases. Am J 

Epidemiol 1996;144(8):782-92. 

[Rec#: 320] 

 375. Munter KH, Schenk JF, Thrun F, et al. The 

"Phoenix" ADR database of the Drug 

Commission of the German Medical 

Profession--a clinically useful 

approach to optimize evidence-based 

medicine in Germany. Semin Thromb 

Hemost 1999;25(1):57-64. 

[Rec#: 321] 

 376. Murray D. Electronic records solved this 

practice's problems. Med Econ 

1996;73(10):135-6, 142-4, 147. 

[Rec#: 823] 

 377. Neame R. Smart cards--the key to trustworthy 

health information systems . BMJ 

1997;314(7080):573-7. 

[Rec#: 825] 

 378. Nenov V, Klopp J. Remote analysis of 

physiological data from neurosurgical 

ICU patients. J Am Med Inform 

Assoc 1996;3(5):318-27. 

[Rec#: 322] 

 379. Nerenz DR. Capacities and limitations of 

information systems as data sources 

on quality of care at the end of life. J 

Pain Symptom Manage 

2001;22(3):773-83. 

[Rec#: 323] 

 380. Nicolosi E, Molino F, Molino G. 

Computerized approach to active and 

objective quality control of patient 

care. Int J Biomed Comput 

1996;42(1-2):59-66. 

[Rec#: 826] 

 381. Nilsson G, Ahlfeldt H, Strender LE. 

Computerisation, coding, data 

retrieval and related attitudes among 

Swedish general practitioners-a 

survey of necessary conditions for a 

database of diseases and health 

problems. Int J Med Inf 

2002;65(2):135-43. 

[Rec#: 325] 

 382. Nixon SJ, Munro MM, Sands I, et al. Is 

Intranet technology the low cost, 

quick fix solution for Scotland's 

clinical information technology 

chaos? Results of a pilot study. Scott 

Med J 2000;45(5):140-3. 

[Rec#: 566] 
G-23

 383. Njalsson T. On content of practice. The 

advantage of computerized 

information systems in family 

practice. Scand J Prim Health Care 

1995;13 Suppl 1:1-102. 

[Rec#: 827] 

 384. Norrie P, Blackwell RE. Constructing a 

financial case for a computerized 

patient data management system 

(PDMS) in a cardiothoracic intensive 

care unit. Med Inform Internet Med 

2000;25(3):161-9. 

[Rec#: 567] 

 385. O'Brien L. Setting the standard. West Coast 

healthcare network mandates ERP 

system to ensure business 

standardization in four states. Health 

Manag Technol 2003;24(9):18-21. 

[Rec#: 568] 

 386. O'Connell RT, Cho C, Shah N, et al. Take 

note(s): differential EHR satisfaction 

with two implementations under one 

roof. J Am Med Inform Assoc 

2004;11(1):43-9. 

[Rec#: 917] 

 387. O'Connor AM, Rostom A, Fiset V, et al. 

Decision aids for patients facing 

health treatment or screening 

decisions: systematic review. Bmj 

1999;319(7212):731-4. 

[Rec#: 326] 

 388. O'Connor AM, Stacey D, Entwistle V, et al. 

Decision aids for people facing health 

treatment or screening decisions. 

Cochrane Database Syst Rev 

2003;(2):CD001431. 

[Rec#: 327] 

 389. O'Connor K. Information management for 

managed care. Med Group Manage J 

1995;42(6):52-4, 56, 75. 

[Rec#: 829] 

 390. O'Neil AC, Petersen LA, Cook EF, et al. 

Physician reporting compared with 

medical-record review to identify 

adverse medical events. Ann Intern 

Med 1993;119(5):370-376. 

[Rec#: 29] 

 391. Ohno-Machado L, Wang SJ, Mar P, et al. 

Decision support for clinical trial 

eligibility determination in breast 

cancer. Proc AMIA Symp 1999;340-

4. 

[Rec#: 100] 

 392. Oostendorp A, Hoekstra S, Aarts J. Exploring 

factors required for successful 

implementation of healthcare 

informatics. Stud Health Technol 

Inform 1998;51:146-51. 

[Rec#: 867] 

 393. Ornstein S, Bearden A. Patient perspectives on 

computer-based medical records. J 

Fam Pract 1994;38(6):606-10. 

[Rec#: 900] 

 394. Ornstein SM. Translating research into 

practice using electronic medical 

records the PPRNet-TRIP project: 

primary and secondary prevention of 

coronary heart disease and stroke. 

Top Health Inf Manage 

2001;22(2):52-8. 

[Rec#: 330] 

 395. Ornstein SM, Garr DR, Jenkins RG. A 

comprehensive microcomputer-based 

medical records system with 

sophisticated preventive services 

features for the family physician. J 

Am Board Fam Pract 1993;6 (1):55-

60. 

[Rec#: 713] 

 396. Orson J. Using the Internet to enhance 

evidence-based practice. Br J 

Perioper Nurs 2000;10(8):421-7. 

[Rec#: 331] 

 397. Overhage JM, Middleton B, Miller RA, et al. 

Does national regulatory mandate of 

provider order entry portend greater 

benefit than risk for health care 

delivery? The 2001 ACMI debate. 

The American College of Medical 

Informatics. J Am Med Inform Assoc 

2002;9(3):199-208. 

[Rec#: 101] 
G-24

 398. Overhage JM, Tierney WM, McDonald CJ. 

Design and implementation of the 

Indianapolis Network for Patient Care 

and Research. Bull Med Libr Assoc 

1995;83(1):48-56. 

[Rec#: 681] 

 399. Owen RR, Thrush CR, Cannon D, et al. Use of 

electronic medical record data for 

quality improvement in schizophrenia 

treatment. J Am Med Inform Assoc 

2004;11(5):351-7. 

[Rec#: 945] 

 400. Owens DK. Analytic tools for public health 

decision making. Med Decis Making 

2002;22(5 Suppl):S3-10. 

[Rec#: 332] 

 401. Ozbolt JG. From minimum data to maximum 

impact: using clinical data to 

strengthen patient care. MD Comput 

1997;14(4):295-301. 

[Rec#: 831] 

 402. Pace MA. At home with coding. Western 

healthcare system improves revenue 

cycle management via home coders. 

Health Manag Technol 

2003;24(9):22-3. 

[Rec#: 570] 

 403. Pakpahan R, Balas EA, Boren SA. 

Computable decision modules for 

patient safety in child health care. 

Proc AMIA Symp 2002;592-6. 

[Rec#: 333] 

 404. Papaconstantinou C, Theocharous G, 

Mahadevan S. An expert system for 

assigning patients into clinical trials 

based on Bayesian networks. J Med 

Syst 1998;22(3):189-202. 

[Rec#: 105] 

 405. Parente ST. Proprietary data systems: help or 

hindrance? Health Aff (Millwood) 

1997;16(5):218-20. 

[Rec#: 979] 

 406. Parker B, Harman A. No boundaries. 

Advanced telecom solutions enable 

secure storage and transfer of 

electronic medical records over great 

distances. Health Manag Technol 

2003;24(8):37-8. 

[Rec#: 571] 

 407. Parker RM, Ratzan SC, Lurie N . Health 

literacy: a policy challenge for 

advancing high-quality health care. 

Health Aff (Millwood) 

2003;22(4):147-53. 

[Rec#: 957] 

 408. Parmigiani G. Measuring uncertainty in 

complex decision analysis models. 

Stat Methods Med Res 

2002;11(6):513-37. 

[Rec#: 334] 

 409. Pavlin JA. Investigation of disease outbreaks 

detected by "syndromic" surveillance 

systems. J Urban Health 2003;80 (2 

Suppl 1):i107-14. 

[Rec#: 106] 

 410. Pavlopoulos S, Berler A, Kyriacou E, et al. 

Design and development of a 

multimedia database for emergency 

telemedicine. Technol Health Care 

1998;6(2-3):101-10. 

[Rec#: 335] 

 411. Pawlson G. Information technologies and the 

health care workforce. Health Aff 

(Millwood) 2002;21(6):266; author 

reply 267-8. 

[Rec#: 960] 

 412. Pennachio DL. Saving way more than a buck 

or two. Med Econ 2003;80(11):84, 

87. 

[Rec#: 572] 

 413. Persson M, Bohlin J, Eklund P . Development 

and maintenance of guideline-based 

decision support for pharmacological 

treatment of hypertension. Comput 

Methods Programs Biomed 

2000;61(3):209-19. 

[Rec#: 336] 

 414. Petersen LA, Brennan TA, O'Neil AC, et al. 

Does housestaff discontinuity of care 

increase the risk for preventable 

adverse events? Ann Intern Med 

1994;121(11):866-72. 

[Rec#: 107] 
G-25

 415. Petrick N, Sahiner B, Chan HP, et al. Breast 

cancer detection: evaluation of a 

mass-detection algorithm for 

computer-aided diagnosis -- 

experience in 263 patients. Radiology 

2002;224(1):217-24. 

[Rec#: 108] 

 416. Philp I, Goddard A, Connell NA, et al. 

Development and evaluation of an 

information system for quality 

assurance. Age Ageing 

1994;23(2):150-153. 

[Rec#: 33] 

 417. Piccoli A. Elementary clinical decision 

analysis in evidence-based 

nephrology. J Nephrol 

2000;13(6):419-32. 

[Rec#: 338] 

 418. Pifer EA, Smith S, Keever GW. EMR to the 

rescue. An ambulatory care pilot 

project shows that data sharing equals 

cost shaving. Healthc Inform 

2001;18(2):111-4. 

[Rec#: 573] 

 419. Pincus HA, McQueen L. The computer age: 

beware the loss of the narrative. Gen 

Hosp Psychiatry 1996;18(4):284-5. 

[Rec#: 833] 

 420. Pliskin N, Glezerman M, Modai I, et al. 

Spreadsheet evaluation of 

computerized medical records: the 

impact on quality, time, and money. J 

Med Syst 1996;20(2):85-100. 

[Rec#: 834] 

 421. Pollack R. What works. Computerized patient 

record cuts redundant documentation, 

improves charting. Health Manag 

Technol 1997;18(9):29. 

[Rec#: 835] 

 422. Poon EG, Blumenthal D, Jaggi T, et al. 

Overcoming barriers to adopting and 

implementing computerized 

physician order entry systems in U.S. 

hospitals. Health Aff (Millwood) 

2004;23(4):184-90. 

[Rec#: 952] 

 423. Powsner SM, Wyatt JC, Wright P. 

Opportunities for and challenges of 

computerisation. Lancet 

1998;352(9140):1617-22. 

[Rec#: 574] 

 424. Pringle M, Robins S, Brown G. Computers in 

the surgery. The patient's view. Br 

Med J (Clin Res Ed) 

1984;288(6413):289-91. 

[Rec#: 902] 

 425. Pringle M, Robins S, Brown G. Topic 

analysis: an objective measure of the 

consultation and its application to 

computer assisted consultations. Br 

Med J (Clin Res Ed) 

1985;290(6484):1789-91. 

[Rec#: 687] 

 426. Protti D. What can the American electronic 

health record (EHR) pioneers tell us 

about what it takes to be successful? 

Healthc Manage Forum 

2002;15(2):33-5. 

[Rec#: 575] 

 427. Purves I. Computerised guidelines in primary 

health care: reflections and 

implications. Stud Health Technol 

Inform 1995;16:57-74. 

[Rec#: 340] 

 428. Rader RW, Toalson TW. Ready or not, handheld prescribing is coming. J Okla 

State Med Assoc 2001;94(9):403-8. 

[Rec#: 342] 

 429. Raghupathi W, Tan J. Strategic uses of 

information technology in health 

care: a state-of-the-art survey. Top 

Health Inf Manage 1999;20(1):1-15. 

[Rec#: 343] 

 430. Ralston JD, Revere D, Robins LS, et al. 

Patients' experience with a diabetes 

support programme based on an 

interactive electronic medical record: 

qualitative study. BMJ 

2004;328(7449):1159. 

[Rec#: 921] 

 431. Ramos JD, Rai-Chaudhuri A, Neill RW. 

International online discussion lists 

on chronic myelogenous leukaemia. 

BMJ 2004;328(7449):1177-8. 
G-26

[Rec#: 927] 

 432. Ramsaroop P, Ball MJ. The "bank of health". 

A model for more useful patient 

health records. MD Comput 

2000;17(4):45-8. 

[Rec#: 577] 

 433. Ramsey SD, McIntosh M, Etzioni R, et al. 

Simulation modeling of outcomes and 

cost effectiveness. Hematol Oncol 

Clin North Am 2000;14(4):925-38. 

[Rec#: 344] 

 434. Ranke MB, Dowie J. KIGS and KIMS as tools 

for evidence-based medicine. Horm 

Res 1999;51 Suppl 1:83-6. 

[Rec#: 345] 

 435. Reed H. Streamlining patient access with a 

paperless system. NAHAM Manage J 

1997;23(4):12, 15. 

[Rec#: 836] 

 436. Rehm S, Kraft S. Electronic medical records: 

the FPM vendor survey . Fam Pract 

Manag 2001;8(1):45-54. 

[Rec#: 579] 

 437. Renner K. Electronic medical records in the 

outpatient setting (Part 1) . Med 

Group Manage J 1996;43(3):52, 54, 

56-7 passim. 

[Rec#: 838] 

 438. Renner K. Electronic medical records in the 

outpatient setting (Part 2) . Med 

Group Manage J 1996;43(4):60-5, 82. 

[Rec#: 839] 

 439. Retchin SM, Wenzel RP. Electronic medical 

record systems at academic health 

centers: advantages and 

implementation issues. Acad Med 

1999;74(5):493-8. 

[Rec#: 580] 

 440. Rettig RA. Are patients a scarce resource for 

academic clinical research? Health 

Aff (Millwood) 2000;19(6):195-205. 

[Rec#: 968] 

 441. Rhodes H, Dougherty M. Practice brief. 

Document imaging as a bridge to the 

EHR. J Ahima 2003;74(6):56A-56G. 

[Rec#: 347] 

 442. Ridsdale L, Hudd S. Computers in the 

consultation: the patient's view . Br J 

Gen Pract 1994;44(385):367-9. 

[Rec#: 904] 

 443. Robinson A, Thomson RG. The potential use 

of decision analysis to support shared 

decision making in the face of 

uncertainty: the example of atrial 

fibrillation and warfarin 

anticoagulation. Qual Health Care 

2000;9(4): 238-44. 

[Rec#: 349] 

 444. Robinson TN, Patrick K, Eng TR, et al. An 

evidence-based approach to 

interactive health communication: a 

challenge to medicine in the 

information age. Science Panel on 

Interactive Communication and 

Health. Jama 1998;280(14):1264-9. 

[Rec#: 350] 

 445. Rogoski RR. The ABCs of CPRs and EMRs. 

Definitions, semantics, product 

differences and the need for 

demonstrated results hinder widescale adoption . Health Manag 

Technol 2002;23(5):14-6, 19. 

[Rec#: 583] 

 446. Rogoski RR. Having it your way. Three 

physicians who were early adopters 

and who remain vocal EMR 

advocates share their insights about 

the utilization, challenges and 

benefits of EMRs. Health Manag 

Technol 2003;24(5):12-6. 

[Rec#: 585] 

 447. Rogoski RR. Opening the floodgates of 

usability. Clinical information 

systems allow free flow of patient 

data to clinicians when and where 

they need it. Health Manag Technol 

2003;24(9):12-7. 

[Rec#: 586] 
G-27

 448. Rogoski RR. You say tomato and... While the 

practice management landscape has 

blossomed with progressive 

technology, still differences exist in 

issues of EMR integration, and 

HIPAA endures as the looming 

challenge of the decade . Health 

Manag Technol 2002;23(10):24-6. 

[Rec#: 584] 

 449. Roland MO, Zander LI, Evans M, et al. 

Evaluation of a computer assisted 

repeat prescribing programme in a 

general practice. Br Med J (Clin Res 

Ed) 1985;291(6493):456-8. 

[Rec#: 690] 

 450. Rollman BL, Gilbert T, Lowe HJ, et al. The 

electronic medical record: its role in 

disseminating depression guidelines 

in primary care practice. Int J 

Psychiatry Med 1999;29(3):267-86. 

[Rec#: 351] 

 451. Roos NP, Black C, Roos LL, et al. Managing 

health services: how administrative 

data and population-based analyses 

can focus the agenda. Health Serv 

Manage Res 1998;11(1):49-67. 

[Rec#: 353] 

 452. Roos NP, Black C, Roos LL, et al. Managing 

health services: how the Population 

Health Information System 

(POPULIS) works for policymakers. 

Med Care 1999;37(6 Suppl):JS27-41. 

[Rec#: 354] 

 453. Roper WL, Mays G. A maturing market for 

health information technology? 

Health Aff (Millwood) 

1999;18(2):255-6. 

[Rec#: 972] 

 454. Rosenfeld S , Zeitler E, Mendelson D. 

Financial Incentives: Innovative 

Payment for Health Information 

Technology. 2004. [Rec#: 941] 

 455. Rosenquist PB, Colenda CC, Briggs J, et al. 

Riding a Trojan horse: computerized 

psychiatric treatment planning using 

managed care principles. Manag Care 

Q 1996;4(4):89-95. 

[Rec#: 356] 

 456. Rosenstein AH. Using information 

management to implement a clinical 

resource management program. Jt 

Comm J Qual Improv 

1997;23(12):653-66. 

[Rec#: 359] 

 457. Rosse C. Information technology: value to 

patients. Health Aff (Millwood) 

1999;18(2):256. 

[Rec#: 971] 

 458. Rosser WW, McDowell I, Newell C. Use of 

reminders for preventive procedures 

in family medicine. CMAJ 

1991;145(7):807-14. 

[Rec#: 148] 

 459. Ruffin M. Information Technology, Part 1: A 

way to streamline medical practice. 

Physician Exec 1996;22(7):37-40. 

[Rec#: 840] 

 460. Russell J, Greenhalgh T, Boynton P, et al. Soft 

networks for bridging the gap 

between research and practice: 

illuminative evaluation of CHAIN. 

BMJ 2004;328(7449):1174. 

[Rec#: 926] 

 461. Sackett DL, Straus SE. Finding and applying 

evidence during clinical rounds: the 

"evidence cart". Jama 

1998;280(15):1336-8. 

[Rec#: 360] 

 462. Safran C, Rind DM, Davis RB, et al. Effects of 

a knowledge-based electronic patient 

record in adherence to practice 

guidelines. MD Comput 

1996;13(1):55-63. 

[Rec#: 908] 

 463. Samsa GP, Reutter RA, Parmigiani G, et al. 

Performing cost-effectiveness 

analysis by integrating randomized 

trial data with a comprehensive 

decision model: application to 

treatment of acute ischemic stroke. J 

Clin Epidemiol 1999;52(3):259-71. 

[Rec#: 361] 
G-28

 464. Sanders GD, Hagerty CG, Sonnenberg FA, et 

al. Distributed decision support using 

a web-based interface: prevention of 

sudden cardiac death. Med Decis 

Making 1999;19(2):157-66. 

[Rec#: 362] 

 465. Sandrick K. Calculating ROI for CPRs. Health 

Manag Technol 1998;19(6):16-20. 

[Rec#: 589] 

 466. Sarasin FP. Decision analysis and the 

implementation of evidence-based 

medicine. Qjm 1999;92(11):669-71. 

[Rec#: 363] 

 467. Sargent JA. Identifying the value of computers 

in dialysis. Blood Purif 

2002;20(1):11-9. 

[Rec#: 590] 

 468. Satava RM. Medical applications of virtual 

reality. J Med Syst 1995;19(3):275-

80. 

[Rec#: 364] 

 469. Sausser GD. Thin is in: web-based systems 

enhance security, clinical quality. 

Healthc Financ Manage 2003;57 

(7):86-9. 

[Rec#: 591] 

 470. Scalise D. CPOE (computerized physician 

order entry). An executive's guide. 

Hosp Health Netw 2002;76(6):41-6. 

[Rec#: 592] 

 471. Schadow G, Russler DC, McDonald CJ. 

Conceptual alignment of electronic 

health record data with guideline and 

workflow knowledge. Int J Med Inf 

2001;64(2-3):259-74. 

[Rec#: 593] 

 472. Scharf L. Revising nursing documentation to 

meet patient outcomes. Nurs Manage 

1997;28(4):38-9. 

[Rec#: 693] 

 473. Schiff GD, Klass D, Peterson J, et al. Linking 

laboratory and pharmacy: 

opportunities for reducing errors and 

improving care. Arch Intern Med 

2003;163(8):893-900. 

[Rec#: 113] 

 474. Schiff GD, Rucker TD. Computerized 

prescribing: building the electronic 

infrastructure for better medication 

usage. JAMA 1998;279(13):1024-9. 

[Rec#: 114] 

 475. Schleyer T, Spallek H. Dental informatics. A 

cornerstone of dental practice. J Am 

Dent Assoc 2001;132(5):605-13. 

[Rec#: 365] 

 476. Schmidt IK, Svarstad BL. Nurse-physician 

communication and quality of drug 

use in Swedish nursing homes. Soc 

Sci Med 2002;54(12):1767-77. 

[Rec#: 115] 

 477. Schmitt KF, Wofford DA. Financial analysis 

projects clear returns from electronic 

medical records. Healthc Financ 

Manage 2002;56(1):52-7. 

[Rec#: 594] 

 478. Schmitz HH. A protocol for evaluating 

hospital information systems. Hosp 

Health Serv Adm 1977;22(1):45-56. 

[Rec#: 35] 

 479. Schneider EC, Eisenberg JM. Strategies and 

methods for aligning current and best 

medical practices. The role of 

information technologies. West J 

Med 1998;168(5 ):311-8. 

[Rec#: 366] 

 480. Schneider EC, Riehl V, Courte-Wienecke S, et 

al. Enhancing performance 

measurement: NCQA's road map for 

a health information framework. 

National Committee for Quality 

Assurance. JAMA 

1999;282(12):1184-90. 

[Rec#: 595] 

 481. Schneider JH. Online personal medical 

records: are they reliable for 

acute/critical care? Crit Care Med 

2001;29(8 Suppl):N196-201. 

[Rec#: 367] 

 482. Schuster DM, Hall SE, Couse CB, et al. 

Involving users in the implementation 

of an imaging order entry system. J 

Am Med Inform Assoc 

2003;10(4):315-21. 

[Rec#: 116] 
G-29

 483. Sciamanna CN, Ford DE, Flynn JA, et al. An 

evidence-based interactive computer 

program to assist physicians in 

counseling smokers to quit. MD 

Comput 1999;16(5):54-60. 

[Rec#: 369] 

 484. Scott JS. EDI plays major role in coordinating 

care. Health Manag Technol 

1995;16(2):36-8, 40. 

[Rec#: 841] 

 485. Sennett C. Moving ahead, measure by 

measure. Health Aff (Millwood) 

1998;17(4):36-7. 

[Rec#: 978] 

 486. Seroussi B, Bouaud J, Dreau H, et al. ASTI: a 

guideline-based drug-ordering system 

for primary care. Medinfo 2001;10(Pt 

1):528-32. 

[Rec#: 370] 

 487. Shafazand S, Shigemitsu H, Weinacker AB. A 

brave new world: remote intensive 

care unit for the 21st century. Crit 

Care Med 2000;28(12):3945-6. 

[Rec#: 117] 

 488. Shaffer DW, Kigin CM, Kaput JJ, et al. What 

is digital medicine? Stud Health 

Technol Inform 2002;80:195-204. 

[Rec#: 371] 

 489. Shaikh I. What works. New VA facility uses 

paperless medical records systems to 

compete. Health Manag Technol 

1997;18(7):28. 

[Rec#: 842] 

 490. Shani M. The impact of information on 

medical thinking and health care 

policy. Int J Med Inf 2000;58-59:3-

10. 

[Rec#: 373] 

 491. Shea S, Starren J, Weinstock RS, et al. 

Columbia University's Informatics for 

Diabetes Education and Telemedicine 

(IDEATel) Project: rationale and 

design. J Am Med Inform Assoc 

2002;9(1):49-62. 

[Rec#: 119] 

 492. Shiffman RN, Brandt CA, Liaw Y, et al. A 

design model for computer-based 

guideline implementation based on 

information management services. J 

Am Med Inform Assoc 1999;6(2):99-

103. 

[Rec#: 374] 

 493. Shojania KG , Duncan BW, McDonald KM, et 

al. Making health care safer: a critical 

analysis of patient safety practices. 

Evid Rep Technol Assess (Summ) 

2001;(43):i-x, 1-668. 

[Rec#: 992] 

 494. Shojania KG, Duncan BW, McDonald KM, et 

al. Safe but sound. Patient safety 

meets evidence-based medicine. 

JAMA 2002;288(4):508-513. 

[Rec#: 169] 

 495. Sicotte C, Denis JL, Lehoux P . The computer 

based patient record: a strategic issue 

in process innovation. J Med Syst 

1998;22(6):431-43. 

[Rec#: 597] 

 496. Sim I, Gorman P, Greenes RA, et al. Clinical 

decision support systems for the 

practice of evidence-based medicine. 

J Am Med Inform Assoc 

2001;8(6):527-34. 

[Rec#: 376] 

 497. Sim I, Sanders GD, McDonald KM. Evidencebased practice for mere mortals: the 

role of informatics and health services 

research. J Gen Intern Med 

2002;17(4):302-8. 

[Rec#: 377] 

 498. Sirio CA, Segel KT, Keyser DJ, et al. 

Pittsburgh Regional Healthcare 

Initiative: a systems approach for 

achieving perfect patient care. Health 

Aff (Millwood) 2003;22(5):157-65. 

[Rec#: 956] 

 499. Sittig DF, Stead WW. Computer-based 

physician order entry: the state of the 

art. J Am Med Inform Assoc 

1994;1(2):108-23. 

[Rec#: 121] 
G-30

 500. Siwicki B. Outpatient electronic records. 

Tackling the challenges. Health Data 

Manag 1997;5(5):66-8, 79, 74 

passim. 

[Rec#: 843] 

 501. Smith B. Computer-based patient records 

support organizational redesign . 

Hosp Technol Ser 1997;16(7):2-4. 

[Rec#: 845] 

 502. Smith D, Newell LM. Deploying the EMR: 

Part 2, Implementation. J Med Pract 

Manage 2003;18(6):304-8. 

[Rec#: 598] 

 503. Smith J. The case for capitalism in healthcare. 

Health Manag Technol 

2001;22(6):68, 67. 

[Rec#: 599] 

 504. Solomon GL, Dechter M. Are patients pleased 

with computer use in the examination 

room? J Fam Pract 1995;41(3):241-4. 

[Rec#: 910] 

 505. Sonksen P, Williams C. Information 

technology in diabetes care 'Diabeta': 

23 years of development and use of a 

computer-based record for diabetes 

care. Int J Biomed Comput 

1996;42(1-2):67-77. 

[Rec#: 846] 

 506. Soper WD. Why I love my EMR. Fam Pract 

Manag 2002;9(9):35-8. 

[Rec#: 600] 

 507. Soto CM, Kleinman KP, Simon SR. Quality 

and correlates of medical record 

documentation in the ambulatory care 

setting. BMC Health Serv Res 

2002;2(1):22. 

[Rec#: 601] 

 508. Staccini P, Joubert M, Quaranta JF, et al. 

Modelling health care processes for 

eliciting user requirements: a way to 

link a quality paradigm and clinical 

information system design. Int J Med 

Inf 2001;64(2-3):129-42. 

[Rec#: 602] 

 509. Staender S, Davies J, Helmreich B, et al. The 

anaesthesia critical incident reporting 

system: an experience based database. 

Int J Med Inf 1997;47(1-2):87-90. 

[Rec#: 379] 

 510. Stair TO. Reduction of redundant laboratory 

orders by access to computerized 

patient records. J Emerg Med 

1998;16(6):895-7. 

[Rec#: 122] 

 511. Stammer L. Chart pulling brought to its knees. 

Healthc Inform 2001;18(2):107-8. 

[Rec#: 603] 

 512. Stanfill PH, Weber DG, Wayne C. Clinical 

patient information capture: the 

Columbia/HCA cancer care database 

experience. J Oncol Manag 

1998;7(2):17-21. 

[Rec#: 380] 

 513. Starr P. Smart technology, stunted policy: 

developing health information 

networks. Health Aff (Millwood) 

1997;16 (3):91-105. 

[Rec#: 982] 

 514. Starren J, Hripcsak G, Sengupta S, et al. 

Columbia University's Informatics for 

Diabetes Education and Telemedicine 

(IDEATel) project: technical 

implementation. J Am Med Inform 

Assoc 2002;9(1):25-36. 

[Rec#: 120] 

 515. Stausberg J, Koch D, Ingenerf J, et al. 

Comparing paper-based with 

electronic patient records: lessons 

learned during a study on diagnosis 

and procedure codes. J Am Med 

Inform Assoc 2003;10 (5):470-7. 

[Rec#: 604] 

 516. Stefanelli M. Knowledge management to 

support performance-based medicine. 

Methods Inf Med 2002;41(1):36-43. 

[Rec#: 381] 

 517. Stevens B. Development and testing of a 

pediatric pain management sheet . 

Pediatr Nurs 1990;16(6):543-8. 

[Rec#: 694] 
G-31

 518. Stricklin ML, Struk CM. Point of care 

technology: a sociotechnical approach 

to home health implementation. 

Methods Inf Med 2003;42(4):463-70. 

[Rec#: 605] 

 519. Struk C. Critical steps for integrating 

information technology in home care: 

one agency's experience. Home 

Healthc Nurse 2001;19(12):758-65. 

[Rec#: 382] 

 520. Sujansky WV. The benefits and challenges of 

an electronic medical record: much 

more than a "word-processed" patient 

chart. West J Med 1998;169(3):176-

83. 

[Rec#: 383] 

 521. Sullivan M. The CPR's hidden impact: 

reducing malpractice risk. Healthc 

Inform 1996;13(6):36, 38, 40. 

[Rec#: 849] 

 522. Swanson T, Dostal J, Eichhorst B, et al. 

Recent implementations of electronic 

medical records in four family 

practice residency programs. Acad 

Med 1997;72(7):607-12. 

[Rec#: 851] 

 523. Sylvestri MF. Health care informatics: the key 

to successful disease management. 

Med Interface 1996;9(5):94-6, 98-9. 

[Rec#: 852] 

 524. Szynal D. Doctors become palm readers. 

Health Data Manag 2001;9(6):20, 23, 

25. 

[Rec#: 606] 

 525. Taigman M. Ending the paper trail. Electronic 

documentation in EMS. Emerg Med 

Serv 2003;32(3):75-8. 

[Rec#: 607] 

 526. Takeda H, Endoh H. Commentary on 'health 

care in the information society. A 

prognosis for the year 2013'. Int J 

Med Inf 2002;66(1-3):107-11. 

[Rec#: 386] 

 527. Takeda H, Matsumura Y, Nakajima K, et al. 

Health care quality management by 

means of an incident report system 

and an electronic patient record 

system. Int J Med Inf 2003;69(2-

3):285-93. 

[Rec#: 608] 

 528. Tang PC. AMIA advocates national health 

information system in fight against 

national health threats. J Am Med 

Inform Assoc 2002;9(2):123-4. 

[Rec#: 124] 

 529. Tang PC, Fafchamps D, Shortliffe EH. 

Traditional medical records as a 

source of clinical data in the 

outpatient setting. Proc Annu Symp 

Comput Appl Med Care 1994;575-9. 

[Rec#: 125] 

 530. Tang PC, LaRosa MP, Gorden SM . Use of 

computer-based records, 

completeness of documentation, and 

appropriateness of documented 

clinical decisions. J Am Med Inform 

Assoc 1999;6(3):245-51. 

[Rec#: 126] 

 531. Tang PC, LaRosa MP, Newcomb C, et al. 

Measuring the effects of reminders 

for outpatient influenza 

immunizations at the point of clinical 

opportunity. J Am Med Inform Assoc 

1999;6(2):115-21. 

[Rec#: 127] 

 532. Teich JM, Glaser JP, Beckley RF, et al. The 

Brigham integrated computing 

system (BICS): advanced clinical 

systems in an academic hospital 

environment. Int J Med Inf 

1999;54(3):197-208. 

[Rec#: 390] 

 533. Teich JM, Spurr CD, Schmiz JL, et al. 

Enhancement of clinician workflow 

with computer order entry. Proc Annu 

Symp Comput Appl Med Care 

1995;459-463. 

[Rec#: 40] 

 534. Teich JM, Wagner MM, Mackenzie CF, et al. 

The informatics response in disaster, 

terrorism, and war. J Am Med Inform 

Assoc 2002;9(2):97-104. 
G-32

[Rec#: 129] 

 535. Terry K. EMRs boost efficiency, too. Med 

Econ 2002;79(7):38-40, 43. 

[Rec#: 611] 

 536. Terry K. EMRS cost too much? This group 

says no way! Med Econ 

2002;79(7):34-9. 

[Rec#: 612] 

 537. Terry K. EMRs. What you need to know. Med 

Econ 2003;80(9):TCP5-TCP14. 

[Rec#: 613] 

 538. Terry K. How connectivity is changing 

practice. Med Econ 2000;77(13):166-

70, 173. 

[Rec#: 609] 

 539. Terry K. The move to put office orders online. 

Med Econ 2003;80(15):23-4. 

[Rec#: 614] 

 540. Terry K. Will the Internet finally put an end to 

paperwork? Med Econ 

2000;77(13):152-4, 159-60, 163-5. 

[Rec#: 610] 

 541. Terry NP. An eHealth diptych: the impact of 

privacy regulation on medical error 

and malpractice litigation. Am J Law 

Med 2001;27(4):361-419. 

[Rec#: 391] 

 542. Thompson C, McCaughan D, Cullum N, et al. 

Research information in nurses' 

clinical decision-making: what is 

useful? J Adv Nurs 2001;36(3):376-

88. 

[Rec#: 393] 

 543. Thomson R, Robinson A, Greenaway J, et al. 

Development and description of a 

decision analysis based decision 

support tool for stroke prevention in 

atrial fibrillation. Qual Saf Health 

Care 2002;11(1):25-31. 

[Rec#: 394] 

 544. Tierney WM. Feedback of performance and 

diagnostic testing: lessons from 

Maastricht. Med Decis Making 

1996;16 (4):418-9. 

[Rec#: 41] 

 545. Tierney WM. Improving clinical decisions and 

outcomes with information: a review. 

Int J Med Inf 2001;62(1):1-9. 

[Rec#: 42] 

 546. Tierney WM, Murray MD, Gaskins DL, et al. 

Using computer-based medical 

records to predict mortality risk for 

inner-city patients with reactive 

airways disease. J Am Med Inform 

Assoc 1997;4(4):313-21. 

[Rec#: 48] 

 547. Tierney WM, Overhage JM, McDonald CJ. 

Demonstrating the effects of an 

IAIMS on health care quality and 

cost. J Am Med Inform Assoc 

1997;4(2 Suppl):S41-6. 

[Rec#: 853] 

 548. Tierney WM, Overhage JM, Takesue BY, et 

al. Computerizing guidelines to 

improve care and patient outcomes: 

the example of heart failure. J Am 

Med Inform Assoc 1995;2(5):316-22. 

[Rec#: 697] 

 549. Tierney WM, Takesue BY, Vargo DL, et al. 

Using electronic medical records to 

predict mortality in primary care 

patients with heart disease: prognostic 

power and pathophysiologic 

implications. J Gen Intern Med 1996; 

11(2):83-91. 

[Rec#: 49] 

 550. Timmermans DR, Sprij AJ, de Bel CE. The 

discrepancy between daily practice 

and the policy of a decision-analytic 

model: the management of fever of 

unknown origin. Med Decis Making 

1996;16(4):357-66. 

[Rec#: 396] 

 551. Tomasulo P. PD(X)MD: a new clinical 

information tool for patient care. Med 

Ref Serv Q 2003;22(2):77-83. 

[Rec#: 397] 

 552. Tomes JP. Legal issues in the acquisition of 

computerized medical records and 

payment systems. J Health Care 

Finance 1996;22(4):88-92. 

[Rec#: 854] 
G-33

 553. Tsui FC, Espino JU, Dato VM, et al. Technical 

description of RODS: a real-time 

public health surveillance system. J 

Am Med Inform Assoc 

2003;10(5):399-408. 

[Rec#: 130] 

 554. Tully M. The impact of information 

technology on the performance of 

clinical pharmacy services. J Clin 

Pharm Ther 2000;25(4):243-9. 

[Rec#: 617] 

 555. Urden LD. Development of a nurse executive 

decision support database. A model 

for outcomes evaluation. J Nurs Adm 

1996;26(10):15-21. 

[Rec#: 399] 

 556. Utz SW. Computerized documentation of case 

management. From diagnosis to 

outcomes. Nurs Case Manag 

1998;3(6):247-54. 

[Rec#: 618] 

 557. Valenta AL, Wigger U. Early results of user 

profiles: physicians' opinions on the 

use of information technology. Proc 

AMIA Annu Fall Symp 1996;65-9. 

[Rec#: 914] 

 558. van Ginneken AM. The computerized patient 

record: balancing effort and benefit. 

Int J Med Inf 2002;65(2):97-119. 

[Rec#: 620] 

 559. van Ginneken AM, Stam H, van Mulligen EM, 

et al. ORCA: the versatile CPR. 

Methods Inf Med 1999;38(4-5):332-

8. 

[Rec#: 621] 

 560. Vissers MC, Greve JW, Gouma DJ, et al. 

Introduction of a computerised 

protocol in clinical practice: is there 

anything to gain? Eur J Surg 

1997;163(4):245-54. 

[Rec#: 400] 

 561. von Grey M. Focus on voice recognition. 

Technology moves into the 

mainstream. Health Manag Technol 

2000;21 (4):20-1. 

[Rec#: 622] 

 562. Wagner EH. The role of patient care teams in 

chronic disease management . BMJ 

2000;320(7234):569-72. 

[Rec#: 131] 

 563. Wagner EH, Austin BT, Von Korff M. 

Organizing care for patients with 

chronic illness. Milbank Q 

1996;74(4):511-44. 

[Rec#: 132] 

 564. Walker LM. At Kaiser, a vision of the 

connected future. Med Econ 

2002;79(3):52-4, 58. 

[Rec#: 623] 

 565. Walsh SH. The clinician's perspective on 

electronic health records and how 

they can affect patient care. BMJ 

2004;328(7449):1184-7. 

[Rec#: 931] 

 566. Wang YZ, Wang S. Automation of clinical 

and patient records. South Med J 

1999;92(1):16-22. 

[Rec#: 624] 

 567. Wanlass RL, Reutter SL, Kline AE. 

Communication among rehabilitation 

staff: "mild," "moderate," or "severe" 

deficits? Arch Phys Med Rehabil 

1992;73(5):477-81. 

[Rec#: 135] 

 568. Ward MD. e-Medx, Inc.'s e-POC for cancer. 

Case Manager 2001;12(2):29. 

[Rec#: 403] 

 569. Ward MD. InformaCare disease management 

system. Case Manager 

2002;13(3):30-1. 

[Rec#: 404] 

 570. Ward SR. Health smart cards: merging 

technology and medical information . 

Med Ref Serv Q 2003;22(1):57-65. 

[Rec#: 625] 

 571. Waring N. To what extent are practices 

'paperless' and what are the 

constraints to them becoming more 

so? Br J Gen Pract 2000;50(450):46-

7. 

[Rec#: 626] 
G-34

 572. Warren JR, Posey B, Thornton T, et al. Can 

computer autoacquisition of medical 

information meet the needs of the 

future? A feasibility study in direct 

computation of the fine grained 

electronic medical record. Proc 

AMIA Symp 1999;445-9. 

[Rec#: 627] 

 573. Watlington AG. Realizing system benefits: 

one planned step at a time. Comput 

Healthc 1989;10(2):32-4, 36. 

[Rec#: 50] 

 574. Watson S. Electronic records pay off twice. 

Health Data Manag 1996;4(3):112-4. 

[Rec#: 855] 

 575. Weaver FM, Hatzakis M, Evans CT, et al. A 

comparison of multiple data sources 

to identify vaccinations for veterans 

with spinal cord injuries and 

disorders. J Am Med Inform Assoc 

2004;11 (5):377-9. 

[Rec#: 946] 

 576. Weekley JS, Smith BJ, Pradhan M. The 

intersection of health informatics and 

evidence-based medicine: computerbased systems to assist clinicians. 

Med J Aust 2000;173(7):376-8. 

[Rec#: 405] 

 577. Weiner M, Callahan CM, Tierney WM, et al. 

Using information technology to 

improve the health care of older 

adults. Ann Intern Med 2003; 139(5 

Pt 2):430-6. 

[Rec#: 51] 

 578. Weingarten M, Andrew WF. The CPR: 

balancing quality and cost. Healthc 

Inform 1996;13(8):44, 46, 48. 

[Rec#: 858] 

 579. Weingarten SR, Henning JM, Badamgarav E, 

et al. Interventions used in disease 

management programmes for patients 

with chronic illness-which ones 

work? Meta-analysis of published 

reports. BMJ 2002;325(7370):925. 

[Rec#: 136] 

 580. Weinstein MC, O'Brien B, Hornberger J, et al. 

Principles of good practice for 

decision analytic modeling in healthcare evaluation: report of the ISPOR 

Task Force on Good Research 

Practices--Modeling Studies. Value 

Health 2003;6(1):9-17. 

[Rec#: 406] 

 581. Welton JM, Jarr S. Automating and improving 

the data quality of a nursing 

department quality management 

program at a university hospital. Jt 

Comm J Qual Improv 

1997;23(12):623-35. 

[Rec#: 407] 

 582. Westra B, Raup G. Computerized charting--an 

essential tool for survival. Caring 

1995;14(8):57-61. 

[Rec#: 859] 

 583. Whitlock WL, Brown A, Moore K, et al. 

Telemedicine improved diabetic 

management. Mil Med 

2000;165(8):579-84. 

[Rec#: 137] 

 584. Willey CJ, Struckhoff B. Physicians use EMR 

to improve quality. Esse Health takes 

lead in primary care technology, 

quality. Med Group Manage J 

1999;46(6):18-21, 24-7. 

[Rec#: 629] 

 585. Williams B. The untapped potential of 

electronic medical records. Pt. 1--

Road-testing the promises of EMR 

technology. Tenn Med 

2001;94(4):119-21. 

[Rec#: 630] 

 586. Williams B. The untapped potential of 

electronic medical records. Pt. 2--

Solving the problems of EMR 

technology. Tenn Med 

2001;94(5):161-4. 

[Rec#: 631] 

 587. Williams C, Whitfield G. Written and 

computer-based self-help treatments 

for depression. Br Med Bull 

2001;57:133-44. 

[Rec#: 408] 
G-35

 588. Wong BK, Sellaro CL, Monaco JA. 

Information systems analysis 

approach in hospitals: a national 

survey. Health Care Superv 

1995;13(3):58-64. 

[Rec#: 52] 

 589. Wong ET, Abendroth TW. Reaping the 

benefits of medical information 

systems. Acad Med 1996;71(4):353-

7. 

[Rec#: 409] 

 590. Wood L, Coulson R. Revitalizing the General 

Practice Research Database: plans, 

challenges, and opportunities. 

Pharmacoepidemiol Drug Saf 

2001;10(5):379-83. 

[Rec#: 633] 

 591. Worth ER, Patrick TB. Do electronic mail 

discussion lists act as virtual 

colleagues? Proc AMIA Annu Fall 

Symp 1997;325-9. 

[Rec#: 139] 

 592. Wu AW, Pronovost P, Morlock L . ICU 

incident reporting systems. J Crit 

Care 2002;17(2):86-94. 

[Rec#: 410] 

 593. Wu R, Peters W, Morgan MW. The next 

generation of clinical decision 

support: linking evidence to best 

practice. J Healthc Inf Manag 2002; 

16(4):50-5. 

[Rec#: 411] 

 594. Wulff HR. Clinical medicine in the age of the 

computer. Aust N Z J Med 

2000;30(4):503-5. 

[Rec#: 412] 

 595. Young AS, Mintz J, Cohen AN, et al. A 

Network-Based System to Improve 

Care for Schizophrenia: The Medical 

Informatics Network Tool (MINT). J 

Am Med Inform Assoc 

2004;11(5):358-67. 

[Rec#: 947] 

 596. Zielstorff RD, Barnett GO, Fitzmaurice JB, et 

al. A decision support system for 

prevention and treatment of pressure 

ulcers based on AHCPR guidelines. 

Proc AMIA Annu Fall Symp 

1996;562-6. 

[Rec#: 722] 

 597. Zoccali C. The importance of large, high 

quality clinical databases in 

nephrology. J Nephrol 2001;14 Suppl 

4:S89-93. 

[Rec#: 415] 

 598. Zurhellen W. The coming of age of the 

electronic office. Pediatr Ann 

2001;30(5):289-97. 

[Rec#: 635] 

 599. Zvarova J, Tomeckova M, Stefek M, et al. 

Decision support and data analysis 

tools for risk assessment in primary 

preventive study of atherosclerosis . 

Stud Health Technol Inform 1997;43 

Pt B:625-8. 

[Rec#: 416]