Evidence Report/Technology Assessment

Number 220

Health Information 

Exchange

Evidence-Based

Practice

Health Information 

Technology
Evidence Report/Technology Assessment

Number 220

Health Information Exchange 

Prepared for: 

Agency for Healthcare Research and Quality

U.S. Department of Health and Human Services 

540 Gaither Road 

Rockville, MD 20850 

www.ahrq.gov 

Contract No. 290-2012-00014-I 

Prepared by:

Pacific Northwest Evidence-based Practice Center

Portland, OR 

Investigators: 

William Hersh, M.D. 

Annette Totten, Ph.D. 

Karen Eden, Ph.D. 

Beth Devine, Ph.D., Pharm.D., M.B.A. 

Paul Gorman, M.D. 

Steve Kassakian, M.D. 

Susan S. Woods, M.D. 

Monica Daeges, B.A. 

Miranda Pappas, M.A. 

Marian S. McDonagh, Pharm.D. 

AHRQ Publication No. 15(16)-E002-EF 

December 2015
ii

This report is based on research conducted by the Pacific Northwest Evidence-based Practice 

Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), 

Rockville, MD (Contract No. 290-2012-00014-I). The findings and conclusions in this document 

are those of the authors, who are responsible for its contents; the findings and conclusions do not 

necessarily represent the views of AHRQ. Therefore, no statement in this report should be 

construed as an official position of AHRQ or of the U.S. Department of Health and Human 

Services.

The information in this report is intended to help health care decisionmakers—patients and 

clinicians, health system leaders, and policymakers, among others—make well informed

decisions and thereby improve the quality of health care services. This report is not intended to 

be a substitute for the application of clinical judgment. Anyone who makes decisions concerning 

the provision of clinical care should consider this report in the same way as any medical 

reference and in conjunction with all other pertinent information, i.e., in the context of available 

resources and circumstances presented by individual patients. 

AHRQ or U.S. Department of Health and Human Services endorsement of any derivative 

products that may be developed from this report, such as clinical practice guidelines, other 

quality enhancement tools, or reimbursement or coverage policies, may not be stated or implied. 

This report may periodically be assessed for the currency of conclusions. If an assessment is 

done, the resulting surveillance report describing the methodology and findings will be found on 

the Effective Health Care Program Web site at www.effectivehealthcare.ahrq.gov. Search on the 

title of the report. 

This report is made available to the public under the terms of a licensing agreement between the 

author and the Agency for Healthcare Research and Quality. This report may be used and 

reprinted without permission except those copyrighted materials that are clearly noted in the 

report. Further reproduction of those copyrighted materials is prohibited without the express 

permission of copyright holders.

Persons using assistive technology may not be able to fully access information in this report. For 

assistance contact EffectiveHealthCare@ahrq.hhs.gov. 

Suggested citation: Hersh W, Totten A, Eden K, Devine B, Gorman P, Kassakian S, Woods SS, 

Daeges M, Pappas M, McDonagh MS. Health Information Exchange. Evidence 

Report/Technology Assessment No. 220. (Prepared by the Pacific Northwest Evidence-based 

Practice Center under Contract No. 290-2012-00014-I.) AHRQ Publication No. 15(16)-E002-EF. 

Rockville, MD: Agency for Healthcare Research and Quality; December 2015. 

www.effectivehealthcare.ahrq.gov/reports/final.cfm. DOI: https://doi.org/10.23970/

AHRQEPCERTA220.

None of the investigators have any affiliations or financial involvement that conflicts with 

the material presented in this report.
iii

Preface 

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based 

Practice Centers (EPCs), sponsors the development of systematic reviews to assist public- and 

private-sector organizations in their efforts to improve the quality of health care in the United 

States. These reviews provide comprehensive, science-based information on common, costly 

medical conditions, and new health care technologies and strategies. 

Systematic reviews are the building blocks underlying evidence-based practice; they focus 

attention on the strength and limits of evidence from research studies about the effectiveness and 

safety of a clinical intervention. In the context of developing recommendations for practice, 

systematic reviews can help clarify whether assertions about the value of the intervention are 

based on strong evidence from clinical studies. For more information about AHRQ EPC 

systematic reviews, see www.effectivehealthcare.ahrq.gov/reference/purpose.cfm. 

AHRQ expects that these systematic reviews will be helpful to health plans, providers, 

purchasers, government programs, and the health care system as a whole. Transparency and 

stakeholder input are essential to the Effective Health Care Program. Please visit the Web site 

(www.effectivehealthcare.ahrq.gov) to see draft research questions and reports or to join an 

email list to learn about new program products and opportunities for input. 

We welcome comments on this systematic review. They may be sent by mail to the Task 

Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road, 

Rockville, MD 20850, or by email to epc@ahrq.hhs.gov. 

Richard G. Kronick, Ph.D. Arlene S. Bierman, M.D., M.S. 

Director Director

Agency for Healthcare Research and Quality Center for Evidence and Practice Improvement 

Agency for Healthcare Research and Quality

Stephanie Chang, M.D., M.P.H. 

Director, EPC Program

Center for Evidence and Practice Improvement

Agency for Healthcare Research and Quality

Edwin Lomotan, M.D.

Task Order Officer

Center for Evidence and Practice Improvement 

Agency for Healthcare Research and Quality
iv

Acknowledgments 

The authors gratefully acknowledge the following individuals for their contributions to this 

project: Andrew Hamilton, M.L.S., M.S., for conducting literature searches and Spencer Dandy, 

B.S., for assistance with preparing this report (both are located at the Oregon Health & Science 

University) and Jon White, M.D., Task Order Officer at the Agency for Healthcare Research and 

Quality.

Key Informants 

In designing the study questions, the EPC consulted several Key Informants who represent 

the end-users of research. The EPC sought the Key Informant input on the priority areas for

research and synthesis. Key Informants are not involved in the analysis of the evidence or the 

writing of the report. Therefore, in the end, study questions, design, methodological approaches, 

and/or conclusions do not necessarily represent the views of individual Key Informants. 

Key Informants must disclose any financial conflicts of interest greater than $10,000 and any 

other relevant business or professional conflicts of interest. Because of their role as end-users, 

individuals with potential conflicts may be retained. The TOO and the EPC work to balance, 

manage, or mitigate any conflicts of interest.

The list of Key Informants who participated in developing this report follows:

Julia Adler-Milstein, Ph.D.

Assistant Professor, School of Information

Health Management and Policy, School of 

Public Health

University of Michigan

Ann Arbor, MI

Michael Barr, M.D., M.B.A., FACP

Executive Vice President of the Quality 

Measurement & Research Group

National Committee for Quality Assurance

Washington, DC

Richard Dave deBronkart, S.B.

Patient Advocate

Nashua, NH

Erin Holve, Ph.D., M.P.P., M.P.H.

Senior Director

AcademyHealth

Washington, DC

Gilad Kuperman, M.D., Ph.D.

Associate Professor, Biomedical Informatics

Columbia University

New York, NY

Deven McGraw, J.D., M.P.H.

Director, Health Privacy Project

Center for Democracy & Technology

Washington, DC

Kory Mertz, B.A.

Project Officer, Policy Analyst

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC

Christopher Muir, M.A.

Director, State Health Information Exchange 

Program

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC
v

Susan Otter, B.A.

Oregon State Coordinator for Health 

Information Technology 

Oregon Health Authority, Office of the 

Director

Portland, OR

Vaishali Patel, Ph.D., M.P.H.

Senior Advisor, Office of Economic 

Analysis, Evaluation and Modeling

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC

Bret Schillingstad, M.D.

Physician/Clinical Informatics

Epic

Madison, WI

Lee Stevens, B.A.

Program Manager for the State Health 

Information Exchange Program

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC

Technical Expert Panel

In designing the study questions and methodology at the outset of this report, the EPC 

consulted several technical and content experts. Broad expertise and perspectives were sought. 

Divergent and conflicted opinions are common and perceived as healthy scientific discourse that 

results in a thoughtful, relevant systematic review. Therefore, in the end, study questions, design, 

methodologic approaches, and/or conclusions do not necessarily represent the views of 

individual technical and content experts.

Technical Experts must disclose any financial conflicts of interest greater than $10,000 and 

any other relevant business or professional conflicts of interest. Because of their unique clinical 

or content expertise, individuals with potential conflicts may be retained. The TOO and the EPC 

work to balance, manage, or mitigate any potential conflicts of interest identified.

The list of Technical Experts who participated in developing this report follows: 

Julia Adler-Milstein, Ph.D.

Assistant Professor

School of Information Health Management 

and Policy, School of Public Health

University of Michigan

Ann Arbor, MI

Thomas Agresta, M.D., M.B.I.*

Professor of Family Medicine, Department 

of Family Medicine

University of Connecticut School of 

Medicine

Farmington, CT

Melinda Buntin, Ph.D.

Professor and Chair, Department of Health 

Policy

Vanderbilt University School of Medicine

Nashville, TN

Patricia Dykes, Ph.D.*

Assistant Professor, Harvard Medical 

School

The Center for Patient Safety Research and 

Practice

The Center for Nursing Excellence

Brigham & Women's Hospital

Boston, MA
vi

Erin Holve, Ph.D., M.P.P., M.P.H.

Senior Director

AcademyHealth

Washington, DC

Vaishali Patel, Ph.D., M.P.H.

Senior Advisor, Office of Economic 

Analysis, Evaluation and Modeling

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC

Joshua R Vest, Ph.D., M.P.H.*

Assistant Professor, Department of 

Healthcare Policy and Research 

Weill Cornell Medical College 

New York, NY

Daniel Vreeman, P.T., D.P.T., M.S.C.*

Associate Director of Terminology Services 

Regenstrief Institute, Inc. 

Associate Research Professor 

Indiana University School of Medicine 

Indianapolis, IN

Johanna Westbrook, Ph.D., M.H.A., 

FACHI, FACMI,

Grad.Dip.App.Epidemiol, B.App.Sc.*

Professor of Health Informatics, Australian 

Institute of Health Innovation 

Faculty of Medicine and Health Sciences

Macquarie University

Sydney, Australia

*This TEP member also provided review of the draft report.

Peer Reviewers

Prior to publication of the final evidence report, EPCs sought input from independent Peer 

Reviewers without financial conflicts of interest. However, the conclusions and synthesis of the 

scientific literature presented in this report do not necessarily represent the views of individual 

reviewers. 

Peer Reviewers must disclose any financial conflicts of interest greater than $10,000 and any 

other relevant business or professional conflicts of interest. Because of their unique clinical or 

content expertise, individuals with potential nonfinancial conflicts may be retained. The TOO 

and the EPC work to balance, manage, or mitigate any potential nonfinancial conflicts of interest 

identified. 

The list of Peer Reviewers follows:

Jessica S. Ancker, Ph.D., M.P.H.

Associate Professor, Department of 

Healthcare Policy and Research

Weill Cornell Medical College

New York, NY

Patricia Flatley Brennan, Ph.D., R.N. 

Moehlman Bascom Professor School of

Nursing

Industrial & Systems Engineering

Wisconsin Institute for Discovery 

University of Wisconsin–Madison 

Madison, WI

Christopher Chute, M.D., D.P.H.

Bloomberg Distinguished Professor of 

Health Informatics

Professor of Medicine, Public Health, and 

Nursing 

Chief Health Research Information Officer, 

Johns Hopkins Medicine 

Johns Hopkins University

Baltimore, MD
vii

Keven Johnson, M.D., M.S.

Professor and Chair, Department of 

Biomedical Informatics

Vanderbilt University School of Medicine

Nashville, TN

Rainu Kaushal, M.D., M.P.H.

Professor and Chairman, Department of 

Healthcare Policy and Research

Weill Cornell Medical College

New York, NY

Clement McDonald, M.D. 

Director, Lister Hill National Center for 

Biomedical Communications 

National Library of Medicine, National 

Institutes of Health 

Bethesda, MD

Ann McKibbon, B.Sc., M.L.S., Ph.D.

Associate Professor, Department of Clinical 

Epidemiology & Biostatistics

Director, eHealth MSc Program

Member, Health Information Research Unit

McMaster University

Ontario, Canada

Christopher Muir, M.A.

Director, State Health Information Exchange 

Program

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC

Matthew Swain, M.P.H.

Senior Analyst, Strategy and Performance

Office of the National Coordinator for 

Health Information Technology

Department of Health and Human Services

Washington, DC
viii

Health Information Exchange 

Structured Abstract

Objectives. This review sought to systematically review the available literature on health 

information exchange (HIE), the electronic sharing of clinical information across the boundaries 

of health care organizations. HIE has been promoted as an important application of technology in 

medicine that can improve the efficiency, cost-effectiveness, quality, and safety of health care 

delivery. However, HIE also requires considerable investment by sponsors, which have included 

governments as well as health care organizations. This review aims to synthesize the currently 

available research addressing HIE effectiveness, use, usability, barriers and facilitators to actual 

use, implementation, and sustainability, and to present this information as a foundation on which 

future implementation, expansion, and research can be based. 

Data sources. A research librarian designed and conducted searches of electronic databases,

including MEDLINE® (1990 to February 2015), PsycINFO® (1990 to February 2015),

CINAHL® (1990 through February 2015), the Cochrane Central Register of Controlled Trials 

(through January 2015), the Cochrane Database of Systematic Reviews (through January 2015), 

the Database of Abstracts of Reviews of Effects (through the first quarter of 2015), and the 

National Health Sciences Economic Evaluation Database (through the first quarter of 2015). The 

searches were supplemented by reviewing reference lists and the table of contents of journals not 

indexed in the databases we searched. 

Review methods. Two investigators reviewed abstracts and the selected full-text articles for 

inclusion based on predefined criteria. Discrepancies were resolved through discussion and 

consensus, with a third investigator making the final decision as needed. Data were abstracted 

from each included article by one person and verified by another. All analyses were qualitative, 

and they were customized according to the topic. 

Results. We included 136 studies overall, with 34 on effectiveness, 26 of which reported 

intermediate clinical, economic, or patient outcomes, and 8 that reported on clinical perceptions 

of HIE. We also found 58 studies on the use of HIE, 22 on usability and other facilitators and 

barriers to actual use of HIE, 45 on facilitators or barriers to HIE implementation, and 17 on 

factors related to sustainability of HIE. 

No studies of HIE effectiveness reported impact on primary clinical outcomes (e.g., mortality 

and morbidity) or identified harms. Low-quality evidence somewhat supports the value of HIE 

for reducing duplicative laboratory and radiology test ordering, lowering emergency department 

costs, reducing hospital admissions (less so for readmissions), improving public health reporting, 

increasing ambulatory quality of care, and improving disability claims processing. In studies of 

clinician perceptions of HIE, most respondents attributed positive changes to HIE, such as 

improvements in coordination, communication, and knowledge about the patient. However in 

one study clinicians reported that the HIE did not save time and may not be worth the cost.

Studies of HIE use found that HIE adoption has increased over time, with 76 percent of U.S. 

hospitals exchanging information in 2014, an 85-percent increase since 2008 and a 23-percent 

increase since 2013. HIE systems were used by 38 percent of office-based physicians in 2012,

while use remains low, less than 1 percent, among long-term care providers. 
ix

Within organizations with HIE, the number of users or the number of visits in which the HIE 

was used was generally very low. The degree of usability of an HIE was associated with 

increased rates of use but was not associated with effectiveness outcomes. The most commonly 

cited barriers to HIE use were lack of critical mass electronically exchanging data, inefficient 

workflow, and poorly designed interface and update features. Information was insufficient to 

allow us to assess usability by HIE function or architecture. 

Studies provided information on both external environmental and internal organizational 

characteristics that affect implementation and sustainability. General characteristics of the HIE 

organization (e.g., strong leadership) or specific characteristics of the HIE system were the most 

frequently cited facilitators, while disincentives such as competition or lack of a business case 

for HIE were the most frequently identified barriers. 

Limitations. The scope of studies identified was limited compared with the actual uses and 

capabilities of HIE. The outcomes measured and methods of measurement and analysis, for 

example, were limited and narrowly defined; the issue of potential confounders was not 

addressed in most studies of effectiveness, and harms were not adequately studied. There was a 

high degree of heterogeneity in study designs, outcomes, HIE types, and settings across the 

studies, limiting the ability to synthesize the evidence; no quantitative analyses were possible. 

The applicability of this evidence base is uncertain because the HIE systems studied were so 

diverse, and many in existence have not contributed to research in this field. 

Conclusions. The full impact of HIE on clinical outcomes and potential harms is inadequately 

studied, although evidence provides some support for benefit in reducing use of some specific 

resources and achieving improvements in quality-of-care measures. Use of HIE has risen over 

time, and is highest in hospitals and lowest in long-term care settings. However, use of HIE 

within organizations that offer it is still low. Barriers to HIE use include lack of critical mass 

participating in the exchange, inefficient workflow, and poorly designed interface and update 

features. Studies have identified numerous facilitators and barriers to implementation and 

sustainability, but the studies have not ranked or compared their impact. To advance our 

understanding of HIE, future studies need to address comprehensive questions, use more 

rigorous designs, use a standard for describing types of HIE, and be part of a coordinated 

systematic approach to studying HIE. 
x

Contents

Executive Summary.................................................................................................................ES-1

Introduction....................................................................................................................................1

Background................................................................................................................................1

Scope of Review and Key Questions.........................................................................................2

Methods...........................................................................................................................................7

Topic Development and Refinement .........................................................................................7

Literature Search Strategy..........................................................................................................7

Process for Study Selection .......................................................................................................8

Populations...........................................................................................................................8

Intervention and Comparators..............................................................................................8

Outcomes by Key Question .................................................................................................8

Timing..................................................................................................................................9

Settings.................................................................................................................................9

Study Design........................................................................................................................9

Data Abstraction and Data Management .................................................................................10

Assessment of Methodological Risk of Bias of Individual Studies.........................................10

Data Synthesis and Organization of Report.............................................................................11

Grading the Body of Evidence for Each Key Question...........................................................11

Assessing Applicability ...........................................................................................................12

Peer Review and Public Commentary .....................................................................................12

Results...........................................................................................................................................13

Results of Literature Searches .................................................................................................13

Description of Included Studies...............................................................................................13

Key Question 1. Is HIE effective in improving clinical, economic, and population 

outcomes? ..........................................................................................................................14

Key Question 2. What harms have resulted from HIE?...........................................................14

Key Question 3. Is HIE effective in improving intermediate outcomes such as patient 

and provider experience, perceptions, or behavior; health care processes; or the 

availability, completeness, or accuracy of information? ...................................................14

Key Points..........................................................................................................................14

Detailed Synthesis..............................................................................................................14

Key Question 4. What are the current level of use and primary uses of HIE? ........................26

Key Points..........................................................................................................................26

Detailed Synthesis..............................................................................................................27

Key Question 5. How does the usability of HIE impact effectiveness or harms 

for individuals and organizations? .....................................................................................57

Key Question 6. What facilitators and barriers impact use of HIE?........................................57

Key Points..........................................................................................................................57

Detailed Synthesis..............................................................................................................57

Key Question 7. What facilitators and barriers impact implementation of HIE?....................70

Key Question 8. What factors influence sustainability of HIE?..............................................70

Key Points..........................................................................................................................70

Detailed Synthesis..............................................................................................................70

Discussion......................................................................................................................................80

Key Findings............................................................................................................................80
xi

Strength of Evidence................................................................................................................81

Findings in Relationship to What Is Already Known..............................................................82

Applicability ............................................................................................................................83

Limitations of the Evidence Base ............................................................................................85

Future Research Needs ............................................................................................................85

Conclusions..............................................................................................................................88

References.....................................................................................................................................89

Abbreviations and Acronyms ...................................................................................................100

Tables

Table A. Summary of evidence ................................................................................................ES-2

Table 1. Studies of HIE included for assessing outcomes.............................................................16

Table 2. Patient and clinician perceptions of HIE .........................................................................24

Table 3. Factors that may affect outcomes ....................................................................................25

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data 

exchanged, and characteristics of successfully participating organizations 

(United States–wide studies)..........................................................................................................31

Table 5. Level of use and primary uses of HIE: transfer of records between integrated 

delivery systems.............................................................................................................................38

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, 

by regional or statewide initiatives................................................................................................42

Table 7. Level of use and primary uses of HIE: extent of use, types of information 

exchanged, and adoption in international or multinational settings ..............................................53

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use...................58

Table 9. Barriers and facilitators of actual HIE use grouped by theme.........................................67

Table 10. Facilitators to implementation and sustainability of HIE ..............................................73

Table 11. Barriers to implementation and sustainability of HIE ...................................................77

Table 12. Summary of evidence ....................................................................................................80

Figures

Figure 1. Analytic framework..........................................................................................................4

Figure 2. Literature flow diagram..................................................................................................13

Figure 3. Rasmussen sociotechnical analysis framework..............................................................87

Appendixes

Appendix A. Search Strategies

Appendix B. Inclusion and Exclusion Criteria

Appendix C. List of Included Studies

Appendix D. List of Excluded Studies

Appendix E. Study Design Terminology

Appendix F. Evidence Table

Appendix G. Risk of Bias Assessment Criteria

Appendix H. Strength of Evidence Criteria

Appendix I. Quality Assessment Tables
ES-1

Executive Summary

Background 

Health information exchange (HIE) is the sharing of electronic clinical data across 

organizations.1 The idea that records should follow patients wherever they receive care has been 

promoted as a cornerstone of efforts to improve the coordination, efficiency, and effectiveness of 

health services. The underlying belief is that ultimately patients would benefit if all relevant 

information were available to the various health care providers involved in treating them and 

working to maintain their health. However, realizing this vision is challenging because health 

care is currently provided by a diversity of organizations and providers with disparate 

information systems. A substantial investment of resources is needed to develop an environment 

that allows health care information to follow the patient.

Governments at all levels, as well as health systems and individual organizations, have and 

are continuing to make the significant investment of time and resources to achieve the goals of 

HIE. For example, in the United States, the Health Information Technology for Economic and 

Clinical Health (HITECH) Act, part of the American Recovery and Reinvestment Act of 2009, is 

providing up to $29 billion in incentive funding for the adoption and “meaningful use” of 

electronic health records by hospitals and health professionals. The HITECH Act designated an 

additional $564 million for investment by States or State-designated entities to establish HIE 

capability among health care providers and hospitals in their jurisdictions. Understandably, all 

stakeholders are interested in assuring that there is a return on this investment. These efforts have 

resulted in substantial growth of HIE across the United States.2

The purpose of this review was to identify, summarize, and synthesize the available research 

about HIE. The scope of the review was purposely broad and includes studies about four topics: 

(1) effectiveness, (2) use of HIE, (3) usability and barriers and facilitators to use, and 

(4) implementation and sustainability. 

Methods

This review was completed by the Pacific Northwest Evidence-based Practice Center in 

fulfillment of a contract from the Agency for Healthcare Research and Quality through the 

Effective Health Care Program. We used the Program’s standard methods and procedures,3

which are similar to those established by the Institute of Medicine for systematic reviews.

4 A 

detailed description of the methods is available in the review protocol and in the full report, both 

available at www.effectivehealthcare.ahrq.gov. 

After finalizing the Key Questions to be considered in our review, we looked for reports of 

research on HIE. We searched several bibliographic citation databases (e.g., MEDLINE®) with 

support from two specialized reference librarians, and we searched Web sites and tables of 

contents of publications that are not indexed in citation databases. Studies identified through 

these searches were reviewed for eligibility by two investigators. We included any study with 

data about an actual HIE designed to be used for clinical or public health decisionmaking. We 

included many different types of studies in order to provide a comprehensive review of research 

on HIE effectiveness, use, usability, implementation, and sustainability. Given this broad scope, 

the included studies varied widely in design and quality. We did not include studies of exchanges 

of data for research only, or studies about hypothetical or future HIEs. Data from included 
ES-2

studies were abstracted from the articles, and this information was summarized in tables and 

narratives. 

Results

Overview

The major results are summarized in Table A and described in this section.

Table A. Summary of evidence

Topic Number and Type of 

Included Studies Main Findings Primary Limitations of the Evidence

Effectiveness 34 total:

20 retrospective cohort

3 RCT

2 cross-sectional

2 case series

8 survey (1 survey study 

was an RCT)

Low-quality evidence somewhat 

supports the value of HIE for 

reducing duplicative laboratory 

and radiology test ordering, 

lowering ED costs, reducing 

hospital admissions (less so for 

readmissions), improving public 

health reporting, increasing 

ambulatory quality of care, and 

improving disability claims 

processing. No studies of harm 

were reported.

Studies were of a small number of the 

functioning HIE implementations, with 

similarity to unstudied ones unknown, 

possibly limiting generalizability.

Studies looked at limited outcomes,

considering the intended scope of the 

impact of HIE.

Use 58 total:

25 survey

13 audit log

9 retrospective 

database

7 mixed methods

2 focus groups 

1 time-motion

1 geocoding

The proportion of hospitals and 

ambulatory care practices that 

have adopted HIE is increasing.

Currently, rates of HIE use within 

organizations with HIE are 

generally low.

While there are relatively high-quality 

national and regional surveys and 

reports that track the expansion of 

HIE among health care organizations, 

there is not a corresponding 

comprehensive effort to track changes 

in rates of use within organizations.

Usability and 

factors affecting 

use 

22 total:

9 multiple-site case 

study

11 cross-sectional

2 before-after

The most commonly cited barriers 

to HIE use were lack of critical 

mass electronically exchanging 

data (8 studies); inefficient 

workflow (10 studies); poorly 

designed interface and update 

features (7 studies).

Studies of usability did not relate it to 

effectiveness and do not permit 

comparisons across settings or types 

of HIE.

Studies had limitations, such as 

incomplete description of the 

functionality and architecture of the 

systems, making comparison by type 

difficult.
ES-3

Table A. Summary of evidence (continued)

Topic Number and Type of 

Included Studies Main Findings Primary Limitations of the Evidence

Implementation 

and 

sustainability 

52 total:

26 cross-sectional

17 multiple-site case 

study

2 before-after

3 retrospective cohort

2 prospective cohort

2 time series

Most facilitators of implementation 

cited in research were

characteristics of HIE projects or 

the internal environment of the 

organizations implementing HIE,

such as leadership. Most of the 

identified barriers to 

implementation were external 

environmental factors, such as 

concerns about competition.

Factors related to sustainability 

were similar to those identified for 

implementation. 

The research has not been designed 

to allow ranking or comparisons of the

relative impact of different barriers

and facilitators.

The definition and appropriate 

measures of sustainability of HIE are 

not yet agreed upon, and the majority 

of projects are relatively recent.

ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial

We reviewed 5,211 abstracts and 849 full-text articles. Of these, we included 136 studies that 

addressed one or more of our Key Questions. The data in the following sections come from a 

body of literature in which studies of 12 different HIE implementations are the most frequent 

even though they represent a small proportion of the HIEs functioning in the United States. 

Fewer studies were based on national surveys/datasets, and a comparatively small number of 

studies were conducted in other countries. Most of this literature has been published since 2006. 

Most studies were retrospective cohort studies (analysis of existing data comparing a certain 

outcome with and without HIE) or cross-sectional studies. We included several multisite case 

studies that consisted of qualitative analysis of data from several sources, including responses 

from interviews, questionnaires, or focus groups. Other less common research designs included 

before-and-after studies and time-series studies, which looked at what happened before and after 

HIE implementation. Only two randomized trials (in 3 publications) were identified. In general, 

the risk of bias for these studies was high, with some rated as moderate, although not all study 

designs were rated, and the overall strength of evidence was assessed as low or insufficient for 

most outcomes. 

Effectiveness

We identified 34 studies that associated HIE with various outcomes, with 26 assessing the 

impact of HIE on resource use and 8 reporting on user perceptions of HIE impact. Studies that 

examined whether HIE improved resource use defined this as: (1) reduced ordering of laboratory

tests, radiology exams, and costs, especially in the emergency department (ED); (2) reduced 

hospital admissions, hospital readmissions, and consultations; (3) successful public health use; or 

(4) improvement in quality of care or service delivery. The overall strength of evidence was low, 

as most studies were retrospective and reported on narrow questions, such as reduction in test 

ordering or consultations, and not larger overall clinical and financial impacts. Furthermore, the 

retrospective design of most of the studies raised the potential for confounding factors impacting 

their conclusions.

Studies of reduced laboratory tests, radiology exams, and costs showed the most consistent 

associated benefits. Four U.S. studies found reductions in ED orders of lab tests and radiology 

exams,5-8 and three more found reductions in radiology alone.

9-11 A United States–based 
ES-4

ambulatory study found a reduced rate of increase in laboratory testing and no impact on 

imaging,12 while a Finland-based study found that orders for lab tests increased while orders for 

imaging decreased.

13 Two studies found that HIE reduced overall ED costs.5,6

The studies of admissions and readmissions had inconsistent findings, with some reporting 

that HIE reduced admissions6,7,14-16 or readmissions,17 while others reported no effect.18-21

Similarly, the findings related to consultations or referrals were mixed, with one study reporting 

fewer consultations and cost savings7 and another reporting an increase in referrals by both 

primary care physicians and specialists.13 We did not pool the results using meta-analysis, as the 

patient populations differed across studies.

Studies of other resource-use outcomes more consistently identified benefits. Studies of 

quality of care found that physicians providing preventive services who used HIE performed 

better on quality measures.

22,23 Studies also reported that HIE could help identify frequent ED 

users24 but did not lead to improvement of medication adherence.25 One study found that HIE 

reduced the time needed to evaluate Social Security claims.26 Another found a positive 

association between general patient satisfaction in hospitals and whether the hospital had 

implemented HIE.27

In studies that asked users of HIE to report on their perception of its impact, all found at least 

some benefit, although some uncovered negative aspects as well. Physicians were more satisfied 

with electronic than paper lab reports;28 more physicians preferred HIE that pushed data to them 

than HIE that required them to pull the data with a query;29 and physicians believed electronic 

reports of ED use improved followup30,31 and that HIE improved ambulatory care practice 

efficiency.

32,33 However, physicians in one study responded that having HIE provide pharmacy 

information in the ED improved knowledge but did not reduce time spent to provide service and 

was not worth the cost.34 Patients reported that they preferred having records transferred via HIE 

over transferring paper records themselves.35

Although most studies of the effectiveness of HIE reported positive results, the literature as a 

whole was not comprehensive and few studies were of high quality. HIE is usually broad based 

and designed to affect practice and numerous outcomes; however, evaluation studies have 

focused on only one or a small number of uses or potential effects. Additionally, even in cases in 

which the results were positive, the effect sizes were not large or able to be assessed given the 

information provided. For example, ED savings are hard to evaluate if the overall budget for the 

ED is not known. (See evidence tables in Appendix F of the full report for detailed results.)

Additionally, many studies employed simple study designs that impede risk-of-bias assessment 

(thus lowering our confidence in the study results). Given these limitations, it is not possible to 

conclude with any certainty that HIE has consistently been effective in improving health 

outcomes.

Use of Health Information Exchange

We identified 58 studies that described either the level of use of HIE or the primary uses of 

HIE. Of these, 15 studies evaluated HIE use nationally in the United States and 2 studies 

evaluated HIE use across integrated delivery systems. About half (30 studies) of these studies 

analyzed the extent to which HIE was implemented in a State or across a region, but these were 

concentrated in New York (10 studies), Texas (5 studies), and Tennessee (5 studies). Six studies 

evaluated HIE in other countries and three in multiple countries, two of which included the 

United States.
ES-5

Nationwide surveys in the United States suggest that HIE use has risen substantially among 

hospitals since 2008. Use of HIE was reported by 11 percent of hospitals in 2009,36 while more 

current estimates range from 30 to 58 percent.37-39 Recent data from the Office of the National 

Coordinator for Health Information Technology (ONC) suggest that more than three-quarters 

(76%) of non-Federal acute care hospitals electronically exchanged laboratory results, radiology 

reports, clinical care summaries, and/or medication lists with an outside provider.2 This 

represents an 85-percent increase since 2008 and a 23-percent increase since 2013. Close to 7 in 

10 hospitals (69%) electronically exchanged health information with ambulatory providers 

outside of their organization, representing a 92-percent increase since 2008 and a 21-percent

increase since 2013. Results from the National Ambulatory Medical Care Survey (2013) 

concluded that 39 percent of office-based physicians reported having HIE capability with other 

providers or hospitals.40 Limited data suggest that use of technology in general and HIE 

specifically is very low (> 1%) in long-term care settings.41,42

Between 2004 and 2009, regional health information organization (RHIO) was the term used 

to describe HIE organizations; several of the included studies used this term. All RHIOs are 

involved in HIE by definition, but both their reach and composition vary. In 2008 and 2009, 

RHIOs included 14 percent of U.S. hospitals and 3 percent of ambulatory care practices.43 A

study of public health departments found that 36 percent had no RHIO in their jurisdiction and 

12 percent had no relationship with the RHIO in their area.44 Of those with a RHIO in their area, 

40 percent were actually exchanging information.44 In RHIOs, the entities most commonly 

providing data are hospitals (83%), followed by ambulatory settings (60%); the entities most 

commonly receiving data are ambulatory settings (95%), followed by hospitals (83%), public 

health departments (50%), and payers (44%).45

Studies of HIE in integrated delivery systems included exchanges among the Department of 

Defense, Department of Veterans Affairs (VA), and the private sector. In an initial test in one 

city, 73 percent of patients could be located across the system and exchanges were executed two 

to three times a week.46 A larger 12-site expansion experiment resolved some issues in matching 

patients but reported that the VA received information from private organizations for 9 percent 

of the matched patients.47

While organizational involvement and capacity for HIE are increasing, the data about actual 

use of HIE when it is possible were limited and suggested that HIE is still not integrated into 

usual care. For example, studies from the MidSouth e-Health Alliance suggested low use of HIE 

overall (from 2.6% to 9.5% of visits in 2008 and 2009),

48 with higher use for ED visits (15%) 

and return clinic visits (19%).49 In another example, data collected in the Central Texas HIE 

from 2006 to 2011, HIE use was low—used in only 2.3 percent of encounters.50

Usability and Other Barriers and Facilitators to Use

We reviewed 22 studies that examined either usability or other barriers and facilitators to 

actual HIE use. The evidence was insufficient to compare usability by type of HIE function 

(query-based, or pull, vs. directed, or push, exchange) or by type of architecture (centralized or 

not).

We found five surveys on HIE usability, and most defined usability as it relates to function 

and/or measured satisfaction with exchanging health information.

29,32,51-53 Perceptions of 

usability were related to actual use. One study reported higher scores on a measure of satisfaction 

with user interface related to more frequent use,52 and another reported that users endorsing 

statements that the HIE was useful and easy to learn to operate had higher levels of weekly HIE 
ES-6

use.54 Providers who used HIE also reported increased satisfaction and improved relationships 

with care partners.

53,55A related negative finding was that providers had high expectations for 

HIE before implementation and reported some ongoing unmet needs once HIE was operational.53

Barriers and facilitators to use of HIE were identified using cross-sectional and multiple-site 

case studies that drew on data from several sources (e.g., interviews, focus groups, and 

observations). Barriers and facilitators identified fell under three broad topics: lack of critical 

mass electronically exchanging data, workflow, and interface. Several facilitators showed 

promise in promoting electronic health data exchange: obtaining more complete patient 

information; thoughtful implementation and workflow; and well-designed user interface and data 

presentation. 

Lack of critical mass was a key issue: if providers do not find useful data from HIE, they are 

less likely to use HIE in the future. Data were incomplete because of issues of incomplete patient 

information that related to the setting (more complete in an ED and less in a homeless center) or 

challenges in matching patients across systems.46,47,56-61 Privacy, legal concerns, and

requirements that patients opt in or opt out to sharing data all reduced the completeness of data, 

and approaches to address these factors could lead to more comprehensive data and increased 

use. Differences in how HIE was incorporated into workflow and daily operations also affected 

use.32,47,49,51,53,54,56,60-62 Studies found that when proxy nonphysician users accessed the system 

and provided relevant information to the doctors, the system was used more frequently.48,49

Studies based on observations found that different providers used the exchange differently, with 

nurses seeking information on hospital admissions or other care mentioned by the patients, while 

physicians also used the exchange to complete their understanding of the patient history and to 

facilitate decisionmaking.63 The interface and features of the systems were also cited as 

encouraging or hindering use. User opinions differed in terms of whether they wanted more or 

less information, based both on desire for more content61 and on interface issues, such as the 

need to scroll or click through multiple pages.54,56,60 In addition, users reported that the systems 

slowed down as data were expanded to include more patients and information or that new 

information was not added to centralized systems quickly enough (so that going to records in 

separate systems was quicker).54

Implementation and Sustainability 

We identified 52 studies that aimed to identify factors that affect implementation and 

sustainability. Forty-five studies identified facilitators to implementation (which we grouped into 

8 categories) and barriers (which we grouped into7 categories). While fewer studies (17 studies) 

considered sustainability, we sorted the positive and negative influences on sustainability so that 

they overlapped with our categories of facilitators and barriers to implementation. Studies were 

not designed to rank factors and did not provide enough data to allow us to assess the 

comparative impact of different factors on implementation and sustainability.

Facilitators for implementation focused predominately on the characteristics of the 

implementing organization or of the HIE system the organizations were planning to implement. 

The most frequently cited category we labeled General Structure of the organizations 

implementing HIE, and included specifics such as leadership26,64-66 and prior experience with or 

readiness for information technology (IT) projects.53,67 Another category that facilitated 

implementation, HIE-Specific Structures, included governance26 and participatory 

approaches.23,68-71 Organizations implementing HIE shifted their mission or focus (category 

labeled Orientation Shift) toward collaboration72 and continuity of care,

73 and those that were 
ES-7

successful were able to shift from piloting minimal HIE functions to a robust system quickly.74

Organizations successful in implementing HIE also provided support for the implementation, 

such as training,75,76 and focused on selected outcomes, such as meeting a community need.77

Key Functions is our category of facilitators that included HIE designs that reflected workflow,

69

and functions that could be integrated into care processes47,76,78,79 were also considered 

facilitators for implementation. The one type of external factor cited as a facilitator was policy in 

the form of Federal and State laws and mandates,78,80 as well as grants from Federal and State 

governments that supported preliminary HIE activities and subsidized participating 

organizations.

67

Barriers to implementation overlapped with facilitators but included more categories of 

external factors. External Policy included laws and grants that were identified as barriers when 

their timelines or changes in requirements imposed burdens on organizations that could mitigate 

the support they provided for implementation.

65,81 The most frequently cited category of barriers

was Disincentives, including the issue of financial viability67,75,78,82,83 and the mismatch between 

those who invest in HIE and those who benefit.67,84,85 The Technology Environment was another 

category; characteristics that hindered implementation included lack of standards44,86 and limited 

interoperability across organizations.78,87,88 Three categories of barriers were related to the 

organization and its efforts to establish HIE: the Lack of Necessary Components, such as 

physician engagement;

72 the Fit between the goals and timeline of the organization and HIE 

projects;

89,90 and the need for resources to address complex problems with User Interface and 

Functionality.

47

Fewer studies considered sustainability. Positive influences included factors identified as 

being associated with both implementation and sustainability, such as leadership by a health 

information organization91 and provision of direct financial benefit to HIE participants.84,92 The 

most commonly cited negative influences on sustainability were competition and the difficulty in 

making the business case for HIE.93-96 Other hindrances to sustainability identified were 

structural factors, such as a mismatch between the geographic coverage of the HIE and the 

service area,96 governance issues and lack of trust,96,97 and lack of engagement of participating 

organizations and their providers.77 One study documented that most HIE projects have overly 

optimistic timelines and that the lack of time and missed deadlines worked against 

sustainability.74

Implications 

HIE represents a significant component of health care reform efforts. HIE is one of the major 

applications of health IT and requires significant resources. Thus it is not surprising that 

numerous studies have been published about HIE. However, this body of literature is limited in 

several ways. Most of the studies are not designed to sufficiently control for risk of bias, and 

they focus on relatively narrow outcomes when assessing the impact of a broad-based, complex,

systemic intervention such as HIE. While the studies of use, usability, implementation, and 

sustainability provide information on context and allow some insight into trends, in general they 

do not permit any comparative assessment or ranking of the importance of different barriers or 

facilitators. Additionally these studies do not provide sufficient technical detail to compare HIE

systems by function or architecture.

Although it may not be the purview of research to decide if HIE should be funded as 

infrastructure (as with a utility) or as a part of business operations, the notion that HIE should 

improve efficiency and quality of care, including clinical and economic benefits, is not 
ES-8

overwhelmingly supported by the available evidence. Positive findings are encouraging, but both 

the level of the impact and some inconsistencies in results preclude any definitive conclusion.

Additionally, while surveys suggest that use of HIE is spreading, the scope of use within 

organizations is still limited, implementation is slow, and sustainability seems less than assured. 

Exactly what is needed for HIE to be effective is also difficult to discern from a body of 

literature that does not include many comparative studies and that does not seem to build on prior 

results to create a succession of increasingly relevant studies. We hope that this will improve as 

HIE implementations become more mature and more robust study designs are used. Future 

research should consist of prospective studies, carried out in mature HIE settings, assessing 

patients who are likely to benefit from HIE and comparing appropriate outcomes for the use or 

nonuse of HIE. The prospective collection of data from diverse settings where HIE is used, 

classified by a detailed taxonomy of research type, system implementation, and usage type, could 

allow for prospective cohort studies that could identify aspects of HIE associated with beneficial 

outcomes.

Despite these concerns, expansion of HIE seems likely, and research could better serve this 

effort by developing and pursuing a more deliberate research agenda designed to capture the full 

potential impact of HIE and identify the comparative role of specific factors related to use, 

usability, implementation, and ultimately, sustainability.

Conclusions

The full impact of HIE on clinical outcomes and potential harms is inadequately studied, 

although evidence provides some support for benefit in reducing use of some specific resources 

and improving quality-of-care measures. Use of HIE has risen over time, and is highest in 

hospitals and lowest in long-term care settings. However, use of HIE within organizations that 

offer it is still low. Barriers to HIE use include lack of critical mass exchanging data, inefficient 

workflow, and poorly designed interface and update features. Factors we identified as facilitating 

HIE implementation included general characteristics of the organization and specific 

characteristics of the HIE system. Barriers focused more on the external environment, and 

disincentives made up the largest category of barriers. Sustainability was less frequently studied;

the most frequently cited negative influences were competition and the lack of a business case 

for HIE.

To advance our understanding of HIE, future studies need to address comprehensive 

questions, use more rigorous designs, and be part of a coordinated systematic approach to 

studying HIE.
ES-9

References

1. Health Information Exchange.What is HIE? 

Washington, DC: Department of Health and 

Human Services. www.healthit.gov/

providers-professionals/health-informationexchange/what-hie. Accessed April 18, 

2014. 

2. Swain M, Charles D, Patel V, et al. Health 

Information Exchange among U.S. Nonfederal Acute Care Hospitals: 2008-2014. 

ONC Data Brief No. 24. Washington DC: 

Office of the National Coordinator for 

Health Information Technology; 2015. 

www.healthit.gov/sites/default/files/databrief/ONC_DataBrief24_HIE_Final.pdf.

Accessed April 19, 2015. 

3. Methods Guide for Effectiveness and 

Comparative Effectiveness Reviews. AHRQ 

Publication No. 10(14)-EHC062-EF. 

Rockville, MD: Agency for Healthcare 

Research and Quality. January 2014. 

Chapters available at 

www.effectivehealthcare.ahrq.gov. 

Accessed April 18, 2014. PMID: 21433403.

4. National Research Council. Finding What 

Works in Health Care: Standards for 

Systematic Reviews. Washington, DC: The 

National Academies Press; 2011.

5. Tzeel A, Lawnicki V, Pemble KR. The 

business case for payer support of a 

community-based health information 

exchange: a Humana pilot evaluating its 

effectiveness in cost control for plan 

members seeking emergency department 

care. Am Health Drug Benefits. 

2011;4(4):207-15. PMID: 25126351.

6. Frisse ME, Johnson KB, Nian H, et al. The 

financial impact of health information 

exchange on emergency department care. J 

Am Med Inform Assoc. 2012;19(3):328-33. 

PMID: 22058169.

7. Carr CM, Gilman CS, Krywko DM, et al. 

Observational study and estimate of cost 

savings from use of a health information 

exchange in an academic emergency 

department. J Emerg Med. 2014;46(2):250-

6. PMID: 24071033.

8. Winden TJ, Boland LL, Frey NG, et al. Care 

everywhere, a point-to-point HIE tool: 

utilization and impact on patient care in the 

ED. Appl Clin Inform. 2014;5(2):388-401. 

PMID: 25024756.

9. Bailey JE, Wan JY, Mabry LM, et al. Does 

health information exchange reduce 

unnecessary neuroimaging and improve 

quality of headache care in the emergency 

department? J Gen Intern Med. 

2013;28(2):176-83. PMID: 22648609.

10. Bailey JE, Pope RA, Elliott EC, et al. Health 

information exchange reduces repeated 

diagnostic imaging for back pain. Ann 

Emerg Med. 2013;62(1):16-24. PMID: 

23465552.

11. Lammers EJ, Adler-Milstein J, Kocher KE. 

Does health information exchange reduce 

redundant imaging? Evidence from 

emergency departments. Med Care. 

2014;52(3):227-34. PMID: 24374414.

12. Ross SE, Radcliff TA, Leblanc WG, et al. 

Effects of health information exchange 

adoption on ambulatory testing rates. J Am 

Med Inform Assoc. 2013;20(6):1137-42. 

PMID: 23698257.

13. Mäenpää T, Asikainen P, Gissler M, et al. 

Outcomes assessment of the regional health 

information exchange: a five-year follow-up 

study. Methods Inf Med. 2011;50(4):308-18. 

PMID: 21336419.

14. Ben-Assuli O, Shabtai I, Leshno M. The 

impact of EHR and HIE on reducing 

avoidable admissions: controlling main 

differential diagnoses. BMC Med Inform 

Decis Mak. 2013;13:49. PMID: 23594488.

15. Vest JR, Kern LM, Campion TR Jr, et al. 

Association between use of a health 

information exchange system and hospital 

admissions. Appl Clin Inform. 

2014;5(1):219-31. PMID: 24734135.

16. Ben-Assuli O, Shabtai I, Leshno M. Using 

electronic health record systems to optimize 

admission decisions: the Creatinine case 

study. Health Informatics J. 2015;21(1):73-

88. PMID: 24692078.
ES-10

17. Vest JR, Kern LM, Silver MD, et al. The 

potential for community-based health 

information exchange systems to reduce 

hospital readmissions. J Am Med Inform 

Assoc. 2015 Mar;22(2):435-42. Epub 2014 

Aug 6. PMID: 25100447.

18. Vest JR. Health information exchange and 

healthcare utilization. J Med Syst. 

2009;33(3):223-31. PMID: 19408456.

19. Tzeel A, Lawnicki V, Pemble KR. "Hidden" 

value: how indirect benefits of health 

information exchange further promote 

sustainability. Am Health Drug Benefits. 

2012;5(6):333-40. PMID: 24991331.

20. Lang E, Afilalo M, Vandal AC, et al. Impact 

of an electronic link between the emergency 

department and family physicians: a 

randomized controlled trial. CMAJ. 

2006;174(3):313-8. PMID: 16399880.

21. Jones SS, Friedberg MW, Schneider EC. 

Health information exchange, Health 

Information Technology use, and hospital 

readmission rates. AMIA Annu Symp Proc. 

2011;2011:644-53. PMID: 22195120.

22. Kern LM, Barrón Y, Dhopeshwarkar RV, et 

al. Health information exchange and 

ambulatory quality of care. Appl Clin 

Inform. 2012;3(2):197-209. PMID: 

23646072.

23. Nagykaldi ZJ, Yeaman B, Jones M, et al. 

HIE-i-health information exchange with 

intelligence. J Ambulatory Care Manage. 

2014;37(1):20-31. PMID: 24309392.

24. Shapiro JS, Johnson SA, Angiollilo J, et al. 

Health information exchange improves 

identification of frequent emergency 

department users. Health Aff. 

2013;32(12):2193-8. PMID: 24301405.

25. Willis JM, Edwards R, Anstrom KJ, et al. 

Decision support for evidence-based 

pharmacotherapy detects adherence 

problems but does not impact medication 

use. Stud Health Technol Inform. 

2013;183:116-25. PMID: 23388267.

26. Feldman SS, Horan TA. Collaboration in 

electronic medical evidence development: a 

case study of the Social Security 

Administration's MEGAHIT System. Int J 

Med Inf. 2011;80(8):e127-40. PMID: 

21333588.

27. Vest JR, Miller TR. The association between 

health information exchange and measures 

of patient satisfaction. Appl Clin Inform. 

2011;2(4):447-59. PMID: 23616887.

28. Chang KC, Overhage JM, Hui SL, et al. 

Enhancing laboratory report contents to 

improve outpatient management of test 

results. J Am Med Inform Assoc. 

2010;17(1):99-103. PMID: 20064809.

29. Campion TR Jr, Ancker JS, Edwards AM, et 

al. Push and pull: physician usage of and 

satisfaction with health information 

exchange. AMIA Annu Symp Proc. 

2012;2012:77-84. PMID: 23304275.

30. Altman R, Shapiro JS, Moore T, et al. 

Notifications of hospital events to outpatient 

clinicians using health information 

exchange: a post-implementation survey. 

Inform Prim Care. 2012;20(4):249-55. 

PMID: 23890336.

31. Afilalo M, Lang E, Léger R, et al. Impact of

a standardized communication system on 

continuity of care between family physicians 

and the emergency department. CJEM. 

2007;9(2):79-86. PMID: 17391577.

32. Machan C, Ammenwerth E, Schabetsberger 

T. Evaluation of the electronic transmission 

of medical findings from hospitals to 

practitioners by triangulation. Methods Inf 

Med. 2006;45(2):225-33. PMID: 16538293.

33. Maass MC, Asikainen P, Mäenpää T, et al. 

Usefulness of a Regional Health Care 

Information System in primary care. A case 

study. Comput Methods Programs Biomed. 

2008;91(2):175-81. PMID: 18514363.

34. Kaushal R, Dhopeshwarkar R, Gottlieb L, et 

al. User experiences with pharmacy benefit 

manager data at the point of care. J Eval 

Clin Pract. 2010;16(6):1076-80. PMID: 

20666888.

35. Park H, Lee S-I, Kim Y, et al. Patients' 

perceptions of a health information 

exchange: a pilot program in South Korea. 

Int J Med Inf. 2013;82(2):98-107. PMID: 

22658777.

36. Adler-Milstein J, Bates DW, Jha AK. U.S. 

regional health information organizations: 

progress and challenges. Health Aff. 

2009;28(2):483-92. PMID: 19276008.
ES-11

37. Adler-Milstein J, Jha AK. Health 

information exchange among U.S. hospitals: 

who's in, who's out, and why? Healthcare. 

2014;2(1):26-32. 

38. Audet A-M, Squires D, Doty MM. Where 

are we on the diffusion curve? Trends and 

drivers of primary care physicians' use of 

health information technology. Health Serv 

Res. 2014;49(1 Pt 2):347-60. PMID: 

24358958.

39. Furukawa MF, Patel V, Charles D, et al. 

Hospital electronic health information 

exchange grew substantially in 2008-12. 

Health Aff. 2013;32(8):1346-54. PMID: 

23918477.

40. Furukawa MF, King J, Patel V, et al. 

Despite substantial progress in EHR 

adoption, health information exchange and 

patient engagement remain low in office 

settings. Health Aff. 2014:1-8. PMID: 

25104827.

41. Hamann DJ, Bezboruah KC. Utilization of 

technology by long-term care providers: 

comparisons between for-profit and 

nonprofit institutions. J Aging Health. 

2013;25(4):535-54. PMID: 23509114.

42. Caffrey C, Park-Lee E. Use of electronic 

health records in residential care 

communities. NCHS Data Brief. 

2013(128):1-8. PMID: 24152578.

43. Adler-Milstein J, Bates DW, Jha AK. A 

survey of health information exchange 

organizations in the United States: 

implications for meaningful use. Ann Intern 

Med. 2011;154(10):666-71. PMID: 

21576534.

44. Hessler BJ, Soper P, Bondy J, et al. 

Assessing the relationship between health 

information exchanges and public health 

agencies. J Public Health Manag Pract. 

2009;15(5):416-24. PMID: 19704310.

45. Adler-Milstein J, McAfee AP, Bates DW, et 

al. The state of regional health information 

organizations: current activities and 

financing. Health Aff. 2008;27(1):w60-9. 

PMID: 18073225.

46. Bouhaddou O, Bennett J, Cromwell T, et al. 

The Department of Veterans Affairs, 

Department of Defense, and Kaiser 

Permanente Nationwide Health Information 

Network exchange in San Diego: patient 

selection, consent, and identity matching. 

AMIA Annu Symp Proc. 2011;2011:135-43. 

PMID: 22195064.

47. Byrne CM, Mercincavage LM, Bouhaddou 

O, et al. The Department of Veterans 

Affairs' (VA) implementation of the Virtual 

Lifetime Electronic Record (VLER): 

findings and lessons learned from Health 

Information Exchange at 12 sites. Int J Med 

Inf. 2014;83(8):537-47. PMID: 24845146.

48. Johnson KB, Gadd CS, Aronsky D, et al. 

The MidSouth eHealth Alliance: use and 

impact in the first year. AMIA Annu Symp 

Proc. 2008:333-7. PMID: 18999184.

49. Johnson KB, Unertl KM, Chen Q, et al. 

Health information exchange usage in 

emergency departments and clinics: the 

who, what, and why. J Am Med Inform 

Assoc. 2011;18(5):690-7. PMID: 21846788.

50. Vest JR, Zhao H, Jasperson J, et al. Factors 

motivating and affecting health information 

exchange usage. J Am Med Inform Assoc. 

2011;18(2):143-9. PMID: 21262919.

51. Hyppönen H, Reponen J, Lääveri T, et al. 

User experiences with different regional 

health information exchange systems in 

Finland. Int J Med Inf. 2014;83(1):1-18. 

PMID: 24200753.

52. Gadd CS, Ho Y-X, Cala CM, et al. User 

perspectives on the usability of a regional 

health information exchange. J Am Med 

Inform Assoc. 2011;18(5):711-6. PMID: 

21622933.

53. Messer LC, Parnell H, Huffaker R, et al. The 

development of a health information 

exchange to enhance care and improve 

patient outcomes among HIV+ individuals 

in rural North Carolina. Int J Med Inf. 

2012;81(10):e46-55. PMID: 22898321.

54. Myers JJ, Koester KA, Chakravarty D, et al. 

Perceptions regarding the ease of use and 

usefulness of health information exchange 

systems among medical providers, case 

managers and non-clinical staff members 

working in HIV care and community 

settings. Int J Med Inf. 2012;81(10):e21-9. 

PMID: 22854159.
ES-12

55. Massy-Westropp M, Giles LC, Law D, et al. 

Connecting hospital and community care: 

the acceptability of a regional data linkage 

scheme. Aust Health Rev. 2005;29(1):12-6. 

PMID: 15683350.

56. Kierkegaard P, Kaushal R, Vest JR. How 

could health information exchange better 

meet the needs of care practitioners? Appl 

Clin Inform. 2014;5(4):861-77. PMID: 

25589903.

57. McCullough JM, Zimmerman FJ, Bell DS, 

et al. Electronic health information exchange 

in underserved settings: examining 

initiatives in small physician practices & 

community health centers. BMC Health 

Serv Res. 2014;14:415. PMID: 25240718.

58. Hincapie AL, Warholak TL, Murcko AC, et 

al. Physicians' opinions of a health 

information exchange. J Am Med Inform 

Assoc. 2011;18(1):60-5. PMID: 21106994.

59. Ozkaynak M, Brennan PF. Revisiting 

sociotechnical systems in a case of 

unreported use of health information 

exchange system in three hospital 

emergency departments. J Eval Clin Pract. 

2013;19(2):370-3. PMID: 22420774.

60. Thorn SA, Carter MA, Bailey JE. 

Emergency physicians' perspectives on their 

use of health information exchange. Ann 

Emerg Med. 2014;63(3):329-37. PMID: 

24161840.

61. Rudin R, Volk L, Simon S, et al. What 

affects clinicians' usage of health 

information exchange? Appl Clin Inform. 

2011;2(3):250-62. PMID: 22180762.

62. Nohr C, Kristensen M, Andersen SK, et al. 

Shared experience in 13 local Danish EPR 

projects: the Danish EPR Observatory. Stud 

Health Technol Inform. 2001;84(Pt 1):670-

4. PMID: 11604822.

63. Unertl KM, Johnson KB, Lorenzi NM. 

Health information exchange technology on 

the front lines of healthcare: workflow 

factors and patterns of use. J Am Med 

Inform Assoc. 2012;19(3):392-400. PMID: 

22003156.

64. Feldman SS, Schooley LB, Bhavsar PG. 

Health information exchange 

implementation: lessons learned and critical 

success factors from a case study. JMIR 

Med Inform. 2014;2(2):e19. 

65. Merrill JA, Deegan M, Wilson RV, et al. A 

system dynamics evaluation model: 

implementation of health information 

exchange for public health reporting. J Am 

Med Inform Assoc. 2013;20(e1):e131-8. 

PMID: 23292910.

66. Phillips AB, Wilson RV, Kaushal R, et al. 

Implementing health information exchange 

for public health reporting: a comparison of 

decision and risk management of three 

regional health information organizations in 

New York state. J Am Med Inform Assoc. 

2014;21(e1):e173-7. PMID: 23975626.

67. Dullabh P, Hovey L. Large scale health 

information exchange: implementation 

experiences from five states. Stud Health 

Technol Inform. 2013;192:613-7. PMID: 

23920629.

68. Herwehe J, Wilbright W, Abrams A, et al. 

Implementation of an innovative, integrated 

electronic medical record (EMR) and public 

health information exchange for HIV/AIDS. 

J Am Med Inform Assoc. 2012;19(3):448-

52. PMID: 22037891.

69. Nykänen P, Karimaa E. Success and failure 

factors in the regional health information 

system design process--results from a 

constructive evaluation study. Methods Inf 

Med. 2006;45(1):85-9. PMID: 16482376.

70. Dullabh P, Ubri P, Hovey L. The State HIE 

Program Four Years Later: Key Findings on 

Grantees’ Experiences from a Six-State 

Review. Case Study Report. (Prepared by 

NORC under Contract No. 

HSSP2337010T/OS33547.) Office of the 

National Coordinator for Health Information 

Technology; 2014. 

www.healthit.gov/sites/default/files/CaseStu

dySynthesisGranteeExperienceFinal_12101

4.pdf. Accessed April 22, 2014.

71. Pagliari C, Gilmour M, Sullivan F. 

Electronic Clinical Communications 

Implementation (ECCI) in Scotland: a 

mixed-methods programme evaluation. J 

Eval Clin Pract. 2004;10(1):11-20. PMID: 

14731147.

72. Saff E, Lanway C, Chenyek A, et al. The 

Bay Area HIE. A case study in connecting 

stakeholders. J Healthc Inf Manag. 

2010;24(1):25-30. PMID: 20077922.
ES-13

73. Unertl MK, Johnson BK, Gadd SC, et al. 

Bridging organizational divides in health 

care: an ecological view of health 

information exchange. JMIR Med Inform. 

2013;1(1):e3. PMID: 25600166.

74. Morris G, Afzal S, Bhasker M, et al. QueryBased Exchange: Key Factors Influencing 

Success and Failure. Office of the National 

Coordinator for Health Information 

Technology; 2012. 

www.healthit.gov/sites/default/files/query_b

ased_exchange_final.pdf. Accessed April 

10, 2015. 

75. Ross SE, Schilling LM, Fernald DH, et al. 

Health information exchange in small-tomedium sized family medicine practices: 

motivators, barriers, and potential 

facilitators of adoption. Int J Med Inf. 

2010;79(2):123-9. PMID: 20061182.

76. Silvester BV, Carr SJ. A shared electronic 

health record: lessons from the coalface. 

Med J Aust. 2009;190(11 Suppl):S113-6. 

PMID: 19485857.

77. Goldwater J, Jardim J, Khan T, et al. 

Emphasizing public health within a health 

information exchange: an evaluation of the 

District of Columbia’s Health Information 

Exchange Program. EGEMS (Wash DC). 

2014;2(3):1090. PMID: 25848607.

78. Fontaine P, Zink T, Boyle RG, et al. Health 

information exchange: participation by 

Minnesota primary care practices. Arch 

Intern Med. 2010;170(7):622-9. PMID: 

20386006.

79. Steward WT, Koester KA, Collins SP, et al. 

The essential role of reconfiguration 

capabilities in the implementation of HIVrelated health information exchanges. Int J 

Med Inf. 2012;81(10):e10-20. PMID: 

22841703.

80. Adjerid I, Padman R. Impact of health 

disclosure laws on health information 

exchanges. AMIA Annu Symp Proc. 

2011;2011:48-56. PMID: 22195054.

81. Fairbrother G, Trudnak T, Christopher R, et 

al. Cincinnati Beacon Community Program 

highlights challenges and opportunities on 

the path to care transformation. Health Aff. 

2014;33(5):871-7. PMID: 24799586.

82. Adler-Milstein J, Bates DW, Jha AK. 

Operational health information exchanges 

show substantial growth, but long-term 

funding remains a concern. Health Aff 

(Millwood). 2013;32(8):1486-92. PMID: 

23840051.

83. Dixon B, Miller T, Overhage M. Barriers to 

achieving the last mile in health information 

exchange: a survey of small hospitals and 

physician practices. J Healthc Inf Manag. 

2013;27(4):55-8. 

84. Grossman JM, Kushner KL, November EA. 

Creating sustainable local health information 

exchanges: can barriers to stakeholder 

participation be overcome? Res Briefs. 

2008(2):1-12. PMID: 18496926.

85. Vest JR. More than just a question of 

technology: factors related to hospitals'

adoption and implementation of health 

information exchange. Int J Med Inf. 

2010;79(12):797-806. PMID: 20889370.

86. Schabetsberger T, Ammenwerth E, 

Andreatta S, et al. From a paper-based 

transmission of discharge summaries to 

electronic communication in health care 

regions. Int J Med Inf. 2006;75(3-4):209-15. 

PMID: 16112892.

87. Dobalian A, Claver ML, Pevnick JM, et al. 

Organizational challenges in developing one 

of the Nationwide Health Information 

Network trial implementation awardees. J 

Med Syst. 2012;36(2):933-40. PMID: 

20703640.

88. Overhage JM, Evans L, Marchibroda J. 

Communities' readiness for health 

information exchange: the National 

Landscape in 2004. J Am Med Inform 

Assoc. 2005;12(2):107-12. PMID: 

15561785.

89. Genes N, Shapiro J, Vaidya S, et al.

Adoption of health information exchange by 

emergency physicians at three urban 

academic medical centers. Appl Clin 

Inform. 2011;2(3):263-9. PMID: 23616875.

90. Lobach DF, Kawamoto K, Anstrom KJ, et 

al. Proactive population health management 

in the context of a regional health 

information exchange using standards-based 

decision support. AMIA Annu Symp Proc. 

2007:473-7. PMID: 18693881.
ES-14

91. Kern LM, Wilcox AB, Shapiro J, et al. 

Community-based health information 

technology alliances: potential predictors of 

early sustainability. Am J Manag Care. 

2011;17(4):290-5. PMID: 21615199.

92. Kern LM, Wilcox A, Shapiro J, et al. Which 

components of health information 

technology will drive financial value? Am J 

Manag Care. 2012;18(8):438-45. PMID: 

22928759.

93. Kern LM, Barrón Y, Abramson EL, et al. 

HEAL NY: promoting interoperable health 

information technology in New York State. 

Health Aff. 2009;28(2):493-504. PMID: 

19276009.

94. McGowan JJ, Jordan C, Sims T, et al. Rural 

RHIOs: common issues in the development 

of two state-wide health information 

networks. AMIA Annu Symp Proc. 

2007:528-32. PMID: 18693892.

95. Miller AR, Tucker C. Health information 

exchange, system size and information silos. 

J Health Econ. 2014;33:28-42. PMID: 

24246484.

96. Vest JR, Grinspan ZM, Kern LM, et al. 

Using a health information exchange system 

for imaging information: patterns and 

predictors. AMIA Annu Symp Proc. 

2013;2013:1402-11. PMID: 24551416.

97. Rudin RS, Simon SR, Volk LA, et al. 

Understanding the decisions and values of 

stakeholders in health information 

exchanges: experiences from Massachusetts. 

Am J Public Health. 2009;99(5):950-5. 

PMID: 19299671.
1

Introduction

Background

The use of health information technology (IT) has the potential to improve the quality, safety, 

and efficiency of health care in the United States and around the world.

1 Health IT can support 

patient care delivery activities such as communications, results reporting, order entry, care 

planning, and documentation. Examples of health IT applications include electronic health 

records (EHR), clinical decision support such as alerts and reminders, computerized provider 

order entry, electronic access to clinical practice guidelines and evidence databases, consumer 

health informatics applications, telemedicine, and electronic exchange of health information.

In recent years, the Health Information Technology for Economic and Clinical Health 

(HITECH) Act has accelerated EHR adoption in ambulatory and hospital settings across the 

United States. The HITECH Act, part of the American Recovery and Reinvestment Act of 2009, 

is providing up to $29 billion in incentive funding for the adoption and “meaningful use” of 

EHRs by hospitals and physicians. As a result of HITECH funding, 94 percent of non-Federal 

hospitals,2 78 percent of hospital-based physicians,3 84 percent of emergency departments, and 

73 percent of hospital outpatient departments in the United States have adopted EHRs.4 The 

motivation to increase the use of EHRs is grounded in evidence that health IT may improve the 

quality, safety, efficiency, and satisfaction with care, as has been reported in recent systematic 

reviews.5-8

A key challenge to effective use of health IT, however, is that most U.S. residents, especially 

those with multiple conditions, receive care across a number of settings. Among 3.7 million 

patients hospitalized in Massachusetts during a 5 year period, 31 percent were admitted to two or 

more hospitals (57% of all visits) and 1 percent were admitted to five or more hospitals (10% of 

all visits).9 Similarly, an analysis of 2.8 million patients seen by an emergency department in 

Indiana found that 40 percent had data at multiple institutions.10 These data silos present a 

challenge if we are to meet the goal stated by former Agency for Healthcare Research and 

Quality (AHRQ) Director Dr. Carolyn Clancy that, “data should follow the patient” wherever 

they get their care.11

To enable data to follow patients wherever they receive care, attention is now focused on

health information exchange (HIE), defined as the reliable and interoperable electronic sharing 

of clinical information among physicians, nurses, pharmacists, other health care providers, and

patients across the boundaries of health care institutions, health data repositories, states, and 

other entities who are not within a single organization or among affiliated providers.

12 The 

HITECH Act recognized that EHR adoption alone is insufficient to realize the full promise of 

health IT, allocating $563 million for States or State-designated entities to establish HIE

capability among health care providers and hospitals in their jurisdictions.

13 In the meantime, a 

growing number of private organizations have undertaken HIE.14 Ideally, HIE across health care 

organizations should facilitate care coordination and transitions between settings, improve 

patient safety, and reduce duplicate testing.

The Office of the National Coordinator for Health IT (ONC) has defined three forms of 

HIE:13
2

• Directed exchange: sending and receiving secure information electronically between care 

providers

• Query-based exchange: provider-initiated requests for information on a patient from other 

providers

• Consumer-mediated exchange: patients aggregating and controlling the use of their health 

information among care providers.

ONC also uses the words “push” to describe directed exchange and “pull” to describe querybased exchange.15

In general, HIE is defined as the electronic exchange of patient data across health care 

organizations. This excludes exchange of information that is predominantly paper-based as well 

as queries of remotely accessed systems (e.g., a clinician in one health care system seeking 

information residing in a system of another health care organization accessed over the Internet). 

Many also advocate that HIE be used as a verb or activity-based noun, and not as an entity or 

organization, even though many HIE implementations and/or the organizations implementing 

them call themselves “HIEs.”16,17

An early example of HIE was the work of Dr. Clement McDonald, who pioneered HIE in 

Indiana starting in the 1990s.18 This led to the formation of the Indiana Health Information 

Exchange, one of the largest and most successful HIE efforts in the United States.19 Other early 

efforts to implement HIE, including some high-profile efforts, were less successful.

20 Although 

the rationale for HIE has been viewed as critical,

21 the path to achieve it has in some respects

been more difficult than EHR adoption,22,23 in no small part due to the lack of sustainable 

business models.

24,25 Nonetheless, HIE adoption has grown as a result of the HITECH Act.26

Another barrier to HIE has been the development and adoption of health IT standards to 

ensure interoperability among systems. This has driven ONC, the lead U.S. government agency 

for health IT, to prioritize interoperability in its most recent strategic plan for health IT.27 ONC 

has also launched a process to establish an interoperability roadmap for guiding implementation 

of standards and interoperability, which also has the potential to facilitate adoption and 

improvement of HIE.28 An additional barrier to HIE described by ONC is “information

blocking,” which is the unintentional or deliberate prevention of information exchange between 

health IT systems.

29

Evaluating the effectiveness of HIE (and health IT generally) has been challenging.

30 HIE is 

a technology that is intermediate to improving care delivery, allowing clinicians and others 

improved access to patient data to inform decisions and facilitate appropriate use of testing and 

treatment. HIE is not specific to any health issue or diagnosis. HIE implementations have often

been supported by one-time start-up funding, without long-term support to sustain the programs 

long enough for evaluation. 

The promise for HIE to improve health care delivery is substantial, but adoption in its various 

forms has been complex and costly. It is therefore critical to be able to determine if HIE does 

improve health or intermediate outcomes as well as to systematically assess comparative 

approaches, barriers, return on investment, and sustainability of HIE.

Scope of Review and Key Questions

The review undertaken is timely and necessary—our knowledge of and experience with the 

HIE literature demonstrates an evidence base that is scattered across disciplines and in various 

formats. There are three previously published systematic reviews that focus exclusively on

HIE.

31-33 One of these reviews is almost a half-decade old,31 another focused only on U.S.-based 
3

and clinical-only (i.e., not public health) activities,32 and a third assessed only care outcomes and 

not larger issues of facilitators, barriers, and sustainability.33

In requesting this review, AHRQ’s goal is a report focused on systematically identifying and 

synthesizing evidence on the extent to which HIE can effectively improve a variety of outcomes,

and to determine if it is possible to say how the impact varies by different approaches to HIE.

This is due in part to AHRQ having funded a large portfolio of research in health IT and HIE,

34

and having published an extensive guide to evaluating HIE projects.35 This report also aims to

identify evidence on levels of use, and usability of HIE, as well as facilitators of and barriers to 

implementation, use, and sustainability of HIE. The analytic framework (Figure 1) and Key 

Questions used to guide this review are shown below. The analytic framework shows the target 

populations, interventions, and health outcomes examined, with numbers corresponding to the 

Key Questions.
4

Figure 1. Analytic framework

KQ = Key Question
5

This report focuses on the following Key Questions:

Key Question 1. Is HIE effective in improving clinical (e.g., mortality and morbidity), economic

(e.g., costs and resource use, the value proposition for HIE), and population (e.g., syndromic 

surveillance) outcomes?

Key Question 1a. Does effectiveness vary by type of HIE?

Key Question 1b. Does effectiveness vary by health care settings and systems?

Key Question 1c. Does effectiveness vary by IT system characteristics?

Key Question 1d. What evidence exists that the lack of HIE leads to poorer outcomes? 

Key Question 2. What harms have resulted from HIE? (e.g., violations of privacy, errors in 

diagnosis or treatment from too much, too little or inaccurate information, or patient or provider 

concerns about HIE)?

Key Question 2a. Do harms vary by type of HIE?

Key Question 2b. Do harms vary by health care settings and systems?

Key Question 2c. Do harms vary by the IT system characteristics?

Key Question 3. Is HIE effective in improving intermediate outcomes such as patient and 

provider experience, perceptions, or behavior; health care processes; or the availability, 

completeness, or accuracy of information?

Key Question 3a. Does effectiveness in improving intermediate outcomes vary by type of 

HIE?

Key Question 3b. Does effectiveness in improving intermediate outcomes vary by health care 

settings and systems?

Key Question 3c. Does effectiveness in improving intermediate outcomes vary by IT system 

characteristics?

Key Question 3d. What evidence exists that the lack of HIE leads to poorer intermediate 

outcomes?

Key Question 4. What are the current level of use and primary uses of HIE?

Key Question 4a. Do level of use and primary uses vary by type of HIE?

Key Question 4b. Do level of use and primary uses vary by health care settings and systems, 

or provider type?

Key Question 4c. Do level of use and primary uses vary by IT system characteristics?

Key Question 4d. Do level of use and primary uses vary by data source?

Key Question 5. How does the usability of HIE impact effectiveness or harms for individuals 

and organizations?

Key Question 5a. How usable are various types of HIE?

Key Question 5b. What specific usability factors impact the effectiveness or harms from 

HIE?

Key Question 5c. How does usability vary by health care settings or systems?

Key Question 6. What facilitators and barriers impact use of HIE?

Key Question 6a. Do facilitators and barriers that impact use vary by type of HIE?
6

Key Question 6b. Do facilitators and barriers that impact use vary by health care settings and 

systems?

Key Question 6c. Do facilitators and barriers that impact use vary by IT system 

characteristics?

Key Question 7. What facilitators and barriers impact implementation of HIE?

Key Question 7a. Do facilitators and barriers that impact implementation vary by type of 

HIE?

Key Question 7b. Do facilitators and barriers that impact implementation vary by health care 

settings and systems?

Key Question 7c. Do facilitators and barriers that impact implementation vary by IT system 

characteristics?

Key Question 8. What factors influence sustainability of HIE?
7

Methods

This systematic review follows the methods of the Agency for Healthcare Research and 

Quality (AHRQ) “Methods Guide for Effectiveness and Comparative Effectiveness Reviews.”36

Topic Development and Refinement 

The initial draft Key Questions were first provided by AHRQ, who requested this review as 

part of its effort to assess the impact of the AHRQ’s health information technology (IT) portfolio 

and set future direction for the field. The Key Questions and scope were further revised and 

developed by the review team with input from a group of stakeholders (Key Informants) 

convened for this review to provide diverse perspectives as well as content and methodological 

expertise. The Key Informants consisted of experts in health IT, applied informatics, clinical 

care, health policy and patient advocacy. Key Informants disclosed financial and other conflicts 

of interest prior to participation. The AHRQ Task Order Officer and the investigators reviewed 

the disclosures and determined that the Key Informants had no conflicts of interest that precluded 

participation. 

The project team, with input from a Technical Expert Panel (TEP) convened for this review, 

further developed the approach to this review. The TEP added expertise in informatics research 

and systematic reviews to the perspectives that were represented by the Key Informants. The 

Key Informants and TEP members are listed in the front matter. The protocol was then posted 

for public comment from February 6 to February 26, 2014. Based on public comments, we 

further revised the Key Questions and scope. The final protocol was developed and posted on 

July 21, 2014 on the AHRQ Web site at: http://effectivehealthcare.ahrq.gov/index.cfm/searchfor-guides-reviews-and-reports/?productid=1943&pageaction=displayproduct. The protocol was 

subsequently revised to document a change in the numbering of the Key Questions and reposted.

The original protocol was also registered in the PROSPERO international database of 

prospectively registered systematic reviews.37

Literature Search Strategy

A research librarian conducted searches in Ovid MEDLINE (1990 to February 2015), 

PsycINFO (1990 to February 2015), CINAHL (1990 through February 2015), the Cochrane 

Central Register of Controlled Trials (through January 2015), Cochrane Database of Systematic 

Reviews (through January 2015), the Database of Abstracts of Reviews of Effects, and the 

National Health Sciences Economic Evaluation Database (through the first quarter of 2015). See

Appendix A for the detailed search strategies. Searches were peer reviewed by a second librarian

with systematic review experience who offered suggestions and confirmed accuracy. Searches 

were designed to retrieve publications from January 1, 1990 forward, which reflects the timing of 

initial implementations of health information exchange (HIE) in the United States. Our search 

strategy was based on a broad terms and we evaluated this approach in several ways including 

determining if it successfully identified examples of several types of studies. During our 

literature scan we screened a sample of citations from two additional databases: Business 

Premier and the Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library; 

neither screen resulted in identification of relevant articles and the databases were not searched 

further. Searches were supplemented with hand searches of reference lists of relevant studies and 

the table of contents of journals not indexed in the databases searched (e.g., Generating Evidence 
8

and Methods to improve patient outcomes [eGEMs]), as well as searches of gray literature 

sources (e.g., reports and analyses on Web sites of key organizations). 

In addition, Scientific Information Packets were requested from organizations likely to have 

data on research or evaluations of health information exchange (HIE) that have not been 

published or indexed in citation databases. These organizations had the opportunity to submit 

data using the portal for submitting Scientific Information Packets on the Effective Health Care 

Program Web site. One submission was received from the California Health Care Foundation.

Process for Study Selection

The criteria for inclusion and exclusion of studies was based on the Key Questions and the 

populations, interventions, comparators, outcomes, timing, types of studies and setting (PICOTS) 

defined in the protocol (Appendix B). Papers were selected for review if they reported data about 

HIE (as defined below), had data relevant to a Key Question, and met the other pre-specified

inclusion criteria. Studies of nonhuman subjects and studies with no original data were excluded. 

Abstracts were independently reviewed by two investigators for inclusion. Full-text articles were 

obtained for all studies that any investigator identified as potentially meeting inclusion criteria. 

Two investigators independently reviewed all full-text articles for final inclusion. Sample sets of 

abstracts and full text articles were reviewed by the entire team at key points in the review 

process to establish norms. Inclusion was restricted to English-language articles. A list of the

included studies appears in Appendix C; a list of excluded studies and primary reasons for 

exclusion can be found in Appendix D. Discrepancies were resolved through discussion and 

consensus during team meetings with investigators.

Populations

Study population included any individual or group of health care providers, patients, 

managers, health care institutions, or regional organizations.

Intervention and Comparators

We defined HIE as the electronic sharing of clinical information among users such as

clinicians, patients, administrators, or policymakers, across the boundaries of health care 

institutions, health data repositories, States, and others, typically not within a single organization 

or among affiliated providers, while protecting the integrity, privacy, and security of the 

information. We did not include in this definition of HIE the exchange of information within a 

single organization or entity (e.g., exchange within a network such as Kaiser Permanente or the 

Veteran’s Administration or exchange across roles such as patient and clinician communications 

within a provider organization).

Comparators included were time period prior to HIE implementation, different locations 

(geographic or organizational without HIE) or situations in which HIE is not available (akin to 

“usual care” in a clinical study), comparisons across types of HIE, and comparisons of the 

characteristics of the different settings, health care system, and IT systems in which HIE is used.

Outcomes by Key Question

Key Question 1: Effectiveness was defined in terms of clinical outcomes (e.g., mortality and 

morbidity), economic outcomes (e.g., costs and resource use, the value proposition for HIE) and 
9

population outcomes (e.g., syndromic surveillance for the identification of trends or clusters).

Each study was assessed for its type of outcome and results in terms of the following attributes: 

• Location – geographic

• Health care setting – e.g., emergency department, outpatient, health system

• HIE type – query versus directed

• Outcome category

• Direction of result – benefit versus mixed versus none.

Key Question 2: Harms included unintended negative consequence or adverse events 

experienced by individuals, institutions, or organizations. Harms from HIE may include negative 

outcomes or the risk of negative outcomes resulting from information that is wrong, not provided 

in a timely manner, or in formats that inhibit its identification, comprehension, and use. Harms 

may result from too much information or insufficient information, or include negative impacts on 

attitudes (e.g., patient privacy concerns or clinician liability concerns).

Key Question 3: Intermediate outcomes included clinician and patient experiences and 

perceptions; changes in individual behavior or care delivery processes; and changes in the 

availability, completeness or accuracy of information.

Key Question 4: Level of use was a measure of the usage of HIE use by individuals, health 

care institutions, or regional organizations.

Key Question 5: Usability focused on the function of the HIE in terms of the interaction 

between users and HIE and their ability or capacity to navigate and accomplish tasks.

Key Question 6: Facilitators and barriers were the drivers and challenges to use of HIE in

the workflow and decisions of patients, clinicians, or organizations. 

Key Question 7: Implementation of HIE was defined as the realization of an HIE project 

such that the exchange of data is operational.

Key Question 8: Sustainability was long-term maintenance, development, and improvement 

or expansion of HIE, after the implementation period.

Timing 

No prespecified minimum duration of time was required between implementation of HIE to 

the measurement of outcomes.

Settings

Settings included any aspect of the location or venue in which health information is 

exchanged for the purpose of improving health or health care that is hypothesized to impact 

effectiveness, use, usability, or sustainability. This included the type(s) of clinical environments 

(e.g., ambulatory care, hospital, nursing home), payment/reimbursement model(s) (e.g., fee-forservice, managed care, risk/value-based model such as an accountable care organization), and 

legislative requirements (e.g., participation in HIE required to participate in Medicaid). Also 

included were studies in public health organizations and settings; those using HIE data for 

clinical research were excluded.

Study Design

Our approach to decisions about what designs and units of analysis to include varied across 

the Key Questions, reflecting the fact that different types of research was needed to answer 

different types of questions.
10

For questions on efficacy, effectiveness, and harms a “best evidence” approach was used. 

Randomized controlled trials (RCTs) were included as the top-tier evidence. If insufficient 

evidence was found of this type, observational studies (defined as cohort studies comparing at 

least two HIE systems, case-control studies, and time-series studies) were explored. 

For questions on use, usability, implementation, and sustainability, observational studies and 

qualitative research were included. We also included detailed case studies of multiple HIE

organizations or sites. For studies of use and usability we included examinations both on the 

individual level and organizational level, while implementation and sustainability were defined 

as organizational level activities.

Systematic reviews were considered as sources of studies to be reviewed for possible 

inclusion. High quality reviews with information directly relevant to our Key Questions were 

eligible for inclusion in this review as evidence. High-quality reviews were defined as those 

assessed as being at low risk of bias, according to the Assessing the Methodological Quality of 

Systematic Reviews-AMSTAR quality assessment tool.36,38

We excluded studies that modeled the potential impact of HIE or that presented, discussed, or 

evaluated hypothetical situations about HIE not yet implemented. Also excluded were 

descriptive narratives or “lessons learned” essays that were not based on collecting clinical, 

survey, or interview data from identified users or stakeholders. We restricted inclusion to 

English-language articles, but reviewed English language abstracts of non-English language 

articles to identify studies that would otherwise meet inclusion criteria. 

See Appendix E for the study design terminology used in this review. 

Data Abstraction and Data Management

After studies were selected for inclusion, data were abstracted into categories including but

not limited to: (a) general information such as study design, year, setting, geographic location, 

and duration; (b) characteristics of the HIE implementation such as the form (directed exchange, 

query-based exchange, consumer-mediated exchange), the number and types of participating 

organizations, the type of user interface (e.g., push vs. pull), and the types of information 

included; and (c) key contextual information to be used to identify facilitators and barriers to 

HIE use as well as to assess applicability of the results. At a minimum, we included details about 

the type(s) of clinical environments (e.g., ambulatory care, hospital, nursing home), 

payment/reimbursement model(s) (e.g., fee-for-service, managed care setting, risk/value-based 

model such as an accountable care organization), and relevant outcomes. Abstracted information 

is included in Appendix F and is also available in the Systematic Review Data Repository.

Assessment of Methodological Risk of Bias of Individual 

Studies 

Assessment of risk of bias of trials and observational studies was based on recommendations 

in the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews.

36 Two 

investigators independently assessed risk of bias for all effectiveness studies. Differences were

resolved by discussion and consensus and reviewed by the team of investigators. Individual 

studies were rated as “low,” “moderate,” or “high” risk of bias. The criteria and interpretation of 

these ratings are described in our protocol and in Appendix G. 

For studies of surveys, interviews, and focus groups we did not give a formal overall risk of 

bias rating; however, we did record information about sampling, completion rates, the 
11

development of the questions, and the appropriateness of the analysis. This information informed 

our descriptions of the studies and assessment of both the strength of evidence and the specific 

needs for future research. Appendix G includes a list of the information we recorded. Risk of 

bias was not assessed for case studies, mixed methods studies, or studies based on computer 

system logs.

Data Synthesis and Organization of Report

We constructed evidence tables identifying the study characteristics, results of interest, and 

risk of bias assessment for all included studies with summary tables to highlight the main 

findings. For each study, we recorded the type of HIE when described, information on the 

sample and response rate when reported, and types of stakeholders. We reviewed and highlighted

studies by using a hierarchy of evidence approach, where the best evidence was the focus of our 

synthesis for each Key Question. 

We found heterogeneity in the interventions and outcomes measured, including how similar 

outcomes were measured and reported, such that we did not conduct meta-analyses. We 

combined studies in the synthesis of the results based on the similarity of the type of HIE, the 

implementation of the HIE, outcomes measured, and results reported. Where studies were not 

similar in these areas we provided the results of the invidvidual studies without combining them. 

The evidence for Key Questions 1, 2, and 3 were summarized and presented together as there 

were few studies that reported on primary clinical outcomes and no studies that explicitly 

analyzed harms. Many studies that reported resource usage (primary economic outcomes) were 

actually reporting on clinical process outcomes, such as use of testing or prevention of hospital 

admissions. We included studies of perceptions of HIE only if an actual operational HIE 

implementation was analyzed. For Key Question 4 there were two categories of studies: large, 

mostly national surveys that examined HIE use on a macro level (e.g., which organizations did or 

did not use HIE); and studies that examined how HIE was used within organizations. We 

presented the evidence for Key Questions 5 (usability) and 6 (barriers and facilitators to use)

jointly as some studies addressed both sets of questions together. 

Similarly, we presented the results for Key Questions 7 and 8 together because conceptually, 

organizations consider sustainability when deciding whether or not to adopt an innovation or 

implement a new practice and conversely sustainability is at least partially dependent on the form 

and success of implementation. As a result, there is significant overlap in the research. Many of 

the studies we identified either addressed implementation and sustainability, or addressed 

implementation as well as the topics covered by other Key Questions – impact, use, or 

usage/usability. The focus of the results section for Key Questions 7 and 8 is on categories of 

facilitators and barriers. We grouped the factors identified in the literature into categories in 

order to provide a summary.

Grading the Body of Evidence for Each Key Question 

The strength of evidence for key outcomes was rated only for effectiveness and harms 

outcomes in Key Questions 1, 2, and 3 using the four categories recommended in the AHRQ 

Methods Guide.

36

• A “high” grade indicates high confidence that the estimate of effect lies close to the true 

effect for this outcome. The body of evidence has few or no deficiencies and the findings 

are stable (i.e., another study would not change the conclusions). 
12

• A “moderate” grade indicates moderate confidence that the estimate of effect lies close to 

the true effect for this outcome. The body of evidence has some deficiencies and findings 

are likely to be stable, but some doubt remains. 

• A “low” grade indicates low confidence that the estimate of effect lies close to the true 

effect for this outcome. The body of evidence has major or numerous deficiencies (or 

both) and additional evidence is needed before concluding either that the findings are 

stable or that the estimate of effect is close to the true effect. 

• An “insufficient” grade indicates inability to estimate an effect or no confidence in the 

estimate of effect for this outcome, no evidence is available or the body of evidence has 

unacceptable deficiencies, precluding reaching a conclusion. 

For a more detailed description of the methods and domains used to rate strength of evidence, 

see Appendix H. 

Other outcomes (e.g., perceptions in Key Question 3) and outcomes for Key Questions 4 

through 8 were not formally evaluated for strength of evidence.

Assessing Applicability 

Applicability is defined as the extent to which the effects observed in published studies are 

likely to reflect the expected results when a specific intervention is applied to the population of 

interest under “real-world” conditions.

36 It is an indicator of the extent to which research 

included in a review might be useful for informing clinical decisions in specific situations.

Applicability depends on the particular question and the needs of the user of a review. There is 

no generally accepted universal rating system for applicability. In addition, applicability depends 

in part on context. Therefore, a rating of applicability (such as “high” or “low”) was not assigned 

because applicability may differ based on the user of a review. Rather, factors important for 

understanding the applicability of studies were recorded, such as differences in the organizations 

(e.g., payment/reimbursement model, range of services provided, governance structure, IT 

systems) and people (e.g., profession, type of relationship with the organization, tenure) affected 

by the creation and implementation of the HIE that was the subject of study, the scope of the 

HIE, the clinical settings involved, and the geographic area (e.g., states, regions or countries) in 

which the studies were performed.

Peer Review and Public Commentary

Experts in HIE, individuals representing important stakeholder groups, and Technical Expert 

Panel members were invited to provide external peer review of this systematic review. The

AHRQ Task Order Officer and a designated Evidence-based Practice Center Associate Editor

also provided comments and editorial review. To obtain public comment, the draft report was

posted on the AHRQ Web site for 4 weeks from March 12 to April 8, 2015. A disposition of

comments report detailing the authors' responses to the peer and public review comments will be

made available after AHRQ posts the final systematic review on the public Web site.
13

Results

Results of Literature Searches

Results of the literature search and selection process are summarized in the literature flow 

diagram (Figure 2). Database searches resulted in 5,211 potentially relevant citations. After dual 

review of abstracts and titles, 849 articles were selected for full-text review. After dual review of 

full text articles, 136 studies were included. Data extraction and risk of bias assessment tables for 

included studies are available in Appendixes F and I.

Figure 2. Literature flow diagram

HIE = health information exchange; KQ= Key Question

*Cochrane databases include the Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, 

Database of Abstracts of Reviews of Effects, and National Health Sciences Economic Evaluation Database. †Identified from reference lists, hand searching, suggested by experts, and other sources. ‡Publications may address more than one Key Question, studies may have multiple publications.

Description of Included Studies

Of the 136 studies included in this review, two randomized controlled trials (RCTs) 

described in three papers and 32 observational and survey studies addressed Key Questions 1, 2, 
14

and 3, pertaining to the effectiveness of improving clinical, economic, population, and 

intermediate outcomes. Most were conducted in the United States, although eight were from 

Europe, Canada, Israel, and South Korea. Most studies reported clinical or public health process, 

economic, or population outcomes, while no studies reported harms of health information 

exchange (HIE). The majority were assessed to be of low risk of bias but also contained lowquality, mostly retrospective evidence. We identified 58 studies that addressed Key Question 4, 

pertaining to the use of HIE. The majority were conducted in the United States and were low risk 

of bias or could not be rated due to study design. Twenty-two studies were identified that

addressed Key Questions 5 and 6, pertaining to usability and facilitators and barriers to use. Most 

were assessed to be of moderate risk of bias and were conducted in the United States, Austria,

and Australia. A total of 52 studies addressed Key Questions 7 and 8, related to HIE 

implementation and sustainability. These studies used varying types of qualitative methods; for 

those that could be assessed for risk of bias, most were found to have a high risk of bias.

Key Question 1. Is HIE effective in improving clinical, economic, and 

population outcomes?

Key Question 2. What harms have resulted from HIE? 

Key Question 3. Is HIE effective in improving intermediate outcomes such 

as patient and provider experience, perceptions, or behavior; health care 

processes; or the availability, completeness, or accuracy of information?

Key Points

• HIE has been studied in far fewer places than it has been implemented, resulting in a 

research literature skewed toward a relatively small number of sites.

• Although the potential uses of HIE are broad, most studies reported on narrow questions, 

such as reduction in test ordering or consultations, and not larger overall clinical and 

financial impacts. Furthermore, most of these studies were conducted retrospectively, 

making cause and effect difficult to ascertain.

• The strength of evidence for HIE in improving clinical, economic, or population 

outcomes was low.

• Most studies also reported positive results, raising concerns about publication bias.

Detailed Synthesis

We identified 34 studies that assessed some sort of outcome from HIE use (Table 1). 

Mapping to our original Key Questions, a total of 26 studies were deemed to report clinical

(intermediate), economic, or population outcomes (Key Question 1), while eight were found to 

report on perceptions of outcomes (Key Question 3). However, no studies evaluated primary 

clinical outcomes from HIE (e.g., mortality and morbidity - Key Question 1), and none explicitly 

assessed harms (Key Question 2). Additionally, some studies reported outcomes for more than 

one of the outcomes in the Key Questions. For these reasons, we present the results of Key 

Questions 1 through 3 together below. 
15

The most common study design for assessing outcomes was retrospective cohort, typically 

with HIE use associated with some specific outcome factor.39-56 The next most common design 

was survey, which was usually focused on perception of outcomes.57-64 Two studies were RCTs, 

one of a particular directed information exchange (2 published papers, 1 on clinical outcomes65

and the other on perceptions66) and the other of a clinical decision support intervention using 

data from an HIE implementation.

67 Two studies used cross-sectional analyses of large databases 

to compare those having access to HIE with those without access.68,69 Two other studies used a 

case series methodology, one of which involved asking clinicians if HIE access avoided 

undesirable resource use, and then calculating the costs saved70 and the other that retrospectively 

analyzed data to determine duplicative testing averted.

71

The identified studies were performed mostly in the United States, but we identified eight 

studies from five other countries (Austria,62 Canada,

65,66 Finland,46,61 Israel,41,72 and South 

Korea63). Of the 26 U.S. studies, three assessed multiple HIE implementations in two states (1 

study)69 and the entire country (2 studies).64,68 The remaining 23 studies were conducted (1 study 

per State unless otherwise noted) in Colorado,50 Indiana (3 studies),42,49,59 Louisiana,

47

Massachusetts,60 Minnesota71, North Carolina,67 New York (6 studies),45,51,55-58 Oklahoma,48

South Carolina,70 Tennessee (3 studies),39,40,44 Texas,54 Virginia,43 and Wisconsin (2 studies).52,53

The number of studies and their locations in the United States represent a small fraction of 

those reporting to be operational, sustainable, or innovating according to the eHealth Initiative 

Annual Data Exchange Survey, which reported a total of 84 such HIE implementations in 201373

and 106 in 2014.74 In other words, while a substantial number of HIE implementations exist in 

the United States, only a small number have been subject to evaluation. This low number of 

studies relative to HIE efforts also makes it difficult to generalize factors about aspects of them, 

such as location, HIE type, and setting, with results of research.

In Table 1, we present the results of these studies by outcome category, classifying the 

study’s geographic location, health care setting, HIE type (query vs. directed), and general 

direction of the results. Due mainly to study design and performance or reporting limitations, and 

the lack of ability to combine results, the strength of this body of evidence was rated as low.
16

Table 1. Studies of HIE included for assessing outcomes

Study Location Setting HIE Type Study Type

Risk of 

Bias

Direction 

of 

Result(s)

Outcome(s) Assessed Results

Laboratory Testing or Cost of Testing

Mäenpää et al., 

201146

Tampere, 

Finland

Outpatient Query Retrospective

cohort

Low Negative Lab test ordering Increased lab 

testing

Ross et al., 201350 Mesa County, 

Colorado

Outpatient Query Retrospective

cohort

Low Beneficial Rate of increase in lab 

testing

Reduced rate of 

increase in lab 

testing

Carr et al., 201470 Charleston, 

South Carolina

ED Query Case series Moderate Beneficial Lab testing Reduced lab testing

Frisse et al., 

201244

Memphis, 

Tennessee

ED Query Retrospective

cohort

Moderate Beneficial Lab testing Reduced lab testing

Tzeel et al., 201152 Milwaukee, 

Wisconsin

ED Query Retrospective

cohort

Low Beneficial ED visit costs Decreased with HIE 

use; driven by 

reduced testing

Winden, et al., 

201471

Minnesota ED Query Case series Moderate Beneficial Lab testing Reduction of 

duplicate lab testing

Radiology Testing

Bailey et al., 

201339

Memphis, 

Tennessee

ED Query Retrospective

cohort

Low Beneficial Use of neuroimaging Reduced imaging

Bailey, et al., 

201340

Memphis, 

Tennessee

ED Query Retrospective

cohort

Low Beneficial Use of back imaging Reduced imaging

Carr et al., 201470 Charleston, 

South Carolina

ED Query Case series Moderate Beneficial Use of radiology testing Reduced imaging

Frisse et al., 

201244

Memphis, 

Tennessee

ED Query Retrospective

cohort

Moderate Beneficial Use of radiology testing Reduced imaging

Lammers, AdlerMilstein, and 

Kocher, 201469

California and 

Florida

ED Varied Cross-sectional Low Beneficial Reimaging in ED Reduced imaging

among those who 

implemented HIE
17

Table 1. Studies of HIE included for assessing outcomes (continued)

Study Location Setting HIE Type Study Type

Risk of 

Bias

Direction 

of 

Result(s)

Outcome(s) Assessed Results

Radiology Testing (continued)

Mäenpää et al., 

201146

Tampere, 

Finland

Outpatient Query Retrospective

cohort

Low Beneficial Use of radiology testing Reduced imaging

Ross et al., 201350 Mesa County, 

Colorado

Outpatient Query Retrospective

cohort

Low None Use of radiology testing No impact on 

imaging

Tzeel et al., 201152 Milwaukee, 

Wisconsin

ED Query Retrospective 

cohort

Low Beneficial ED visit costs Decreased with HIE 

use; driven by 

reduced testing

Winden, et al.,

201471

Minnesota ED Query Case series Moderate Beneficial Use of radiology testing Reduction of 

duplicate imaging

Hospital Admissions

Ben-Assuli, 

Shabtai, and 

Leshno, 201341

Israel HMO Query Retrospective

cohort

Low Beneficial Hospital admissions Decreased with HIE 

use

Ben-Assuli, 

Shabtai and 

Leshno, 201572

Israel HMO Query Retrospective 

cohort

Low Beneficial Hospital admissions Decreased with HIE 

use

Frisse et al., 

201244

Memphis, 

Tennessee

ED Query Retrospective 

cohort

Low Beneficial Hospital admissions Decreased with HIE 

use

Carr et al., 201470 Charleston South 

Carolina

ED Query Case series Moderate Beneficial Hospital admissions Decreased with HIE 

use

Tzeel et al., 201253 Milwaukee, 

Wisconsin

ED Query Retrospective

cohort

Low Mixed Hospital admissions

Length of Stay

Increased 

admissions but 

decreased LOS

Vest, 200954 Austin, Texas ED Query Retrospective

cohort

Low Beneficial Hospital admissions for 

ambulatory-sensitive 

diagnoses in indigent 

patients

Increased with use 

of HIE

Vest et al., 2014 56 Rochester, New 

York

ED Query Retrospective

cohort

Low Beneficial Hospital admissions Reduced with HIE
18

Table 1. Studies of HIE included for assessing outcomes (continued)

Study Location Setting HIE Type Study Type

Risk of 

Bias

Direction 

of 

Result(s)

Outcome(s) Assessed Results

Hospital/ED Readmissions

Lang et al., 200665 Montreal, 

Canada

ED Directed RCT Moderate None ED return visits No difference

Vest et al., 2014 55 Rochester, New 

York

ED Query Retrospective

cohort

Low Beneficial Hospital readmissions Decreased with HIE 

use

Jones, Friedberg 

and Schneider, 

201168

U.S. All Varied Cross-sectional Low None Hospital readmissions No difference

Referrals and/or Consultations

Carr et al., 201470 Charleston, 

South Carolina

ED Query Case series Moderate Beneficial Consultation Reduced with HIE 

use

Mäenpää et al., 

201146

Tampere, 

Finland

Outpatient Query Retrospective

cohort

Low Mixed Referral ordering Increased referrals 

with HIE

Emergency Department Costs

Frisse et al., 

201244

Memphis, 

Tennessee

ED Query Retrospective

cohort

Low Beneficial Overall cost Decreased with HIE 

use

Tzeel et al., 201152 Milwaukee, 

Wisconsin

ED Query Retrospective

cohort

Low Beneficial ED visit costs Decreased with HIE 

use; driven by 

reduced lab testing

Public Health Reporting

Magnus et al., 

201247

Louisiana Public health Directed Retrospective

cohort

Low Beneficial Followup care for HIV 

patients

Improved with HIE

Dixon, McGowan 

and Grannis, 

201142

Indiana Public health Directed Retrospective

cohort

Low None Completeness of public 

health reporting

Incomplete due to 

poor quality of 

clinical data

Overhage et al., 

200849

Indiana Public health Directed Retrospective

cohort

Low Beneficial Identification and 

completeness of 

notifiable disease 

reporting

Increased notifiable 

diseases found and 

completeness of 

data for diseases 

found
19

Table 1. Studies of HIE included for assessing outcomes (continued)

Study Location Setting HIE Type Study Type

Risk of 

Bias

Direction 

of 

Result(s)

Outcome(s) Assessed Results

Quality of Ambulatory Care

Kern et al., 201245 Hudson Valley, 

New York

Outpatient Query Retrospective

cohort

Low Beneficial Clinical quality 

measures

Increased with HIE 

Nagykaldi et al., 

201448

Norman and 

Oklahoma City,

Oklahoma

Outpatient Query Retrospective

cohort

Moderate Beneficial Clinical quality 

measures

Increased with HIE 

Willis et al., 201367 North Carolina Outpatient Query RCT Moderate Beneficial Documentation and 

medication 

reconciliation

Increased with HIE

Other Aspects of HIE

Feldman and 

Horan, 201143

Virginia Government Directed Retrospective

cohort

Moderate Beneficial Case processing time 

for SSD determination

Decrease in mean 

case processing 

time

Shapiro et al., 

201351

New York ED Query Retrospective

cohort

Moderate Beneficial Identification of 

frequent ED users

Increased with HIE

Vest and Miller, 

201164

U.S. Hospital Varied Cross-sectional Low Beneficial Patient satisfaction with

hospital care

Higher in 

implemented than 

adopted hospitals

CDS = clinical decision support; CQI = continuous quality improvement; ED = emergency department; HIE = health information exchange; HMO = health maintenance 

organization; LOS = length of stay; PCP = primary care provider; PH = public health; RCT = randomized, controlled trial; SSD = Social Security Disability; 

VA = Veterans Affairs; vs. = versus
20

With the exception of two RCTs (in 3 publications) and one other study with a prospective 

design, most studies used retrospective designs, usually with an approach examining the 

association of HIE use with one or more clinical variables. All of these studies focused on the 

direct effect of HIE, usually in reducing resource use or costs, without determining its larger 

impact (e.g., overall total or proportion of spending in an emergency department [ED] vs. the 

total dollar amounts that HIE appeared to save). None of the studies analyzed individual episodes 

of care to determine clinical appropriateness of possible changes brought about by HIE use.

The prospective studies also had limitations. The RCTs were focused on highly specific uses 

of HIE, namely directed exchange of ED reports in one and pharmacotherapy clinical decision 

support in another. Of note, however, was that neither study showed benefit of HIE. The other 

prospective study was limited by methodology of physicians self-reporting of resources not 

utilized when HIE was used, with no followup or validation of their decisions, or analysis of 

more holistic views of clinical outcomes or costs.

While most of these studies had reasonable internal validity, questions of external validity

remain, especially since the intervention (HIE) was only one of many potential influences on

clinical outcome (i.e., many more factors go into clinical outcomes than the decision to consult 

an HIE system on a patient). As a result, most studies with appropriate retrospective methods are 

listed as having low or moderate risk of bias due to their proper internal validity but there are still 

significant concerns about external validity.

Improving Resource Use

Laboratory Testing

Six studies addressed laboratory testing, with five finding a benefit of HIE in reducing 

overall tests, although estimates of impact on cost were mixed.44,46,50,52,70,71 Four of these studies 

took place in the ED setting, all showing some aspect of reduced testing and cost savings. Two 

studies found overall reduced laboratory testing, with one reporting an odds ratio (OR) of testing 

among patients for whom HIE was accessed to be 0.880 (95% confidence interval [CI], 0.828 to 

0.935)44 and the other noting 23 percent fewer lab testing procedures (statistical significance not 

reported) in a propensity-matched group of patients for whom HIE could have been used.52 A 

third study logged physician self-reports of laboratory testing averted with use of HIE in the ED, 

with savings over 3 months of $462 calculated from tests reportedly not ordered.70 A fourth 

study found 96 instances of duplicate lab testing averted in 1,488 patient encounters that were 

retrospectively analyzed.71Two studies were conducted in ambulatory settings, against a 

backdrop of increased overall laboratory testing. One U.S. study found that after HIE 

implementation, there was a reduction in the rising rate of testing, without overall cost savings.

50

In contrast, a study in Finland found increased laboratory testing during the period of HIE 

implementation (19.0% for primary care physicians and 7.0% for specialist physicians per total 

patient appointments).46 As with all retrospective studies, the four studies of laboratory testing

could have been complicated by confounders, while the prospective study did not validate 

physician self-reporting of tests avoided or measure overall costs of care for the ED encounter or 

subsequent utilization.
21

Radiology Testing

Nine studies assessed radiology testing, with all but one reporting an association of reduced

testing with HIE.39,40,44,46,50,52,69-71 Six of these studies also examined laboratory testing and are 

described previously,

44,46,50,52,71,75 and three additional ED studies assessed only imaging.39,40,69

The ED studies showed a variety of findings. One study found that for all radiologic imaging, 

there was reduction of head computed tomography (CT) imaging, (OR of 0.913, 95% CI, 0.842 

to 0.991) as well as body CT imaging (OR 0.886, 95% CI, 0.828 to 0.948) but no significant 

changes in echocardiogram, chest x-ray, or ankle x-ray testing across 12 EDs.44 Another study

demonstrated 22 percent decreased diagnostic radiology ordering and 52 percent reduced CT 

scan ordering (statistical significance not reported) when HIE was used in the ED.52 Two 

additional studies assessed neuroimaging for headache39 and repeat imaging for back pain in 

EDs.40 For neuroimaging, HIE usage was associated with decreased diagnostic imaging (OR 

0.38; 95% CI, 0.29 to 0.50) and increased adherence to evidence-based guidelines (OR 1.33; 

95% CI, 1.02 to 1.73), although there was no significant change in overall costs. HIE usage was 

associated with reduced repeat imaging for back pain (OR 0.36; 95% CI, 0.18 to 0.71), but no 

change in cost due to higher use of CT scans with HIE access. A prospective case series study

reported $161K in savings over 3 months through averted radiologic testing in EDs,70 while a 

retrospective case series found 453 duplicate radiology testing in 1,488 patient encounters

retrospectively analyzed.

71

One cross-sectional study looked at repeat imaging in the ED in two states (California and 

Florida), finding reduced probability of repeat CT (-8.7%; 95% CI, -14.7% to -2.7%), ultrasound 

(-9.1%; 95% CI, -17.2% to -1.1%), and chest x-ray (-13.0%; 95% CI, -18.3% to -7.7%) ordering 

in hospitals that had HIE participation as reported in the Healthcare Information and 

Management Systems Society Analytics Database of hospital information technology (IT) 

functionality.69

In ambulatory settings, one U.S. study showed no statistically significant reduction in the rate 

of radiologic testing.

50 However, a Finland-based study showed a reduction in radiologic testing

(16.4% reduction for primary care physicians and 11.0% reduction for specialist physicians).46

Hospital Admissions

Eight studies assessed the role of HIE in reducing hospital admissions, with inconsistent 

findings.

41,44,53,54,56,65,70,72 Two studies (described above) found a reduction in hospital admissions 

and lower costs using methods previously described. The bulk of the $1.07 million annual 

savings due to HIE found in one study resulted from reduced admissions.44 Another study also 

reported $118K in savings from averted admissions over a 3-month period.70 Two studies in an 

Israeli health maintenance organization found that viewing the medical history via an electronic 

health record (EHR) decreased possibly redundant admissions, with even greater reductions 

when information was accessed using HIE.41,72 A study in New York found that viewing 

information reduced odds of admission (OR 0.70; 95% CI, 0.52 to 0.95).56

Other studies, however, found no benefit from HIE in terms of avoiding hospital admissions. 

An RCT of directed HIE in Canada providing family physicians electronic reports of ED visits 

versus paper-based reports resulted in no difference in hospital admissions or return visits to the 

ED.65 Other studies found that HIE was associated with increased admissions for ambulatorysensitive diagnoses54 and a 28 percent increased rate of admissions, although such admissions 

had reduced length of stay with 771 fewer bed days per 1,000 health plan members over 16 

months.

53
22

Two studies assessed HIE in reducing hospital readmissions. One study found that assessing 

information in an HIE implementation was associated with reduced odds of hospital readmission 

(OR 0.43; 95% CI, 0.27 to 0.70)55 while another found that U.S. hospitals participating in HIE in 

2007 did not have lower readmission rates for acute myocardial infarction, pneumonia, or heart 

failure.68

Referrals and Consultations

Two studies, described previously, assessed HIE for reducing referrals and/or consultations. 

The prospective ED case series reported reduced consultations, leading to savings of $3,990 over 

3 months.70 The Finland-based ambulatory study, however, found that HIE was associated with 

increased referrals by primary care physicians (43.6%) and specialists (12.8%).46

ED Cost

Another two studies addressed reducing overall ED costs per patient, with both finding 

reductions when HIE was available. One study found that an HIE system encompassing 12 EDs 

resulted in net annual savings (total savings minus operating costs) of $1.07 million, with 

reduced hospital admissions accounting for 97.6 percent of the reduction.44 Another study found 

that for a propensity-matched group of patients for whom HIE could have been used, the group 

for whom HIE was used had $29 per ED visit less expenditures.52 Neither study reported overall 

ED expenditures, making it unknown what proportion of overall ED spending was impacted by 

HIE.

Public Heath Reporting

Three studies assessed HIE in public health settings, all of which were conducted in the 

United States.

42,47,49 Two examined the completeness of notifiable disease reporting data. One 

study compared usual (“spontaneous”) public health reporting with automated lab reporting 

through the HIE, finding a 4.4-fold higher rate of reporting for the HIE-based approach, with 

cases identified an average of 7.9 days earlier.49 The other study showed equal or improved 

completeness of reporting for a variety of data fields in notifiable disease reports, although 

completeness was reduced for some fields (e.g., laboratory units of measure, normal range, and 

abnormal flag) due to inadequacies in the clinical data entering the HIE.42 Another study found 

that a public health HIE led to increased identification of needed followup care of 419 HIV 

patients and 85 percent of them having actual followup care.47

Quality of Care

Three studies looked at the value of HIE in improving quality of care in ambulatory 

settings.45,48,67 One study assessed a benchmark group of clinical quality measures believed to be 

amenable to HIE usage among users and nonusers of an HIE portal. Users of the portal had a 

higher proportion of physicians exceeding mean clinical quality measure performance at baseline 

(57% vs. 48%) that increased after the HIE became available (64% vs. 49%), with the increase 

for portal users before and after availability of the HIE statistically significant (p<0.001).45 An 

RCT of HIE data used in a clinical decision support intervention was able to detect medication 

adherence problems in eight categories of drugs but did not show any benefit in improving 

adherence by patients in taking medications prescribed based on evidence-based guidelines.

67

Another study of six physician practices found improved documentation and delivery of 

preventive services for mammography screening (21.1% to 57.1%, p<0.01), colonoscopy 
23

screening (31.7% to 53.8%, p<0.01), pneumococcal vaccine administration (39.1% to 50.6%, 

p<0.01), and influenza vaccine administration (22.7% to 41.7%, p<0.01).48 The study also found 

that medication reconciliation completion improved from 35.3 percent to 44.9 percent (p<0.001).

Other Aspects of HIE

Three studies assessed other aspects of HIE. One study found a 30 percent reduction in 

evaluation time for Social Security Disability claims.43 Another found that HIE data led to a 20.3 

percent increase in identifying frequent ED users compared with site-specific data.51 An 

additional study focused on hospital-based HIE, finding that communication and satisfaction 

(based on the Hospital Consumer Assessment of Healthcare Providers and Systems survey) were 

higher in hospitals that implemented HIE compared with those that proposed to implement 

HIE.64

Although the risk of bias in most studies was low, the resulting evidence from them was 

mostly of low quality. This low strength evidence mostly favored the value of HIE in reducing 

resource use and costs, especially in the ED. However, these studies used mostly retrospective 

designs that cannot account for how HIE was used and its impact on the overall care of the 

patient beyond the immediate setting where it was used.

Perceptions

A number of studies evaluated clinician or patient perceptions of HIE (Table 2).

57-64,66 Three 

studies assessed clinician perceptions of HIE in the ED setting. One study followed up an RCT 

on the provision of an electronic versus mailed report after an ED visit,65 with family physicians 

reporting improved patient management and followup in ED settings. 66 Another study also 

found that primary care physicians reported enhanced awareness and improved communication 

and followup with primary care physicians after ED admission/discharge.57 An additional study 

found that providing pharmacy information to physicians in the ED improved knowledge and 

gaps but was not felt to reduce time or be worth the cost.60

Other studies assessed perceptions in the outpatient setting. Two studies found that HIE was 

perceived to improve ambulatory care function, resulting in faster acquisition and treatment 

decisions61 and improved care and decreased work for filing and archiving discharge reports that 

were sent.62

Some studies looked at specific aspects of HIE. One study found that physicians were more 

satisfied with electronic lab reports than with paper-based reports.59 Another queried physicians 

on push versus pull HIE, with respondents reporting satisfaction with both, although more so 

with push over pull.58 An additional study assessed patient satisfaction when records were 

transferred via HIE, finding it to be improved over patients delivering paper records 

themselves.63

Clinician perceptions of the value of HIE, where studied, were generally positive. How such 

perceptions translate into improved care is unknown. This body of evidence was of low strength.
24

Table 2. Patient and clinician perceptions of HIE 

Study Location Setting HIE 

Type

Study

Type, 

Data 

Source

Risk of 

Bias

Direction 

of 

Result(s)

Perception(s) Assessed Results

Afilalo, et al., 

200766

Montreal, 

Canada

ED Directed RCT, 

survey

Moderate Beneficial Outcomes improved, better 

patient management

Improved with HIE

Altman et al., 

2012 57

New York ED Directed Crosssectional 

survey

Moderate Beneficial PCP notification of ED 

admission/discharge

Enhanced awareness 

and improved 

communication and 

followup

Campion et al., 

201258

Rochester and

Buffalo, New 

York

Outpatient Both Crosssectional, 

survey

Moderate Beneficial Physician satisfaction of push 

vs. pull

Satisfied with both, 

more with push than 

pull

Chang et al., 

201059

Indiana Outpatient Query Crosssectional, 

survey

Moderate Beneficial Physician satisfaction with 

electronic lab reports

Favorable, including 

over traditional reports

Kaushal et al., 

201060

Massachusetts ED Directed Crosssectional, 

survey

High Mixed Impact of providing pharmacy 

information

Improved knowledge 

and gaps but not felt 

to reduce time or be 

worth the cost

Maass et al., 

200861

Finland Outpatient Query Crosssectional, 

survey

High Beneficial Improvements in care When HIE used, 

faster results 

acquisition and 

treatment decision

Machan, 

Ammenwerth,

and 

Schabetsberger, 

200662

Tyrol, Austria Outpatient Directed Crosssectional, 

survey

Low Beneficial Physician satisfaction with 

discharge reports sent

Improved care and 

decreased work for 

filing and archiving

Park et al., 

201363

South Korea Outpatient Directed Crosssectional, 

survey

Low Beneficial Patient perceptions of data 

transferred

Increased satisfaction 

for patients whose 

records transferred via 

HIE

ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial; U.S. = United States 
25

Factors Associated With Outcomes

To determine whether effectiveness of HIE varied by location, health care setting, or 

outcome type, we rated each study outcome by whether HIE was found to have some beneficial 

effect or not. As shown in Table 3, the preponderance of studies showed that HIE use for 

different functions, in various settings, and of varying types was mostly positive. While the 

number of positive versus negative studies was not an indicator of the overall direction of the 

evidence, we did note that for each “negative” study, there is at least one “positive one.

For “Type of HIE,” there was no clear pattern of findings to suggest that one type is clearly 

better than another, even indirectly. The two RCTs we found were described in three papers. 

Two of these reported outcomes, one for each RCT, both of which showed no benefit for the HIE 

intervention.65,67 A perceptions study of one of the RCTs found perceptions of improved patient 

outcomes and their management.66 These are in contrast with the observational study designs 

where 96 percent found beneficial effects of HIE. This is somewhat typical in comparing RCT 

and observational study results, likely due to confounding. For HIE setting, only ambulatory and 

ED have enough studies to evaluate patterns, with outpatient settings less likely to find beneficial 

results compared with studies in ED settings, but again based on indirect comparisons only.

The sparseness of studies across geographic settings does not allow for identification of patterns, 

although across most studies in the United States, the findings were positive.

Table 3. Factors that may affect outcomes

Factor Studies of 

Outcomes

Studies of 

Perceptions

Studies 

Reported as 

Beneficial

Studies 

Reported as No 

Benefit

Total

Study Type

Retrospective cohort 20 19 1 20

Randomized controlled trial 2 1 1 2 3

Cross-sectional 2 1 1 2

Case series 2 2 2

Survey* 8 8 8

Setting

All 1 1 1

Emergency department 13 3 13 3 16

Government 1 1 1

HMO 2 2 2

Hospital 1 1 1

Outpatient 5 5 9 1 10

Public health 3 3 3

Location

U.S. multistate 3 2 1 3

Colorado 1 1 1

Indiana 2 1 3 3

Louisiana 1 1 1
26

Table 3. Factors that may affect outcomes (continued)

Factor Studies of 

Outcomes

Studies of 

Perceptions

Studies 

Reported as 

Beneficial

Studies 

Reported as No 

Benefit

Total

Massachusetts 1 1 1

Minnesota 1 1 1

North Carolina 1 1 1

New York 4 2 6 6

Oklahoma 1 1 1

South Carolina 1 1 1

Tennessee 3 3 3

Texas 1 1 1

Virginia 1 1 1

Wisconsin 2 1 1 2

Austria 1 1 1

Canada 1 1 1 1 2

Finland 1 1 1 1 2

Israel 2 2 2

South Korea 1 1 1

HIE Type

Directed 5 5 8 2 10

Query 18 2 19 1 20

Multiple 3 1 3 1 4

HIE = health information exchange; HMO = health maintenance organization; vs. = versus

*1 survey study was also an RCT.

Key Question 4. What are the current level of use and primary uses of HIE? 

Key Points

• More than three-quarters (76%) of non-Federal acute care hospitals electronically 

exchanged laboratory results, radiology reports, clinical care summaries, and/or 

medication lists with any outside providers in 2014. This represented an 85 percent

increase since 2008 and a 23 percent increase since 2013. Close to seven in 10 hospitals 

(69%) electronically exchanged health information with ambulatory providers outside of 

their organization, representing a 92 percent increase since 2008 and a 21 percent

increase since 2013.

• A variety of HIE models are employed across settings. Hospitals and ambulatory care 

providers both provide and use data; while laboratory services provide data and 

community clinics use data. At least 50 percent of these organizations are reaching an 

advanced stage of use of core functionalities; many supporting health care reform 

initiatives and advanced analytics. 

• Use varies by type of health care professional, with higher use by nurses and clerks, when 

compared with physicians. Patient engagement remains low.

• Use is increasing in ambulatory care practices, with a 2013 estimate of 38 percent of 

practices using HIE. Characteristics of higher HIE use being larger practice size, practice 
27

owned by a health system (vs. physician owned), and multispecialty (vs. single specialty) 

practice.

• HIE use in long-term care settings is low (<1%), with the consistent pattern of nonprofits 

enjoying wider use than for-profit entities. Less than four in ten residential care facilities 

that use EHRs also exchange health information. 

• Results of regional and statewide studies that evaluate HIE use in inpatient, outpatient, 

community clinic, or EDs suggest that HIE is used for few patients; the extent of HIE use 

is low. Results of international/multi-national studies suggest the same finding. 

• HIE use was in its infancy in the 2000s but has been steadily increasing since then. 

– A recently released 2015 report from the ONC suggests that the United States is 

making great progress in exchanging health information.

– HIE is particularly useful in the ED and in the ambulatory setting to alert 

providers to inpatient or ED events recently experienced by patients. 

• Patients also seem willing to consent to data exchange, as long as the benefits of doing so 

are clear to them.

Detailed Synthesis

We identified 58 studies that described the levels of use and primary uses of HIE (Tables 4-

7). Several methods were used by investigators to answer questions about HIE use, including 

surveys (25 studies),25,26,73,74,76-96 analyses of HIE audit-logs (13 studies),40,45,54,97-106

retrospective database analyses (9 studies),107-115 and mixed methods (7 studies).116-122 Two 

studies used focus group methods,123,124 one study used time-motion methods,

61 and another used

geo-coding.125

Over one-half of the studies (30 of 58) analyzed HIE implementations over a regional or 

statewide area,45,54,76,77,83-86,88,90,92,96-106,112,118-120,123-126 while an additional 15 evaluated HIE use 

nationally.25,26,78-81,87,91,93,107-111,113 Of those that evaluated use regionally or over a statewide area, 

10 studies evaluated HIE implementations in the State of New York,45,76,77,96-98,102,106,112,125 five

in Texas,54,101,103-105 five in Tennessee,40,86,99,118,119 two in Indiana,88,92 and two in Minnesota.85,90

Five studies evaluated HIE in a single State (Massachusetts,

123 North Carolina,100 Wisconsin,84

Northeastern Ohio,120 and Louisiana124).

Two studies evaluated HIE use across integrated delivery systems. One exchanged data 

between the Department of Veterans Affairs (VA), the Department of Defense (DoD), and nonFederal care organizations,116 and the other between the VA and Kaiser Permanente.82 Seven 

studies evaluated HIE use outside of the United States61,89,94,114,115,121,122 and two in multiple 

countries including the United States.95,117

The majority of studies evaluated HIE use across inpatient and ambulatory care settings.

Seven studies were limited to evaluations of HIE use in hospitals,76,88,96,107,108,111,117 three of these

used data from the American Hospital Association (AHA).

107,108,111 Four studies evaluated HIE 

use that involved exchange of data with nursing homes or residential care facilities; two using

data from the National Nursing Home Survey and the National Survey of Residential Care 

Facilities,93,113 the other two using data from New York State.77,112 Three studies focused on 

evaluating HIE use in the ED; all of these exchanged data regionally.40,99,100 Two studies focused 

on evaluating HIE use in office settings using data from the National Ambulatory Medical Care 

Survey,

91,110 three others used within State data, one from Indiana92 and two from Minnesota.85,90
28

The majority of studies assessed overall use of the HIE, while two assessed the use of HIE 

for repeated imaging in the ED,40,102 and two evaluated HIE for prevention or tracking of 

infections.83,88

Twenty-seven studies included data collected in 2010 or more recently;25,26,73,74,77,83,88,90-

98,102,106,108-113,120,124,125 the majority of studies used data collected in 2009 or earlier. Fifteen 

studies used a query-based HIE;40,54,86,97-99,101-105,118-120,125 the other studies either did not specify, 

or multiple HIE implementations were included.

Twenty-nine of the studies were rated as being at low risk of bias;25,26,40,54,76-

81,83,86,88,91,93,94,100,101,103-105,107-111,113,121,125 nine at moderate risk of bias;84,85,90,92,95,96,102,112,122 six at 

high risk of bias;61,87,89,114,117,120 and fourteen were not rated due to the type of study design (data 

from audit-logs or qualitative studies).45,73,74,82,97-99,106,115,116,118,119,123,124

Level of Use and Primary Uses: Type of HIE

The majority of the studies used a variety of types of HIE, and did not describe these in 

detail. Data describing the type of HIE, according to the classification system promulgated by the 

Office of the National Coordinator (direct, query-based, or consumer-mediated) were limited to 

studies wherein a specific HIE was evaluated. Of these, query-based HIE systems were noted for 

evaluations of the MidSouth e-Health Alliance (MSeHA),

40,86,99,118,119 the Central Texas HIE (ICare),54,101,103-105 the Health Care Efficiency and Affordability Law for New Yorkers Capital 

Grant Program (HEAL-NY),97,98,102,125 and the Northeast Ohio Public Health Care System.120

Level of Use and Primary Uses: Health Care Settings and Systems

This summary of HIE use by health care setting and systems (Key Question 4b) has been 

combined with the summary by IT system characteristics (Key Question 4c), and data sources 

(Key Question 4d) to provide the summary below. Little meaningful information was found on 

the use of HIE by provider type (also Key Question 4b) so, when available, this information is 

also incorporated into this section.

Participation in HIE, Types of Data Exchanged, Characteristics of 

Successfully Participating Organizations (United States–Wide Surveys)

Six studies used survey methods to investigate the frequency of data exchange and types of 

data exchanged across regional health information organizations (RHIOs) across the United 

States (Table 4).

25,78-81,87 Across these studies, between 138 and 207 organizations met the 

definition of a RHIO; while between 20 and 81 RHIOs provided data. These data, collected from 

2006 through 2012, suggest that entities most commonly providing data are hospitals (83%), 

followed by ambulatory settings (60%); and that the entities most commonly receiving data were 

ambulatory settings (95%), followed by hospitals (83%), public health departments (50%), and 

payers (44%).81 Using survey data collected in 2007, Hessler, et al. focused on the exchange 

between RHIO and State and local public health departments, and found that of 138 public health 

agencies, 50 (36%) had no RHIO in their jurisdiction; 16 (12%) had no relationship with a 

RHIO, and 26 (40%) were exchanging information. Twelve of 20 RHIOs were exchanging 

information; seven of these (35%) with public health entities.87 The types of data most frequently 

exchanged were laboratory test results (84% to 90%),78,81,87 inpatient data (70%), medication 

histories (70%), and outpatient data (60%).78,81 In 2008 and 2009, of 75 operational RHIOs, 

covering 14 percent of U.S. hospitals and 3 percent of ambulatory practices, only 13 supported 

the criteria for meaningful use criteria of the Health Information Technology for Economic and 
29

Clinical Health Act (3% of hospitals and <1% of ambulatory practices),79 while by 2012, there 

had been a 61 percent increase in the number of operational RHIOs, from 75 to 119.25

Two additional surveys were conducted by the eHealth Initiative 73,74 One-hundred, ninetynine of 315 identified HIE organizations completed the 2013 annual survey. These HIE entities 

were a mix of community-based, State-based, and health care delivery organizations. Results 

indicate there is no single dominant model of HIE. Ninety organizations use a ‘Direct’ standardsbased protocol for securely exchanging data, mostly for transitions in care. Patient opt out was 

the most common consent model, although patient engagement remains low amongst 

organizations exchanging data. Eighty-four organizations had reached an advanced stage of 

operation or innovation; most took 2 years to become operational. Among organizations that 

responded in both 2011 and 2013, 27 more had reached stages 5 (operating), 6 (sustaining), or 7

(innovating) on the eHealth Initiative’s maturity scale, in 2013. Hospitals and ambulatory care 

providers are the stakeholders most commonly providing/viewing data; independent laboratories 

also commonly provide data. Community and public health clinics commonly view data. HIE

organizations are focusing on functionalities to support health care reform initiatives and 

advanced analytics. 

The number of HIE organizations identified and that responded in 2014 was lower than in 

2013, with 126 of 267 identified responding in 2014.74 Again, there was a mix of communitybased, State-based, and health care delivery organization-based HIE entities responding. Data 

were provided by hospitals, ambulatory care providers, laboratories, and community/public 

health clinics. Data were accessed by ambulatory care providers, hospitals, community/public 

health clinics, and behavioral or mental health providers. Findings suggest an 11 percent increase 

over 2013 in the proportion of organizations that have reached stage 6 (operating) or higher (106 

organizations). Uses of HIE included support for an accountable care organization to improve 

patient outcomes, for a patient centered medical home, for a State Innovation Model, and for a 

bundled payment initiative. Results suggest data exchange is reaching a point of stability and 

acceptance, and that organizations are settling on a set of core services offerings.26

Nine studies investigated HIE use retrospectively, using U.S.-wide survey data collected for 

other purposes, with an information technology add-on.26,91,107-111,113 Four of these used data 

from the AHA,

26,107,108,111 two from the National Ambulatory Medical Care Survey,

(NAMCS),91,110 and one each from the Commonwealth Fund Health Policy Surveys,109 the 

National Nursing Home Survey/National Survey of Residential Care Facilities,

111 and, another 

from the National Survey of Residential Care Facilities.93

These studies investigated overall participation in HIE use. Results suggest that HIE use by 

hospitals has risen from 11 percent (2009)78 to between 30 percent and 58 percent more 

recently.

108,109,111 Results from the recently released ONC brief suggest that more than threequarters (76%) of non-Federal acute care hospitals electronically exchanged laboratory results, 

radiology reports, clinical care summaries, and/or medication lists with any outside providers in 

2014. This represents an 85 percent increase since 2008 and a 23 percent increase since 2013. 

Close to seven in 10 hospitals (69%) electronically exchanged health information with 

ambulatory providers outside of their organization, representing a 92 percent increase since 2008 

and a 21 percent increase since 2013.26 Characteristics associated with higher use are nonprofit 

status, presence of an EHR system, larger market share, and larger practices.107-109,111 Results 

from the NAMCS (2011) suggest that the majority of office-based physicians reported being able 

to both send and receive data; 64 percent of these exchanges were through an EHR vendor and 

28 percent through a hospital system. Activities included viewing laboratory results and 
30

incorporating these into the EHR, and exchanging clinical summaries with patients. Primary care 

providers were more likely to use HIE than specialists.91 Results from the NAMCS (2013) 

suggest that 39 percent of office-based physicians reported having HIE capability with other 

providers or hospitals. Characteristics of higher HIE use were larger practice size, practice 

owned by a health-system (vs. physician owned), and multispecialty (vs. single specialty) 

practice.110 Data from the National Nursing Home Survey (2004) and the National Survey of 

Residential Care Facilities Survey, both from the Centers for Disease Control and Prevention, 

indicate that HIE use in these settings is low, with the consistent pattern of nonprofits enjoying 

wider use than for-profit entities.113 Finally, recent data from the National Survey of Residential 

Care Facilities suggest that 23 percent of residential care communities that use EHRs also 

exchanged health information. Nearly 25 percent could exchange with pharmacies and 17 

percent with physicians.93
31

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies)

Study Geographic 

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Adler-Milstein, 

et al., 200881

U.S.-wide RHIOs Varies Crosssectional 

survey

Low Participation in RHIO

Types of data 

exchanged.

-Most common entities providing and receiving data:

83% of hospitals; 67%-95% of ambulatory settings;

50% of public health departments; 44% of payers.

-Types of data exchanged: Test results: 60%-90%; 

Inpatient data: 70%; Medication histories: 70%; 

Outpatient data: 60%; Images: 56%.

Adler-Milstein, 

Bates, and 

Jha, 200978

U.S.-wide RHIOs Varies Crosssectional 

survey

Low Types of data 

exchanged.

-Types of data exchanged: Test results: 84%; 

Inpatient data: 70%; Medication histories: 66%; 

Outpatient data: 64%.

Adler-Milstein, 

Landefeld, and 

Jha, 201080

U.S.-wide RHIOs Varies Crosssectional 

survey

Low Characteristics of 

successful 

participation.

-Likelihood of being operational associated with 

exchanging narrow set of data and involving broad 

group of stakeholders

Adler-Milstein, 

Bates, and Jha 

201179

U.S.-wide RHIOs Varies Crosssectional 

survey

Low Number of 

operational RHIOs 

supporting stage 1 

meaningful use; 

number financially 

viable.

-75 operational RHIOs, covering 14% of U.S. 

hospitals and 3% of ambulatory practices.

-13 RHIOs support stage 1 meaningful use 

(covering 3% of hospitals and 0.9% of ambulatory 

practices).

Adler-Milstein, 

Bates, and 

Jha, 201325

U.S.-wide RHIOs Varies Crosssectional 

survey

Low Participation in RHIO.

Types of data 

exchanged.

Characteristic of 

successful 

organization.

-61% increase from 2011 (75 to 119 RHIOs).

-Types of data exchanged: Test results: 82%; 

Summary records: 79%; Discharge records: 66%; 

Clinical summaries: 61%

-Predominant organization was nonprofit.

Hessler, et al., 

200987

U.S.-wide RHIOs Varies Crosssectional 

survey

High Participation in RHIO. -RHIOs:

-12/20 (60%) are exchanging information

-7/20 (35%) with Public Health

-Type of data exchanged most frequently: Test 

results: 86%.

-Public health agencies:

-50 (36%) have no RHIO in jurisdiction.

-16 (12%) have no relationship with RHIO.

-26 (40%) are exchanging information.
32

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies) (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

eHealth 

Initiative, 

201373

U.S.-wide All Varies Crosssectional 

survey

Not rated 

due to study 

design

Participation in HIE. 

Stage of maturity.

Key findings.

-84 organizations had reached 'advanced' stage of 

operation, sustainability, or innovation.

-27 more had reached stages 5 (operating), 6 

(sustaining), or 7 (innovating) on the eHealth 

Initiative's HIE maturity scale in 2013 than in 2011.

-Hospitals and ambulatory care providers most 

commonly providing/viewing data, followed by 

laboratories and community public health clinics.

-Most took 2 years to become operational.

Key findings:

1) Exchanges are focusing on functionalities to 

support health reform and advance analytics.

2) Patient engagement remains low amongst

organizations exchanging data.

Swain, et al.,

201526

U.S.-wide NonFederal 

acute care 

hospitals 

and 

outside 

providers

Varies Retrospective 

database 

analysis of 

AHA data

Low HIE use between 

hospitals and 

hospitals;

HIE use between 

hospitals and outside 

providers;

Types of data 

exchanged (Labs, 

radiology, meds, 

clinical care 

summaries)

More than three-quarters (76%) of non-Federal 

acute care hospitals electronically exchanged 

laboratory results, radiology reports, clinical care 

summaries, and/or medication lists with any outside 

providers. This represents an 85% increase since 

2008 and a 23% increase since 2013. Close to 

seven in ten hospitals (69%) electronically 

exchanged health information with ambulatory 

providers outside of their organization, representing 

a 92% increase since 2008 and a 21% increase 

since 2013.
33

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies) (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

eHealth 

Initiative, 

201474

U.S.-wide All Varies Crosssectional 

survey

Not rated 

due to study 

design

Participation HIE.

Stage of maturity. 

Key findings. 

Provides data: 112 hospitals, 100 ambulatory care 

providers, 56 laboratories, 52 community/public 

health clinics.

Accesses data: 111 Ambulatory care providers, 104 

hospitals, 75 community/public health clinics, 65 

behavioral or mental health providers.

Key findings: 106 had reached stage 6 (sustaining) 

or higher on the eHealth Initiative's HIE maturity 

scale (an increase of 11% over 2013). 

64 support an accountable care organization; 52 

support a Patient Centered Medical Home; 21 

support a State Innovation Model; 12 support a 

bundled payment initiative.

Looking to the future

1) Data exchange is reaching a point of stability and 

acceptance.

2) Organizations are settling on a set of core service 

offerings.

3) As organizations mature, they will offer new and 

innovative services (public health has already 

leveraged HIE; alert notification services may help 

accountable care organizations to track patients).

Adler-Milstein, 

DesRoches,

and Jha, 

2011107

U.S.-wide Hospitals Varies Crosssectional 

review of 

database 

analysis of 

AHA data

Low Participation in HIE.

Characteristics of 

successful 

organizations.

11% of hospitals engaged in HIE.

Use significantly higher for private/nonprofit status, 

greater market bed share, teaching status, large 

size, presence of cardiac ICU, and presence of EHR 

system.

Adler-Milstein 

and Jha, 

2014108

U.S.-wide Hospitals Varies Crosssectional 

Measurement 

of HIE usage 

among U.S. 

hospitals

Low Participation in HIE.

Characteristics of 

successful 

organizations.

30% of hospitals engaged in HIE.

Use significantly higher for private/non-profit status; 

greater market bed share, in less competitive 

market. 

Varies widely by State.
34

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies) (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Furukawa, et 

al., 2013111

U.S.-wide Hospitals Varies Cross –

sectional 

survey

Low Participation HIE.

Types of data 

exchanged. 

Characteristics of 

successful 

organizations.

-In 2012, 58% of hospitals exchanging data, 41% 

increase over 2008, (p<0.01).

-In 2012, 51% of hospitals exchanging with 

unaffiliated ambulatory providers, 36% with other 

hospitals outside their organization.

-In 2012, 52%, 53%, 35% and 33% exchanging 

images, laboratory tests, care summaries, 

prescription lists with outside providers, respectively 

(39%, 51%, 40%, 55% increase, respectively)

-After adjusting for hospital and area characteristics, 

hospitals with basic EHR and participation in health 

information organizations (HIOs) had highest rates 

of exchange activity.

-In 2012, 80% of hospital with EHR and HIO were 

exchanging, 71% with HIO but no EHR were 

exchanging; 60% with EHR but no HIO were 

exchanging.

-All consistent across different providers types and 

clinical information types.

-Hospital characteristics associated with lower 

exchange rates were rural, for-profit, locations with 

greater Medicare part A spending.
35

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies) (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Patel, et al., 

201391

U.S.-wide Ambulator

y Care

Varies Cross –

sectional 

survey

Low Participation in HIE.

Types of data 

exchanged. 

Characteristics of 

successful 

organizations.

-31% of offices could share clinical summaries.

-Of these, 76% could both send and receive.

-64% of these exchanges were through an EHR 

vendor; 28% through a hospital-based system. 

-55% e- prescribe, 67% view laboratory results, 42%

incorporate lab results into EHR.

-State differences: the capacity to electronically 

exchange clinical summaries with patients varied 

from 55% (Minnesota) to 18% (Louisiana). 

-Proportion of physicians who exchange clinical 

summaries with other providers varied from 61% 

(Wisconsin) to 15% (Alabama).

-Adoption of EHR strongest practice characteristic 

associated with exchange capacity, p<.001.

-EHR vendors have wide range of capacities for 

exchange: 24% to 77%.

-Primary care providers more likely to exchange vs. 

specialists.

Furukawa, et 

al., 2014110

U.S.-wide Ambulator

y care

Varies Cross –

sectional 

survey

Low Participation in HIE. 

Characteristics of 

successful 

organizations.

-39% of office-based physicians reported having 

HIE capability with other providers or hospitals.

-Characteristics of higher HIE use were larger 

practice size (vs. solo), practices owned by healthsystems (vs. physician owned); multispecialty 

practices (vs. single specialty).

Audet, Squires,

and Doty, 

2014109

U.S.-wide Ambulator

y care

Varies Crosssectional 

analysis of 

database

Low Participation in HIE. 

Characteristics of 

successful 

organizations.

-32% of physicians engage in HIE.

-Use significantly higher for practices that have 

higher proportion for formal IT support, are part of 

an integrated system, larger practices, presence of

EHR system, and receiving financial incentives.

-Use significantly increased since 2009.
36

Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully 

participating organizations (United States–wide studies) (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Hamann and 

Bezboruah, 

2013113

U.S.-wide Nursing 

Homes

Varies Crosssectional 

analysis of 

two survey 

databases

Low Participation in HIE. 

Characteristics of 

successful 

organizations.

For profit vs. nonprofit:

-Percent residential care facilities using HIE: 0.14% 

vs. 0.21%; p<0.00.

Number of partners in HIE: 0.32% vs. 0.42%; 

p=0.02.

-For profits less likely to participate in HIE; OR 

0.663, p<0.001.

-Supports hypothesis and proposed framework for 

why non-profits are more likely to use health IT.

Caffrey and 

Park-Lee,

201393

U.S.-wide National 

Survey of 

Residenti

al Care 

Facilities

Varies Crosssectional 

survey

Low Use of HIE among 

residential care 

communities that use 

EHRs

23% used computerized systems for exchanging 

health information with pharmacies; 17% with 

physicians; 20% with other health or long-term care 

providers, such as hospitals and nursing homes.

AHA = American Hospital Association; e = electronic; EHR = electronic health record; HIE = health information exchange; HIO = health information organization; 

ICU = intensive care units; IT= information technology; NAMCS = National Ambulatory Medical Care Survey; RHIO = regional health information organization 

U.S. = United States of America; vs. = versus
37

Transfer of Records Between Integrated Delivery Systems

The VA and DoD use the Virtual Lifetime Electronic Record (VLER) system for eHealth 

exchange with the private sector, in the Nationwide Health Information Network (NwHIN) – a 

‘network of networks’. This is a federated, pull (query-based) model for transfer of records 

between integrated delivery systems, using an opt in consent approach by patients. The NwHIN 

allows users to pull in data from other organizations (Table 5). In an early study, Bouhaddou et 

al. investigated the transfer of records across three integrated delivery systems in San Diego, 

California; the VA, DoD, and Kaiser Permanente Southern California. They found that 264 of 

363 of patients (73%) who opted in and provided valid authorization could be correlated across 

integrated delivery systems.82 In a recent, much larger study, Byrne et al. enrolled 12 sites. Of 

the 64,237 veterans who provided authorization and opted in, less than 0.01 percent opted in and

subsequently opted out. The proportion of data matched between exchange partners ranged from 

12 percent to 88 percent. The highest matching rates were accomplished using social security 

numbers in the matching algorithm. Data were retrieved for 2,724 unique VA patients with the 

exchange partner, and for 1,764 unique VA providers reviewing exchange partner data.116
38

Table 5. Level of use and primary uses of HIE: transfer of records between integrated delivery systems

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Bouhaddou, 

et al., 201182

San Diego, 

California

Nationwide 

Health 

Information 

Network 

(NwHIN; VA, 

DoD, Kaiser 

Permanente)

VLER Crosssectional study 

of patient 

records

Not rated 

due to 

study 

design

Transfer of records 

between integrated 

delivery systems.

Of 363 patients who opted in and provided 

valid authorization, 264 could be correlated 

across integrated delivery systems, with 

exchange of records between KP and VA, 2-3 

per week.

Byrne, et al., 

2014116

U.S. VA, DoD, 

private 

sector

VLER Crosssectional study 

of patient 

records

Not rated 

due to 

study 

design

Transfer of records 

between integrated 

delivery systems.

-64,237 veterans provided authorization and 

opted in.

-31,080 (48%; range 12%-88%).

-Highest matching rates with exchange 

partners using social security number in their 

algorithm.

-5,524 inbound disclosers to VA from 

exchange partners (18/100 matched). 

-13,913 outbound disclosures to exchange 

partner.

-Data retrieved for 2,724 unique VA patients 

with exchange partner.

-1,764 unique VA providers reviewing 

exchange partner data.

-9% of veterans for whom there was ≥1 

disclosure to VA matched with exchange 

partner.

DoD = Department of Defense; HIE = health information exchange; KP = Kaiser Permanente; NwHIN = Nationwide Health Information Network; SSN = social security number; 

U.S. = United States of America; VA = Veterans Affairs; VLER = Virtual Lifetime Electronic Record
39

Participation in HIE and Extent of Use: Regional or Statewide Initiatives

Nine studies described the use of HIE in the State of New York. Five of these used audit 

logs,45,97,98,102,106 two used surveys,76,77 one used a database of clinical data,112 and one geocoding125 (Table 6). Most of the HIE implementations are query-based. The studies of audit logs 

indicate frequent queries,97,98 and an increasing proportion of physicians accessing HIE over time 

(33% to 43% over 18 months).45 Separately, of 63,305 patients enrolled from three hospitals, an 

average of 238 clinical event alerts were provided per day to notify ambulatory care providers of 

inpatient or ED admissions for their patients; a total of 42,818 events were detected over a 6-

month timeframe.

106 Primary HIE users varied by study. In one study, primary users were nonclinical staff in the outpatient setting and clinicians in the inpatient setting,97 while in another, 86

percent of sessions were with staff in an ED.102

Abramson et al. conducted three statewide surveys in New York, two in 205 hospitals76 and 

the other in 632 nursing homes.77 In each, they investigated participation in HIE and the 

exchange of data. In hospitals, their results suggest that between 2009 and 2012 the percent of 

respondent hospitals participating in HIE and exchanging data, increased from 23 percent to 79 

percent. In 2012, institutions exchanged data more frequently with other hospitals (71%) and 

ambulatory care providers (69%), than with long-term care facilities (45%) and home health 

agencies (38%).96 Among nursing homes 54 percent participate in HIE, with 31 percent of 

providers exchanging information outside the system. HIE use was highest when nursing homes 

had an EHR. The types of data exchanged were pharmacy (42%), labs (39%), and hospital data

(39%). The seventh study was a retrospective database analysis of clinical data that described a 

geriatric care coordination program that used a Clinical Event Notification system to request 

information from nursing homes when patients were seen in the ED.

112 The authors suggested 

that use of the Clinical Event Notification functionality may have facilitated avoidance of 18

percent of hospital admissions, as these admissions lasted less than 48 hours. As not all studies 

described the type of HIE in detail, we were unable to draw any conclusions based on the type of 

HIE utilized. Finally, using a novel study design, Onyile et al. estimated the proportion of 

patients in the New York Clinical Information Exchange (now Healthix) system by mapping the 

most current zip code for each patient to the appropriate U.S. county. They found that 88 percent

of patients in the system live within 30 minutes of New York’s Times Square.125

A series of five studies investigated HIE use in a query-based Central Texas HIE. I-Care is 

an HIE implementation comprised of hospital systems, public and private clinics, and 

governmental agencies operating federally qualified health centers.54,101,103-105 Four of these

studies were conducted across several facility member sites, with a fifth study across two sites.101

For adult patients seen in the ED, use was low; in 57 percent of patients54 and only 2.3 percent of 

encounters.105 In a subset of two sites that did not have an EHR (but that mandated use of the 

HIE), the HIE was accessed in 21 percent of the encounters.101 Across these studies, HIE use was 

higher for those with a greater number of ED visits and hospitalizations,54,101,105 older age, a 

greater number of chronic conditions,

101,105 females, and those with fragmented care.101 HIE use 

was lower for blacks and Hispanics, visits for alcohol use, injury, poisoning, an unfamiliar 

patient, and a busier than average day.105 Similar results were found in the study that focused on 

children seen in the ED; use was greater for those less than 1 year old, who had more frequent 

encounters in the past, and a greater number of diagnoses. Use was lower if the patient was 

unfamiliar, or if the day was busier than average.104 In a companion study that investigated how 

use of HIE varies by job type and organization in an indigent care setting, Vest et al. found that 
40

the most frequent users were those whose positions were administrative, followed by social 

services, physicians, nurses, public health professionals, and pharmacy professionals. The 

hospital was the workplace for 50 percent of users, followed by adult ED, ambulatory care, 

public health agency, mental health agency, and children’s ED. Most clinical access took place 

in the ED and in public/mental health agencies. In the majority of use sessions, users accessed 

the system in a minimal fashion; almost all use was administrative.103

Of the five studies conducted in the MSeHA, based in Memphis, Tennessee, three used auditlogs,

40,99,118 one was a cross-sectional survey,86 and one used mixed methods.119 MSeHA is an 

HIE implementation that facilitates data exchange across EDs and community-based ambulatory 

clinics. In 2007, across these studies, HIE use was low, being used for 12.5 percent of the study 

population.40 In another, HIE was viewed in the ED for between 3 percent and 10 percent of 

visits.99 In a third, HIE was used for only 15 percent of return ED visits and 19 percent of return 

clinic visits; yet users reported the HIE provided additional information about histories and 

prevented repeat tests or procedures.118 In the separate cross-sectional survey of 151 users, 43 

percent reported using HIE less than 1 hour per week, 39 percent between 1 and 4 hours, and 18 

percent, greater than 4 hours per week.86 In a separate study of workflow, nurses accessed HIE 

when prompted by patients about a recent hospitalization, while providers accessed HIE for 

reasons beyond simply identifying a recent hospitalization. HIE access occurred at various points 

of care. Workflow patterns evolved over time, due to revisions in access policies and staffing 

changes.119 Across these studies, use was higher when the HIE was accessed by nurses and 

clerks versus physicians.99,118

Separately, Dixon et al. conducted an online survey of 63 infection preventionists in six 

states with HIE, to gauge the awareness and engagement of these preventionists in using HIE for 

public health surveillance. One-half of their respondents were unaware of their organization’s

involvement in HIE, and only 10 percent reported their organizations used the HIE.83

Nine additional studies describe HIE use at the State-level, two studies each from Indiana 

and Minnesota, and one each from Wisconsin, North Carolina, Massachusetts, Northeastern 

Ohio, and Louisiana.84,85,88,90,92,100,120,123,124 These studies used data from 2005123 through 2013.90

Methods of data collection included surveys,84,85,88,90,92,120 interviews,85,124 focus groups,123,124

and audit-logs.

100,120 Each study makes a useful contribution to the HIE literature. 

In an Indiana study of a coordinated antibiotic-resistance infection tracking, alerting, and 

prevention system, of the several thousand patients for whom email alerts were sent, 

approximately one-quarter were identified as having had documentation in a different hospital 

system of a previous infection with methicillin-resistant staphylococcus aureus or vancomycinresistant enterococcus. Capture of this type of laboratory data was found useful.88 Other Indiana 

investigators found real-time alerting helpful in prompting followup,

92 as did investigators in 

Louisiana.124 Patients were generally accepting of data sharing, as long as patient benefit was 

evident.124 In a study of small practices (<20 physicians) in Minnesota, results revealed that no 

practice was fully involved in a regional HIE and that HIE was not part of most practices’ shortterm strategic plans.85 In a study more recently conducted in Minnesota, intended to monitor 

progress toward meeting the legislative requirement that all health care providers have an 

interoperable EHR by January 2015, investigators found that over one-half of respondents 

exchanged data with affiliated or unaffiliated hospitals.90 The Tripathi et al. study was unique in 

that researchers conducted focus groups with patients who lived in three communities that 

piloted the Massachusetts HIE. All three communities agreed to share all EHR data except text 

notes, consult letters, and scanned reports. Consumer opt in was the preferred consent method, as 
41

it is in VLER. Strategies identified to drive consumer opt in included educating patients and 

providers about the enhanced convenience and lower costs of HIE.123 Lobach et al. investigated 

the impact of the HIE on sentinel events for Medicaid patients in Durham County, North 

Carolina. In an analysis of almost 12,000 patients enrolled, they found that 19 percent 

experienced a sentinel event over a 6-month period. They concluded that the HIE was useful in 

population health management using HIE.100 In a description of HIE implementations in 

Wisconsin, Foldy found that 78 percent (21 of 27) of organizations had HIE projects, some 

operational, others planned. Most were surveillance systems, delivering data to central registries, 

but a growing number served clinicians and patients.84 Kaelber et al. investigated HIE use in the 

Northeast Ohio Public health care system, Care Everywhere. Of the 18 percent (74 of 412) of 

physicians who responded to the survey, approximately one-third of ED physicians, one-fifth of 

primary care physicians, and one-tenth of specialty care physicians used HIE. Use was highest 

when patients were older, with more comorbidities, Medicare/Medicaid insured, or black.120

These results reflect the variation in the implementation and impact of HIE, providing data that 

are not necessarily generalizable to other settings. These data suggest that small practices are not 

adopting HIE, while larger health systems are. They further suggest that HIE may be useful in 

exchanging data in the ED, and for surveillance of infectious diseases, that patients and providers 

view HIE favorably, and that patients can and do ”buy-in” to the concept of HIE when the 

benefits are evident.
42

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Abramson, et 

al., 201276

New York 

State

Hospitals Varies Crosssectional 

survey

Low Participation in HIE.

Exchange of data

-23% of respondent hospitals participate and 

exchange data.

-37% participate but do not exchange data.

-40% do not participate

Abramson, et 

al., 201477

New York 

State

Nursing 

homes

Varies Cross-sectional 

survey

Low Participation in HIE. 

Exchange of data

-54% participate in HIE.

-OR=2.26 more likely to exchange when have 

EHR.

-When EHR used, 60% exchange with providers 

within system; 31% exchange with providers 

outside system.

-HIE highest for pharmacies (42%), labs (39%), 

and hospitals (39%).

Abramson, et 

al., 201496

New York 

State

Hospitals Varies Cross-sectional 

survey

Moderate Use of HIE (sent or 

received). Type of 

institution 

information is 

shared with.

79% (n=102) of respondents reported actively 

exchanging any electronic patient-level clinical data 

with an entity outside their institution in 2012 vs. 

60% in 2009

Institutions exchanged data with:

Hospitals outside system: 71% (n=72)

Ambulatory providers outside system: 69% (n=70)

Long term care facilities: 45% (n=46)

Home health agencies: 38% (n=39)

Most commonly exchanged data were radiology 

reports, followed by laboratory results, medication

lists and clinical histories.

Kern, et al., 

201245

Hudson 

Valley, New 

York

Hospitals 

and 

Laboratories

MedAllies 

Portal

Cross-sectional 

study of audit 

logs

Not rated Extent of use. Percent of MDs using portal: 33% months 1-6 vs. 

42% months 7-12 vs. 43% months 13-18.

-Mean days logged-in per month by MD: 8 (SD: 6).
43

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Campion, et 

al., 201398

Binghamton, 

New York

RHIO (2 

hospitals and 

13 ambulatory 

clinics)

Southern 

Tier 

HealthLink 

RHIO; 

Query

Cross-sectional 

audit logs

Not rated Extent of use. -202,365 auto queries; 54% to hospitals, 46% to 

clinics.

-145,668 unique patient encounters.

-81, 687 consented patients.

-41% of patients had at least one supported 

encounter.

Campion, et 

al., 201397

New York 

State

3 RHIOs 

(hospital and 

outpatient)

Query Cross-sectional 

audit log

Not rated Extent of use. -System access occurred in 60% to 82% of 

practice sites registered to use system, 

depending on community.

-In communities A and B, users were non-clinical 

staff in outpatient settings; in community C, 

users were inpatient clinicians.

-Proportions of patients whose data were 

accessed varied between 5%-60%.

-Most frequently accessed data were patient 

summaries, followed by laboratory tests and 

imaging data.

Vest, et al., 

2013102

Rochester, 

New York

RHIO (hospital 

and outpatient) 

and claims 

from health 

plans

Query Case-control 

study of auditlog files

Low Extent of use.

Patient and 

provider 

characteristics 

associated with use 

of an HIE system to 

access radiology 

report.

-Each source organization sent average of 971 

(range: 6 to 8,002) documents to 49 (3 to 106) 

other organizations.

-User organizations accessed average of 49 (1 

to 8,444) documents from 6 (1 to 17) source 

organizations.

-Overall number of radiology reports retrieved in 

outpatient setting was 17 times greater than 

number of reports retrieved in the ED and 

inpatient settings combined (23,201 outpatient 

vs. 1,333 ED and 313 inpatient).

-86,152 user sessions with associated claims 

files represented the activity of 1,119 different 

users representing 145 different workplace 

locations; 86% of sessions were with staff; 4% 

were with physicians.
44

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Moore, et al., 

2012106

New York RHIO; New 

York Clinical 

Information 

Exchange 

(NYCLIX; 

outpatient).

Not stated Cross-sectional 

audit log

Not rated Extent of use to 

alert ambulatory 

providers to patient 

events (patients 

admitted to or 

discharged from the 

hospital or ED).

Over 6 months:

-42,818 events detected, on average 238 events 

per day.

-≥1 event: 6,913 patients.

-1 event: 1,879 patients.

-≥10 events: 623 patients

-Mean number of events in inpatients who had 

an event: 7.7 events.

-Mean number of events in all patients: 0.7 

events.

Gutteridge, et 

al., 2014112

New York RHIO (ED) Healthix Cross-sectional 

database 

analysis 

Moderate Extent of use for 

clinical event 

Notification.

-5,722 patients enrolled.

-497 unique notifications sent for 206 patients.

-219 of 497 (44%) for ED visits.

-121 of 497 (55%) during normal business 

hours.

-Hospital admissions resulted from 45% of ED 

visits; 18% of these lasted <48 hours, 

suggesting they were avoidable

Onyile, et al., 

2013125

New York New York 

Clinical 

Information 

Exchange 

(NYCLIX)

Query Cross-sectional 

analysis of zip 

code data

Low Mapped most 

current zip code for 

each unique patient 

to the appropriate 

U.S. county; 

calculated distance 

from each zip code 

to Times Square.

-12 visits/ 100 patients within 30 miles; 

-0.4 visits/ 100 patients at 100 miles; 

-88% of patients live within 30 miles of Times 

Square.
45

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Vest, 200954 Texas Central Texas 

HIE (I-Care).

Query Retrospective 

cohort study of 

audit logs

Low Association 

between HIE use 

and resource use.

Factors that predict 

HIE use.

-All levels of HIE information access were 

associated increased expected ED visits and 

ambulatory care sensitive hospitalizations, vs. 

no information accessed.

-HIE used more for those that used the system 

more, or were sicker. 

-HIE not accessed for 43% of individuals

-Ultimately, these results imply that HIE 

information access did not transform care in the 

ways many would expect.

After adjusting for confounding factors the 

following factors increased the odds of HIE 

information access:

OR 1.03 for increasing age.

OR 1.13 for increasing number of chronic 

conditions.

OR 1.63 for at least one prior year clinic visit.

OR 1.96 for an ED visit in prior year.

OR 2.02 for being hospitalized in 2004.

Vest, et al., 

2011104

Texas Central Texas: 

I-Care (EDs at 

11 facilities 

participating in 

HIE)

Query Case-control 

study of audit 

log files

Low Extent of use for 

indigent children: 

association 

between 

basic/novel HIE use 

and resource 

use/patient 

characteristics. 

Novel usage=more 

screens.

System was accessed for 15,586 of 179,445 

encounters (~9%); 

Basic HIE access:

OR ~1.5 for over 1 vs. under 1 year old.

OR ~1.5 for primary care visits in last 12 

months. 

OR ~1.5-2 for ED visits in last 12 months.

OR ~1.3 for hospitalized.

OR ~1.05 for #diagnoses.

OR ~0.46 if unfamiliar with patient.

OR ~0.65 if busier than average.

Novel HIE access:

OR ~1.3 for over 1 vs. under 1 year old.

OR ~2 for primary care visits in last 12 months.

OR not significant for ED visits in last 12 months.

OR ~1.15 for hospitalized.

OR ~1.05 for #diagnoses.

OR ~0.19 if unfamiliar with patient.

OR NS if busier than average.
46

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Vest, et al., 

2011105

Texas Central Texas: 

I-Care

(EDs at 10 

facilities 

participating in 

HIE)

Query Case-control 

study of audit 

log files

Low Extent of use for 

indigent adults: 

association 

between basic HIE 

use and resource 

use/patient 

characteristics. 

-No access of system for 97.7% of encounters. 

-Users accessed the I-Care system for 2.3% of 

the 271,305 encounters.

-Basic HIE usage (42,527) 41% of instances.

-Sample was predominately Hispanic, younger, 

and a higher proportion of charity care 

recipients.

After adjustment:

OR ~0.76 to 0.89 (lower HIE access) for African 

American and Hispanics.

HIE access higher for unknown or charity care.

OR 4.7 vs. 2.6 for unknown payer.

OR ~1.25 to 1.5 (higher access) for more ED 

visits, hospitalizations.

HIE access lower for alcohol use, injury, 

poisoning, unfamiliar patient, busier than 

average day.

Vest and 

Jasperson, 

2012103

Texas Central Texas: 

I-Care

(hospital and 

outpatient

Query Case-control 

study of auditlog files

Low Extent of use; 

HIE use by job 

type, workplace.

Usage patterns.

-297 users, 113 unique job titles, collapsed into 

administration (59% of users), social services 

(~15% of users), physician (~12% of users),

nurse (~6% of users), public health (~6% of 

users), and pharmacy (~1% of users).

-Workplaces: ambulatory care (~9% of users), 

ED (~18% of users), children’s ED (3% of 

users), hospital (53% of users), public health 

agency (8% of users), or mental health agency 

(8% of users).

-In more than 6 out of 10 sessions, users 

accessed the system in a minimal fashion. 

-Average pattern length was 2.89 screens 

(range 1-83 screens); 66% of all user sessions 

had a pattern length of only two screens.

-Use was 94% administrative, roughly evenly 

distributed across workplaces but for dominance 

of hospital accesses (~38%).

-Most clinical access took place in ED and 

public/mental health.
47

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Vest, et al., 

2012101

Texas Central Texas: 

I-Care

(outpatient-2 

urban safety 

net clinics)

Query Case-control 

study of auditlog files

Low Extent of use. 

Association 

between HIE use 

and patient 

characteristics

-HIE accessed for 21% of encounters.

-7,101 encounter-based, 1,227 retrospective.

In adjusted model, access associated with:

OR 1.12 for female.

OR 1.16 for > 40 years.

OR 1.19 of has chronic diseases.

OR 1.13 if had ED visit in last 3 months.

OR 1.33 if hospitalized in last 4 months.

OR 1.52 if received fragmented care.

Johnson, et 

al., 200899

Tennessee MidSouth eHealth Alliance 

(5 EDs)

Query Multiple site 

case studies of 

audit-log files 

and qualitative 

feedback 

Not rated Extent of use in ED.

Percent of users 

who logged in.

HIE viewed in 3% of all visits and 10% of visits 

where patient had visit to another site in past 30 

days.

Percent of total users who logged on ranged 

from 0 in one site where the high was 12% to 

75% by unit clerks in a site that had high use by 

other professions.

Bailey, et al., 

201340

Tennessee MidSouth eHealth Alliance Query Retrospective 

cohort study of 

log data

Low Extent of use. 

Repeat ED visits in 

which HIE was 

accessed vs. repeat 

visits in which HIE 

was not used for 

lumbar or thoracic 

imaging.

HIE use was low, at 12.5% of study population.

Gadd, et al., 

201186

Tennessee MidSouth eHealth AllianceQuery Cross-sectional 

survey

Low Extent of use. -151/162 users (93%)

Average usage per week:

<1 hour: =65 (43%)

Between 1 and 4 hours: 58 (39%)

≥4 hours: 27 (18%)
48

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Johnson, et 

al., 2011118

Tennessee MidSouth eHealth Alliance 

(12 EDs and 9 

safety net 

clinics)

Query Multiple site 

case studies, 

audit logs, 

Comment 

cards,

Feedback in 

system,

Interviews,

Observations,

ED claims

Not rated Extent of use.

Type of data 

accessed.

Provider log on 

rates.

Participant opt out 

rates.

-Access increased from 4% to 7% of patient 

encounters over 24 months, ranged from 1% to 

16 % across sites.

-15% for return ED visits and 19% for return 

clinic visits.

-HIE access higher where nurses and clerks 

involved and lowest where MD only accessed.

-Patient opt out rates were 1-3%.

-Primary user reported consequence of HIE: 

provided additional history (29%); prevented 

repeat test or procedure (20%).

Unertl, 

Johnson, and 

Lorenzi, 

2012119

Tennessee MidSouth eHealth Alliance 

(6 EDs and 8 

ambulatory 

clinics)

Query Multiple site 

case studies, 

direct 

observation at 

14 sites, 

informal 

interviews at 

sites, 9 semi 

structured 

telephone 

interviews

2009

Not rated Workflow patterns, 

by job description.

Cross organizational patterns; 2 workflow 

models identified

1. Nurse workflow: prompted by patient reporting 

recent hospitalization event during intake, HIE 

access by nurse or assistant, printed discharge 

summary, added to chart

2. Physician workflow: HIE accessed by provider 

(doctor or nurse practitioner) for greater reasons 

beyond hospitalization; HIE access occurred at 

various points of care; HIE review of more 

information including history

-Other observations: clerks tracked biopsy 

results; workflow patterns evolved over time, due 

to factors such as access policies or staffing 

changes; residents logged into other EMR due 

to lack of HIE access.

-Reasons to access HIE: visit to another 

hospital; issues of patient trust; communication 

challenges; referrals.

Dixon, Jones,

and Grannis, 

201383

6 states HIE Varies Cross-sectional 

survey

Low Extent of use.

Awareness and 

engagement of 

infection 

preventionists in 

HIE for public 

health surveillance.

-10% of infection preventionists reported their 

organizations were formally engaged in HIE. 

-49% were unaware of organizational 

involvement in HIE.

-<5% reporting via secure email, web-based 

entry, through EHR, or through HIE.

-72% in organizations with EHR

-20% involved in implementation of EHR
49

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Kho, et al., 

201388

Indiana Indiana 

network for 

Patient Care.

5 hospital 

systems (17 

hospitals).

Not stated Retrospective 

cohort study 

with companion 

survey

Low Extent of use.

Coordinated 

antibiotic-resistant 

infection tracking, 

real-time alerting, 

and prevention

In 3 years:

-12,748 email alerts sent on 6,270 unique 

patients.

-23% (MRSA) and 22% (VRE) had previous 

history identified at a different hospital system.

-Of 10 infection preventionists surveyed, most 

recommended to add automated capture of 

laboratory data.

Anand, et al., 

201292

Indiana Primary care 

physician 

offices.

Indiana 

HIE

Cross-sectional 

survey

Moderate Extent of use.

Effect of real-time 

alerting from ED, on 

physician action 

-35% found information helpful vs. 20% not 

helpful.

-24% made followup call to patient vs. 4% sent 

attached letter

Fontaine, et 

al., 201085

Minnesota 9 primary care 

practices with 

fewer than 20 

physicians.

Not stated Cross-sectional 

surveys & 

interviews

Moderate Extent of use. No practice was fully involved in a regional HIE. 

HIE was not part of most practices’ short-term 

strategic plans.

Soderberg 

and 

Laventure, 

201390

Minnesota 1,623 clinics Varies Cross-sectional 

survey

Moderate Extent of use. 

To monitor 

progress toward 

meeting the 

legislative 

requirement that all 

health care 

providers have an 

interoperable EHR 

by January 2015.

-54% exchange data with affiliated hospitals.

-36% with unaffiliated hospitals.

-Common challenges for HIE: limited capacity of 

others to exchange, lack of technical support or 

expertise, competing priorities, cost and privacy 

concerns.
50

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Foldy, 200784 Wisconsin HIE

organizations

Not 

specified; 

varies

Cross-sectional 

survey

Moderate Extent of use. 

Description of 

projects, stages, 

users, 

organizational 

home, governance, 

scope.

-21 of 27 organizations had HIE.

-21 organizations sponsored 16 (76%) 

operational and 11 (52%) planned HIE

organizations projects. Most were surveillance 

systems, but a growing proportion served 

clinicians and patients. 

-Most advanced HIE project had 40% of 

respondents in implementation and 40% in 

operation phases. 

-44% delivered data only to central registries, 

50% delivered to providers and registries.

-63% based in government organizations.

Lobach, et al., 

2007100

North Carolina RHIO Northern 

Piedmont 

Community 

Care 

Network 

(outpatient) 

Retrospective 

cohort study

Low Extent of use.

Frequency and 

types of sentinel 

events.

-Of 11,899 continuously enrolled patients from a 

single county over a six-month period, 2,285 

unique patients (19%) experienced 7,226 

sentinel health events.

Frequency of types of events:

-43 hospital admissions for asthma.

-76 hospital admissions for diabetes.

-2,546 low-severity ED visits.

-1,728 ≥2 missed appointments in 60 days.

Tripathi, et 

al., 2009123

Massachusetts Massachusetts 

eHealth 

Collaborative

Not stated Multiple site 

case studies, 

consumer focus 

groups

Not rated Type of patient 

consent;

Types of data to 

share.

Discussion of experience/lessons learned:

1. Decision on consent: opt in chosen due to 

State law stricter than Federal HIPAA law; use of 

centralized data repository; and consumer 

feedback.

2. All 3 communities agreed on what to share -

all EHR data except text notes, consult letters 

and scanned reports.

3. Consumer focus groups identified themes to 

drive HIE/opt in: promote convenience and 

costs, promote with providers, State benefits up 

front, confront risks, use professional marketing.

4. Consumer opt in across 2 smaller 

communities were 88% and 92%.
51

Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Herwehe, et

al., 2012124

Louisiana Louisiana 

Public Health 

Information 

Exchange

Not stated Cross-sectional 

focus groups, 

interviews, 

message logs

Not rated Extent of use.

Counts of real-time 

alerts and 

responses.

Perceptions of 

patients.

In the 2 year period 2/1/2009 to 1/31/2011:

-488 registrations of 345 unique patients with 

HIV identified.

-Clinicians responded to 73% of alerts and 

documented actions on note that was shared 

with public health.

-Results include statement that 'no negative 

feedback has been received from providers' with 

no detail.

-Summary of patient interviews found general 

acceptance of data sharing as long as there was 

patient benefit and a preference for care in the 

health care verses the public health system.

-Challenges: concerns about data ownership 

and ethics and disparate data systems, but 

these are reported as challenges they were able 

to address.

Kaelber, et 

al., 2013120

Ohio Northeast 

Ohio Public 

Health Care 

System (10 

hospitals and 

affiliated 

practices using 

Care 

Everywhere)

Query Cross-sectional 

surveys and 

audit logs

High Extent of use.

Characteristics of 

patients.

Perceptions of 

users.

Usage of HIE:

-Overall: 1.3%.

-ED: 3.6%.

-Primary care: 2%.

Specialty care: 0.5%.

-Usage highest among patients who were older, 

with more co-morbid illness, Medicare/Medicaid 

insured, and black.

-Self-reported impact was more efficient care 

(93%), time savings (85%), prevented 

admissions (15%), decreased tests ordered 

(84%), decreased imaging ordered (74%), and 

improved care in other ways (82%)

ED = emergency department; EHR = electronic health record; EMR = electronic medical records; HIE = health information exchange; HIPAA = Health Insurance Portability and 

Accountability Act; MD = medical doctor; MRSA = methicillin-resistant Staphylococcus aureus; NS = not significant; NYCLIX = New York Clinical Information Exchange; 

OR = odds ratio; RHIO = regional health information organization; U.S. = United States of America; VRE = vancomycin-resistant enterococci; vs. = versus
52

Extent of Use, Types of Information Exchanged, and Adoption in 

International or Multinational Settings

Six studies that evaluate the use of HIE in non-U.S. settings met our inclusion criteria, one in 

Australia,114 one in South Korea,

89 one in Scotland,122 one in England,121 two in Finland61,115

(Table 7). Three multi-country studies,

94,95,117 two that included data from the United States,95,117

comprise the last three studies in this group. Lee et al. found that the data most commonly

transmitted differed by setting. From the hospital it was working diagnosis; from the clinic, it 

was clinical findings. The most useful data were laboratory or imaging data.89 Silvester and Carr 

found that commitment and interest in adoption increased over time.114 Mäenpää et al. also found 

a steady increase in uses over time by physicians, nurses and administrative staff. 115 Maass et al. 

conducted a unique time-motion study of HIE-facilitated care of 20 diabetic patients, and found 

that of 20 visits, four involved use of HIE, with one facilitating a faster treatment decision and 

three providing access to the most recent test results.61 Investigating use in the National Health 

System in Scotland122 and England,121 Pagliari and Greenhalgh, respectively, both found use to 

be relatively low, although Pagliari’s study is now older (2004). Finally, Jha et al. assessed HIE 

adoption by physicians and hospitals in six developed countries (United States, United Kingdom, 

Canada, Germany, the Netherlands, Australia, and New Zealand), and reported varying results, 

but they did find generally low use due to a variety of identified barriers that prevented fuller 

adoption. In the United States, fewer than 12 percent of organizations were exchanging data on 

less than 1 percent of involved populations.117 In a more recent study conducted in Australia, 

Canada, France, Germany, the Netherlands, New Zealand, Norway, Switzerland, the United 

Kingdom, and the United States, Schoen found that the percent of primary care physicians 

reporting HIE capabilities ranged from a low of 14 percent in Canada to a high of 55 percent in 

New Zealand; use in the United States was reported to be 31 percent.

95 In a study that included 

the 27 European Union countries plus Croatia, Iceland, Norway, and Turkey, Codagnone used a 

factor analysis to create a composite metric that ranged between 0 and 4 to measure the extent of

exchange of health information.94 The metric suggested low to moderate use, with an average 

score across the 31 countries of 1.88. These early reports suggest that HIE in developed countries 

was in the initial stages of use in the early years of the 21st century, and is increasing slowly over 

time. 
53

Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or 

multinational settings

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Lee, et al., 

201289

Seoul, Korea Hospital and 35 

clinics

Not 

specified

Before-after 

surveys

High Types of 

information 

exchanged.

Most commonly transmitted information differed by 

setting:

-From hospital was working diagnosis: 99% vs. 71% 

for clinic, p<0.0001.

-From clinic it was clinical findings: 80%, but this did 

not differ from hospital.

-Most useful was laboratory or imaging in both 

settings but it was more frequently rated as useful by 

hospitals (88% and 7% of cases p<0.0001)

Silvester, et 

al., 2009114

Brisbane, 

Australia

RHIO Not 

specified

Before-after 

database 

analysis of 

clinical 

information

High Extent of use. -Mean events uploaded for each patient record 

during 12 months: 9.7

-Increased HIE use by nurses.

-Number of patients registered increased from 474 

(July 2007) to 1,320 (June 2008).

-Increased commitment to use.

-Interest to adopt by others.

Maass, et al., 

200861

Finland RHIO Not 

specified

Crosssectional 

survey of HIEfacilitated care 

of 20 diabetic 

patients

High Extent of use. Of 20 visits, 4 involved use of information system, 

with 1 allowing faster treatment decision and 3 

providing access to latest test results.
54

Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or multinational 

settings (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Mäenpää, et 

al., 2012115

Finland RHIO Not 

specified

Retrospective 

cohort of audit 

logs

Not rated Extent of use. - HIE utilization rates increased annually in all 10 

federations of Municipalities. 

-Viewing of reference information increased steadily 

in each professional group over the 5-year study 

period.

-No associations detected between use of HIE and 

test ordering outcomes.

Frequency of laboratory test and imaging increased. 

The higher the numbers of emergency visits and 

appointments, the higher the numbers of emergency 

referrals to specialized care, viewed

references, and HIE usage among the groups of 

different health care professionals.

Pagliari, et 

al., 2004122

Scotland Primary and 

secondary care

Varies Crosssectional

survey and 

database 

review

Moderate 6 electronic 

deliverables: 

1) outpatient 

booking;

2) referrals;

3) results 

reporting; 

4) discharge 

correspondence 

5) clinic letters; 

6) clinic email 

Access: 

To referral system (47%), results reporting (37%), 

outpatient booking (3%) 

Use:

Results reporting (36%), referral (18%); clinic email 

(9%); outpatient booking (2%)

Hospital wards able to send e-discharges: 10%;

Wards generating and sending e-discharges: 7%;

Surveys - of responding practices:

Use of Lab results (93%); referrals (58%); 

discharges (42%); outpatient booking (16%).

90% reported daily or weekly use. 

Clinicians most common users of reporting/ referrals;

Administrative /clerical staff most common users of 

discharge/ booking.
55

Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or multinational 

settings (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Greenhalgh,

et al., 2010121

England Primary care 

out-of-hours 

and walk-in 

centers

Varies Crosssectional 

database 

review and 

ethnographic

field notes

Low Use of the 

summary care 

record (SCR)

SCR accessed in 4% of all encounters;

SCR accessed in 21% of encounters where an SCR 

was available;

When available, clinicians accessed SCR 0% to 84% 

of time; main determinants of success were clinician 

characteristics (not specified);

Jha, et al., 

2008117

U.S., 

U.K., 

Canada, 

Germany, 

the 

Netherlands, 

Australia, 

New Zealand

Physicians and 

hospitals

Varies Crosssectional, 

mixed 

methods 

literature 

review, 

surveys and 

interviews

High HIE adoption in 

developed 

countries.

Australia: Early pilots, but no major investment. Lack 

of unified patient ID an issue.

Canada: Province-wide efforts, particularly Alberta; 

national—early development of ‘Health Infoway’ but 

little info exchanged.

Germany: Most computers with records not 

connected; Germans have smart cards, but only 

administrative data now.

The Netherlands: National ‘SwithPoint’ pilot with 20% 

of population, plan full implementation in 2008.

New Zealand: Planning stage, have unified patient 

ID, focus of discharge, laboratory and pathology 

reports to general practitioners.

U.K.: National Programme, but mostly small amount 

of data exchanged in more minor programs.

U.S.: RHIOs, but <12% of organizations exchanging 

data and <1% of population involved.
56

Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or multinational 

settings (continued)

Study Geographic

Location Setting HIE Type Study Type

Risk of 

Bias

Outcome(s)

Assessed Results

Schoen, et 

al., 201295

Australia, 

Canada, 

France, 

Germany, 

the 

Netherlands, 

New 

Zealand, 

Norway, 

Switzerland, 

U.K., and 

U.S.

Primary care Varies Crosssectional 

survey

Moderate Ability to 

electronically

exchange 

patient 

summaries and 

test results with 

doctors outside 

their practice

Percent of primary care physicians reporting HIE 

capabilities:

Australia: 27%

Canada: 14%

France: 39%

Germany: 22%

The Netherlands: 49%

New Zealand: 55%

Norway: 45%

Switzerland: 49%

United Kingdom: 38%

U.S.: 31%

In the U.S. capacity for electronic exchange of 

patient information was concentrated in larger 

practices and those in integrated health systems 

(50% of physicians reported HIE vs. 23% of 

physicians not part of integrated practices p<0.05)

Codagnone, 

et al., 201494

27 countries 

in the 

European 

Union plus 

Croatia, 

Iceland, 

Norway and 

Turkey

Varies Varies Crosssectional 

surveys and 

interviews

Low Factor analysis 

to reveal a 

composite 

measure of HIE 

use 

On a scale between 0 to 4, Denmark score the 

highest (3.04), while the EU27 plus 4 scored 1.88.

HIE = health information exchange; ID = identification; RHIO = regional health information organization; U.K. = United Kingdom; U.S. = United States of America; vs. = versus
57

Key Question 5. How does the usability of HIE impact effectiveness or 

harms for individuals and organizations?

Key Question 6. What facilitators and barriers impact use of HIE?

Key Points

• The 22 studies of usability did not relate usability to effectiveness or harm. 

• The evidence was insufficient to compare usability by type of function (query-based or 

pull vs. directed or pushed exchange) or by type of architecture (centralized or not).

• The most frequent users rated usability higher than infrequent users.

• Sites with proxy users (e.g., nurses, registrars) in the workflow reported the highest HIE 

use.

• The three most commonly cited barriers to HIE use were: lack of critical mass using 

exchanges (8 studies); inefficient workflow (10 studies); and poorly designed interface 

and update features (7 studies).

• Several facilitators showed promise in promoting electronic health data exchange: 

obtaining more complete patient information (6 studies); thoughtful implementation and 

workflow (12 studies); and well-designed user interface and data presentation (7 studies). 

Detailed Synthesis

We identified nine multiple site case studies,82,99,116,118,119,127-130 11 cross-sectional 

studies,

58,62,86,94,131-137 and two before-after studies (Table 8).138,139 Because these studies do not 

include a comparison with a non-HIE organizational site, risk of bias is not reported but is 

described when the details provided sufficient detail. No studies provided results on harm. All 

but five of the studies described experience with exchanging health information in the United 

States.62,94,133,134,139
58

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Bouhaddou, et al., 

201182

Multiple site case 

studies of patient 

records, consent; 

usage. 

Nationwide Health Informatics Network 

(NHIN) via CONNECT gateway allows 

users to pull in data from other 

organizations. The VA and DoD used 

the VLER systems for eHealth 

exchange with private sector. Transfer 

of records between integrated delivery 

systems; National query-based.

Consent was opt in for the VA and 

Kaiser and opt out for DoD.

Patient identifier and 

demographic data, 

rates of consent

Of 363 patients who opted in and provided valid authorization, 264 could be 

correlated; exchange of records between KP and VA 2-3 per week. Older 

patients were more likely to consent for HIE.

Byrne, et al., 2014116

Multiple site case 

studies. Quantitative 

data on Veteran 

participation and 

provider usage, 

interviews with both. 

HIE between VA, DoD, non-Federal 

care organizations. The NHIN. The VA 

and DoD used the VLER systems for 

eHealth exchange with private sector.

Federated pull (query-based) model 

Transfer of records between integrated 

delivery systems; National querybased.

Consent was opt in for the VA and 

Kaiser and opt out for DoD. 

Veterans’ 

authorization 

preferences, system 

dashboard. 73 

provider interviews, 50 

veteran interviews and 

documents from 

meetings 

-Used opt in model for patients and 81% of veterans agreed that each 

patient has a choice

-Matching of patients varied from 12-88% dependent on whether the 

exchange partner used social security number

-None of the veterans interviewed were aware if their providers were using 

HIE, the user-interfaces at the sites face the provider not the patient

-Providers increased usage after training on VLER system

-Providers noted barriers of missing data, additional sign-on and need for 

better integration with workflow

Campion, et al., 

201258

Cross-sectional survey 

of physician 

satisfaction with push 

vs. pull HIE

HealtheLink, Rochester New York

RHIO. Direct exchange (push) of local 

lab and radiology results; query-based 

(pull) searching for lab and radiology 

results across greater Buffalo and 

Rochester area

Online survey 

responses from 112 of 

584 invited physicians 

(19% response rate).

Only 99 completed 

survey.

80% used push HIE and 53% used pull HIE. A greater proportion of MDs 

reported using push HIE always or most of the time (68%) vs. pull HIE 

(19%), (p=0.001). MDs more satisfied with push HIE than pull HIE (p<0.05).

Codagnone, et al., 

201494

Cross-sectional survey 

and interviews of 

general practitioners 

using eHealth that 

included HIE.

Varies as this was an international 

survey

Survey of 9196 

general practitioners 

who used computers 

in 31 European 

countries. 2 Focus 

group sessions.

From focus group sessions, authors reported on usability that HIE remains 

at the “transactional” level and doesn’t yet support information sharing 

across healthcare tiers. There were quite a few general practitioners not yet 

using HIE. Additionally, concern about interoperability, lack of system 

resilience, lack of data standards and concern about security were barriers 

to adoption and use.
59

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Finnell and Overhage, 

2010131

Cross-sectional survey 

of EMS providers and 

analysis of use of HIE

Indiana Network for Patient Care 

(INPC). Community-wide EMR and 

active surveillance of reportable 

conditions, real-time electronic lab 

reporting. Query-based with a 

centralized model. Consent was opt out 

for both providers and patients.

Online survey 

responses from 58 of 

180 invited medics 

(32% response rate), 

Database analysis of 

use of INPC per 

contact.

Over a six month study period, requests for patient data via HIE increased 

from 15% to 26% per patient contact. The majority of medics surveyed felt 

the HIE information was an important for delivering quality patient care, 

particularly for patients who can’t communicate their health history. Medics 

who didn’t use HIE cited network difficulties that delayed receiving the INPC 

abstract.

Gadd, et al., 201186

Cross-sectional survey 

of HIE use and 

usability

MSeHA in Memphis Tennessee.

Consolidated data from multiple 

hospital emergency departments and 

community-based ambulatory clinics.

Query-based exchange with a 

decentralized system architecture with 

secure vaults managed by each 

organization. Consent was opt out.

Email survey 

responses from 165 of 

237 health care 

professionals (70% 

response rate).

-3 usability factors were positively predictive of system usage: overall 

reactions (p<0 0.01), learning (p<0.05), and system functionality (p<0.01)

-Users commented that HIE needs more tech support and could use more 

types of data

Hincapie, et al., 

2011132

Cross-sectional, focus 

groups of physicians

AMIE based on MA-Share created for 

the NHIN that is a federated querybased exchange model. Medication 

history, lab test results, and discharge 

summaries.

Focus group meetings 

of 29 physicians on 

HIE quality of care, 

workflow, cost

Benefits included identification of "doctor shopping", avoiding duplicate 

testing, and increased efficiency for gathering information; disadvantage 

was limited availability of data. 

Hypponen , et. al, 

2014133

Cross-sectional survey

of Finnish physicians 

on HIE success

Varied depending on type of regional 

health informational exchange system.

Type 1: master patient index required 

separate login to centralized database.

Type 2: web distribution model. Limited 

group of referring physicians could see 

hospital info. Type 3: regional virtual 

model. Clinician used an integrated 

system that includes all inpatient and 

outpatient information. Clinician has 

access to electronic patient record at 

other institution. Consent was opt in for 

Type 3.

Survey included 1693 

physician respondents 

aged less than 65 

years. 1079 

specialized care; 614 

primary care

Users of three local EHR systems preferred electronic HIE to paper to a 

larger extent than users of other EHR systems. Experiences with an 

integrated RHIE system (type 3) were more positive than those with other 

types or RHIE systems. Users of Type 1 reported lengthy log-in process 

and information took too long to receive. Recommended that HIE 

organizations address interoperability and interface issues, technical and 

data standards when designing system. Data format at one institution 

should be compatible with format of other institutions. Authors also 

commented that those who had access to all information via their own HER 

may not have realized that they were using HIE.
60

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Johnson, et al., 200899

Multiple site case 

studies.

Quantitative analysis 

of audit-log files; 

qualitative analysis of 

feedback of system. 

MSeHA in Memphis, Tennessee.

Consolidated data from multiple 

hospital emergency departments and 

community-based ambulatory clinics.

Query-based exchange with a 

decentralized system architecture with 

secure vaults managed by each 

organization. Consent was opt out.

Audit logs, 

demographics of 

users, feedback from 

users

-MSeHA was used for 3% of all visits

-The site with the highest usage had registrars looking up HIE data when 

patient arrived at the ED

-The site that mostly serves pediatric patients used MSeHA the least vs. 

other sites

Johnson, et al., 

2011118

Multiple site case 

studies. Quantitative 

analysis of audit data; 

qualitative: semistructured interviews 

and direct 

observations. 

MSeHA in Memphis Tennessee.

Consolidated data from multiple 

hospital emergency departments and 

community-based ambulatory clinics.

Query-based exchange with a 

decentralized system architecture with 

secure vaults managed by each 

organization. Consent was opt out.

Audit logs, 

feedback in system 

(12% of all patient 

visits with HIE), 

interviews,

observations

ED claims

HIE access was higher where nurses and clerks involved and lowest where 

MD only access, patient opt out rates were 1-3%. 

Kierkegaard, Kaushal 

and Vest, 2014127

Multi-site case study. 

Qualitative, interviews 

with users and 

nonusers of HIE. 

3 RHIO sites with query-based 

exchanges in New York: 2 federated 

models, 1 centralized model.

Automated delivery of imaging and lab 

results to provider EHRs for two 

exchanges, automated CCD (one 

system). The one system that didn’t 

have automated delivery included 

secure messaging and event 

(admission) notification

2 day site visits, onsite 

and telephone 

interviews with HIE 

users and non-users, 

observations of 

workflow

-MDs had low tolerance for search failures

-Where clerks were not trained or supported, fewer patients consented

-MDs often delegated the HIE task

-Login process perceived as a burden and system was slow.
61

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Machan, 

Ammenwerth, and 

Schabetsberger, 

200662

Cross-sectional. 

Qualitative semistructured, problemcentric interviews 

followed by crosssectional survey on 

usage. 

TILAK, health@net in Tyrol region of 

Austria. Transmission of discharge 

letters and clinical findings from 

hospitals to general practitioners.

Direct exchange via email that was 

automatically integrated to physicians’ 

computer system.

Interview with 4 

providers followed by 

cross-sectional survey 

of 104 of 242 (43%) 

providers on HIE use

-Overall satisfaction positive for 66.4%, with 83.7% agreeing to receiving all 

reports electronically, 82.7% reporting less work for filing and archiving, and 

78.8% agreeing it led to improved quality of care

-Barriers were reported, e.g., reports not meeting physician's needs 

-One facilitator is automatic filing of HIE information in patient EHR

Massy-Westropp, et 

al., 2005134

Cross-sectional 

satisfaction survey 

and 2 staff focus 

group sessions

Exchange in Adelaide, South Australia 

linking a public teaching hospital, ED 

and aged home-based care community

services organization. When admitted 

to the hospital, the patient was added 

to a daily inpatient list received by the 

home-based providers who could log 

into secure website to run live reports 

of matched inpatients.

Satisfaction survey 

responses from 55 of 

132 nurses, clinicians 

and allied health staff, 

2 focus group 

sessions with staff

Those who had embraced the use of the integration tools were significantly 

more likely to rate Integration higher than those who were not using it as 

often (p<0.001).

In the discussion they estimated a 20% savings in staff time.

McCullough, et al., 

2014135

Cross-sectional. Key 

informant interviews 

with stakeholders at 

practices and health 

centers

2 states: California, Minnesota.

California: Collaborate HIE system, a 

Query-based exchange from three 

hospitals, 90 providers, and 

laboratories.

Minnesota: CentraHealth exchange 

between Federally Qualified health 

Centers and hospitals. This system 

was in implementation at time of study.

24 interviews with 

clinicians, 

administrators, and 

office staff users

Identified barriers: Lack of well-functioning area-level exchange, 

challenge achieving a critical mass of users, need strong relationships with 

exchange partners, incompatible Health IT used, data ownership and 

provider liability concerns about who sees the data, can’t find data on 

patients.

Identified benefits: Improved productivity at initial visit, improved 

completeness of records, avoidance of duplicative services of patient 

financial risk Improved nonvisit consults
62

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Messer, et al., 2012138

Before-after study of 

organizational 

readiness to change, 

needs assessment 

interviews and prepost quantitative 

survey of HIV provider 

users

North Carolina HIV information 

cooperative regional health information 

organization (CHIC RHIO). 1 large 

academic med center and 5 AIDS 

service organizations. Used 

CAREWare from HRSA. Query-based 

exchange where each participating 

organization managed its own 

database.

Interviews and 

assessment with 

39 stakeholders; pre 

and post survey of 29 

providers' satisfaction 

with HIE, relationships 

with other providers, 

barriers.

-Qualitative and quantitative approaches provided several “lessons learned”

-It is important to establish clear understanding of privacy and data sharing 

among stakeholders

-Initial concerns about confidentiality diminished over time as trust was built

-Respondents noted it is important to manage expectations upfront

-Clinic staff must use 2 systems the EHR and CAREWare which takes 

effort and increases errors

-There was an unmet need for training for report generation

Myers, et al., 2012128

Multiple site case 

studies. Quantitative: 

emailed survey to 

current and intended 

users; qualitative: 

interviews with current 

HIE users during site 

visits

5 exchanges that were part of the 

Information Technology Networks of 

Care Initiative that included BronxLebanon Hospital Center, Duke 

university; hospitals, the city of 

Paterson, Louisiana State University

Health Care Services Division, New 

York Presbyterian Hospital, St. Mary 

Medical Center Foundation.

Query-based.

Interviews and Webbased survey with 

case managers, 

providers and 

nonclinicians on 

usefulness and ease 

of use.

62 of 102 responded 

(62%)

-Mean composite for ease of use was high (3.9 of 5.0) and no difference by 

role

-Mean composite for usefulness was also high (4.0 of 5.0) and no 

differences by role

-Qualitative: adoption of the HIE and perceptions of its use and usefulness 

varied by occupational role of the patient-care team. Also noticed that case 

workers outside the clinic used the HIE routinely. Those within clinics used 

HIE sporadically.

Nohr, et al., 2001139

Before-after Danish 

study that included 

survey and interviews 

on HIE expectation vs 

experience.

Four types were described: (1) 

common database; (2) Electronic Data

Interchange via structured messages:

copies of data are transferred between 

systems; (3) middleware: software 

between application and database; (4) 

internet technology: data 

communicated via browser.

Survey respondents:

Expected benefits in 

1998 (n=102); 

Experiences in 

benefits in 1999 

(n=57); Expected 

barriers in 1998 

(n=101); Experiences 

in barriers in 99 

(n=99). Group 

interviews per site.

Several organizations have since started workflow analysis to identify 

former hidden procedures and for determining user requirements. One of 

the barriers was that most professionals used to the free-text nature of 

paper records and were now forced into structured format. One of the 

barriers was lack of knowledge about integration principles which left the 

vendors to provide solutions.

One of the facilitators of success was a bottom-up approach with users 

involved during implementation. It is also helpful if the training go beyond 

basic use and provide information on becoming experts in using HIE.

Finally, the organizations were unprepared technically to have a system 

running 24/7. They suggested having back up plans, e.g., mirrored 

databases.
63

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Ozkaynak and 

Brennan, 2013129

Multiple site case 

studies. Direct 

observation, informal 

interviews during 

observation, formal 

semi-structured 

interviews with HIE 

users. 

3 ED sites accessing the EDLinking 

system in Madison, Wisconsin.

Clinicians can choose to use (or not 

use) the exchange.

210 hours direct 

observations, varied 

across shifts, in 5 

rounds, informal 

conversations to 

followup on 

observations, plus 13 

open ended HIE 

interviews.

-The ED providers only used the HIE for 5% of visits

-It was used primarily for patients in chronic pain to detect drug-seeking 

behavior. This information was then used as support to confirm or confront 

patients who may be abusing the system.

Rudin, et al., 2011136

Cross-sectional.

Twenty interviews with 

clinician users, HIE 

staff and 

administrators

Massachusetts eHealth Collaborative.

All nontext portions of medical record.

Could link directly from the EHR to 

existing exchange. Query-based 

exchange. Consent was opt in.

Interviews of 15 

clinician users, 2 HIE 

staff, and 3 

administrators

-Motivators were belief in improved quality of care, time savings, and 

reduced need to answer questions.

-Motivation was moderated by missing data, workflow issues, and usability 

issues (too many clicks required to get to information).

-Missing data was attributed contributing providers not "locking their notes" 

on their EHR.

Thorn, Carter, and 

Bailey, 2014130

Multiple site case 

studies. Interviews 

with ED physicians 

using HIE

HIE name not explicitly stated but may 

be MidSouth eHealth Alliance 

(MSeHA). Query-based exchange.

Consent was opt out.

Individual 

unstructured 

interviews with 15 ED 

physicians

Barrier themes

1. Trouble accessing system, acuity of patient or history not available, team 

members' inability to access.

2. HIE use affected decisions only sometimes, for specific cases (e.g. drug 

seekers).

3. Access challenges, separate login, variability in data being pertinent, 

absence of data types or data on specific patients, user design flaws, and 

lack of technical support.

4. Barriers to usage also included continued practice of defensive medicine, 

desire for autonomy, changing the culture, belief that HIE does not alter 

decisions, health system competition, and reduced revenue, workflow 

disruption.
64

Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)

Author, Year

Study Design

HIE Description

Type of HIE

Patient Consent Process

Evaluation Data Results

Unertl, Johnson, and 

Lorenzi, 2012119

Multiple site case 

studies. Ethnographic 

study, direct 

observation, informal 

interviews during 

observation, formal 

semi-structured 

interviews with HIE 

users. Moderate risk 

of bias

MSeHA in Memphis, Tennessee

Consolidated data from multiple 

hospital emergency departments and 

community-based ambulatory clinics.

Decentralized, query-based exchange.

Consent was opt out.

Observation (180 

hours) in 6 ED and 8 

ambulatory clinics, 

informal interviews 

during observation 

and 9 formal semistructured interviews 

with physicians, 

nurses and IT 

management

-HIE workflow was modeled for each ED site and clinic

-2 models emerged: physician-based and nurse-based

Yeager, et al., 2014137

Cross-sectional.

Qualitative analysis of 

16 interviews with 

healthcare 

stakeholders

LaHIE

Hybrid, centralized and federated HIE 

in Louisiana that includes DIRECT 

messaging between providers.

Providers can share CCDs, lab results, 

and electrocardiogram results.

Interviews with 16 

healthcare 

representatives from 

organizations 

interested in joining 

LaHIE but not yet 

enrolled (n=4), not 

interested in joining 

(n=4), or already 

enrolled (n=8)

Five themes were identified related to usability.

1. Physicians found separate HIE logins required recalling separate 

passwords and delayed receiving information. Suggested having staff 

access HIE prior to visit and bring into patient chart.

2. Training is needed to get a critical mass of providers to contribute.

3. Quality of data in HIE is limited if some only provide discrete data.

4. Physicians expressed concern about liability if the HIE data isn’t 

integrated into the patient chart.

AMIE = Arizona Medical Information Exchange; CCD = Continuity of Care Documents; CHIC = Carolina HIV Information Cooperative; DoD = The Department of Defense; 

e = electronic; ED = emergency department; e.g. = for example; EHR = electronic health record; HIE = health information exchange; HRSA = Health Resources and Services 

Administration; IT = information technology;; KP = Kaiser Permanente; LaHIE= Louisiana health information exchange; MD = medical doctor; MSeHA = Mid-South eHealth 

Alliance; NHIN = The Nationwide Health Information Network; RHIE= regional health information exchange; RHIO = regional health information organization; TILAK = Tiroler 

Landeskrankenanstalten; VA = The Department of Veterans Affairs; VLER = Virtual Lifetime Electronic Record
65

HIE Usability 

Usability was defined in the 1998 International Standards Organization 9241-11 standard as 

“the extent to which a product can be used by specified users to achieve specified goals with 

effectiveness, efficiency and satisfaction in a specified context of use.” We found five surveys on 

HIE usability and most defined usability as it relates to function and/or measured satisfaction 

with exchanging health information.58,62,86,133,138 One multiple site case study reported usability 

as composite measures of: ease of use and usefulness, described below for current and intended 

users of five HIE systems.128 The composite score for perceived ease of use (which included 

level of agreement for 10 statements on use) averaged 3.9 on a 5.0 scale where 5.0 was “strongly 

agree.”128 For example users were asked to provide level of agreement for, “Learning to operate 

‘the HIE’ was easy for me.” Similarly, the same respondents averaged 4.0 of 5.0 on the 

perceived usefulness composite score, which was also based on responses to 10 statements. The 

survey sample included 24 case managers, 21 medical providers, and 17 nonclinician staff 

members and perceptions about usability did not vary by role. This emailed survey achieved a 62 

percent (62 of 102) response rate and the inter-scale agreement reliability; Cronbach Alpha 

ranged from 0.57 to 0.93.

Usability features were also examined in relation to actual use in one cross-sectional study of 

health care professionals electronically exchanging health data through the MSeHA.86 Health 

professionals were emailed the survey and responded to questions about actual use and usability 

features that included questions from the Questionnaire for User Interface Satisfaction (QUIS)

7.0 instrument in six areas: overall reactions, screen, terminology and system information, effort 

required to learn the system, system capabilities, and system functionality. Multivariate analyses 

revealed that average weekly use of the MSeHA was associated with higher scale scores in:

overall reactions (OR 1.50, p<0.01), learning (OR 1.32, p<0.05), and system functionality (OR 

1.34, p<0.01). The reported psychometrics for the survey questionnaire (inter-scale agreement 

reliability on the QUIS scales: Cronbach’s Alpha ranging from 0.74 to 0.91) and response rate 

(165 of 237, 70%) were good, reducing concern about bias and increasing ability to generalize.

HIE Satisfaction 

Satisfaction with HIE, a measure of usability, was examined in one cross-sectional study134

and one before-after study 138 One additional cross-sectional study that stratified satisfaction by 

types of HIE is described later.58 Using a pre-post survey study design (n=29), physicians at one 

clinic and five AIDS service organizations in North Carolina reported increased satisfaction after 

the Carolina HIV Information Cooperative (CHIC) RHIO was implemented.138 Participants 

reported improved satisfaction with ease of data exchanged and improved patient care after using 

CAREWare software. The respondents also perceived that CAREWare was a good use of 

resources. They also reported improved relations with HIV care partners after implementing the 

RHIO. By contrast, before implementation, the providers had high expectations for how 

exchanging information would affect their work and reported some unmet expectations 

afterward. 

In a second study on satisfaction of HIE users in Adelaide, South Australia,134 users who 

embraced the use of the data exchange integration tools were significantly more likely to rate 

integration higher than those who were not using it as often (p<0.001). This result echoes a more 

recent study that found frequent users are more pleased with the usability of an HIE system than 

infrequent users.86 The response rate for the Massy-Westropp study was 24 percent (55 of 132). 
66

While both satisfaction studies134,138 provide descriptive evidence from surveys that users were 

satisfied with usability, neither provided sufficient details in the methods sections to eliminate 

bias or a comparison that would enable generalization. 

Usability of HIE by Type

We also examined whether certain functionality (direct exchange or push vs. query-based

exchange) was more usable. Directed exchange is provider-to-provider electronic exchange of 

patient information to coordinate care.

32 In this type of exchange, the data are electronically sent 

to the recipient’s EHR or clinical inbox.58 In query-based systems, the user accesses an exchange 

system, queries for information (e.g., ED, hospital admissions, or discharges) on a particular 

patient and pulls data from multiple health care organizations.58 This is important particularly for 

unplanned care (e.g., patient comes into the ED).

13 We also attempted to evaluate usability by 

type of architecture (e.g., whether the query-based system used a centralized or federated model).

However, few publications provided this level of technical detail to make a comparison.

Additionally, the authors used a variety of terms and descriptions which made it difficult to 

classify usability by architecture. When the authors provided detail on architecture, it was 

included in Table 8.

Only one cross-sectional study evaluated clinician satisfaction with exchanging health 

information using query-based (pull) or direct exchange (push).58 In this comparison study, 

clinicians had access to “pushed” health data (laboratory and radiology) through certified EHRs;

physicians who ordered tests could designate other physicians to receive the test results. The 

physicians in this study could also query (pull), using a secure web portal, for test results, patient 

demographics and transcribed reports provided by physicians, hospitals, laboratories and 

radiology centers across the greater Buffalo and Rochester areas of New York. More providers 

reported using electronically pushed data exchange (80%) than pulled exchange of health 

information (53%). A greater proportion of physicians reported using pushed data exchange 

always or most of the time (68%) compared with pulled exchange (19%, p=0.001). The 

physicians were more satisfied when data were pushed than pulled (p<0.05).

In summary, we found insufficient data to compare usability by type or architecture of the 

electronic data exchange.

Facilitators and Barriers Impacting HIE Use 

We identified many barriers and facilitators to electronic health data exchange in the 

literature. Evaluations of the MSeHA provide the most complete evidence on barriers and 

facilitators of use86,99,118,119,130 but other studies echoed similar barriers.62,82,94,116,127-129,131-

133,135,136,139 Barriers and facilitators were assessed with qualitative approaches in these studies 

which were difficult to assess for risk of bias and generalizability. In this section, the barriers 

mentioned most often are presented in partnership with affiliated facilitators (Table 9).
67

Table 9. Barriers and facilitators of actual HIE use grouped by theme

Barriers Studies of 

Barriers Facilitators Studies of Facilitators

Lack of Critical Mass

• Patients concerned about 

privacy and security

• Poor matching or patients

• Providers stop using querybased system when can’t find 

patients

• Incomplete patient 

information

• Patients outside of the HIE 

catchment area

Bouhaddau, et al., 

201182

Byrne, et al., 

2014116

Hincapie, et al., 

2011132

Kierkegaard, 

Kaushal, and Vest, 

2014127

McCullough, et al., 

2014135

Ozkaynak and

Brennan 2012129

Rudin, et al., 

2011136

Thorn, Carter, and 

Bailey, 2014130

More Complete Patient 

Information 

• Consider opt in vs. opt out 

• Obtain consent at 

registration

• Educate patients on HIE

• Make HIE visible to patients 

(turn screen so they can 

see it during visit).

• Consider when to push and 

when to pull data

Bouhaddou, et al., 

201182

Byrne, et al., 2014116

Campion, et al., 

201258

Kierkagaard, 

Kaushal, and Vest, 

2014127

Messer, et al., 

2012138

Johnson, et al., 

2011118

Inefficient Workflow 

• Separate login to portal – too 

many clicks.

• Unmet expectations

• Policy that prohibits proxy 

users

• Need for more technical 

support

• Need for culture change 

about practice

Byrne, et al., 

2014116

Hypönnen, et al,. 

2013133

Johnson, et al., 

2011118

Kierkegaard, 

Kaushal, and Vest, 

2014127

Machan, 

Ammenwerth, and 

Schabetsberger, 

200662

Messer, et al., 

2012138

Myers, et al., 

2012128

Nohr, et al.,

2001139

Rudin, et al., 

2011136

Thorn, Carter, and 

Bailey, 2014130

Thoughtful implementation 

and workflow

• Identify former hidden 

workflow

• Provide training for 

providers and proxy users

• Manage expectations of 

new HIE

• Develop workflow for 

providers and proxy users.

• Have providers and proxyusers involved in design of 

interface

• Implement a case 

management approach for 

HIE use

• Have champion HIE users

• Have sufficient technical 

support

Byrne, et al., 2014116

Gadd, et al., 201486

Hincapie, et al., 

2011132

Kierkegaard, 

Kaushal, and Vest, 

2014127

Johnson, et al., 

200899

Johnson, et al., 

2011118

Messer, et al., 

2012138

Rudin, et al., 2011136

Thorn, Carter, and 

Bailey, 2014130

Ozkaynak and 

Brennan 2012129

Nohr, et al., 2001139

Unertl, Johnson, and 

Lorenzi, 2012119
68

Table 9. Barriers and facilitators of actual HIE use grouped by theme (continued)

Barriers Studies of 

Barriers Facilitators Studies of Facilitators

Poorly-designed Interface and 

Update Features

• Too much information and

slow response

• Duplicate Information

• Reports in exchange 

workflow may not meet 

needs of the provider

• Competing use with existing 

patient portal with complete 

information

• Lack of notes to set context 

in patient information

• HIE not updated in real time

Hypönnen, et al.,

2013133

Codagnene and 

LupiañezVillanueva, 201494

Kierkegaard, 

Kaushal, and Vest 

2014127

Myers, et al., 

2012128

Thorn, Carter, and 

Bailey, 2014130

Machan, 

Ammenwerth, and 

Schabetsberger, 

200662

Rudin, et al., 

2011136

Well-designed Interface and 

Data Presentation

• Monitor quality of data 

against standards

• Provide clear notifications of 

HIE

• Send brief report first

• Automatic integration with 

existing provider systems

• Include providers and proxy 

users in design of interface

Bryne, et al., 2014116

Hypönnen, et al,. 

2013133

Campion, et al., 

201258

Kierkegaard, 

Kaushal, and Vest, 

2014127

Machan, 

Ammenwerth, and 

Schabetsberger, 

200662

Thorn, Carter, and 

Bailey, 2014130

Myers, et al., 2012128

HIE = Health information exchange; vs. = versus

Addressing Lack of Critical Mass

Concern was expressed in several studies about the need for a critical mass of users and 

populated patient information.

82,116,127,129,130,132,135,136 Underlying reasons for lack of critical mass 

can include several reasons (e.g., the providers aren’t electronically exchanging the data or 

patients have not consented). Patients concerned with privacy and security may not understand 

the benefits and/or may not consent to have their data shared with other providers. Even when 

they do consent, they may not be properly matched to existing data.132 Also, match rate can vary 

by population and setting; for example, the match rate for providers practicing in a homeless 

center was lower, but the match rate for ED physicians was higher.

132 Some contributing 

providers reported legal concerns for sharing patient data and may choose to not participate. The 

end result was that providers searching for patient information may grow frustrated at taking the 

time to search and stop using the system.

To increase the critical mass, several approaches have been suggested. These include 

addressing concern about privacy, careful consideration about the consent process, and a process 

for educating patients.58,82,116,118,127,134,138 To address patient and provider concern about privacy, 

create clear understanding about privacy and data sharing among all stakeholders (providers, 

patients, nonclinician partners) prior to implementation.138 In planning for electronic health 

exchange, several authors noted the importance of deciding whether to have opt out or opt in

consent process for patients.

58,82,99,116,118,119,127,129,136 Of veterans interviewed, 90 percent were 

positive about the VLER HIE system. At the same time, 81 percent felt each person should have 

a choice to opt in and the default should not be automatic participation.

116 Opting in protocols 

seem to yield a high patient participation rate (93% to 97%).58,127,136 When age is considered, 

older patients opt in more often than younger patients.82 The percentage of consented patients 

can be increased with a workflow that includes front staff members being trained to educate and 

consent patients as they first arrive.127 Additionally, patient awareness of provider use of the HIE 

may increase patients perception of the benefits of electronically exchanged data. Patients in the 

VA reported being unaware that providers were using the VLER system to access information 
69

outside of the VA.116 The authors noted that the user interfaces of the VLER are not visible by 

patients because the display faces the providers. We identified one organization that used an opt 

out protocol (MSeHA).118,119 Patients had the option to opt out at every encounter. The opt out 

rate was 1 to 3 percent,118 which is slightly better than programs with an opt in protocol that lose 

3 to 7 percent of patients who do not consent.58,136

Addressing Inefficient Workflow in Electronically Exchanging Data

Often the workflow was inefficient to providers attempting to exchange health 

information.

62,116,118,127,128,130,136,138 Users complained that additional logins and policies against 

proxy users increased the time the provider needed to access the patient information.

Sites with proxy users (registrars, nurses, clerks, and other physicians) who accessed the 

system and then provided the information to the attending physician had the highest access 

rates.99,118 Proxy use was described as a way to save provider time or address needs of limited 

users without privileges.130,132,136 Additionally, some organizations made it difficult to get 

privileges to access exchanged data so those with privileges were called upon to look up 

information for those without.130

An ethnographic qualitative study of the MSeHA identified two role-based workflow 

models: physician-based and nurse-based.119 These investigators completed 180 observation 

hours of six EDs and eight ambulatory clinics using the MSeHA exchange system, informational 

interviews during observation, and nine semi-structured interviews. In the nurse-based model, if 

a patient mentioned a recent hospital visit, the triage nurse or medical assistant would search for 

data primarily looking for summary documents related to recent hospital visits, such as a 

discharge summary, but rarely searched for other medical history. The nurse then printed off the 

information for use by the provider. In the provider-based model, physicians and nurse 

practitioners searched for electronically exchanged information for more reasons than hospital 

visits. These providers browsed online medical history for purposes of decisionmaking. Finally, 

another study of the MSeHA reported that use dropped significantly after a new policy 

prohibited registrars from searching the system at the start of a visit.118 Initially registrars would 

print off a summary sheet of available data. Providers then queried the system, based on the 

summary sheet. When a new policy came in place prohibiting registrars and nursing team 

members from accessing the system for security reasons, use dropped significantly.

During implementation several other strategies were mentioned related to changing current 

workflow: providing training and enough technical support to support the new workflow,86,116

addressing needed culture change,130 and having champion users.99,127 One physician expressed 

in an interview that exchanging data is a change from practice. Physicians “get bogged down 

[with exchanging health information] and just want to see patients”.130 Introducing new 

technology requires addressing the need for change and the resistance that may exist. These 

studies also encouraged sites to manage expectations upfront138 and have a pilot implementation 

prior to launch so users aren’t disappointed.118,132

Addressing Poorly Designed Interface and Update Features

Several design features of the HIE created barriers to use.62,116,127,128,136 While HIE users 

understood why textual notes were not exchanged for confidentiality reasons, this lack of context 

made the information less valuable.136 While some users wanted more information, other users 

wanted shorter reports to avoid having to scroll up and down, click on many pages or go to 

another task. Some complained that the exchange contained too much information that was not 
70

filtered enough to be meaningful for providers.127,128 They reported that reading a paper report 

was much faster than reviewing the exchanged information.

128 This finding was echoed by 

another study that recommended the main findings should be sent first in a brief report.62 The 

design features could be addressed better at the implementation phase by including more 

providers during the design phase.127 Another facilitator is to continually monitor the quality and 

usability of the exchanged data to meet standards and the needs of the users.116 Similarly, as 

more patient data and more types of data were exchanged, users reported that their system 

response slowed suggesting the need to continually review (and reduce) what was being 

exchanged.116

Some users expressed concern with how quickly the patient information was updated and 

found it more efficient to go directly to the partnering clinic or hospital for information than to 

rely on current information in the exchange.

128 Systems that automatically integrate with the 

providers’ EHRs may reduce this concern and also reduce need for users to have to login into 

multiple systems.62,130

Key Question 7. What facilitators and barriers impact implementation of 

HIE? 

Key Question 8. What factors influence sustainability of HIE?

Key Points

• There was a sizable body of research that attempts to identify and categorize the 

facilitators and barriers to implementation and factors that affect the sustainability of HIE

(52 studies).

• This literature identified several categories of characteristics of HIE activities and 

organizations (internal factors) that affect implementation 

o The most commonly identified facilitators were general organizational 

characteristics such as leadership while the most frequently cited barriers were 

disincentives such as lack of financial viability. 

• The research cited policy and external environment influences as affecting 

implementation less frequently than internal factors.

o Laws and mandates that require or support organizations engaging in HIE were 

the most frequently reported external facilitator for implementation.

• The most frequently cited negative influence on sustainability was competition that 

limited the necessary collaboration among organizations required to support HIE.

• Two key positive influences on sustainability were desire for the expected outcomes from 

HIE and the selection of HIE functions most likely to have financial benefits.

Detailed Synthesis

Both implementation and sustainability are organizational level measures of approaches to 

change. While the experiences, attitudes, and priorities of individuals may be important, 

ultimately the decisions to adopt and continue to support HIE activities are made by 

organizations not individuals. For this reason this section focuses on organizational level 

characteristics and factors that affect organizations’ decisions and actions.
71

Implementation involves identifying new practices or technologies; making the decision to 

incorporate them into workflow and processes; and taking the actions necessary to prepare for 

and then initiate adoption of change. Sustainability is essentially the ongoing maintenance of 

what was implemented, but also includes the idea that the practice or technology that was 

implemented must evolve to continue to meet the changing needs of the organization. 

Approaches to understanding implementation and sustainability are rooted in consideration of 

the fit between an organization and the practice or technology as well as the external and internal 

factors that either facilitate or act as barriers to the change. In the case of HIE, health care 

organizations must consider first whether, and then how, to participate in HIE (implementation). 

Once HIE is established the focus shifts to how to maintain, improve, and grow the systems 

(sustainability).

We identified 52 studies that addressed implementation and/or sustainability (Appendix F).

Fifty of the included studies were published in the past 8 years (2006 to 2014). Eight studies 

assessed HIE activities in countries other than the United States, 10 were based on U.S. national 

surveys or data, 10 covered multiple sites in the United States, but the most common were 24

studies that covered single State or regional HIE organizations and their efforts. Six of the 

studies were about HIE in New York, with five about statewide efforts or several RHIOs and one 

about New York City. Three were about HIE in California, but each study was about a difference 

regional HIE organization. No other State or metropolitan region was the subject of more than 

two studies. 

Most of the studies were cross-sectional designs that collected data via surveys and 

interviews and relied on qualitative data analysis. More specifically 26 of the studies were crosssectional,79,84,85,87,94,100,108,124,140-157 17 were multiple site case studies that compared experiences 

across different organizations or sites,82,116,122,123,158-170 two compared outcomes before and after 

HIE,114,138 three were retrospective cohorts,43,44,48 two were prospective cohort studies,171,172 and 

two were time series.173,174 Almost half (23 of 52) of the studies used data from multiple sources, 

while the most common sole sources were interviews (10 studies) and surveys (9 studies). Other 

sources of data included databases (4 studies), audit logs (3 studies), and one each that used 

documents, organizational assessments, and a literature analysis. 

Given the focus of Key Questions 7 and 8 and the sources of data it is not surprising that 

most of the analyses where qualitative (25 studies), including narrative summaries and the 

identification of themes. Twelve studies used quantitative analyses such as descriptive statistics, 

while seven employed more complex multivariate analyses. Eight studies combined qualitative 

and quantitative analyses (mixed methods). 

Variety in study design, data sources, and analytic methods make assessing the quality across 

the 52 studies that address these Key Questions problematic. Quality assessment is frequently 

tied to risk of bias and the criteria are related to how the groups are constructed in cohort studies 

and how quantitative analyses are used to make these comparisons. While there are criteria for 

quality in other types of studies, these are used less frequently and there is not yet widespread 

agreement on the criteria, what is necessary to meet them, or what constitutes the difference 

between levels of quality. We can say that most of the studies in this section either attempted to 

include all sites or participants or included large samples of the population, increasing the 

likelihood that they are representative of the target populations. Also as we excluded purely 

descriptive studies, the qualitative analyses tended to follow established procedures (e.g., 

involvement of multiple researchers in coding) although in several cases the description of 

methods was limited.
72

One or more facilitator or barrier to implementation was identified in 42 studies while 17 

studies reported factors related to sustainability. Some studies addressed by implementation and 

sustainability. We grouped the facilitators into eight categories and the barriers into seven 

categories created based on our interpretation of their similarities. These are described in the text 

below. In Tables 10 and 11 the specific factors included in each category are listed below the 

category in the first column and the studies that report this factor related to implementation or 

sustainability are cited in the second and third column respectively. 

Implementation 

Facilitators

Seven of the eight categories of facilitators for implementation identified in the literature 

(below) are predominately “internal” factors, concerned with the characteristics of the HIE or its 

components, while only one category, external policy, addresses the environment for the HIE. 

General Structural Characteristics

These include leadership,43,144,164,174 prior experience with or readiness for IT projects,

138,158

preexisting membership in a network,

155 or trust and solidarity among practices participating in 

HIE. One evaluation of HIE efforts concluded that, “having IT initiatives underway prior to 

receiving... funding contributed substantially to the states’ readiness and subsequent 

implementation progress.”158

HIE Specific Structures

This category includes findings from seven studies and specific factors were goverance,

43

and participatory approaches that included efforts to encourage user engagement and stakeholder 

buy-in.

48,122,124,150,159 Examples include findings that involving users in development was key to 

implementation150 and that a participatory process and shared decisionmaking permitted the HIE 

to address different values held by participants related to balancing individual rights and public 

health.124

Orientation Shift in HIE Organizations

This is a category that could also be called mission or change in ideology. Two studies found 

that implementation depended on a shift from competition to collaboration,

154 or from ownership 

of data to continuity of care that included realizing the value of external information.170 Another 

important shift is from treating HIE activities as a pilot test to integrating them into a robust 

system integrated in workflow.163 This research highlighted experiences that staying in the pilot 

phase for too long was detrimental to full implementation and increased use.

Design Characteristics

Cited as a facilitator for implementation in six studies. Studies found that a design that 

reflects an understanding of work flow,

150 and designs with smaller scale or more limited scope 

were more likely to be implemented.169,173 The architecture and adaptability of information 

systems were cited as important design characteristics by two studies161,169 with one researcher 

explaining, “Our findings suggest that communities embarking on HIE initiatives would do well 

to examine how particular HIE technical architectures map to their objectives, local context, 

existing relationships, sustainability plans, and vision of both present and possible future 
73

needs.”161 An additional study found that successful HIE organizations used some existing 

standards rather than waiting for more universal standards that are under development.159

Key Functions

This is a category of functions that may seem obvious but that are essential. Four studies 

reported that HIE systems needed to be set up so that use became part of care routines, so that the 

burden and time required of staff was minimized and so that useful data was provided.85,114,116,169

One study concluded: “Implementation outcomes...were shaped substantially by the degree of 

attention dedicated to reworking procedures and practices so that HIE usage becomes routine.”169

Another study highlighted that addressing issues related to providing better quality data and 

integration into workflow allowed successful system-wide deployment.116 However, the capacity 

for advanced use (HIE that provides new tools or information) may be an important facilitator as 

HIE evolves. One study cited the example of HIE providing the foundation for development of a 

system that alerted providers to important patient events leading to both improvements in quality 

of care and contributing to organization goals such as medical home certification.

143

Implementation Support

The need for an organization to provide resources to support the implementation of HIE was 

cited in the results of four studies. Specific types of support cited included technical assistance 

and training infrastructure,114,167 the ability to do extensive testing for data quality,

154 and a 

comprehensive strategy for HIE activities and their implementation.168

Expected Outcomes

Two studies reported that specific expected outcomes were key to implementation. These 

included public awareness of the HIE148 and link to a community need.146 A third study 

highlighted the importance of establishing tangible intermediate goals in order to keep 

participants engaged and foster ongoing support.159

External Policy

Federal and State laws and mandates,

85,140,159 as well as grants,158 were identified as 

facilitators in five studies when they promoted, required, or funded HIE director or foundational 

components such as EHRs. One study of 31 countries in Europe documented that HIE activities 

were more widespread in countries with national healthcare systems verses countries with social 

insurance systems.94

Table 10. Facilitators to implementation and sustainability of HIE

Facilitator

Number of Studies Reporting 

an Implementation 

Facilitator

Number of Studies 

Reporting Sustainability 

Positive Influences

General structure/organization* 8 1

Leadership 443,144,164,174

Prior IT initiatives or IT readiness 2138,158

Network membership 1155

Trust and solidarity 1167

Able to innovate and react quickly 1163
74

Table 10. Facilitators to implementation and sustainability of HIE (continued)

Facilitator

Number of Studies Reporting 

an Implementation 

Facilitator

Number of Studies 

Reporting Sustainability 

Positive Influences

HIE-specific structure* 7 3

Participatory approach/user 

engagement/stakeholder buy-in

548,122,124,150,159

Governance 143

HIE lead by Health Information Organization 1171

Community needs assessment 1171

Marketing to patients 1123

Control over technology 1163

Orientation shift* 4

From competitive to collaboration 1154

From ownership of data to continuity of care 1170

To valuing contribution of external information 1170

From pilot to robust system quickly 1163

Design characteristics* 6 3

Information system architecture/adaptability 2161,169

Smaller scale/limited scope 2169,173

Reflect understand of services and work flow 1150

Use of some existing standards while waiting for 

single standards in long term future

1159

Select function likely to have financial benefit 2147,160

Key functions* 5 1

Make use routine/minimize burden and 

time/provide useful data

485,114,116,169

Advance use (decision support; medical home 

functions)

1143 1147

Implementation support* 4

Comprehensive strategy 1168

Extensive testing for data quality assurance 1154

Technical assistance/training/change 

management

2114,167

Expected outcomes* 3 3

Public awareness 1148

Link to community need (public health use) 1146

Tangible intermediate goals 1159

Savings exceed costs 144

Quality of care 1153
75

Table 10. Facilitators to implementation and sustainability of HIE (continued)

Facilitator

Number of Studies Reporting 

an Implementation 

Facilitator

Number of Studies 

Reporting Sustainability 

Positive Influences

External policy* 5 1

Laws and mandates 385,140,159 1140

Federal and State grants 1158

Type of Healthcare System (National, Social 

Insurance, transition)

194

HIE = health information exchange; IT= information technology

*Bold indicates overall category of facilitator.

Barriers

Barriers to HIE implementation cited in the research are not simply the inverse of the 

facilitators. While there is some overlap in the categories, the barriers cited include more 

external, environmental factors. The seven categories of barriers are included in Table 11.

External Policy

This is the one category of barriers that corresponds most directly to a category of 

facilitators. While Federal and State laws and funding and grants were seen as facilitators for 

HIE implementation, changes in Federal policy,164 the fragmented nature of funding (e.g., in 

public health HIE may be funded for some activities and not others),

157 and the uncertainty and 

the timelines for funding were seen as barriers.143,174 One study identified the disconnect between 

State or Federal government goals and local realities as a significant barrier to HIE 

development.166

Disincentives

This is a broad category and the largest, including 20 studies. Four studies reported that 

competition for patients and the difficulty making the business case for HIE are important 

barriers,108,142,151,155 and five additional studies more specifically cited the costs of HIE and the 

lack of financial viability.85,108,141,158,167 In states with mature HIE implementations, where 

presumably the infrastructure was in place, participants cited costs and a lack of understanding of 

the value proposition as the major barrier to participation.141 Three studies identified the fact that 

the organizations that invest in HIE are not always the ones that benefit (e.g., hospitals invest in 

HIE but do not necessarily realize the savings when duplicate tests or admissions are 

avoided).155,158,160 One study cited a trend to set up HIE that supported more administrative tasks 

over clinical tasks as a barrier.94 Two additional studies cited insufficient resources.84,87 In 

addition to financial and resource concerns, five studies identified concerns about data misuse, 

ability to protect privacy, and ethical issues related to sharing data.124,142,148,160,165

Structural Characteristics

This is a category of barriers that includes some parallels in the facilitators—leadership can 

promote HIE, but lack of leadership or effective communication from management can be 

important barriers according to two studies.85,174 While being in a network might facilitate HIE, 

one study concluded that hospitals that are part of larger systems are less likely to participate in 

HIE, perhaps because patients stay in the system and there is less need for external data.149

Another identified barrier is the mismatch between the geographic coverage of the HIE and the 
76

service areas for patients, as would be the case for a hospital with a service area that crosses 

State lines and a State-based HIE.148 Diversity and complexity within and across HIE systems 

were also cited as barriers. One study concluded that the extent of differences made sharing and 

applying lessons learned from one experience to another difficult166 while another stated that 

many types of stakeholders and data result in levels of complexity that can impede 

implementation.165

Technology

The second most frequently cited (13 studies) category of barriers to implementation were 

issues related to technology. More specifically these barriers related to the technological 

environment. Two studies cited the lack of standards or differences in standards across 

organizations in the terms and definitions used in the data as well as the format of data 

sources.87,172 Similarly three studies reported that interoperability across systems was an 

issue,85,142,151 while three more studies specifically mentioned difficulties related to EHR 

interfaces that made exchange difficult or resulted in inappropriate or inaccuracy matching and 

merging.143,154,167 Lack of system resilience, including operating speed and reliability was 

identified in a study of HIE activities in 31 European countries94 while a study in the United 

States cited lack of information system capacity, particularly in smaller organizations. The 

authors of the study in European countries concluded, “we can pinpoint some clear bottlenecks 

in terms of ‘electronically embedded’ system inter-connection with other healthcare players, 

technical inter-operability, system resilience, and security.[...].Limited adoption of Health 

Information Exchange (HIE) is surely also a consequence of such bottlenecks.”94 One study was 

less circumspect in citing problems with vendors and reporting that, “the most significant 

barriers ... were largely due to a long and arduous process of collaborating with commercial 

entities involved in technology design and delivery.”48

Lack of Necessary Components

This was presented as a barrier in five studies. Four studies reported that participants or 

providers were not sufficiently engaged in implementation of the HIE or were not aware of its 

value.84,141,154,158 One study emphasized that physician engagement was important by pointing 

out that physicians are the primary source of care data and suggested that for this reason their 

engagement is the primary determinant of HIE success.154 One study focused on the challenges 

in securing data sharing agreements as a barrier to implementation.143

Fit

This is short hand for the correspondence between an innovation and the potential adopting 

organizations. Lack of fit is a barrier that may not be apparent when the innovation is assessed 

out of context. Two studies found that HIE implementation was deterred when organizations or 

departments were unable or unwilling to integrate HIE into work processes.152,167 Another 

instance where lack of fit is problematic is when expectations are not met. Two studies reported 

that expectation for the data in terms of timeliness and completeness were barriers to 

implementation.100,145 One additional study underlined the fact that timelines were not realistic,

particularly in cases where the technology was to be integrated into quality improvement 

activites.143
77

User Interface and Functionality

Eight studies cited specific user interface and functionality problems as barriers to 

implementation. These included lacking the technology and human resources needed to adapt the 

organization’s software and processes for HIE,141 and the need for training and expertise.142,174

Two studies reported that user problems as fundamental as forgotten logons145 and the technical 

performance of network connections hindered implementation.116 One study reported corrupt 

data as a barrier to HIE,172 while another reported the lack of tests that identify that the ability to 

match patients across systems were a barrier to development.82 One study of an advanced 

application of a system to generate alerts based on HIE data stalled when the providers to notify 

about a patient’s events could not be identified.100

Table 11. Barriers to implementation and sustainability of HIE

Barrier

Number of Studies 

Reporting Implementation

Barriers

Number of Studies 

Reporting Sustainability 

Negative Influences

External policy* 3 1

Laws and regulations 1162

Changes in external (Federal, State) policy 1164

Funding uncertainty and timelines 2143,174

Disincentives* 15 4

Competition/difficult business case 4108,142,151,155 4148,149,156,173

Costs/financial viability 579,85,141,158,167

Organizations that invests does not benefit 3155,158,160

Resources (funding and time) 284,87

Concerns about data misuse, privacy, or ethics 4124,142,148,160

Structure* 4 3

Geographic coverage mismatch with service areas 1148 1156

Lack of leadership and management communication 285,174

Larger hospital systems (less need for external 

exchange)

1149

Focus on long term care 1171

Governance/trust 2153,156

Technology* 9 1

Lack or differences in standards 287,172 1162

EHR interface 3143,154,167

Interoperability across systems 385,142,151

Problems with vendors 148

Lack of necessary components* 5 1

Participant/provider engagement, awareness of value 484,141,154,158 1146

Securing data sharing agreements 1143
78

Table 11. Barriers to implementation and sustainability of HIE (continued)

Barrier

Number of Studies 

Reporting Implementation

Barriers

Number of Studies 

Reporting Sustainability 

Negative Influences

Fit* 5

Inability or willingness to integrate into work 

processes

2152,167

Lack of enough time for development and integration 

into Quality Improvement

1143

Failure to meet expectations that data needs will be 

timely, complete and meet expectations. 

2100,145

User interface and functionality* 8

Tech and HR resources to adapt software and 

processes

1141

Need for training and expertise 2142,174

Corrupt data 1172

User interface and technical performance 2116,145

Ability to match patients 182

Difficulty identifying provider to get alerts generated 

from HIE

1100

EHR = electronic health record; HIE = health information exchange; HR = human resources; IT = information technology

*Bold indicates overall category of barrier.

Subgroup Differences

During our review we attempted to abstract data from the included studies that would allow 

us to determine if the barriers and facilitators to implementation varied by type of HIE, health 

care settings, and systems or IT system characteristics. Most publications did not include this 

information so we were not able to consistently identify any differences.

We also considered that implementation might change over time as HIE becomes more 

common and as new HIE efforts could benefit from the experience of early adapters. At this time 

we do not see any notable changes, but this may be to the relatively short time period (less than a 

decade) covered by the included studies. While the hardware and software that make HIE 

possible have changed significantly in less than a decade, organizational change and clinical 

practice patterns have historically changed more slowly. 

Sustainability

In making a distinction and summarizing the factors identified in the 17 studies that 

considered sustainability separately, we placed studies according to what the researchers/authors 

reported as their focus and we accepted their definitions and/or measures.44,108,123,140,146-

149,153,156,159,160,162,163,166,171,173 As HIE and health IT mature, a definition of successful 

sustainability may be developed and the evidence could them be reanalyzed incorporating such a 

definition. 

The factors that have been found to influence the sustainability of HIE fit into the categories 

created to summarize the facilitators and barrier for implementation, and in some cases it can be 

difficult to make a distinction. This is in part because sustainability is still a future goal rather for 

all but the organizations that were very early adopters of HIE. 
79

We presented the sustainability factors under the most appropriate category on Tables 10 and 

11, but added rows for specific factors when they differ from those identified in studies of 

implementation.

Ten included studies identified factors that are positive influences on sustainability. These 

included having an HIE implementation led by a health information organization as opposed to a 

health care organization171 and having leadership and technology that allowed the HIE 

organization to innovate and react quickly to changes in the market and environment.163

Sustainability was also linked to marketing the HIE to patients,123 to how an HIE system

incorporated a community needs assessment,

171 and if it selected functions likely to financially 

benefit the participants.147,160 One study suggested that HIE implementations with advanced 

functions such as providing decision support are more sustainable147 while another pointed out 

that these functions should add value related to either Stage 2 meaningful use or reform priorities

in order to support sustainability.159 Achieving important expected outcomes such as improved 

quality of care153 and realizing savings that exceed the costs of the HIE system are 

understandably important44 and one study described how most of the HIE organizations it 

examined are developing subscription fee structures to provide ongoing financial support.159 One 

study reported that laws and mandates could promote sustainability as well as implementation of 

HIE. 140

However, laws and mandates, particularly changes in these were also one of the reported 

negative influences on sustainability.162,166 Four studies found that competition and a difficult 

business case for HIE were challenges to sustainability.148,149,156,173 Four structural characteristics 

of HIE were also identified. These included the mismatch between the HIE geographic coverage 

and where patients receive services,156 issues related to governance and trust among the HIE 

collaborators,153,156 and one study found that HIE that focused on long-term care organizations 

were less likely to be sustainable.171 Lack of standards was the only factor directly related to the 

technology for HIE reported among the negative influences and it was reported in only one 

study.162 Lack of sufficient engagement of participants and providers was also reported in one 

study.146

While there was less evidence related to sustainability to report in this review than for 

implementation, the studies to date suggest it is the more complex of two very complex and 

related topics. One researcher suggested this complexity when making the assessment that this 

issue for HIE sustainability are sociological not technological.156 Another suggested 

sustainability may become less a matter of availability of funds and more one of trust and 

responsible stewardship.123 Combined, this result seems to be that sustainability of HIE activities 

is further in the future than many originally thought. As one observer noted “recent history 

suggests that achieving the kind of ubiquitous use among providers or other users that can drive a 

financial value proposition takes time—and likely more time than HIOs have modeled in their 

sustainability plans.”163
80

Discussion

Key Findings

• We found no studies of health information exchange (HIE) that reported the impact on 

clinical outcomes or that identified harms.

• The majority of the included studies reported that HIE improved resource use by 

reducing lab tests, imaging, or hospital admissions and improved quality of care, but the 

strength of evidence was low for all outcomes.

• Studies found that HIE was used by between 30 and 58 percent of hospitals and, 38 

percent of office-based physicians in 2012, while use remains low in long-term care 

settings. 

• Within organizations, studies that looked at the number of users or the number of visits in 

which the HIE is used found generally very low rates of use.

• Studies did not link usability of HIE to effectiveness but they did link it to use. 

• The most commonly cited barriers to HIE use were incomplete patient information, 

inefficient workflow, and poorly designed interface and update features.

• Eight categories of factors facilitated HIE: seven cateogires that are internal 

characteristics while external factors were less frequently cited and we combined these 

into one category.

• Barriers identified in research on HIE implementation focused more on the external 

environment (7 categories). Disincentives was the largest category of barriers.

• Factors that influenced sustainability were similiar to the barriers and facilitiators of 

implementation. The most frequently cited negative influence was competition and the 

lack of a business case for HIE.

Key findings are summarized in Table 12.

Table12. Summary of evidence

Topic

Number of Included 

Studies

Type

Main Findings Primary Limitations of the 

Evidence

Effectiveness 34;

20 Retrospective cohort

3 Randomized controlled 

trial

2 Cross-sectional

2 Case series

8 Survey (1 survey study 

was an RCT)

Low-quality evidence 

somewhat supports the 

value of HIE for reducing 

duplicative laboratory and 

radiology test ordering, 

lowering ED costs, reducing 

hospital admissions (less so 

for readmissions), improving 

public health reporting, 

increasing ambulatory 

quality of care, and 

improving disability claims 

processing. No evidence of

harms was reported.

Studies were from a small number 

of the functioning HIE

implementations, with similarity to 

unstudied ones unknown, possibly 

limiting generalizability.

Studies looked at limited 

outcomes compared with the 

intended scope of the impact of 

HIE.
81

Table12. Summary of evidence (continued)

Topic

Number of Included 

Studies

Type

Main Findings Primary Limitations of the 

Evidence

Use 58;

25 Surveys

13 Audit Logs

9 Retrospective database

7 Mixed methods

2 Focus Groups 

1 Time-motion

1 Geo-Coding

Proportion of hospitals and 

ambulatory care practices 

that have adopted HIE is 

increasing.

Currently, proportion of 

clinicians using HIE and 

proportion of patients or 

episodes associated with 

HIE use are generally low.

While there are relatively high 

quality national and regional 

surveys and reports that are 

tracking the expansion of HIE 

among health care organizations, 

there is not a corresponding 

comprehensive effort to track 

changes in rates of use within 

organizations.

Usability and 

other factors 

affecting use 

22;

9 Multiple site case studies

11 Cross-sectional

2 Before-after

3 most commonly cited 

barriers to HIE use were:

incomplete patient 

information (8 studies); 

inefficient workflow (6 

studies); poorly designed 

interface and update 

features (6 studies).

Studies of usability did not relate it 

to effectiveness and do not permit 

comparisons across settings or 

type of HIE.

Studies had limitations such as 

incomplete reporting on sampling, 

low response rates or selection of 

a narrow setting or patient 

population which minimize 

applicability.

Implementation 

and 

sustainability 

52;

26 Cross-sectional

17 Multiple site case studies

2 Before-after

3 Retrospective cohorts

2 Prospective cohorts

2 Time series

Most facilitators of 

implementation are 

characteristics of the HIE or 

the internal organizational 

environment. Many barriers 

to implementation are 

external, environmental 

factors.

Factors related to 

sustainability overlap with 

those identified for 

implementation. 

Studies do not allow comparison 

of the impact of different barrier 

and facilitators.

The definition and appropriate 

measure of sustainability are not 

yet clear.

ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial

Strength of Evidence

Assessing the overall strength of the evidence for this review was complex, given (1) the 

very broad scope of the review; (2) the large variety of effects and outcomes examined by 

investigators; (3) the diverse types of evidence and study designs; (4) the differing units of 

analysis and intervention (from episodes of care, to individual clinicians or patients, to hospitals 

or clinics, to health systems, to regional or statewide efforts); (5) the multiple contexts of care,

from acute care in emergency department (ED) visits to public health reporting and analysis; (6) 

the variety of technical implementations, even within the broad categories of query-based and 

directed HIE; and (7) the likelihood of reporting bias, expected to be in the direction of positive 

findings, with likely under-reporting of failed or ineffective HIE. In view of these challenges, we 

elected to explicitly and systematically assess the risk of bias and strength of evidence only for 

studies addressing the effectiveness and harms of HIE, our Key Questions 1, 2, and 3.
82

These limitations notwithstanding, a collection of low-quality evidence somewhat supports 

the value of HIE for reducing duplicative laboratory and radiology test ordering, lowering ED

costs, reducing hospital admissions (less so for readmissions), improving public health reporting, 

increasing ambulatory quality of care, and improving disability claims processing. The evidence 

is low-quality because of the retrospective nature of the studies and the limited questions that 

they ask. It is unlikely that additional studies of the kind included in this review will alter the 

overall conclusion that HIE can reduce laboratory and imaging tests associated with episodes of 

care without broadening their scope and using more rigorous designs. Though the preponderance 

of evidence supports positive effects in terms of reduced resource use and improved quality of 

care, it is entirely possible that focused studies with stronger study designs and more 

comprehensive assessment of utilization or clinical outcomes might reach a different conclusion. 

With respect to cost, we did not identify any studies that employed systematic and 

comprehensive economic analysis. Although some of the studies we included projected or 

estimated cost savings based on measured changes in utilization or perceptions of clinicians, 

there were no studies that explicitly measured costs and assessed economic impact in a 

comprehensive fashion. It is fair to say, then, that there was insufficient evidence to reach 

conclusions on the economic impact of HIE.

As stated previously, we found no studies explicitly addressing patient-specific clinical 

outcomes such as morbidity, mortality, or functional status and hence the body of evidence is 

insufficient to determine whether HIE has an impact on patient outcomes.

Findings in Relationship to What Is Already Known

The findings of this review add to the substantial, albeit methodologically challenging, 

evidence base relating to health information technology (IT) generally and HIE in particular. A 

series of comprehensive and systematic reviews of health IT have been published over the last 

decade, including three from a single Evidence-based Practice Center (EPC)5,6,8and one from the 

Office of the National Coordinator7 confronted similar challenges in the diversity and breadth of 

settings, interventions, and outcomes. Overall, these reviews found that the preponderance of 

studies of health IT reported generally positive or “mixed-positive” effects, but with caveats 

about the likelihood of publication bias, methodological limitations of the studies, and the 

concentration of studies coming from a relatively small number of institutions.

The present systematic review of HIE can be compared with two other systematic reviews of 

HIE: one by Rudin et al.32 and another by Rahurkar et al.33 The three systematic reviews used 

generally similar approaches, with similar definitions of HIE and focus on studies of HIE impact, 

excluding system descriptions and simple case studies. The three reviews differ, however, in 

their scope and inclusiveness.

The review by Rahurkar et al. was most narrow in scope, addressing only the impact of HIE 

on “health outcomes,” in which the authors included utilization and cost measures. They 

searched two databases, Scopus and MEDLINE (along with reference mining), included nonU.S. studies, and excluded systematic reviews, qualitative studies, and studies of exchange of 

administrative and financial information.33

The review by Rudin et al. was broader.32 In addition to health and utilization outcomes of 

HIE, they considered studies of patient and provider attitudes, barriers and facilitators to HIE 

use, and financial sustainability. These authors searched three databases, MEDLINE, Web of 

Science, and the Cochrane databases (along with reference mining), and they excluded studies of 
83

public health settings (included by Rahurkar et al.), administrative and financial information 

exchange, non-U.S. studies, and studies of usability.

Our review was the most broad in scope of the three, and the most inclusive in the search for 

evidence. In addition to patient and population health outcomes, economic, utilization process 

outcomes, and barriers and facilitators to implementation and use, our review also included 

studies concerned with use and usability of HIE. We also explicitly searched for reports of harms 

of HIE (although none were found). Our review was also more comprehensive in the search for 

evidence, searching MEDLINE, PsychInfo, CINAHL, the Cochrane databases, Database of 

Abstracts of Reviews of Effects, and the National Health Sciences Economic Evaluation 

Database, as well as reference mining. We also did trial scans of the Business Premier and the 

Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library; databases for any 

potential relevant evidence. In addition, we included non-U.S. studies, and studies that reported 

on public health and surveillance uses as well as exchange of administrative and financial 

information.

The three reviews are based on comparable but not identical evidence bases. The present 

review includes a total of 136 studies. The review by Rudin et al. included 85 studies, 55 of 

which were also included in our review, and the review by Rahurkar et al. included 27 studies, 

18 of which were also included in our review. We examined the references of both of these 

reviews and included any that met our inclusion criteria. 

The overall result is that we examined a more diverse and more inclusive collection of 

evidence, especially with respect to usability and use as well as assessing public health settings, 

but came to largely similar conclusions. Rahurkar et al. performed a multivariable analysis that 

found that study design was the only characteristic associated with finding a beneficial effect, 

with the most rigorous studies being less likely to report benefits of HIE.

The problem of overlap across systematic reviews is an important one and has recently been 

addressed in the methods guides of the Cochrane Collaboration175-177and the Agency for 

Healthcare Research and Quality EPC program.178 When large numbers of systematic reviews 

are conducted, there is inevitable overlap when two reviews are based on the same body of 

evidence. Additional reviews on a subject do not indicate more evidence on the question, only 

more thorough (when independent) examination of the same evidence.

A notable point to be made about the comparison between these reviews is that three review 

groups have now independently searched for and assessed the evidence on the effectiveness of 

HIE and are in agreement on the main conclusions. This raises the level of confidence in the 

conclusions in that the three reviews represent independent replication of one another’s work, 

albeit with the same rather significant limitations in the body of evidence on which the 

conclusions are based.

Applicability

Are the effects reported on in this review, limited as they are, likely to be observed when 

applied under “real world” conditions in health systems, hospitals, and clinics in the United 

States? The greatest confidence in the applicability of these findings comes from the breadth of 

settings – geographic, organizational, and technical – from which they are derived. That is to say, 

for the most part, it can be expected that: (1) near-term resource utilization in the form of 

laboratory and imaging test ordering is likely to be reduced when effective HIE capabilities are 

deployed, while the effect on other utilization and quality indicators is harder to predict; (2) use 

of HIE will be highly dependent on the context of use, perceived value of the information to the 
84

patient care task, and the degree of integration into clinical workflows, including potential 

delegation by clinicians to other members of the health care team depending on the setting; and 

(3) hospital and health system implementation and participation in HIE will be driven by the 

perceived value and return on investment, alignment with organizational goals, internal capacity 

to address technical challenges, and the presence of local and national external financial, 

regulatory, and policy constraints. 

On the other hand, there are limitations to the applicability of the findings (beyond 

limitations to the internal validity already mentioned) having to do with three main concerns: (1) 

concentration of evidence from a relatively small number of sources; (2) use of internally 

developed and refined health IT systems compared with local instances of commercial systems; 

and (3) the exceptionally broad variety of systems, contexts, and purposes of HIE reported in the 

studies included in this review.

First, the concern that the bulk of the evidence about health IT impact arises out of a 

relatively small number of centers has been raised before.5 These centers have been referred to as

“health IT leaders,” which are typically large academic medical centers with internally 

developed health IT systems, implemented incrementally, and refined over a long period of time.

The nature of the health IT systems is in each case unique (being locally developed), and more 

importantly it is difficult to separate the effects of the health IT from the confounding influences 

of the health system itself. Whether findings from these systems can be generalized to the very 

different context of health system and hospital implementations of commercially developed 

systems over shorter periods of time with less internal development and implementation 

infrastructure has been called into question.5 This “health IT leader” effect appears to be reduced 

in more recent updates to the 2006 systematic review by Chaudhry et al. but the issue remains 

important.

6,8 In the present review of HIE the concentration of evidence phenomenon is also 

present, with large numbers of published studies emanating from relatively few areas, this time 

regional implementation programs rather than academic health centers, such as Texas, New 

York, and the MidSouth e-Health Alliance.

Second, separate from the “health IT leader” concern, which has to do with the 

organizational capacity, resources, and mission of these centers, is the issue of internally 

developed systems compared with commercially developed systems. Though no implementation 

is truly “off the shelf” because of customization of local instances of commercial systems, the 

overall model of health IT purchase and installation is quite different from that of incremental 

internal development, implementation, and refinement, such as one sees in systems such as the 

Veterans Affairs or the aforementioned “health IT leader” systems. Related to this concern is a 

finding from other aspects of health IT,176 namely clinical decision support, that systems 

evaluated by their developers tend to achieve more positive outcomes from their evaluation than 

external evaluators. This phenomenon must be assessed with HIE as well.

Third and most important in terms of limiting the applicability of these findings about HIE to 

real-world use is the exceptionally wide variety of systems, purposes, contexts of use, and 

outcomes examined. To address the Key Questions of this systematic review, highly diverse 

evidence has been combined to answer general questions about the overall effectiveness of HIE 

for various outcomes. However, to predict whether specific implementations of HIE in specific 

health care contexts will have favorable impacts on specific desired outcomes is not possible 

from this review and in most cases would not be possible from comparison with individual 

studies because (a) it is unlikely that studies with low risk of bias have been published for most 
85

such specific questions, and (b) in almost all cases these are complex interventions which are 

incompletely specified, with insufficient detail to draw strong meaningful inferences.

179

Limitations of the Evidence Base

The very significant limitations of the evidence base, that is, the individual studies included 

in this review, have been raised in previous systematic reviews of health IT5,6,8 and of HIE.

32

Although increasing in number, the relative proportion of well-conducted studies with rigorous 

designs remains small, and we know from experience in other domains, such as hormone 

replacement therapy, that even a very large number of well-conducted observational studies may 

be found to have misled us when results of rigorous experiments become available.

180 In view of 

this fact, one must continue to proceed with caution when interpreting and applying the results of 

observational studies, even well-conducted ones.

Beyond this, there are three primary concerns about the limitations of the available evidence 

on the impact of HIE (and health IT in general): (1) suitability of study design; (2) execution of 

the studies; and (3) complexity of the interventions with implications for interpretation and for 

generalizability.

First, the evidence in this area addresses a wide variety of questions covering diverse 

domains beyond medical science from computer science, human factors, sociology, organization 

and management and other disciplines. This broad array of questions calls for an equally diverse 

range of study designs. Studies of usability and use require usability engineering methods, 

studies of individual behavior call for methods from anthropology and behavioral sciences, 

studies of organizational change warrant methods drawn from management and systems science, 

while studies of population effects call for the methods of epidemiologists. As Sackett and 

Wennberg noted, “the question being asked determines the appropriate research architecture, 

strategy, and tactics to be used—not tradition, authority, experts, paradigms, or schools of 

thought.”181 A significant limitation of this literature, with its breadth of research questions, is 

the limited toolbox often drawn upon to answer them.

The second main area of limitation is in execution of the studies. Even when strong study 

designs are chosen, their execution may be lacking, whether in sampling strategies, measurement 

methods, or analytic approaches. The unit of analysis problem is but one example. Interventions 

carried out at the level of the health system, hospital, or clinic may be analyzed at the level of the 

patient or episode, without controlling for variation at these multiple levels. Incomplete 

measurement is another: for example where ED test ordering is measured in isolation, ignoring 

the possiblility that the same test might later be ordered in another setting such as urgent care, 

primary care, or in hospital.

The third main area has to do with the complexity of interventions, where the HIE or other 

health IT system itself is necessarily only part of a more complex intervention. The complexity 

of interventions to change the behavior of clinicians or others in the health systems studied 

requires more thorough specification, both in order to adjust for confounders and in order to 

make sense out of how to apply interventions elsewhere. Others have documented the 

inadequacy of specification of the details of complex interventions and called for a more 

systematic and thorough reporting.

179,182

Future Research Needs

Given the limited conclusions that can be reached after review of so much published 

literature on the effects, use, sustainability, and barriers to implementation and use of HIE, what 
86

are the implications for future research? Recognizing that HIE, like health IT in general, will 

almost certainly undergo increasingly widespread implementation in the future, the first aim of 

researchers should be to shift the emphasis from whether HIE systems should be implemented to 

specifically how they should be implemented. The quesion to be answered is not “Does HIE have

positive effects?“ but rather “How can HIE be implemented in order to result in the greatest 

benefit for patients, clinicians, and health systems with the least cost and harm?”

The second aim of researchers on HIE should be to develop greater focus and clarity about 

the level at which interventions are operating and the types and levels at which outcomes are 

measured. The outcomes of interest and the factors influencing them may be quite different at 

different levels of analysis, from specific systems or functionalities of HIE; to individual 

patients, providers, or episodes of care; to health care units such as the ED, primary care practice, 

or hospital ward; to institutions such as hospitals; to aggregates such as health systems; or 

broader regional multi-organization entities or regions. Combining or confusing these levels of 

intervention and levels of analysis only increase the challenges for those who conduct the 

research and for those who wish to interpret and apply it. 

To help achieve an improved focus and clarity, a more formal analytic framework and a more 

descriptive taxonomy are needed. An example of such a framework that could be usefully 

applied in this area is Rasmussen’s socio-technical hierarchy, which specifies the multiple levels 

of a complex sociotechnical system that must be considered together to understand system 

behavior change.183 Examples of its application include Vicente’s analysis of the forces acting at 

multiple levels (Figure 3) to reduce hazards arising from patient controlled analgesia devices184

and Leveson’s Systems—Theoretic Accident Modeling and Processes (STAMP) model for 

understanding system performance and safety.185
87

Figure 3. Rasmussen sociotechnical analysis framework* 184

*© Joint Commission Resources: Joint Commission Journal on Quality and Patient Safety. Volume 29: Issue 11. p.599, (2003). 

Reprinted with permission.

Recognizing that one cannot understand a system by separately analyzing its parts, Rasmussen developed this analytical 

framework that encompasses the full range of dimensions that must be considered together to make sense of socio-technical 

behavior.

Similarly, a formal taxonomy for implementation of complex interventions has been 

proposed which would enable more complete and useful specification of interventions to allow 

better analysis, interpretation, and application.

179,187 This taxonomy should be extended specific 

to HIE to include clinical, technical, and organizational details of the HIE implementation. The 

clinical taxonomy should focus not only on patient outcomes, but also on issues such as health 

disparities related to HIE and health system issues that may improve or undermine use of HIE. 

The technical taxonomy should include aspects of system architecture, messaging and 

terminology standards, and other details. The HIE research community should consider a 

standardized reporting instrument for HIE evaluation comparable to the Consolidated Standards 

of Reporting Trials (CONSORT) statement for randomized controlled trials (RCTs).

188

The third step researchers can take to improve the evidence base for implementation of HIE 

is to broaden the methodologic toolbox applied to these questions. As indicated above, the study 

approach and architecture must be suited to the question being asked, employing methods from 

usability engineering, behavioral sciences, systems engineering, and organizational sciences, 
88

depending on the question being addressed. These would include methods used in engineering 

and quality improvement, as well as in the study of complex adaptive systems. In epidemiology 

it has been proposed that health and health care can be fruitfully studied as complex adaptive 

systems, which require “different methods from the usual epidemiological techniques.”189

Examples include infectious disease epidemiology, smoking,

189 and obesity.190 Because “(i) 

factors at multiple levels, including biological, behavioural and group levels may influence 

health and disease, and (ii) ... the interrelation among these factors often includes dynamic 

feedback and changes over time,” new approaches are needed to complement the classic methods 

of clinical trials which are frequently unsuitable for complex interventions in organizational 

contexts. 

What types of studies should be performed? RCTs are impractical for technologies with 

wide-ranging purposes like HIE. Yet, retrospective studies associating HIE versus non-use for 

outcomes such as test ordering and hospital admissions are very limited in conclusions that can 

be drawn. Research is also challenging because many of the important clinical outcomes that 

could benefit from HIE have many other potential contributing and confounding factors relating 

to the patient, his or her clinicians, the quality of care delivered, the electronic health record, and 

other health IT used, the nature of the health care delivery system, the regulatory environment, 

and many more.

Future studies should be prospective, carried out in mature HIE settings, assessing patients 

who are likely to benefit from HIE and comparing appropriate outcomes for the use or non-use 

of HIE. The prospective collection of data from diverse settings where HIE is used, classified by 

the taxonomy advocated above, could allow for prospective cohort studies that could identify 

aspects of HIE associated with beneficial outcomes. This will likely require an effort comparable 

in scope to national data collection efforts, such as the Patient-Centered Outcomes Research 

Institute Clinical Data Research Network initiative.191 Ideally such an undertaking could be 

synergistic with these other large-scale efforts.

Evaluation should be a requirement for all HIE implementations, certainly those funded by 

grants or other external funding. The challenges of evaluating health IT projects, especially in 

community settings, is well-known,30 but all funders must demand this requirement to grow the 

evidence base. By the same token, funders must provide adequate resources for such evaluations. 

In addition, evaluation should be performed by researchers external to the project to reduce 

potential bias from system developers evaluating their own implementations.176

Conclusions

The full impact of HIE on clinical outcomes and potential harms is insufficiently studied, 

although evidence provides some support for benefit in reducing use of some specific resources 

and achieving improvements in quality of care measures. Use of HIE has increased over time and 

is highest in hospitals and lowest in long-term care settings. However, use of HIE within 

organizations that offer it is still low. Barriers to HIE use include incomplete patient information, 

inefficient workflow, and poorly designed interface and update features, but factors affecting 

implementation and sustainability remain unclear. To advance our understanding of HIE, future 

studies need to address comprehensive questions, use more rigorous designs, and be part of a 

coordinated, systematic approach to studying HIE.
89

References

1. Crossing the Quality Chasm: A New Health 

System for the 21st Century. Washington, 

DC: National Academies Press; 2001.

2. Dustin Charles, Meghan Gabriel, Furukawa 

M. Adoption of Electronic Health Record 

Systems among U.S. Non-federal Acute 

Care Hospitals: 2008-2013. ONC Data 

Brief, no. 16. The Office of the National 

Coordinator for Health Information 

Technology. Washington, DC. Available at: 

www.healthit.gov/sites/default/files/oncdata

brief16.pdf. Accessed March 2, 2014. 

3. Hsiao C-J, Hing E. Use and Characteristics 

of Electronic Health Record Systems 

Among Office-based Physician Practices: 

United States, 2001–2013. NCHS Data 

Brief, No. 143. National Center for Health 

Statistics. Hyattsville, MD. Available at: 

www.cdc.gov/nchs/data/databriefs/db143.ht

m. Accessed March 3, 2015. PMID: 

24439138.

4. Eric Jamoom, Esther Hing. Progress With 

Electronic Health Record Adoption Among 

Emergency and Outpatient Departments: 

United States, 2006–2011. NCHS Data 

Brief, No. 187. National Center for Health 

Statistics. Hyattsville, MD. Available at: 

www.cdc.gov/nchs/data/databriefs/db187.ht

m. Accessed March 3, 2015. PMID: 

25714041.

5. Chaudhry B, Wang J, Wu S, et al. 

Systematic review: impact of health 

information technology on quality, 

efficiency, and costs of medical care. Ann 

Intern Med. 2006;144(10):742-52. PMID:

16702590.

6. Goldzweig CL, Towfigh A, Maglione M, et 

al. Costs and benefits of health information 

technology: new trends from the literature. 

Health Aff (Millwood). 2009;28(2):w282-

93. PMID: 19174390.

7. Buntin MB, Burke MF, Hoaglin MC, et al. 

The benefits of health information 

technology: a review of the recent literature 

shows predominantly positive results. 

Health Aff (Millwood). 2011;30(3):464-71. 

PMID: 21383365.

8. Jones SS, Rudin RS, Perry T, et al. Health 

information technology: an updated 

systematic review with a focus on 

meaningful use. Ann Intern Med. 

2014;160(1):48-54. PMID: 24573664.

9. Bourgeois FC, Olson KL, Mandl KD. 

Patients treated at multiple acute health care 

facilities: quantifying information 

fragmentation. Arch Intern Med. 

2010;170(22):1989-95. PMID: 21149756.

10. Finnell JT, Overhage JM, Grannis S. All 

health care is not local: an evaluation of the 

distribution of emergency department care 

delivered in Indiana. AMIA Annu Symp 

Proc. 2011;2011:409-16. PMID: 22195094.

11. Clancy C. Keynote Adress AMIA 2007 

Anual Symposium. 2007. 

12. What is HIE (Health Information 

Exchange)? Department of Health and 

Human Services. Washington, DC. 

Available at: www.healthit.gov/providersprofessionals/health-informationexchange/what-hie. Accessed April 18, 

2014. 

13. Williams C, Mostashari F, Mertz K, et al. 

From the Office of the National 

Coordinator: the strategy for advancing the 

exchange of health information.[Erratum 

appears in Health Aff (Millwood). 2012 

Mar;31(3):886]. Health Aff. 

2012;31(3):527-36. PMID: 22392663.

14. Health Information Exchange Roadmap: 

The Landscape and a Path Forward. 

National eHealth Collaborative. 

Washington, DC. Available at: 

www.riqi.org/matriarch/documents/NeHC%

20Roadmap%20for%20HIE%20-

%20The%20Landscape%20and%20a%20Pa

th%20Forward.pdf. Accessed June 17, 2015. 

15. Dullabh P, Ubri P, Loganathan S, et al. 

Evaluation of the State Health Information 

Exchange Cooperative Agreement Program: 

State Approaches to Enabling HIE: 

Typology Brief. Office of the National 

Coordinator for Health Information 

Technology. 2014. Available at: 

www.healthit.gov/sites/default/files/statehiet

ypologybrief.pdf. Accessed April 10, 2015. 
90

16. Glasgow A. The nouns and verbs of 

successful HIE. Healthcare IT News. 

Available at: www.healthcareitnews.com/

news/nouns-and-verbs-successful-hie. 

Accessed April 19, 2015. 

17. Mary Mosquera. HIE as a verb: ONC want 

to move quickly on data exchange. 

Healthcare IT News. Available at: 

www.healthcareitnews.com/news/hie-verbonc-wants-move-quickly-data-exchange. 

Accessed April 19, 2015. 

18. McDonald CJ. The barriers to electronic 

medical record systems and how to 

overcome them. J Am Med Inform Assoc. 

1997;4(3):213-21. PMID: 9147340.

19. McDonald C, Overhage J, Barnes M, et al. 

The Indiana Network for Patient Care: a 

working local health information 

infrastructure. Health Aff. 2005;24:1214-20. 

PMID: 16162565.

20. Frohlich J, Karp S, Smith MD, et al. 

Retrospective: lessons learned from the 

Santa Barbara project and their implications 

for health information exchange. Health Aff 

(Millwood). 2007;26(5):w589-91. PMID: 

17670777.

21. Kuperman GJ. Health-information 

exchange: why are we doing it, and what are 

we doing? J Am Med Inform Assoc. 

2011;18(5):678-82. PMID: 21676940.

22. McGowan J, Kuperman G, Olinger L, et al. 

Strengthening Health Information 

Exchange: Final Report HIE Unintended 

Consequences Work Group. Rockville, 

MD: Westat. 2012. Available at: 

www.healthit.gov/sites/default/files/hie_uc_

workgroup_final_report.pdf. Accessed 

January 7, 2014. 

23. Yasnoff WA, Sweeney L, Shortliffe EH. 

Putting health IT on the path to success. 

JAMA. 2013;309(10):989-90. PMID: 

23483170.

24. Adler-Milstein J, Jha AK. Sharing clinical 

data electronically: a critical challenge for 

fixing the health care system. JAMA. 

2012;307(16):1695-6. PMID: 22535851.

25. Adler-Milstein J, Bates DW, Jha AK. 

Operational health information exchanges 

show substantial growth, but long-term 

funding remains a concern. Health Aff 

(Millwood). 2013;32(8):1486-92. PMID: 

23840051.

26. Swain M, Charles D, Patel V, et al. Health 

Information Exchange among U.S. Nonfederal Acute Care Hospitals: 2008-2014. 

ONC Data Brief No. 24. Washington DC: 

The Office of the national Coordinator for 

Health Information Technology (ONC). 

2015. Available at: www.healthit.gov/sites/

default/files/data-brief/ONC_DataBrief24_

HIE_Final.pdf. Accessed April 19, 2015. 

27. The Office of the National Coordinator for 

Health Information Technology (ONC). 

Federal Health IT Strategic Plan 2015-2020. 

2014. Available at: 

www.healthit.gov/sites/default/files/federalhealthIT-strategic-plan-2014.pdf. Accessed 

April 23, 2015.

28. The Office of the National Coordinator for 

Health Information Technology (ONC). 

Connecting Health Care for the Nation. A 

shared nationwide Interoperability 

Roadmap. Draft Version 1.0. 2015. 

Available at: /www.healthit.gov/sites/

default/files/nationwide-interoperabilityroadmap-draft-version-1.0.pdf. Accessed 

April 23, 2015.

29. The Office of the National Coordinator for 

Health Information Technology (ONC). 

Report on Health Information Blocking. 

Washington DC: Department of Health and 

Human Services. 2015. Available at: 

www.healthit.gov/sites/default/files/reports/i

nfo_blocking_040915.pdf. Accessed April 

17, 2015. 

30. Kern LM, Ancker JS, Abramson E, et al. 

Evaluating health information technology in 

community-based settings: lessons learned. J 

Am Med Inform Assoc. 2011;18(6):749-53. 

PMID: 21807649.

31. Hincapie A, Warholak T. The impact of 

health information exchange on health 

outcomes. Appl Clin Inform. 2011;2(4):499-

507. PMID: 23616891.
91

32. Rudin RS, Motala A, Goldzweig CL, et al. 

Usage and effect of health information 

exchange: a systematic review. Ann Intern 

Med. 2014;161(11):803-11. PMID: 

25437408.

33. Rahurkar S, Vest JR, Menachemi N. Despite 

the spread of health information exchange, 

there is little evidence of its impact on cost, 

use, and quality of care. Health Aff 

(Millwood). 2015;34(3):477-83. PMID: 

25732499.

34. Yi R, Samarth A, Dearfield C, et al. 

Synthesis of Lessons Learned in the First 5 

Years of State and Regional Demonstration 

Health Information Exchange Projects. 

AHRQ Publication No. 11-0050-EF. 

Rockville, MD: Agency for Healthcare 

Quality and Research. March, 2011. 

Available at: http://healthit.ahrq.gov/sites/

default/files/docs/citation/synthesis_of_lesso

ns_learned.pdf. Accessed April 18, 2014. 

35. Pan E BC, Damico D, Crimmins M. Guide 

to Evaluating Health Information Exchange 

Projects (Prepared for the Agency for 

Healthcare Research and Quality under 

Contract No.290200900023-I.). AHRQ 

Publication No. 14-0015-EF. Rockville, 

MD: Agency for Healthcare Research and 

Quality. September 2014. Available at: 

http://healthit.ahrq.gov/sites/default/files/doc

s/page/guide-to-evaluating-hie-projects.pdf. 

Accessed April 23, 2015.

36. Methods Guide for Effectiveness and 

Comparative Effectiveness Reviews. AHRQ 

Publication Number 10(14)-EHC062-EF. 

Rockville, MD: Agency for Healthcare 

Research and Quality. January 2014. 

Available at: 

www.effectivehealthcare.ahrq.gov. 

Accessed April 18, 2014. PMID: 21433403.

37. Hersh W, Totten A, Gorman P, et al. Health 

Information Exchange PROSPERO 

2014:CRD42014013285 Available at: 

www.crd.york.ac.uk/PROSPERO/display_re

cord.asp?ID=CRD42014013285#.VTga1fnF

9u1. Accessed April 22, 2014. 

38. Shea BJ, Hamel C, Wells GA, et al. 

AMSTAR is a reliable and valid 

measurement tool to assess the 

methodological quality of systematic 

reviews. J Clin Epidemiol. 

2009;62(10):1013-20. PMID: 19230606.

39. Bailey JE, Wan JY, Mabry LM, et al. Does 

health information exchange reduce 

unnecessary neuroimaging and improve 

quality of headache care in the emergency 

department? J Gen Intern Med. 

2013;28(2):176-83. PMID: 22648609.

40. Bailey JE, Pope RA, Elliott EC, et al. Health 

information exchange reduces repeated 

diagnostic imaging for back pain. Ann 

Emerg Med. 2013;62(1):16-24. PMID: 

23465552.

41. Ben-Assuli O, Shabtai I, Leshno M. The 

impact of EHR and HIE on reducing 

avoidable admissions: controlling main 

differential diagnoses. BMC Med Inform 

Decis Mak. 2013;13:49. PMID: 23594488.

42. Dixon BE, McGowan JJ, Grannis SJ. 

Electronic laboratory data quality and the 

value of a health information exchange to 

support public health reporting processes. 

AMIA Annu Symp Proc. 2011;2011:322-30. 

PMID: 22195084.

43. Feldman SS, Horan TA. Collaboration in 

electronic medical evidence development: a 

case study of the Social Security 

Administration's MEGAHIT System. Int J 

Med Inf. 2011;80(8):e127-40. PMID: 

21333588.

44. Frisse ME, Johnson KB, Nian H, et al. The 

financial impact of health information 

exchange on emergency department care. J 

Am Med Inform Assoc. 2012;19(3):328-33. 

PMID: 22058169.

45. Kern LM, Barrón Y, Dhopeshwarkar RV, et 

al. Health information exchange and 

ambulatory quality of care. Appl Clin 

Inform. 2012;3(2):197-209. PMID: 

23646072.

46. Mäenpää T, Asikainen P, Gissler M, et al. 

Outcomes assessment of the regional health 

information exchange: a five-year follow-up 

study. Methods Inf Med. 2011;50(4):308-18. 

PMID: 21336419.

47. Magnus M, Herwehe J, Gruber D, et al. 

Improved HIV-related outcomes associated 

with implementation of a novel public health 

information exchange. Int J Med Inf. 

2012;81(10):e30-8. PMID: 22883431.
92

48. Nagykaldi ZJ, Yeaman B, Jones M, et al. 

HIE-i-Health Information Exchange With 

Intelligence. J Ambulatory Care Manage. 

2014;37(1):20-31. PMID: 24309392.

49. Overhage JM, Grannis S, McDonald CJ. A 

comparison of the completeness and 

timeliness of automated electronic 

laboratory reporting and spontaneous 

reporting of notifiable conditions. Am J 

Public Health. 2008;98(2):344-50. PMID: 

18172157.

50. Ross SE, Radcliff TA, Leblanc WG, et al. 

Effects of health information exchange 

adoption on ambulatory testing rates. J Am 

Med Inform Assoc. 2013;20(6):1137-42. 

PMID: 23698257.

51. Shapiro JS, Johnson SA, Angiollilo J, et al. 

Health Information Exchange Improves 

Identification Of Frequent Emergency 

Department Users. Health Aff. 

2013;32(12):2193-8. PMID: 24301405.

52. Tzeel A, Lawnicki V, Pemble KR. The 

Business Case for Payer Support of a 

Community-Based Health Information 

Exchange: A Humana Pilot Evaluating Its 

Effectiveness in Cost Control for Plan 

Members Seeking Emergency Department 

Care. Am Health Drug Benefits. 

2011;4(4):207-15. PMID: 25126351.

53. Tzeel A, Lawnicki V, Pemble KR. "Hidden" 

Value: How Indirect Benefits of Health 

Information Exchange Further Promote 

Sustainability. Am Health Drug Benefits. 

2012;5(6):333-40. PMID: 24991331.

54. Vest JR. Health information exchange and 

healthcare utilization. J Med Syst. 

2009;33(3):223-31. PMID: 19408456.

55. Vest JR, Kern LM, Silver MD, et al. The 

potential for community-based health 

information exchange systems to reduce 

hospital readmissions. J Am Med Inform 

Assoc. 2014 PMID: 25100447.

56. Vest JR, Kern LM, Campion TR, Jr., et al. 

Association between use of a health 

information exchange system and hospital 

admissions. Appl Clin Inform. 

2014;5(1):219-31. PMID: 24734135.

57. Altman R, Shapiro JS, Moore T, et al. 

Notifications of hospital events to outpatient 

clinicians using health information 

exchange: a post-implementation survey. 

Inform Prim Care. 2012;20(4):249-55. 

PMID: 23890336.

58. Campion TR, Jr., Ancker JS, Edwards AM, 

et al. Push and pull: physician usage of and 

satisfaction with health information 

exchange. AMIA Annu Symp Proc. 

2012;2012:77-84. PMID: 23304275.

59. Chang KC, Overhage JM, Hui SL, et al. 

Enhancing laboratory report contents to 

improve outpatient management of test 

results. J Am Med Inform Assoc. 

2010;17(1):99-103. PMID: 20064809.

60. Kaushal R, Dhopeshwarkar R, Gottlieb L, et 

al. User experiences with pharmacy benefit 

manager data at the point of care. J Eval 

Clin Pract. 2010;16(6):1076-80. PMID: 

20666888.

61. Maass MC, Asikainen P, Mäenpää T, et al. 

Usefulness of a Regional Health Care 

Information System in primary care. A case 

study. Comput Methods Programs Biomed. 

2008;91(2):175-81. PMID: 18514363.

62. Machan C, Ammenwerth E, Schabetsberger 

T. Evaluation of the electronic transmission 

of medical findings from hospitals to 

practitioners by triangulation. Methods Inf 

Med. 2006;45(2):225-33. PMID: 16538293.

63. Park H, Lee S-i, Kim Y, et al. Patients' 

perceptions of a health information 

exchange: a pilot program in South Korea. 

Int J Med Inf. 2013;82(2):98-107. PMID: 

22658777.

64. Vest JR, Miller TR. The association between 

health information exchange and measures 

of patient satisfaction. Appl Clin Inform. 

2011;2(4):447-59. PMID: 23616887.

65. Lang E, Afilalo M, Vandal AC, et al. Impact 

of an electronic link between the emergency 

department and family physicians: a 

randomized controlled trial. CMAJ. 

2006;174(3):313-8. PMID: 16399880.

66. Afilalo M, Lang E, Léger R, et al. Impact of 

a standardized communication system on 

continuity of care between family physicians 

and the emergency department. CJEM. 

2007;9(2):79-86. PMID: 17391577.
93

67. Willis JM, Edwards R, Anstrom KJ, et al. 

Decision support for evidence-based 

pharmacotherapy detects adherence 

problems but does not impact medication 

use. Stud Health Technol Inform. 

2013;183:116-25. PMID: 23388267.

68. Jones SS, Friedberg MW, Schneider EC. 

Health information exchange, Health 

Information Technology use, and hospital 

readmission rates. AMIA Annu Symp Proc. 

2011;2011:644-53. PMID: 22195120.

69. Lammers EJ, Adler-Milstein J, Kocher KE. 

Does health information exchange reduce 

redundant imaging? Evidence from 

emergency departments. Med Care. 

2014;52(3):227-34. PMID: 24374414.

70. Carr CM, Gilman CS, Krywko DM, et al. 

Observational study and estimate of cost 

savings from use of a health information 

exchange in an academic emergency 

department. J Emerg Med. 2014;46(2):250-

6. PMID: 24071033.

71. Winden TJ, Boland LL, Frey NG, et al. Care 

everywhere, a point-to-point HIE tool: 

utilization and impact on patient care in the 

ED. Appl Clin Inform. 2014;5(2):388-401. 

PMID: 25024756.

72. Ben-Assuli O, Shabtai I, Leshno M. Using 

electronic health record systems to optimize 

admission decisions: The Creatinine case 

study. Health Informatics J. 2015;21(1):73-

88. PMID: 24692078.

73. eHealth Initiative. Result from Survey on 

Health Data Exchange 2013. The Challenge 

to Connect. Available at: https://ehi-railsapp.s3.amazonaws.com/uploads/article/file/

3/eHIResultsFromSurveyonHealthDataExch

ange2013.pdf. Accessed June 17, 2015. 

74. eHealth Initiative. Post HITECH: The 

Landscape of Health Information Exchange. 

Available at:

http://visit.medicity.com/rs/aetnainc/images/

2014%20eHI%20Data%20Exchange%20Su

rvey%20Key%20Findings.pdf. Accessed 

June 17, 2015. 

75. Carr CM, Saef SH, Zhao J, et al. Can data 

from a health information exchange be used 

to describe patients who visit multiple 

emergency departments within a region? 

Acad Emerg Med. 2014;21(5):S141. 

76. Abramson EL, McGinnis S, Edwards A, et 

al. Electronic health record adoption and 

health information exchange among 

hospitals in New York State. J Eval Clin 

Pract. 2012;18(6):1156-62. PMID:

21914089.

77. Abramson EL, McGinnis S, Moore J, et al. 

A statewide assessment of electronic health 

record adoption and health information 

exchange among nursing homes. Health 

Serv Res. 2014;49(1 Pt 2):361-72. PMID: 

24359612.

78. Adler-Milstein J, Bates DW, Jha AK. U.S. 

Regional health information organizations: 

progress and challenges. Health Aff. 

2009;28(2):483-92. PMID: 19276008.

79. Adler-Milstein J, Bates DW, Jha AK. A 

survey of health information exchange 

organizations in the United States: 

implications for meaningful use. Ann Intern 

Med. 2011;154(10):666-71. PMID: 

21576534.

80. Adler-Milstein J, Landefeld J, Jha AK. 

Characteristics associated with regional 

health information organization viability. J 

Am Med Inform Assoc. 2010;17(1):61-5. 

PMID: 20064803.

81. Adler-Milstein J, McAfee AP, Bates DW, et 

al. The state of regional health information 

organizations: current activities and 

financing. Health Aff. 2008;27(1):w60-9. 

PMID: 18073225.

82. Bouhaddou O, Bennett J, Cromwell T, et al. 

The Department of Veterans Affairs, 

Department of Defense, and Kaiser 

Permanente Nationwide Health Information 

Network exchange in San Diego: patient 

selection, consent, and identity matching. 

AMIA Annu Symp Proc. 2011;2011:135-43. 

PMID: 22195064.

83. Dixon BE, Jones JF, Grannis SJ. Infection 

preventionists' awareness of and 

engagement in health information exchange 

to improve public health surveillance. Am J 

Infect Control. 2013;41(9):787-92. PMID: 

23415767.

84. Foldy S. Inventory of electronic health 

information exchange in Wisconsin, 2006. 

WMJ. 2007;106(3):120-5. PMID: 

17642349.
94

85. Fontaine P, Zink T, Boyle RG, et al. Health 

information exchange: participation by 

Minnesota primary care practices. Arch 

Intern Med. 2010;170(7):622-9. PMID: 

20386006.

86. Gadd CS, Ho Y-X, Cala CM, et al. User 

perspectives on the usability of a regional 

health information exchange. J Am Med 

Inform Assoc. 2011;18(5):711-6. PMID: 

21622933.

87. Hessler BJ, Soper P, Bondy J, et al. 

Assessing the relationship between health 

information exchanges and public health 

agencies. J Public Health Manag Pract. 

2009;15(5):416-24. PMID: 19704310.

88. Kho AN, Doebbeling BN, Cashy JP, et al. A 

regional informatics platform for 

coordinated antibiotic-resistant infection 

tracking, alerting, and prevention. Clin 

Infect Dis. 2013;57(2):254-62. PMID: 

23575195.

89. Lee S-I, Park H, Kim J-W, et al. Physicians' 

perceptions and use of a health information 

exchange: a pilot program in South Korea. 

Telemed J E Health. 2012;18(8):604-12. 

PMID: 22352898.

90. Soderberg K, Laventure M. Minnesota 

clinics' adoption, use and exchange of 

electronic health information. Minn Med. 

2013;96(9):45-8. PMID: 24494362.

91. Patel V, Swain MJ, King J, et al. Physician 

capability to electronically exchange clinical 

information, 2011. Am J Manag Care. 

2013;19(10):835-43. PMID: 24304162.

92. Anand V, Sheley ME, Xu S, et al. Real time 

alert system: a disease management system 

leveraging health information exchange. 

Online J Public Health Inform. 2012;4(3) 

PMID: 23569648.

93. Caffrey C, Park-Lee E. Use of electronic 

health records in residential care 

communities. NCHS data brief. 

2013(128):1-8. PMID: 24152578.

94. Codagnone C, Lupiañez-Villanueva F. 

Benchmarking Deployment of eHealth 

among General Practitioners (2013) - Final 

Report. Lukembourg: European 

Commission.

95. Schoen C, Osborn R, Squires D, et al. A 

survey of primary care doctors in ten 

countries shows progress in use of health 

information technology, less in other areas. 

Health Aff (Millwood). 2012;31(12):2805-

16. PMID: 23154997.

96. Abramson EL, Silver M, Kaushal R. 

Meaningful use status and participation in 

health information exchange among New 

York State hospitals: A longitudinal 

assessment. Jt Comm J Qual Patient Saf. 

2014;40(10) 

97. Campion TR, Jr., Edwards AM, Johnson 

SB, et al. Health information exchange 

system usage patterns in three communities: 

practice sites, users, patients, and data. Int J 

Med Inf. 2013;82(9):810-20. PMID: 

23743323.

98. Campion TR, Jr., Vest JR, Ancker JS, et al. 

Patient encounters and care transitions in 

one community supported by automated 

query-based health information exchange. 

AMIA Annu Symp Proc. 2013;2013:175-84. 

PMID: 24551330.

99. Johnson KB, Gadd CS, Aronsky D, et al. 

The MidSouth eHealth Alliance: use and 

impact in the first year. AMIA Annu Symp 

Proc. 2008:333-7. PMID: 18999184.

100. Lobach DF, Kawamoto K, Anstrom KJ, et 

al. Proactive population health management 

in the context of a regional health 

information exchange using standards-based 

decision support. AMIA Annu Symp Proc. 

2007:473-7. PMID: 18693881.

101. Vest JR, Gamm LD, Ohsfeldt RL, et al. 

Factors associated with health information 

exchange system usage in a safety-net 

ambulatory care clinic setting. J Med Syst. 

2012;36(4):2455-61. PMID: 21523428.

102. Vest JR, Grinspan ZM, Kern LM, et al. 

Using a health information exchange system 

for imaging information: patterns and 

predictors. AMIA Annu Symp Proc. 

2013;2013:1402-11. PMID: 24551416.

103. Vest JR, Jasperson JS. How are health 

professionals using health information 

exchange systems? Measuring usage for 

evaluation and system improvement. J Med 

Syst. 2012;36(5):3195-204. PMID: 

22127521.
95

104. Vest JR, Jasperson JS, Zhao H, et al. Use of 

a health information exchange system in the 

emergency care of children. BMC Med 

Inform Decis Mak. 2011;11:78. PMID: 

22208182.

105. Vest JR, Zhao H, Jaspserson J, et al. Factors 

motivating and affecting health information 

exchange usage. J Am Med Inform Assoc. 

2011;18(2):143-9. PMID: 21262919.

106. Moore T, Shapiro JS, Doles L, et al. Event 

detection: a clinical notification service on a 

health information exchange platform. 

AMIA Annu Symp Proc. 2012;2012:635-42. 

PMID: 23304336.

107. Adler-Milstein J, DesRoches CM, Jha AK. 

Health information exchange among US 

hospitals. Am J Manag Care. 

2011;17(11):761-8. PMID: 22084896.

108. Adler-Milstein J, Jha AK. Health 

information exchange among U.S. hospitals: 

Who's in, who's out, and why? Healthcare. 

2014;2(1):26-32. 

109. Audet A-M, Squires D, Doty MM. Where 

are we on the diffusion curve? Trends and 

drivers of primary care physicians' use of 

health information technology. Health Serv 

Res. 2014;49(1 Pt 2):347-60. PMID: 

24358958.

110. Furukawa MF, King J, Patel V, et al. 

Despite Substantial Progress In EHR 

Adoption, Health Information Exchange 

And Patient Engagement Remain Low In 

Office Settings. Health Aff. 2014:1-8. 

PMID: 25104827.

111. Furukawa MF, Patel V, Charles D, et al. 

Hospital Electronic Health Information 

Exchange Grew Substantially In 2008-12. 

Health Aff. 2013;32(8):1346-54. PMID: 

23918477.

112. Gutteridge DL, Genes N, Hwang U, et al. 

Enhancing a Geriatric Emergency 

Department Care Coordination Intervention 

Using Automated Health Information 

Exchange-Based Clinical Event 

Notifications. EGEMS (Wash DC). 

2014;2(3) 

113. Hamann DJ, Bezboruah KC. Utilization of 

technology by long-term care providers: 

comparisons between for-profit and 

nonprofit institutions. J Aging Health. 

2013;25(4):535-54. PMID: 23509114.

114. Silvester BV, Carr SJ. A shared electronic 

health record: lessons from the coalface. 

Med J Aust. 2009;190(11 Suppl):S113-6. 

PMID: 19485857.

115. Mäenpää T, Asikainen P, Gissler M, et al. 

The utilization rate of the regional health 

information exchange: how it impacts on 

health care delivery outcomes. J Public 

Health Manag Pract. 2012;18(3):215-23. 

PMID: 22473113.

116. Byrne CM, Mercincavage LM, Bouhaddou 

O, et al. The Department of Veterans 

Affairs' (VA) implementation of the Virtual 

Lifetime Electronic Record (VLER): 

Findings and lessons learned from Health 

Information Exchange at 12 sites. Int J Med 

Inf. 2014;83(8):537-47. PMID: 24845146.

117. Jha AK, Doolan D, Grandt D, et al. The use 

of health information technology in seven 

nations. Int J Med Inf. 2008;77(12):848-54. 

PMID: 18657471.

118. Johnson KB, Unertl KM, Chen Q, et al. 

Health information exchange usage in 

emergency departments and clinics: the 

who, what, and why. J Am Med Inform 

Assoc. 2011;18(5):690-7. PMID: 21846788.

119. Unertl KM, Johnson KB, Lorenzi NM. 

Health information exchange technology on 

the front lines of healthcare: workflow 

factors and patterns of use. J Am Med 

Inform Assoc. 2012;19(3):392-400. PMID: 

22003156.

120. Kaelber DC, Waheed R, Einstadter D, et al. 

Use and perceived value of health 

information exchange: one public healthcare 

system's experience. Am J Manag Care. 

2013;19(10 Spec No):Sp337-43. PMID: 

24511888.

121. Greenhalgh T, Stramer K, Bratan T, et al. 

Adoption and non-adoption of a shared 

electronic summary record in England: a 

mixed-method case study. BMJ. 

2010;340:c3111. PMID: 20554687.

122. Pagliari C, Gilmour M, Sullivan F. 

Electronic Clinical Communications 

Implementation (ECCI) in Scotland: a 

mixed-methods programme evaluation. J 

Eval Clin Pract. 2004;10(1):11-20. PMID: 

14731147.
96

123. Tripathi M, Delano D, Lund B, et al. 

Engaging patients for health information 

exchange. Health Aff. 2009;28(2):435-43. 

PMID: 19276000.

124. Herwehe J, Wilbright W, Abrams A, et al. 

Implementation of an innovative, integrated 

electronic medical record (EMR) and public 

health information exchange for HIV/AIDS. 

J Am Med Inform Assoc. 2012;19(3):448-

52. PMID: 22037891.

125. Onyile A, Vaidya SR, Kuperman G, et al. 

Geographical distribution of patients visiting 

a health information exchange in New York 

City. J Am Med Inform Assoc. 

2013;20(e1):e125-30. PMID: 23104049.

126. Bailey JE, Wan J, Pope R, et al. Health 

information exchange use reduces avoidable 

diagnostic imaging in the emergency 

evaluation of back pain. J Gen Intern Med. 

2011;26(University of Tennessee Health 

Science Center, Memphis, United 

States):S349-S50. 

127. Kierkegaard P, Kaushal R, Vest JR. How 

could health information exchange better 

meet the needs of care practitioners? Appl 

Clin Inform. 2014;5(4):861-77. 

128. Myers JJ, Koester KA, Chakravarty D, et al. 

Perceptions regarding the ease of use and 

usefulness of health information exchange 

systems among medical providers, case 

managers and non-clinical staff members 

working in HIV care and community 

settings. Int J Med Inf. 2012;81(10):e21-9. 

PMID: 22854159.

129. Ozkaynak M, Brennan PF. Revisiting 

sociotechnical systems in a case of 

unreported use of health information 

exchange system in three hospital 

emergency departments. J Eval Clin Pract. 

2013;19(2):370-3. PMID: 22420774.

130. Thorn SA, Carter MA, Bailey JE. 

Emergency physicians' perspectives on their 

use of health information exchange. Ann 

Emerg Med. 2014;63(3):329-37. PMID: 

24161840.

131. Finnell JT, Overhage JM. Emergency 

medical services: the frontier in health 

information exchange. AMIA Annu Symp 

Proc. 2010;2010:222-6. PMID: 21346973.

132. Hincapie AL, Warholak TL, Murcko AC, et 

al. Physicians' opinions of a health 

information exchange. J Am Med Inform 

Assoc. 2011;18(1):60-5. PMID: 21106994.

133. Hyppönen H, Reponen J, Lääveri T, et al. 

User experiences with different regional 

health information exchange systems in 

Finland. Int J Med Inf. 2014;83(1):1-18. 

PMID: 24200753.

134. Massy-Westropp M, Giles LC, Law D, et al. 

Connecting hospital and community care: 

the acceptability of a regional data linkage 

scheme. Aust Health Rev. 2005;29(1):12-6. 

PMID: 15683350.

135. McCullough JM, Zimmerman FJ, Bell DS, 

et al. Electronic health information exchange 

in underserved settings: examining 

initiatives in small physician practices & 

community health centers. BMC Health 

Serv Res. 2014;14:415. PMID: 25240718.

136. Rudin R, Volk L, Simon S, et al. What 

Affects Clinicians' Usage of Health 

Information Exchange? Appl Clin Inform. 

2011;2(3):250-62. PMID: 22180762.

137. Yeager VA, Walker D, Cole E, et al. Factors 

Related to Health Information Exchange 

Participation and Use. J Med Syst. 

2014;38(8) PMID: 24957395.

138. Messer LC, Parnell H, Huffaker R, et al. The 

development of a health information 

exchange to enhance care and improve 

patient outcomes among HIV+ individuals 

in rural North Carolina. Int J Med Inf. 

2012;81(10):e46-55. PMID: 22898321.

139. Nohr C, Kristensen M, Andersen SK, et al. 

Shared experience in 13 local Danish EPR 

projects: the Danish EPR Observatory. Stud 

Health Technol Inform. 2001;84(Pt 1):670-

4. PMID: 11604822.

140. Adjerid I, Padman R. Impact of health 

disclosure laws on health information 

exchanges. AMIA Annu Symp Proc. 

2011;2011:48-56. PMID: 22195054.

141. Dixon B, Miller T, Overhage M. Barriers to 

achieving the last mile in health information 

exchange: a survey of small hospitals and 

physician practices. J Healthc Inf Manag. 

2013;27(4):55-8. 
97

142. Dobalian A, Claver ML, Pevnick JM, et al. 

Organizational challenges in developing one 

of the Nationwide Health Information 

Network trial implementation awardees. J 

Med Syst. 2012;36(2):933-40. PMID: 

20703640.

143. Fairbrother G, Trudnak T, Christopher R, et 

al. Cincinnati Beacon Community Program 

Highlights Challenges And Opportunities 

On The Path To Care Transformation. 

Health Aff. 2014;33(5):871-7. PMID: 

24799586.

144. Feldman SS, Schooley LB, Bhavsar PG. 

Health Information Exchange 

Implementation: Lessons Learned and 

Critical Success Factors From a Case Study. 

JMIR Med Inform. 2014;2(2):e19. 

145. Genes N, Shapiro J, Vaidya S, et al. 

Adoption of health information exchange by 

emergency physicians at three urban 

academic medical centers. Appl Clin 

Inform. 2011;2(3):263-9. PMID: 23616875.

146. Goldwater J, Jardim J, Khan T, et al. 

Emphasizing Public Health Within a Health 

Information Exchange: An Evaluation of the 

District of Columbia’s Health Information 

Exchange Program. EGEMS (Wash DC).

2014;2(3) 

147. Kern LM, Wilcox A, Shapiro J, et al. Which 

components of health information 

technology will drive financial value? Am J 

Manag Care. 2012;18(8):438-45. PMID: 

22928759.

148. McGowan JJ, Jordan C, Sims T, et al. Rural 

RHIOs: common issues in the development 

of two state-wide health information 

networks. AMIA Annu Symp Proc. 

2007:528-32. PMID: 18693892.

149. Miller AR, Tucker C. Health information 

exchange, system size and information silos. 

J Health Econ. 2014;33:28-42. PMID: 

24246484.

150. Nykänen P, Karimaa E. Success and failure 

factors in the regional health information 

system design process--results from a 

constructive evaluation study. Methods Inf 

Med. 2006;45(1):85-9. PMID: 16482376.

151. Overhage JM, Evans L, Marchibroda J. 

Communities' readiness for health 

information exchange: the National 

Landscape in 2004. J Am Med Inform 

Assoc. 2005;12(2):107-12. PMID: 

15561785.

152. Pirnejad H, Bal R, Berg M. Building an 

inter-organizational communication network 

and challenges for preserving 

interoperability. Int J Med Inf. 

2008;77(12):818-27. PMID: 18579436.

153. Rudin RS, Simon SR, Volk LA, et al. 

Understanding the decisions and values of 

stakeholders in health information 

exchanges: experiences from Massachusetts. 

Am J Public Health. 2009;99(5):950-5. 

PMID: 19299671.

154. Saff E, Lanway C, Chenyek A, et al. The 

Bay Area HIE. A case study in connecting 

stakeholders. J Healthc Inf Manag. 

2010;24(1):25-30. PMID: 20077922.

155. Vest JR. More than just a question of 

technology: factors related to hospitals' 

adoption and implementation of health 

information exchange. Int J Med Inf. 

2010;79(12):797-806. PMID: 20889370.

156. Vest JR, Campion Jr TR, Kaushal R. 

Challenges, Alternatives, and Paths to 

Sustainability for Health Information 

Exchange Efforts. J Med Syst. 2013;37(6):1-

8. PMID: 24141531.

157. Vest JR, Issel LM. Factors related to public 

health data sharing between local and state 

health departments. Health Serv Res. 

2014;49(1 Pt 2):373-91. PMID: 24359636.

158. Dullabh P, Hovey L. Large Scale Health

Information Exchange: Implementation 

Experiences from Five States. Stud Health 

Technol Inform. 2013;192:613-7. PMID: 

23920629.

159. Dullabh P, Ubri P, Hovey L. The State HIE 

Program Four Years Later: Key Findings on 

Grantees’ Experiences from a Six-State 

Review. Office of the National Coordinator 

for Health Information Technology. 2014. 

Available at: www.healthit.gov/sites/

default/files/CaseStudySynthesisGranteeExp

erienceFinal_121014.pdf. Accessed April 

22, 2015.
98

160. Grossman JM, Kushner KL, November EA. 

Creating sustainable local health information 

exchanges: can barriers to stakeholder 

participation be overcome? Res Briefs. 

2008(2):1-12. PMID: 18496926.

161. McCarthy DB, Propp K, Cohen A, et al. 

Learning from Health Information Exchange 

Technical Architecture and Implementation 

in Seven Beacon Communities. EGEMS 

(Wash DC). 2014;2(1):6. 

162. Miller RH. Satisfying patient-consumer 

principles for health information exchange: 

evidence from California case studies. 

Health Aff. 2012;31(3):537-47. PMID: 

22392664.

163. Morris G, Afzal S, Bhasker M, et al. QueryBased Exchange: Key Factors Influencing 

Success and Failure. Office of the National 

Coordinator for Health Information 

Technology. 2012. Available at: 

www.healthit.gov/sites/default/files/query_b

ased_exchange_final.pdf. Accessed April 

10, 2015. 

164. Phillips AB, Wilson RV, Kaushal R, et al. 

Implementing health information exchange 

for public health reporting: a comparison of 

decision and risk management of three 

regional health information organizations in 

New York state. J Am Med Inform Assoc. 

2014;21(e1):e173-7. PMID: 23975626.

165. Pouloudi A. Information technology for 

collaborative advantage in healthcare 

revisited. Information & Management. 

1999;35(6):345-56. 

166. Agency for Healthcare Research and 

Quality. Evolution of State Health 

Information Exchange/A Study of Vision, 

Strategy, and Progress. Available at: 

http://www.avalerehealth.net/research/docs/

State_based_Health_Information_Exchange

_Final_Report.pdf. Accessed November 20, 

2014. 

167. Ross SE, Schilling LM, Fernald DH, et al. 

Health information exchange in small-tomedium sized family medicine practices: 

motivators, barriers, and potential 

facilitators of adoption. Int J Med Inf. 

2010;79(2):123-9. PMID: 20061182.

168. Sicotte C, Paré G. Success in health 

information exchange projects: solving the 

implementation puzzle. Soc Sci Med. 

2010;70(8):1159-65. PMID: 20137847.

169. Steward WT, Koester KA, Collins SP, et al. 

The essential role of reconfiguration 

capabilities in the implementation of HIVrelated health information exchanges. Int J 

Med Inf. 2012;81(10):e10-20. PMID: 

22841703.

170. Unertl MK, Johnson BK, Gadd SC, et al. 

Bridging Organizational Divides in Health 

Care: An Ecological View of Health 

Information Exchange. JMIR Med Inform. 

2013;1(1):e3. 

171. Kern LM, Wilcox AB, Shapiro J, et al. 

Community-based health information 

technology alliances: potential predictors of 

early sustainability. Am J Manag Care. 

2011;17(4):290-5. PMID: 21615199.

172. Schabetsberger T, Ammenwerth E, 

Andreatta S, et al. From a paper-based 

transmission of discharge summaries to 

electronic communication in health care 

regions. Int J Med Inf. 2006;75(3-4):209-15. 

PMID: 16112892.

173. Kern LM, Barrón Y, Abramson EL, et al. 

HEAL NY: Promoting interoperable health 

information technology in New York State. 

Health Aff. 2009;28(2):493-504. PMID: 

19276009.

174. Merrill JA, Deegan M, Wilson RV, et al. A 

system dynamics evaluation model: 

implementation of health information 

exchange for public health reporting. J Am 

Med Inform Assoc. 2013;20(e1):e131-8. 

PMID: 23292910.

175. Becker L, Oxman A. Chapter 22: Overviews 

of reviews. In: Higgins J, Green S, eds. 

Cochrane Handbook for Systematic Reviews 

of Interventions Version 5.1.0 (updated 

March 2011): The Cochrane Collaboration; 

2011.

176. Roshanov PS, Fernandes N, Wilczynski JM, 

et al. Features of effective computerised 

clinical decision support systems: metaregression of 162 randomised trials. BMJ. 

2013;346:f657. PMID: 23412440.

177. Thomson D, Russell K, Becker L, et al. The 

evolution of a new publication type: Steps 

and challenges of producing overviews of 

reviews. Research Synthesis Methods. 

2010;1:198-211. 
99

178. White C, Ip S, McPheeters M. Using 

existing systematic reviews to replace de 

novo processes in conducting Comparative 

Effectiveness Reviews. Methods Guide for 

Comparative Effectiveness Reviews. 

Rockville, MD: Agency for Healthcare 

Research and Quality; 2009.

179. Michie S, Fixsen D, Grimshaw JM, et al. 

Specifying and reporting complex behaviour 

change interventions: the need for a 

scientific method. Implement Sci. 

2009;4:40. PMID: 19607700.

180. Petitti DB. Hormone replacement therapy 

and heart disease prevention: 

experimentation trumps observation. JAMA. 

1998;280(7):650-2. PMID: 9718060.

181. Sackett DL, Wennberg JE. Choosing the 

best research design for each question. BMJ. 

1997;315(7123):1636. PMID: 9448521.

182. Craig P, Dieppe P, Macintyre S, et al. 

Developing and evaluating complex 

interventions: the new Medical Research 

Council guidance. BMJ. 2008;337:a1655. 

PMID: 18824488.

183. Rasmussen J. Risk management in a 

dynamic society: a modelling problem. 

Safety science. 1997;27(2):183-213. 

184. Vicente KJ. What does it take? A case study 

of radical change toward patient safety. Joint 

Commission journal on quality and safety. 

2003;29(11):598-609. PMID: 14619352.

185. Leveson N. Engineering a Safer World. 

Systems Thinking Applied to Safety. 

Cambridge: MIT Press; 2012.

186. Rasmussen J. Risk Management in a 

Dynamic Society: A Modelling Problem.

Safety Science. 1997;27(2/3):183-213. 

187. Michie S, Richardson M, Johnston M, et al. 

The behavior change technique taxonomy 

(v1) of 93 hierarchically clustered 

techniques: building an international 

consensus for the reporting of behavior 

change interventions. Annals of behavioral 

medicine : a publication of the Society of 

Behavioral Medicine. 2013;46(1):81-95. 

PMID: 23512568.

188. Schulz KF, Altman DG, Moher D. 

CONSORT 2010 statement: updated 

guidelines for reporting parallel group 

randomized trials. Ann Intern Med. 

2010;152(11):726-32. PMID: 20335313.

189. Pearce N, Merletti F. Complexity, 

simplicity, and epidemiology. Int J 

Epidemiol. 2006;35(3):515-9. PMID: 

16415326.

190. Galea S, Riddle M, Kaplan GA. Causal 

thinking and complex system approaches in 

epidemiology. Int J Epidemiol. 

2010;39(1):97-106. PMID: 19820105.

191. Fleurence RL, Curtis LH, Califf RM, et al. 

Launching PCORnet, a national patientcentered clinical research network. J Am 

Med Inform Assoc. 2014:amiajnl-2014-

002747. PMID: 24821743.
100

Abbreviations and Acronyms

AHA American Hospital Association

AHRQ Agency for Healthcare Research and Quality

CHIC Carolina HIV Information Cooperative

CI confidence interval

CONSORT Consolidated Standards of Reporting Trials

CT computed tomography

DoD Department of Defense

ED emergency department

eGEMS Generating Evidence and Methods to improve patient outcomes

EHR electronic health record

EPC Evidence-based Practice Center

HEAL-NY Health Care Efficiency and Affordability Law for New Yorkers Capital 

Grant Program

HIE health information exchange

HITECH Health Information Technology for Economic and Clinical Health

I-Care Central Texas HIE

IEEE Institute of Electrical and Electronics Engineers

IT information technology

K Thousand

MSeHA MidSouth e-Health Alliance

NAMCS National Ambulatory Medical Care Survey

NwHIN Nationwide Health Information Network

ONC The Office of the National Coordinator for Health Information 

Technology

OR odds ratio

PICOTS populations, interventions, comparators, outcomes, timing, types of 

studies, and setting

QUIS Questionnaire for User Interface Satisfaction

RCT randomized controlled trials

RHIO regional health information organization

STAMP Systems—Theoretic Accident Modeling and Processes

TEP Technical Expert Panel

VA Veterans Affairs

VLER Virtual Lifetime Electronic Record
A-1

Appendix A. Search Strategies

Database: Ovid MEDLINE® and Ovid OLDMEDLINE® <1990 to 

February 2015> Search Strategy

1 (health information adj5 exchang$).mp. 

2 hie.mp. 

3 exp Medical Records/ 

4 exp Systems Analysis/ 

5 exp Medical Informatics/

6 Information Dissemination/ 

7 3 or 4 or 5 or 6 

8 2 and 7

9 1 or 8 

10 health information organization$.mp. 

11 7 and 10 

12 (hio or hios or rhio or rhios).mp. 

13 7 and 12 

14 ((clinical$ or health$) adj5 (data adj3 exchang$)).mp. 

15 7 and 14 

16 (patient$ adj2 match$).mp. 

17 7 and 16 

18 ((query or querie$) adj3 (base or based or bases or basing) adj5 exchang$).mp. 

19 7 and 18

20 directed exchang$.mp. 

21 7 and 20 

22 ((consumer$ or patient$) adj5 mediat$ adj7 exchang$).mp. 

23 7 and 22 

24 ((health information adj5 tech$) and exchang$).mp.

25 7 and 24 

26 (health information adj7 network$).mp. 

27 7 and 26

28 ((health information or ((electronic$ or computer$) adj2 (health or medic$ or patient$) adj2 

record$) or ehr or emr) adj7 exchang$).mp. 

29 7 and 28

30 (exchang$ adj5 network$).mp.

31 7 and 30 (116)

32 (interoperab$ adj7 standard$).mp. (320)

33 7 and 32 

34 ((inter or between or across) adj3 (organization$ or systems) adj7 network$).mp. 

35 7 and 34 

36 9 or 11 or 13 or 15 or 17 or 19 or 21 or 23 or 25 or 27 or 29 or 31 or 33 or 35

37 Medical Record Linkage/

38 exp systems integration/ 

39 37 and 38 

40 exp Cooperative Behavior/
A-2

41 37 and 40 

42 exp Medical Informatics Applications/

43 37 and 42 

44 10 or 12 or 14 or 16 or 18 or 20 or 22 or 24 or 26 or 28 or 30 or 32

45 43 and 44 

46 36 or 39 or 41 or 45

47 6 and 38 and 42

48 6 and 38 and 40

49 4 and 37 and 40 

50 4 and 37 and 42 

51 6 and 37 and 42 

52 6 and 37 and 40 

53 4 and 38 and 40 

54 46 or 47 or 48 or 49 or 50 or 51 or 52 or 53 

55 limit 54 to english language 

Database: PsycINFO <1990 to February 2015> Search 

Strategy

1 ((healthcare information or health information) adj5 exchang$).mp. [mp=title, abstract, 

heading word, table of contents, key concepts, original title, tests & measures] 

2 exp medical records/

3 exp information systems/ 

4 exp Information Dissemination/

5 exp systems analysis/ 

6 exp information technology/ 

7 exp computer mediated communication/

8 2 or 3 or 4 or 5 or 6 or 7 

9 hie.mp. 

10 8 and 9 

11 1 or 10

12 health information organization$.mp. 

13 (hio or hios or rhio or rhios).mp. 

14 ((clinical$ or health$) adj5 (data adj3 exchang$)).mp. 

15 (patient$ adj2 match$).mp. 

16 8 and 15 

17 ((query or querie$) adj3 (base or based or bases or basing) adj5 exchang$).mp. 

18 directed exchang$.mp. 

19 ((consumer$ or patient$) adj5 mediat$ adj7 exchang$).mp. 

20 ((health information adj5 tech$) and exchang$).mp. 

21 (health information adj7 network$).mp. 

22 ((health information or ((electronic$ or computer$) adj2 (health or medic$ or patient$) adj2 

record$) or ehr or emr) adj7 exchang$).mp. 

23 (exchang$ adj5 network$).mp. 

24 8 and 23 

25 (interoperab$ adj7 standard$).mp. 

26 11 or 12 or 14 or 16 or 17 or 18 or 19 or 20 or 21 or 22 or 24 or 25 
A-3

Databases: EBM Reviews - Cochrane Database of Systematic 

Reviews, Cochrane Central Register of Controlled Trials, 

Database of Abstracts of Reviews of Effects, NHS Economic 

Evaluation Database <1990 to January 2015>

Search Strategy

1 (health information adj5 exchang$).mp. 

2 hie.mp. 

3 ((health or medical) adj3 (record or records)).mp. [mp=title, text, subject heading word] 

4 ((System or systems) adj3 Analysis).mp. [mp=title, text, subject heading word] 

5 ((health$ or medic$) adj5 informatic$).mp. [mp=title, text, subject heading word] 

6 ((informat$ or data) adj5 (link$ or disseminat$ or transfer$ or request$ or share$ or 

sharing)).mp. [mp=title, text, subject heading word] 

7 3 or 4 or 5 or 6 

8 2 and 7 

9 1 or 8

10 health information organization$.mp. 

11 7 and 10 

12 (hio or hios or rhio or rhios).mp. 

13 7 and 12 

14 ((clinical$ or health$) adj5 (data adj3 exchang$)).mp. 

15 7 and 14 

16 (patient$ adj2 match$).mp. 

17 7 and 16 

18 ((query or querie$) adj3 (base or based or bases or basing) adj5 exchang$).mp. 

19 7 and 18 

20 directed exchang$.mp. 

21 7 and 20 

22 ((consumer$ or patient$) adj5 mediat$ adj7 exchang$).mp. 

23 7 and 22 

24 ((health information adj5 tech$) and exchang$).mp. 

25 7 and 24 

26 (health information adj7 network$).mp. 

27 7 and 26 

28 ((health information or ((electronic$ or computer$) adj2 (health or medic$ or patient$) adj2 

record$) or ehr or emr) adj7 exchang$).mp. 

29 7 and 28 

30 (exchang$ adj5 network$).mp. 

31 7 and 30 

32 (interoperab$ adj7 standard$).mp. 

33 7 and 32 

34 ((inter or between or across) adj3 (organization$ or systems) adj7 network$).mp. 

35 7 and 34 

36 ((health$ or medic$) adj3 record adj7 (link$ or disseminat$ or transfer$ or request$ or 

share$ or sharing)).mp. [mp=title, text, subject heading word] 

37 9 or 11 or 13 or 15 or 17 or 19 or 21 or 23 or 25 or 27 or 29 or 31 or 33 or 35 or 36 
B-1

Appendix B. Inclusion and Exclusion Criteria

Table B1. Inclusion and exclusion criteria

Include Exclude

Population All KQs: Any individual or group of health care providers, patients, managers, health care institutions, 

or regional organizations.

All KQs: Not applicable to a U.S. 

population.

Interventions All KQs: Heath Information Exchange . HIE is defined as the electronic sharing of clinical information 

among users such as health care providers, patients, administrators or policy makers across the 

boundaries of health care institutions, health data repositories, States and others, typically not within a 

single organization or among affiliated providers, while protecting the integrity, privacy, and security of 

the information.

All KQs: Hypothetical HIEs, HIE within an 

organization/single setting, independent 

electronic prescription or referral system, 

a single person accessing multiple 

systems, registries, HIE for research, 

marketing or administration, nonelectronic transfers.

Comparators KQ 1-3: Time period prior to HIE implementations, geographic or organizational locations without HIE, 

situations in which HIE is not available, multiple types of HIE, characteristics of the different settings 

and systems in which HIE is used.

KQ 4-8: No comparison required

KQ 1-3: No comparator

KQ 4-8: None
C-1

Appendix C. List of Included Studies

Abramson EL, McGinnis S, Edwards A, et al. Electronic health record adoption and health 

information exchange among hospitals in New York State. J Eval Clin Pract. 2012;18(6):1156-

62. PMID: 21914089.

Abramson EL, McGinnis S, Moore J, et al. A statewide assessment of electronic health record 

adoption and health information exchange among nursing homes. Health Serv Res. 2014;49(1 Pt 

2):361-72. PMID: 24359612.

Abramson EL, Silver M, Kaushal R. Meaningful use status and participation in health 

information exchange among New York State hospitals: A longitudinal assessment. Jt Comm J 

Qual Patient Saf. 2014;40(10) 

Adjerid I, Padman R. Impact of health disclosure laws on health information exchanges. AMIA 

Annu Symp Proc. 2011;2011:48-56. PMID: 22195054.

Adler-Milstein J, Bates DW, Jha AK. U.S. Regional health information organizations: progress 

and challenges. Health Aff. 2009;28(2):483-92. PMID: 19276008.

Adler-Milstein J, Bates DW, Jha AK. A survey of health information exchange organizations in 

the United States: implications for meaningful use. Ann Intern Med. 2011;154(10):666-71. 

PMID: 21576534.

Adler-Milstein J, Bates DW, Jha AK. Operational health information exchanges show substantial 

growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32(8):1486-92. 

PMID: 23840051.

Adler-Milstein J, DesRoches CM, Jha AK. Health information exchange among US hospitals. 

Am J Manag Care. 2011;17(11):761-8. PMID: 22084896.

Adler-Milstein J, Jha AK. Health information exchange among U.S. hospitals: Who's in, who's 

out, and why? Healthcare. 2014;2(1):26-32. 

Adler-Milstein J, Landefeld J, Jha AK. Characteristics associated with regional health 

information organization viability. J Am Med Inform Assoc. 2010;17(1):61-5. PMID: 20064803.

Adler-Milstein J, McAfee AP, Bates DW, et al. The state of regional health information 

organizations: current activities and financing. Health Aff. 2008;27(1):w60-9. PMID: 18073225.

Afilalo M, Lang E, Léger R, et al. Impact of a standardized communication system on continuity 

of care between family physicians and the emergency department. CJEM. 2007;9(2):79-86. 

PMID: 17391577.

Agency for Healthcare Research and Quality. Evolution of State Health Information Exchange/A 

Study of Vision, Strategy, and Progress. Available at: 
C-2

http://www.avalerehealth.net/research/docs/State_based_Health_Information_Exchange_Final_R

eport.pdf. Accessed November 20, 2014. 

Altman R, Shapiro JS, Moore T, et al. Notifications of hospital events to outpatient clinicians 

using health information exchange: a post-implementation survey. Inform Prim Care. 

2012;20(4):249-55. PMID: 23890336.

Anand V, Sheley ME, Xu S, et al. Real time alert system: a disease management system 

leveraging health information exchange. Online J Public Health Inform. 2012;4(3) PMID: 

23569648.

Audet A-M, Squires D, Doty MM. Where are we on the diffusion curve? Trends and drivers of 

primary care physicians' use of health information technology. Health Serv Res. 2014;49(1 Pt 

2):347-60. PMID: 24358958.

Bailey JE, Pope RA, Elliott EC, et al. Health information exchange reduces repeated diagnostic 

imaging for back pain. Ann Emerg Med. 2013;62(1):16-24. PMID: 23465552.

Bailey JE, Wan JY, Mabry LM, et al. Does health information exchange reduce unnecessary 

neuroimaging and improve quality of headache care in the emergency department? J Gen Intern 

Med. 2013;28(2):176-83. PMID: 22648609.

Ben-Assuli O, Shabtai I, Leshno M. The impact of EHR and HIE on reducing avoidable 

admissions: controlling main differential diagnoses. BMC Med Inform Decis Mak. 2013;13:49. 

PMID: 23594488.

Ben-Assuli O, Shabtai I, Leshno M. Using electronic health record systems to optimize 

admission decisions: The Creatinine case study. Health Informatics J. 2015;21(1):73-88. PMID: 

24692078.

Bouhaddou O, Bennett J, Cromwell T, et al. The Department of Veterans Affairs, Department of 

Defense, and Kaiser Permanente Nationwide Health Information Network exchange in San 

Diego: patient selection, consent, and identity matching. AMIA Annu Symp Proc. 

2011;2011:135-43. PMID: 22195064.

Byrne CM, Mercincavage LM, Bouhaddou O, et al. The Department of Veterans Affairs' (VA) 

implementation of the Virtual Lifetime Electronic Record (VLER): Findings and lessons learned 

from Health Information Exchange at 12 sites. Int J Med Inf. 2014;83(8):537-47. PMID: 

24845146.

Caffrey C, Park-Lee E. Use of electronic health records in residential care communities. NCHS 

data brief. 2013(128):1-8. PMID: 24152578.

Campion TR, Jr., Ancker JS, Edwards AM, et al. Push and pull: physician usage of and 

satisfaction with health information exchange. AMIA Annu Symp Proc. 2012;2012:77-84. 

PMID: 23304275.
C-3

Campion TR, Jr., Edwards AM, Johnson SB, et al. Health information exchange system usage 

patterns in three communities: practice sites, users, patients, and data. Int J Med Inf. 

2013;82(9):810-20. PMID: 23743323.

Campion TR, Jr., Vest JR, Ancker JS, et al. Patient encounters and care transitions in one 

community supported by automated query-based health information exchange. AMIA Annu 

Symp Proc. 2013;2013:175-84. PMID: 24551330.

Carr CM, Gilman CS, Krywko DM, et al. Observational study and estimate of cost savings from 

use of a health information exchange in an academic emergency department. J Emerg Med. 

2014;46(2):250-6. PMID: 24071033.

Chang KC, Overhage JM, Hui SL, et al. Enhancing laboratory report contents to improve 

outpatient management of test results. J Am Med Inform Assoc. 2010;17(1):99-103. PMID: 

20064809.

Codagnone C, Lupiañez-Villanueva F. Benchmarking Deployment of eHealth among General 

Practitioners (2013) – Final Report. Luxembourg: European Commission:2014. Available at: 

file:///X:/SOM/Informatics/Informat/EPC%20IV%20TO11%20Health%20Info%20Exchange/4

%20Articles,%20Searches,%20ENL/PDFs/Articles/ON%20HOLD/Codagnone%202013.pdf. 

Accessed December 16, 2014. 

Dixon B, Miller T, Overhage M. Barriers to achieving the last mile in health information 

exchange: a survey of small hospitals and physician practices. J Healthc Inf Manag. 

2013;27(4):55-8. 

Dixon BE, Jones JF, Grannis SJ. Infection preventionists' awareness of and engagement in health 

information exchange to improve public health surveillance. Am J Infect Control. 

2013;41(9):787-92. PMID: 23415767.

Dixon BE, McGowan JJ, Grannis SJ. Electronic laboratory data quality and the value of a health 

information exchange to support public health reporting processes. AMIA Annu Symp Proc. 

2011;2011:322-30. PMID: 22195084.

Dobalian A, Claver ML, Pevnick JM, et al. Organizational challenges in developing one of the 

Nationwide Health Information Network trial implementation awardees. J Med Syst. 

2012;36(2):933-40. PMID: 20703640.

Dullabh P, Hovey L. Large Scale Health Information Exchange: Implementation Experiences 

from Five States. Stud Health Technol Inform. 2013;192:613-7. PMID: 23920629.

Dullabh P, Ubri P, Hovey L. The State HIE Program Four Years Later: Key Findings on 

Grantees’ Experiences from a Six-State Review. Office of the National Coordinator for Health 

Information Technology. 2014. Available at: 

http://www.healthit.gov/sites/default/files/CaseStudySynthesisGranteeExperienceFinal_121014.

pdf. Accessed April 22, 2015.
C-4

eHealth Initiative. Result from Survey on Health Data Exchange 2013. The Challenge to 

Connect. Available at: http://www.ehidc.org/resource-center/reports/view_document/458-

survey-results-results-from-survey-on-data-exchange-2013-data-exchange. Accessed March 3, 

2015. 

eHealth Initiative. Post HITECH: The Landscape of Health Information Exchange. Available at: 

http://www.ehidc.org/resource-center/publications/view_document/461-reports-2014-ehi-dataexchange-survey-key-findings. Accessed January 9, 2014. 

Fairbrother G, Trudnak T, Christopher R, et al. Cincinnati Beacon Community Program 

Highlights Challenges And Opportunities On The Path To Care Transformation. Health Aff. 

2014;33(5):871-7. PMID: 24799586.

Feldman SS, Horan TA. Collaboration in electronic medical evidence development: a case study 

of the Social Security Administration's MEGAHIT System. Int J Med Inf. 2011;80(8):e127-40. 

PMID: 21333588.

Feldman SS, Schooley LB, Bhavsar PG. Health Information Exchange Implementation: Lessons 

Learned and Critical Success Factors From a Case Study. JMIR Med Inform. 2014;2(2):e19. 

Finnell JT, Overhage JM. Emergency medical services: the frontier in health information 

exchange. AMIA Annu Symp Proc. 2010;2010:222-6. PMID: 21346973.

Foldy S. Inventory of electronic health information exchange in Wisconsin, 2006. WMJ. 

2007;106(3):120-5. PMID: 17642349.

Fontaine P, Zink T, Boyle RG, et al. Health information exchange: participation by Minnesota 

primary care practices. Arch Intern Med. 2010;170(7):622-9. PMID: 20386006.

Frisse ME, Johnson KB, Nian H, et al. The financial impact of health information exchange on 

emergency department care. J Am Med Inform Assoc. 2012;19(3):328-33. PMID: 22058169.

Furukawa MF, King J, Patel V, et al. Despite Substantial Progress In EHR Adoption, Health 

Information Exchange And Patient Engagement Remain Low In Office Settings. Health Aff. 

2014:1-8. PMID: 25104827.

Furukawa MF, Patel V, Charles D, et al. Hospital Electronic Health Information Exchange Grew 

Substantially In 2008-12. Health Aff. 2013;32(8):1346-54. PMID: 23918477.

Gadd CS, Ho Y-X, Cala CM, et al. User perspectives on the usability of a regional health 

information exchange. J Am Med Inform Assoc. 2011;18(5):711-6. PMID: 21622933.

Genes N, Shapiro J, Vaidya S, et al. Adoption of health information exchange by emergency 

physicians at three urban academic medical centers. Appl Clin Inform. 2011;2(3):263-9. PMID: 

23616875.
C-5

Goldwater J, Jardim J, Khan T, et al. Emphasizing Public Health Within a Health Information 

Exchange: An Evaluation of the District of Columbia’s Health Information Exchange Program. 

EGEMS (Wash DC). 2014;2(3) 

Greenhalgh T, Stramer K, Bratan T, et al. Adoption and non-adoption of a shared electronic 

summary record in England: a mixed-method case study. BMJ. 2010;340:c3111. PMID: 

20554687.

Grossman JM, Kushner KL, November EA. Creating sustainable local health information 

exchanges: can barriers to stakeholder participation be overcome? Res Briefs. 2008(2):1-12. 

PMID: 18496926.

Gutteridge DL, Genes N, Hwang U, et al. Enhancing a Geriatric Emergency Department Care 

Coordination Intervention Using Automated Health Information Exchange-Based Clinical Event 

Notifications. EGEMS (Wash DC). 2014;2(3) 

Hamann DJ, Bezboruah KC. Utilization of technology by long-term care providers: comparisons 

between for-profit and nonprofit institutions. J Aging Health. 2013;25(4):535-54. PMID: 

23509114.

Herwehe J, Wilbright W, Abrams A, et al. Implementation of an innovative, integrated electronic 

medical record (EMR) and public health information exchange for HIV/AIDS. J Am Med Inform 

Assoc. 2012;19(3):448-52. PMID: 22037891.

Hessler BJ, Soper P, Bondy J, et al. Assessing the relationship between health information 

exchanges and public health agencies. J Public Health Manag Pract. 2009;15(5):416-24. PMID: 

19704310.

Hincapie AL, Warholak TL, Murcko AC, et al. Physicians' opinions of a health information 

exchange. J Am Med Inform Assoc. 2011;18(1):60-5. PMID: 21106994.

Hyppönen H, Reponen J, Lääveri T, et al. User experiences with different regional health 

information exchange systems in Finland. Int J Med Inf. 2014;83(1):1-18. PMID: 24200753.

Jha AK, Doolan D, Grandt D, et al. The use of health information technology in seven nations. 

Int J Med Inf. 2008;77(12):848-54. PMID: 18657471.

Johnson KB, Gadd CS, Aronsky D, et al. The MidSouth eHealth Alliance: use and impact in the 

first year. AMIA Annu Symp Proc. 2008:333-7. PMID: 18999184.

Johnson KB, Unertl KM, Chen Q, et al. Health information exchange usage in emergency 

departments and clinics: the who, what, and why. J Am Med Inform Assoc. 2011;18(5):690-7. 

PMID: 21846788.

Jones SS, Friedberg MW, Schneider EC. Health information exchange, Health Information 

Technology use, and hospital readmission rates. AMIA Annu Symp Proc. 2011;2011:644-53. 

PMID: 22195120.
C-6

Kaelber DC, Waheed R, Einstadter D, et al. Use and perceived value of health information 

exchange: one public healthcare system's experience. Am J Manag Care. 2013;19(10 Spec 

No):Sp337-43. PMID: 24511888.

Kaushal R, Dhopeshwarkar R, Gottlieb L, et al. User experiences with pharmacy benefit 

manager data at the point of care. J Eval Clin Pract. 2010;16(6):1076-80. PMID: 20666888.

Kern LM, Barrón Y, Abramson EL, et al. HEAL NY: Promoting interoperable health 

information technology in New York State. Health Aff. 2009;28(2):493-504. PMID: 19276009.

Kern LM, Barrón Y, Dhopeshwarkar RV, et al. Health information exchange and ambulatory

quality of care. Appl Clin Inform. 2012;3(2):197-209. PMID: 23646072.

Kern LM, Wilcox A, Shapiro J, et al. Which components of health information technology will 

drive financial value? Am J Manag Care. 2012;18(8):438-45. PMID: 22928759.

Kern LM, Wilcox AB, Shapiro J, et al. Community-based health information technology 

alliances: potential predictors of early sustainability. Am J Manag Care. 2011;17(4):290-5. 

PMID: 21615199.

Kho AN, Doebbeling BN, Cashy JP, et al. A regional informatics platform for coordinated 

antibiotic-resistant infection tracking, alerting, and prevention. Clin Infect Dis. 2013;57(2):254-

62. PMID: 23575195.

Kierkegaard P, Kaushal R, Vest JR. How could health information exchange better meet the 

needs of care practitioners? Appl Clin Inform. 2014;5(4):861-77. 

Lammers EJ, Adler-Milstein J, Kocher KE. Does health information exchange reduce redundant 

imaging? Evidence from emergency departments. Med Care. 2014;52(3):227-34. PMID: 

24374414.

Lang E, Afilalo M, Vandal AC, et al. Impact of an electronic link between the emergency 

department and family physicians: a randomized controlled trial. CMAJ. 2006;174(3):313-8. 

PMID: 16399880.

Lee S-I, Park H, Kim J-W, et al. Physicians' perceptions and use of a health information 

exchange: a pilot program in South Korea. Telemed J E Health. 2012;18(8):604-12. PMID: 

22352898.

Lobach DF, Kawamoto K, Anstrom KJ, et al. Proactive population health management in the 

context of a regional health information exchange using standards-based decision support. AMIA 

Annu Symp Proc. 2007:473-7. PMID: 18693881.

Maass MC, Asikainen P, Mäenpää T, et al. Usefulness of a Regional Health Care Information 

System in primary care. A case study. Comput Methods Programs Biomed. 2008;91(2):175-81. 

PMID: 18514363.
C-7

Machan C, Ammenwerth E, Schabetsberger T. Evaluation of the electronic transmission of 

medical findings from hospitals to practitioners by triangulation. Methods Inf Med. 

2006;45(2):225-33. PMID: 16538293.

Mäenpää T, Asikainen P, Gissler M, et al. Outcomes assessment of the regional health 

information exchange: a five-year follow-up study. Methods Inf Med. 2011;50(4):308-18. 

PMID: 21336419.

Mäenpää T, Asikainen P, Gissler M, et al. The utilization rate of the regional health information 

exchange: how it impacts on health care delivery outcomes. J Public Health Manag Pract. 

2012;18(3):215-23. PMID: 22473113.

Magnus M, Herwehe J, Gruber D, et al. Improved HIV-related outcomes associated with 

implementation of a novel public health information exchange. Int J Med Inf. 2012;81(10):e30-8. 

PMID: 22883431.

Massy-Westropp M, Giles LC, Law D, et al. Connecting hospital and community care: the 

acceptability of a regional data linkage scheme. Aust Health Rev. 2005;29(1):12-6. PMID: 

15683350.

McCarthy DB, Propp K, Cohen A, et al. Learning from Health Information Exchange Technical 

Architecture and Implementation in Seven Beacon Communities. EGEMS (Wash DC). 

2014;2(1):6. 

McCullough JM, Zimmerman FJ, Bell DS, et al. Electronic health information exchange in 

underserved settings: examining initiatives in small physician practices & community health 

centers. BMC Health Serv Res. 2014;14:415. PMID: 25240718.

McGowan JJ, Jordan C, Sims T, et al. Rural RHIOs: common issues in the development of two 

state-wide health information networks. AMIA Annu Symp Proc. 2007:528-32. PMID: 

18693892.

Merrill JA, Deegan M, Wilson RV, et al. A system dynamics evaluation model: implementation 

of health information exchange for public health reporting. J Am Med Inform Assoc. 

2013;20(e1):e131-8. PMID: 23292910.

Messer LC, Parnell H, Huffaker R, et al. The development of a health information exchange to 

enhance care and improve patient outcomes among HIV+ individuals in rural North Carolina. Int 

J Med Inf. 2012;81(10):e46-55. PMID: 22898321.

Miller AR, Tucker C. Health information exchange, system size and information silos. J Health 

Econ. 2014;33:28-42. PMID: 24246484.

Miller RH. Satisfying patient-consumer principles for health information exchange: evidence 

from California case studies. Health Aff. 2012;31(3):537-47. PMID: 22392664.

Moore T, Shapiro JS, Doles L, et al. Event detection: a clinical notification service on a health 

information exchange platform. AMIA Annu Symp Proc. 2012;2012:635-42. PMID: 23304336.
C-8

Morris G, Afzal S, Bhasker M, et al. Query-Based Exchange: Key Factors Influencing Success 

and Failure. Office of the National Coordinator for Health Information Technology. 2012

Myers JJ, Koester KA, Chakravarty D, et al. Perceptions regarding the ease of use and usefulness 

of health information exchange systems among medical providers, case managers and nonclinical staff members working in HIV care and community settings. Int J Med Inf. 

2012;81(10):e21-9. PMID: 22854159.

Nagykaldi ZJ, Yeaman B, Jones M, et al. HIE-i-Health Information Exchange With Intelligence. 

J Ambulatory Care Manage. 2014;37(1):20-31. PMID: 24309392.

Nohr C, Kristensen M, Andersen SK, et al. Shared experience in 13 local Danish EPR projects: 

the Danish EPR Observatory. Stud Health Technol Inform. 2001;84(Pt 1):670-4. PMID: 

11604822.

Nykänen P, Karimaa E. Success and failure factors in the regional health information system 

design process--results from a constructive evaluation study. Methods Inf Med. 2006;45(1):85-9. 

PMID: 16482376.

Onyile A, Vaidya SR, Kuperman G, et al. Geographical distribution of patients visiting a health 

information exchange in New York City. J Am Med Inform Assoc. 2013;20(e1):e125-30. PMID: 

23104049.

Overhage JM, Evans L, Marchibroda J. Communities' readiness for health information exchange: 

the National Landscape in 2004. J Am Med Inform Assoc. 2005;12(2):107-12. PMID: 15561785.

Overhage JM, Grannis S, McDonald CJ. A comparison of the completeness and timeliness of 

automated electronic laboratory reporting and spontaneous reporting of notifiable conditions. 

Am J Public Health. 2008;98(2):344-50. PMID: 18172157.

Ozkaynak M, Brennan PF. Revisiting sociotechnical systems in a case of unreported use of 

health information exchange system in three hospital emergency departments. J Eval Clin Pract. 

2013;19(2):370-3. PMID: 22420774.

Pagliari C, Gilmour M, Sullivan F. Electronic Clinical Communications Implementation (ECCI) 

in Scotland: a mixed-methods programme evaluation. J Eval Clin Pract. 2004;10(1):11-20. 

PMID: 14731147.

Park H, Lee S-i, Kim Y, et al. Patients' perceptions of a health information exchange: a pilot 

program in South Korea. Int J Med Inf. 2013;82(2):98-107. PMID: 22658777.

Patel V, Swain MJ, King J, et al. Physician capability to electronically exchange clinical 

information, 2011. Am J Manag Care. 2013;19(10):835-43. PMID: 24304162.

Phillips AB, Wilson RV, Kaushal R, et al. Implementing health information exchange for public 

health reporting: a comparison of decision and risk management of three regional health 

information organizations in New York state. J Am Med Inform Assoc. 2014;21(e1):e173-7. 

PMID: 23975626.
C-9

Pirnejad H, Bal R, Berg M. Building an inter-organizational communication network and 

challenges for preserving interoperability. Int J Med Inf. 2008;77(12):818-27. PMID: 18579436.

Pouloudi A. Information technology for collaborative advantage in healthcare revisited. 

Information & Management. 1999;35(6):345-56. 

Ross SE, Radcliff TA, Leblanc WG, et al. Effects of health information exchange adoption on 

ambulatory testing rates. J Am Med Inform Assoc. 2013;20(6):1137-42. PMID: 23698257.

Ross SE, Schilling LM, Fernald DH, et al. Health information exchange in small-to-medium 

sized family medicine practices: motivators, barriers, and potential facilitators of adoption. Int J 

Med Inf. 2010;79(2):123-9. PMID: 20061182.

Rudin R, Volk L, Simon S, et al. What Affects Clinicians' Usage of Health Information 

Exchange? Appl Clin Inform. 2011;2(3):250-62. PMID: 22180762.

Rudin RS, Simon SR, Volk LA, et al. Understanding the decisions and values of stakeholders in 

health information exchanges: experiences from Massachusetts. Am J Public Health. 

2009;99(5):950-5. PMID: 19299671.

Saff E, Lanway C, Chenyek A, et al. The Bay Area HIE. A case study in connecting 

stakeholders. J Healthc Inf Manag. 2010;24(1):25-30. PMID: 20077922.

Schabetsberger T, Ammenwerth E, Andreatta S, et al. From a paper-based transmission of 

discharge summaries to electronic communication in health care regions. Int J Med Inf. 

2006;75(3-4):209-15. PMID: 16112892.

Schoen C, Osborn R, Squires D, et al. A survey of primary care doctors in ten countries shows 

progress in use of health information technology, less in other areas. Health Aff (Millwood). 

2012;31(12):2805-16. PMID: 23154997.

Shapiro JS, Johnson SA, Angiollilo J, et al. Health Information Exchange Improves 

Identification Of Frequent Emergency Department Users. Health Aff. 2013;32(12):2193-8. 

PMID: 24301405.

Sicotte C, Paré G. Success in health information exchange projects: solving the implementation 

puzzle. Soc Sci Med. 2010;70(8):1159-65. PMID: 20137847.

Silvester BV, Carr SJ. A shared electronic health record: lessons from the coalface. Med J Aust. 

2009;190(11 Suppl):S113-6. PMID: 19485857.

Soderberg K, Laventure M. Minnesota clinics' adoption, use and exchange of electronic health 

information. Minn Med. 2013;96(9):45-8. PMID: 24494362.

Steward WT, Koester KA, Collins SP, et al. The essential role of reconfiguration capabilities in 

the implementation of HIV-related health information exchanges. Int J Med Inf. 

2012;81(10):e10-20. PMID: 22841703.
C-10

Swain M, Charles D, Patel V, et al. Health Information Exchange among U.S. Non-federal Acute 

Care Hospitals: 2008-2014. ONC Data Brief No. 24. Washington DC: The Office of the national 

Coordinator for Health Information Technology (ONC). 2015. Available at: 

http://www.healthit.gov/sites/default/files/data-brief/ONC_DataBrief24_HIE_Final.pdf. 

Accessed April 19, 2015. 

Thorn SA, Carter MA, Bailey JE. Emergency physicians' perspectives on their use of health 

information exchange. Ann Emerg Med. 2014;63(3):329-37. PMID: 24161840.

Tripathi M, Delano D, Lund B, et al. Engaging patients for health information exchange. Health 

Aff. 2009;28(2):435-43. PMID: 19276000.

Tzeel A, Lawnicki V, Pemble KR. The Business Case for Payer Support of a Community-Based 

Health Information Exchange: A Humana Pilot Evaluating Its Effectiveness in Cost Control for 

Plan Members Seeking Emergency Department Care. Am Health Drug Benefits. 2011;4(4):207-

15. PMID: 25126351.

Tzeel A, Lawnicki V, Pemble KR. "Hidden" Value: How Indirect Benefits of Health Information 

Exchange Further Promote Sustainability. Am Health Drug Benefits. 2012;5(6):333-40. PMID: 

24991331.

Unertl KM, Johnson KB, Lorenzi NM. Health information exchange technology on the front 

lines of healthcare: workflow factors and patterns of use. J Am Med Inform Assoc. 

2012;19(3):392-400. PMID: 22003156.

Unertl MK, Johnson BK, Gadd SC, et al. Bridging Organizational Divides in Health Care: An 

Ecological View of Health Information Exchange. JMIR Med Inform. 2013;1(1):e3. 

Vest JR. Health information exchange and healthcare utilization. J Med Syst. 2009;33(3):223-31. 

PMID: 19408456.

Vest JR. More than just a question of technology: factors related to hospitals' adoption and 

implementation of health information exchange. Int J Med Inf. 2010;79(12):797-806. PMID: 

20889370.

Vest JR, Campion Jr TR, Kaushal R. Challenges, Alternatives, and Paths to Sustainability for 

Health Information Exchange Efforts. J Med Syst. 2013;37(6):1-8. PMID: 24141531.

Vest JR, Gamm LD, Ohsfeldt RL, et al. Factors associated with health information exchange 

system usage in a safety-net ambulatory care clinic setting. J Med Syst. 2012;36(4):2455-61. 

PMID: 21523428.

Vest JR, Grinspan ZM, Kern LM, et al. Using a health information exchange system for imaging 

information: patterns and predictors. AMIA Annu Symp Proc. 2013;2013:1402-11. PMID: 

24551416.

Vest JR, Issel LM. Factors related to public health data sharing between local and state health 

departments. Health Serv Res. 2014;49(1 Pt 2):373-91. PMID: 24359636.
C-11

Vest JR, Jasperson JS. How are health professionals using health information exchange systems? 

Measuring usage for evaluation and system improvement. J Med Syst. 2012;36(5):3195-204. 

PMID: 22127521.

Vest JR, Jasperson JS, Zhao H, et al. Use of a health information exchange system in the 

emergency care of children. BMC Med Inform Decis Mak. 2011;11:78. PMID: 22208182.

Vest JR, Kern LM, Campion TR, Jr., et al. Association between use of a health information 

exchange system and hospital admissions. Appl Clin Inform. 2014;5(1):219-31. PMID: 

24734135.

Vest JR, Kern LM, Silver MD, et al. The potential for community-based health information 

exchange systems to reduce hospital readmissions. J Am Med Inform Assoc. 2014 PMID: 

25100447.

Vest JR, Miller TR. The association between health information exchange and measures of 

patient satisfaction. Appl Clin Inform. 2011;2(4):447-59. PMID: 23616887.

Vest JR, Zhao H, Jaspserson J, et al. Factors motivating and affecting health information 

exchange usage. J Am Med Inform Assoc. 2011;18(2):143-9. PMID: 21262919.

Willis JM, Edwards R, Anstrom KJ, et al. Decision support for evidence-based pharmacotherapy 

detects adherence problems but does not impact medication use. Stud Health Technol Inform. 

2013;183:116-25. PMID: 23388267.

Winden TJ, Boland LL, Frey NG, et al. Care everywhere, a point-to-point HIE tool: utilization 

and impact on patient care in the ED. Appl Clin Inform. 2014;5(2):388-401. PMID: 25024756.

Yeager VA, Walker D, Cole E, et al. Factors Related to Health Information Exchange 

Participation and Use. J Med Syst. 2014;38(8) PMID: 24957395.
D-1

Appendix D. List of Excluded Studies

Report on community health information exchanges. Medicine on the Net. 2004;10(3):9-9. 

Exclusion: Wrong study design

Connecting communities: making inroads to exchange electronic healthcare data at the local level. Qual Lett Healthc 

Lead. 2005;17(8):2-10. PMID: 16304880.

Exclusion: Wrong study design

Implementation of SNOMED-CT needed to facilitate interoperable exchange of health information. J AHIMA. 

2005;76(9):30, 2. 

Exclusion: Wrong study design

A primer for building RHIOs. Hosp Health Netw. 2006;80(2):49-56. PMID: 16572948.

Exclusion: Wrong study design

Health information exchange activities continuing to mature, says survey. Healthc Financ Manage. 2007;61(2):11. 

Exclusion: Wrong study design

New computer network helps EDs to reduce redundant test orders: observers see significant savings, benefits in 

patient safety. ED Manag. 2008;20(12):133-4. PMID: 19086738.

Exclusion: Wrong study design

Wisconsin HIE optimizes community care. Communication among ED clinicians and federally qualified health 

centers in the Milwaukee area was improved, including real-time access to patient historical-encounter data. Health 

Manag Technol. 2009;30(12):28-9. PMID: 20043491.

Exclusion: Wrong study design

States with the most health information exchanges. Mod Healthc. 2009;39(27):32. PMID: 19606671.

Exclusion: Wrong study design

By the numbers. States with the most health information exchanges. Based on eHealth initiative's directory of health 

information exchange initiatives. Mod Healthc. 2010;40(14):34. PMID: 20402215.

Exclusion: Wrong study design

Physicians support health information exchange but are concerned about paying monthly fees. AHRQ Research 

Activities. 2010(359):14. 

Exclusion: Wrong study design

Information exchange yields better decisions. ED Manag. 2010;22(9):103-4. PMID: 20853581.

Exclusion: No data relevant to a Key Question

Social Security report details $2 million return on HIE. For the Record (Great Valley Publishing Company, Inc). 

2010;22(4):6. 

Exclusion: Wrong study design

Findings from site visit to community clinic health network in san Diego, CA. Available at: 

http://aspe.hhs.gov/sp/reports/2010/chcit2010/SanDiego.html. Accessed November 10, 2014. 

Exclusion: Wrong study design

States with the most health information exchanges. Based on ehealth initiative's map of health information exchange 

activity in the U.S. Mod Healthc. 2011;41(21):32. PMID: 21714447.

Exclusion: Wrong study design

By The Numbers: States with the most health information exchanges. Mod Healthc. 2011;41(21):32. 
D-2

Exclusion: Wrong study design

High-tech approach to medication reconciliation saves time, bolsters safety at hospital in northern Virginia. ED 

Manag. 2011;23(10):117-9. PMID: 21972757.

Exclusion: Not HIE

Survey shows health information exchange on the rise. For the Record (Great Valley Publishing Company, Inc). 

2011;23(15):5. 

Exclusion: Wrong study design

States with the most health information exchanges: Based on eHealth initiative's map of health information 

exchange activity in the U.S. Mod Healthc. 2012;42(24):34. PMID: 22957359.

Exclusion: Wrong study design

What is HIE (Health Information Exchange)? Department of Health and Human Services. Washington, DC. 

Available at: http://www.healthit.gov/providers-professionals/health-information-exchange/what-hie. Accessed 

April 18, 2014. 

Exclusion: No data relevant to a Key Question

Health Information Exchange Roadmap: The Landscape and a Path Forward. National eHealth Collaborative. 

Washington, DC. Available at: http://www.nationalehealth.org/hie-roadmap. Accessed April 18, 2014. 

Exclusion: No data relevant to a Key Question

Study: More Hospitals Joining Health Information Exchanges. J AHIMA. 2012;83(11):13. 

Exclusion: Wrong study design

Electronic tools for health information exchange: An evidence-based analysis. Ont Health Technol Assess Ser. 

2013;13(11):1-76. PMID: 2419479.

Exclusion: Systematic review not meeting our requirements 

Health Information Exchange May Reduce Hospital Admissions. For the Record (Great Valley Publishing 

Company, Inc). 2014;26(5):32. 

Exclusion: Wrong study design

Aas IHM, Geitung JT. Choosing networks for picture archiving and communication systems and teleradiology. J 

Telemed Telecare. 2003;9 Suppl 1:S27-9. PMID: 12952712.

Exclusion: Not HIE

Aas IM. The organizational challenge for health care from telemedicine and e-health. Oslo: Work Res Inst. 2007

Exclusion: Not HIE

Adler-Milstein J, DesRoches CM, Furukawa MF, et al. More than half of US hospitals have at least a basic EHR, 

but stage 2 criteria remain challenging for most. Health Aff (Millwood). 2014;33(9):1664-71. PMID: 25104826.

Exclusion: Not HIE

Afzal S, Morris G, Palmer S. Health Information Exchange Services in Support of Disaster Preparedness and 

Emergency Medical Response: Assessment of Opportunity in California and the Gulf Coast: Office of the National 

Coordinator for Health Information Technology; 2014. 

Exclusion: Wrong study design

Agarwal M, Bourgeois J, Sodhi S, et al. Updating a patient-level ART database covering remote health facilities in 

Zomba district, Malawi: Lessons learned. Public Health Action. 2013;3(2):175-9. 

Exclusion: Not HIE

Ahern DK, Kreslake JM, Phalen JM. What is eHealth (6): perspectives on the evolution of eHealth research. J Med 

Internet Res. 2006;8(1):e4. PMID: 16585029.
D-3

Exclusion: Not HIE

Ahmed S, Bartlett SJ, Ernst P, et al. Effect of a web-based chronic disease management system on asthma control 

and health-related quality of life: study protocol for a randomized controlled trial. Trials. 2011;12:260. PMID: 

22168530.

Exclusion: Wrong study design

Allen A, Des Jardins TR, Heider A, et al. Making it local: Beacon communities use health information technology to 

optimize care management. Popul Health Manag. 2014;17(3):149-58. PMID: 24476558.

Exclusion: Wrong study design

Allen C, Des Jardins TR, Heider A, et al. Data Governance and Data Sharing Agreements for Community-Wide 

Health Information Exchange: Lessons from the Beacon Communities. EGEMS (Wash DC). 2014;2(1)

Exclusion: No data relevant to a Key Question

Allen KA. Parent and Provider Decision-Making for Infants with HIE, Duke University; 2012.

Exclusion: Not HIE

Allender S, Nichols M, Foulkes C, et al. The development of a network for community-based obesity prevention: 

the CO-OPS Collaboration. BMC Public Health. 2011;11:132. PMID: 21349185.

Exclusion: Not HIE

Ancker JS, Edwards AM, Miller MC, et al. Consumer perceptions of electronic health information exchange. Am J 

Prev Med. 2012;43(1):76-80. PMID: 22704751.

Exclusion: Not HIE

Ancker JS, Miller MC, Patel V, et al. Sociotechnical challenges to developing technologies for patient access to 

health information exchange data. J Am Med Inform Assoc. 2014;21(4):664-70. PMID: 24064443.

Exclusion: No data relevant to a Key Question

Ancker JS, Silver M, Miller MC, et al. Consumer experience with and attitudes toward health information 

technology: a nationwide survey. J Am Med Inform Assoc. 2013;20(1):152-6. PMID: 22847306.

Exclusion: Not HIE

Anderson JG. Social, ethical and legal barriers to e-health. Int J Med Inf. 2007;76(5-6):480-3. PMID: 17064955.

Exclusion: Not HIE

Andrade SE, Davis RL, Cheetham TC, et al. Medication Exposure in Pregnancy Risk Evaluation Program. Matern 

Child Health J. 2012;16(7):1349-54. PMID: 22002179.

Exclusion: Not HIE

Angiollilo J, Fleischman W, Kuperman G, et al. Improving identification of hospital readmissions using a regional 

health information exchange. Acad Emerg Med. 2012;19:S50. 

Exclusion: Wrong study design

Angst CM. Protect my privacy or support the common-good? Ethical questions about electronic health information 

exchanges. J Bus Ethics. 2009;90(Suppl 2):169-78. 

Exclusion: Wrong study design

Angulo C, Crespo P, Maldonado JA, et al. Non-invasive lightweight integration engine for building EHR from 

autonomous distributed systems. Int J Med Inform. 2007;76 Suppl 3:S417-24. PMID: 17600763.

Exclusion: Wrong study design

Appleby C. NYCLIX: New York HIE life. An expansive HIE network has taken shape in the nation's most densely 

populated urban area. Healthc Inform. 2010;27(10):29-31. PMID: 21049716.

Exclusion: Wrong study design
D-4

Appleby C. Surfing the HIE. The Santa Cruz information exchange experience offers lessons on what works. 

Healthc Inform. 2010;27(6):68-9. PMID: 20593734.

Exclusion: Wrong study design

Arar NH, Wen L, McGrath J, et al. Communicating about medications during primary care outpatient visits: the role 

of electronic medical records. Inform Prim Care. 2005;13(1):13-22. PMID: 15949171.

Exclusion: Not HIE

Asangansi I, Braa K. The emergence of mobile-supported national health information systems in developing 

countries... MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics, Part 1. Stud Health 

Technol Inform. 2010;160:540-4. 

Exclusion: No data relevant to a Key Question

Aschman DJ, Abshire TC, Shapiro AD, et al. A community-based partnership to promote information infrastructure 

for bleeding disorders. Am J Prev Med. 2011;41(6 Suppl 4):S332-7. PMID: 22099355.

Exclusion: No data relevant to a Key Question

Ash JS, Guappone KP. Qualitative evaluation of health information exchange efforts. J Biomed Inform. 2007;40(6 

Suppl):S33-9. PMID: 17904914.

Exclusion: Wrong study design

Ashley L, Jones H, Forman D, et al. Feasibility test of a UK-scalable electronic system for regular collection of 

patient-reported outcome measures and linkage with clinical cancer registry data: The electronic Patient-reported 

Outcomes from Cancer Survivors (ePOCS) system. BMC Med Inform Decis Mak. 2011;11:66. PMID: 22029686.

Exclusion: Wrong study design

Badia CM, Duenas AE, Martinez OM, et al. My health log. Eur J Intern Med. 2011;22:S56-S7. 

Exclusion: Wrong study design

Bah S, Alharthi H, El Mahalli AA, et al. Annual Survey on the Level and Extent of Usage of Electronic Health 

Records in Government-related Hospitals. Perspect Health Inf Manag. 2011;8(4):1-12. PMID: 22016668.

Exclusion: Not HIE

Bailey JE, Wan J, Pope R, et al. Health information exchange use reduces avoidable diagnostic imaging in the 

emergency evaluation of back pain. J Gen Intern Med. 2011;26(University of Tennessee Health Science Center, 

Memphis, United States):S349-S50. 

Exclusion: Wrong study design

Bailey JE, Yu X, Ward RD, et al. Effect of health information exchange on hospital admissions for chest pain. J 

Investig Med. 2012;60(1):465-6. 

Exclusion: Wrong study design

Balas A, Al Sanousi A. Interoperable electronic patient records for health care improvement. Stud Health Technol 

Inform. 2009;150:19-23. PMID: 19745258.

Exclusion: Wrong study design

Balasingham I, Ihlen H, Leister W, et al. Communication of medical images, text, and messages in inter-enterprise 

systems: a case study in Norway. IEEE Trans Inf Technol Biomed. 2007;11(1):7-13. PMID: 17249398.

Exclusion: Wrong study design

Balka E, Tolar M, Coates S, et al. Socio-technical issues and challenges in implementing safe patient handovers: 

Insights from ethnographic case studies. Int J Med Inf. 2013;82(12):e345-e57. PMID: 23218926.

Exclusion: Not HIE
D-5

Ball MJ, Gold J. Banking on health: Personal records and information exchange. J Healthc Inf Manag. 

2006;20(2):71-83. PMID: 16669591.

Exclusion: Wrong study design

Ballard J, Rosenman M, Weiner M. Harnessing a health information exchange to identify surgical device adverse 

events for urogynecologic mesh. AMIA Annu Symp Proc. 2012;2012:1109-18. PMID: 23304387.

Exclusion: Wrong study design

Bansal M, Grannis S, Kansky J, et al. Evaluating cost differences among operational teams supporting the Indiana 

health information exchange. Value Health. 2009;12(3):A87. 

Exclusion: No data relevant to a Key Question

Bansler JP, Havn E. Pilot implementation of health information systems: Issues and challenges. Int J Med Inf. 

2010;79(9):637-48. PMID: 20576466.

Exclusion: Wrong study design

Bara D, McPhillips-Tangum C, Wild EL, et al. Integrating child health information systems in public health

agencies. J Public Health Manag Pract. 2009;15(6):451-8. PMID: 19823148.

Exclusion: Not HIE

Barbarito F, Pinciroli F, Mason J, et al. Implementing standards for the interoperability among healthcare providers 

in the public regionalized Healthcare Information System of the Lombardy Region. J Biomed Inform. 

2012;45(4):736-45. PMID: 22285983.

Exclusion: Wrong study design

Barrows RC, Jr., Ezzard J. Technical architecture of ONC-approved plans for statewide health information 

exchange. AMIA Annu Symp Proc. 2011;2011:88-97. PMID: 22195059.

Exclusion: No data relevant to a Key Question

Basch P. Will interoperable HIT lead to a net gain or to a net loss for physicians? (2/23/2005). Health Aff 

(Millwood). 2005;Suppl Web Exclusives:W5-S-1-W5-S-3; author reply W5-S-3-W5-S-6. PMID: 16440450.

Exclusion: Wrong study design

Bassi J, Lau F. Measuring value for money: a scoping review on economic evaluation of health information systems. 

J Am Med Inform Assoc. 2013;20(4):792-801. PMID: 23416247.

Exclusion: Not HIE

Bates DW, Gawande AA. Improving Safety with Information Technology. N Engl J Med. 2003;348(25):2526-34. 

PMID: 12815139.

Exclusion: Not HIE

Beaulieu-Volk D. EHRs' interoperability challenge. HIE expansion aimed at helping providers exchange health 

information safely, but not all services created equally. Med Econ. 2014;91(6):50-3. PMID: 25219166.

Exclusion: Wrong study design

Beckjord EB, Rechis R, Nutt S, et al. What Do People Affected by Cancer Think About Electronic Health 

Information Exchange? Results From the 2010 LIVESTRONG Electronic Health Information Exchange Survey and 

the 2008 Health Information National Trends Survey. J Oncol Pract. 2011;7(4):237-41. PMID: 22043188.

Exclusion: Wrong study design

Bell DS, Cima L, Seiden DS, et al. Effects of laboratory data exchange in the care of patients with HIV. Int J Med 

Inf. 2012;81(10):e74-82. PMID: 22906370.

Exclusion: Not HIE

Ben-Assuli O, Shabtai I, Leshno M. Using electronic health record systems to optimize admission decisions: The 

Creatinine case study. Health Informatics J. 2014. PMID: 24692078.
D-6

Exclusion: More recent data available

Ben-Assuli O, Shabtai I, Leshno M, et al. EHR in emergency rooms: Exploring the effect of key information 

components on main complaints. J Med Syst. 2014;38(4). PMID: 24687240.

Exclusion: No comparison group

Benford MS, Slack CB. Development of a statewide maternal and child health information network... MATCH. 

Comput Nurs. 1989;7(1):9-14. PMID: 2924201.

Exclusion: Wrong study design

Bergmann J, Bott OJ, Pretschner DP, et al. An e-consent-based shared EHR system architecture for integrated 

healthcare networks. Int J Med Inform. 2007;76(2-3):130-6. PMID: 16971171.

Exclusion: Wrong study design

Berry JG, Goldmann DA, Mandl KD, et al. Health information management and perceptions of the quality of care 

for children with tracheotomy: a qualitative study. BMC Health Serv Res. 2011;11:117. PMID: 21605385.

Exclusion: Not HIE

Beynon-Davies P, Lloyd-Williams M. When health information systems fail. Top Health Inf Manage. 

1999;20(1):66-79. PMID: 10539424.

Exclusion: Wrong study design

Biondich PG, Grannis SJ. The Indiana Network for Patient Care: an integrated clinical information system informed 

by over thirty years of experience. J Public Health Manag Pract. 2004:S81-6. PMID: 15643364.

Exclusion: No data relevant to a Key Question

Bipartisan Policy Center. Clinician Perspectives on Electronic Health Information Sharing for Transitions of Care 

2012. Available at: https://www.acponline.org/running_practice/technology/bpc_clinician_survey_100312.pdf. 

Accessed November 10, 2014. 

Exclusion: Wrong study design

Black CD, Burchill CA, Roos LL. The Population Health Information System: data analysis and software. Med 

Care. 1995;33(12 Suppl):DS127-31. PMID: 7500665.

Exclusion: Wrong study design

Blaya JA, Shin SS, Yagui MJA, et al. A web-based laboratory information system to improve quality of care of 

tuberculosis patients in Peru: Functional requirements, implementation and usage statistics. BMC Med Inform Decis 

Mak. 2007;7. PMID: 17963522.

Exclusion: Not HIE

Blobel B. Standards and solutions for architecture based, ontology driven and individualized pervasive health. Stud 

Health Technol Inform. 2012;177:147-57. PMID: 22942047.

Exclusion: Not HIE

Bohren BF, Hadzikadic M. Turning medical data into decision-support knowledge. Proc Annu Symp Comput Appl 

Med Care. 1994:735-9. PMID: 7950022.

Exclusion: Not HIE

Bonney W. Determinants in the acceptance of Health Level Seven (HL7) version 3 messaging standard. Diss Abstr 

Int. 2013;73(12-B(E)).

Exclusion: Not HIE

Boockvar KS, Livote EE, Goldstein N, et al. Electronic health records and adverse drug events after patient transfer. 

Qual Saf Health Care. 2010;19(5):e16. PMID: 20724395.

Exclusion: Not HIE
D-7

Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic 

review to taxonomy and interventions. BMC Health Serv Res. 2010;10:231. PMID: 20691097.

Exclusion: Not HIE

Bouhaddou O, Bennett J, Teal J, et al. Toward a Virtual Lifetime Electronic Record: the Department of Veterans 

Affairs experience with the Nationwide Health Information Network. AMIA Annu Symp Proc. 2012;2012:51-60. 

PMID: 23304272.

Exclusion: No data relevant to a Key Question

Bouhaddou O, Cromwell T, Davis M, et al. Translating standards into practice: experience and lessons learned at the 

Department of Veterans Affairs. J Biomed Inform. 2012;45(4):813-23. PMID: 22285982.

Exclusion: Not HIE

Bouhaddou O, Warnekar P, Parrish F, et al. Exchange of computable patient data between the Department of 

Veterans Affairs (VA) and the Department of Defense (DoD): terminology mediation strategy. J Am Med Inform 

Assoc. 2008;15(2):174-83. PMID: 18096911.

Exclusion: No data relevant to a Key Question

Bourn M, Davies CA. A prodigious information systems failure. Top Health Inf Manage. 1996;17(2):34-44. PMID: 

10162539.

Exclusion: Wrong study design

Bowen R, Carey S, Carter P, et al. HIE management and operational considerations. J AHIMA. 2011;82(5):56-61. 

PMID: 21667869.

Exclusion: Wrong study design

Brailer DJ. Connection tops collection: peer-to-peer technology lets caregivers access necessary data, upon request, 

without using a repository. Health Manag Technol. 2001;22(8):28-9. PMID: 11499130.

Exclusion: Wrong study design

Brailer DJ. From Santa Barbara to Washington: a person's and a nation's journey toward portable health information. 

Health Aff. 2007;26(5):w581-8. PMID: 17670776.

Exclusion: Wrong study design

Branger PJ, van't Hooft A, van der Wouden JC, et al. Shared care for diabetes: supporting communication between 

primary and secondary care. Int J Med Inf. 1999;53(2-3):133-42. PMID: 10193883.

Exclusion: Not HIE

Brattheim B, Faxvaag A, Toussaint P. When information sharing is not enough. Stud Health Technol Inform. 

2011;169:359-63. PMID: 21893773.

Exclusion: Wrong study design

Brelstaff G, Moehrs S, Anedda P, et al. Internet patient records: new techniques. J Med Internet Res. 2001;3(1):E8. 

PMID: 11720950.

Exclusion: Not HIE

Brennan CP. Managed care and health information networks. J Health Care Finance. 1995;21(4):1-5. PMID: 

7583779.

Exclusion: No data relevant to a Key Question

Brocht DF, Abbott PA, Smith CA, et al. A clinic on wheels. A paradigm shift in the provision of care and the 

challenges of information infrastructure. Comput Nurs. 1999;17(3):109-13. PMID: 10341475.

Exclusion: Wrong study design

Brokel JM. Capture, exchange and use data, information and knowledge within electronic health records. Iowa 

Nurse Reporter. 2007;20(1):1, 25, 7. 
D-8

Exclusion: Wrong study design

Brokel JM. Regional health information organization (RHIO) to exchange data. Iowa Nurse Reporter. 2007;20(2):4-

5. 

Exclusion: Wrong study design

Brokel JM. Iowa e-health project: planning for health information exchange with nursing standardized language 

with health information technology tools. Iowa Nurse Reporter. 2009;22(3):1. 

Exclusion: Wrong study design

Brokel JM, Harrison MI. Redesigning care processes using an electronic health record: a system's experience. Jt 

Comm J Qual Patient Saf. 2009;35(2):82-92. PMID: 19241728.

Exclusion: Not HIE

Brown CVR, Foulkrod KH, Sadler HT, et al. Autologous blood transfusion during emergency trauma operations. 

Arch Surg. 2010;145(7):690-4. PMID: 20644133.

Exclusion: Not HIE

Brown JS, Holmes JH, Shah K, et al. Distributed health data networks: a practical and preferred approach to multiinstitutional evaluations of comparative effectiveness, safety, and quality of care. Med Care. 2010;48(6 Suppl):S45-

51. PMID: 20473204.

Exclusion: No data relevant to a Key Question

Brown ML, Riley GF, Potosky AL, et al. Obtaining long-term disease specific costs of care: application to Medicare 

enrollees diagnosed with colorectal cancer. Med Care. 1999;37(12):1249-59. PMID: 10599606.

Exclusion: Not HIE

Buntin MB, Burke MF, Hoaglin MC, et al. The benefits of health information technology: a review of the recent 

literature shows predominantly positive results. Health Aff (Millwood). 2011;30(3):464-71. PMID: 21383365.

Exclusion: Systematic review not meeting our requirements 

Burkle T, Schweiger R, Altmann U, et al. Transferring data from one EPR to another: content--syntax--semantic. 

Methods Inf Med. 1999;38(4-5):321-5. PMID: 10805022.

Exclusion: Wrong study design

Burstin H, Clancy C. Primary care experience: crossing the chasm between theory and practice. J Gen Intern Med. 

2004;19(10):1064-5. PMID: 15482561.

Exclusion: Not HIE

Butler B. Health Information Exchange between Jails and Their Communities: A Bridge That Is Needed under 

Healthcare Reform. Perspect Health Inf Manag. 2014:1-6. PMID: 24808809.

Exclusion: Wrong study design

Caldwell D. Health information exchange. MLO Med Lab Obs. 2012;44(11):46. PMID: 23173526.

Exclusion: Wrong study design

Caldwell D. Management Q&A. Health information exchange. MLO Med Lab Obs. 2012;44(11):46. 

Exclusion: Wrong study design

Callen J, Paoloni R, Li J, et al. Perceptions of the effect of information and communication technology on the 

quality of care delivered in emergency departments: a cross-site qualitative study. Ann Emerg Med. 2013;61(2):131-

44. PMID: 23083964.

Exclusion: No data relevant to a Key Question

Callen JL, Braithwaite J, Westbrook JI. Contextual implementation model: a framework for assisting clinical 

information system implementations. J Am Med Inform Assoc. 2008;15(2):255-62. PMID: 18096917.
D-9

Exclusion: Not HIE

Carr CM, Krywko DM, Moore HE, et al. The impact of a health information exchange on the management of 

patients in an urban academic emergency department: An observational study and cost analysis. Ann Emerg Med. 

2012;60(4):S15. 

Exclusion: Wrong study design

Carr CM, Saef SH, Zhao J, et al. Can data from a health information exchange be used to describe patients who visit 

multiple emergency departments within a region? Acad Emerg Med. 2014;21(5):S141. 

Exclusion: Not HIE

Carr D, Howells A, Chang M, et al. An integrated approach to stakeholder engagement. Healthc Q. 2009;12 Spec 

No Ontario:62-70. PMID: 19458512.

Exclusion: Not HIE

Carr K, Bangalore D, Benin A, et al. Leveraging the benefits of Health Information Technology to support 

healthcare delivery model redesign. J Healthc Inf Manag. 2006;20(1):31-41. PMID: 16429957.

Exclusion: Not HIE

Cebul RD, Love TE, Jain AK, et al. Electronic health records and quality of diabetes care. N Engl J Med. 

2011;365(9):825-33. PMID: 21879900.

Exclusion: Not HIE

Centers for Disease C, Prevention. State electronic disease surveillance systems --- United States, 2007 and 2010. 

MMWR Morb Mortal Wkly Rep. 2011;60(41):1421-3. PMID: 22012115.

Exclusion: Not HIE

Centorrino F, Mark TL, Talamo A, et al. Health and economic burden of metabolic comorbidity among individuals 

with bipolar disorder. J Clin Psychopharmacol. 2009;29(6):595-600. PMID: 19910727.

Exclusion: Not HIE

Champagne T. The development of community-based health information exchanges: A comparative assessment of 

organizational models. Diss Abstr Int. 2014;75(5-B(E)):No Pagination Specified. 

Exclusion: Wrong study design

Chan TC, Killeen JP, Castillo EM, et al. San diego safety net health information exchange. Ann Emerg Med. 

2011;58(4):S310. 

Exclusion: Wrong study design

Chang I, Hwang H-G, Hung M-C, et al. Factors affecting cross-hospital exchange of Electronic Medical Records. 

Information & Management. 2009;46(2):109-15. 

Exclusion: Not HIE

Chau PYK, Hu PJH. Information technology acceptance by individual professionals: A model comparison approach. 

Decision Sciences. 2001;32(4):699-718. 

Exclusion: Not HIE

Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, 

efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-52. PMID: 16702590.

Exclusion: Systematic review not meeting our requirements 

Chen C, Garrido T, Chock D, et al. The Kaiser Permanente electronic health record: Transforming and streamlining 

modalities of care. Health Aff. 2009;28(2):323-33. PMID: 19275987.

Exclusion: Not HIE
D-10

Chen R, Enberg G, Klein GO. Julius--a template based supplementary electronic health record system. BMC Med 

Inform Decis Mak. 2007;7:10. PMID: 17474997.

Exclusion: No data relevant to a Key Question

Cheung K-C, van der Veen W, Bouvy ML, et al. Classification of medication incidents associated with information 

technology. J Am Med Inform Assoc. 2014;21(e1):e63-70. PMID: 24064444.

Exclusion: Not HIE

Cimino JJ, Frisse ME, Halamka J, et al. Consumer-mediated health information exchanges: The 2012 ACMI debate. 

J Biomed Inform. 2014;48:5-15. PMID: 24561078.

Exclusion: Wrong study design

Ciriello JN, Kulatilaka N. Smart health community: the hidden value of health information exchange. Am J Manag 

Care. 2010;16(12 Suppl HIT):SP31-6. PMID: 21314218.

Exclusion: Wrong study design

Clancy GP, Duffy FD. Going "all in" to transform the Tulsa community's health and health care workforce. Acad 

Med. 2013;88(12):1844-8. PMID: 24128637.

Exclusion: Wrong study design

Clayton PD, Narus SP, Huff SM, et al. Building a comprehensive clinical information system from components. The 

approach at Intermountain Health Care. Methods Inf Med. 2003;42(1):1-7. PMID: 12695790.

Exclusion: Wrong study design

Coffman T, Porter JP, Frisse ME. Reducing HIE costs through real-time data feed visualizations. AMIA Annu Symp 

Proc. 2008:913. PMID: 18999214.

Exclusion: Not HIE

Coiera E. Building a National Health IT System from the middle out. J Am Med Inform Assoc. 2009;16(3):271-3. 

PMID: 19407078.

Exclusion: Wrong study design

Collaborative MT. Advanced technologies to lower health care costs and improve quality: Executive summary. 

Available at: http://mehi.masstech.org/sites/mehi/files/documents/AdvancedTechnologies2004.pdf. Accessed April 

22, 2015. 

Exclusion: More recent data available

Collin S, Reeves BC, Hendy J, et al. Implementation of computerised physician order entry (CPOE) and picture 

archiving and communication systems (PACS) in the NHS: quantitative before and after study. BMJ. 

2008;337(a939):1-8. PMID: 18703655.

Exclusion: Not HIE

Collins SA, Bakken S, Vawdrey DK, et al. Model development for EHR interdisciplinary information exchange of 

ICU common goals. Int J Med Inf. 2011;80(8):e141-9. PMID: 20974549.

Exclusion: Not HIE

Conn J. RHIOs make it work. Data-sharing project connects three networks. Mod Healthc. 2006;36(7):22. PMID: 

16515062.

Exclusion: Wrong study design

Constantinides P, Barrett M. Large-scale ICT innovation, power, and organizational change: The case of a regional 

health information network. J Appl Behav Sci. 2006;42(1):76-90. 

Exclusion: Wrong study design

Corado C, Cashy J, Kho A, et al. Fragmented care among stroke patients at 4 Chicago hospitals. Stroke. 2014;45

Exclusion: Not HIE
D-11

Cormont S, Vandenbussche P-Y, Buemi A, et al. Implementation of a platform dedicated to the biomedical analysis 

terminologies management. AMIA Annu Symp Proc. 2011;2011:1418-27. PMID: 22195205.

Exclusion: Not HIE

Corporation RHI. Overlay Regional Health Information Exchange (HIE) Systems: The Sustainable Business Model 

for Health Care Information Technology in the United States. Available at: 

http://ruralhealthit.com/downloads/Overlay_Regional_Health_Information_Exchange_Systems.pdf. Accessed April 

22, 2015. 

Exclusion: Wrong study design

Costa C, Ferreira C, Bastiao L, et al. Dicoogle - an open source peer-to-peer PACS. J Digit Imaging. 

2011;24(5):848-56. PMID: 20981467.

Exclusion: Not HIE

Cresswell K, Sheikh A. The NHS Care Record Service (NHS CRS): recommendations from the literature on 

successful implementation and adoption. Inform Prim Care. 2009;17(3):153-60. PMID: 20074427.

Exclusion: Wrong study design

Crosson JC, Ohman-Strickland PA, Cohen DJ, et al. Typical electronic health record use in primary care practices 

and the quality of diabetes care. Ann Fam Med. 2012;10(3):221-7. PMID: 22585886.

Exclusion: Not HIE

Crounse B. A compelling, sustainable business model for RHIO’s. Available at: 

http://blogs.msdn.com/healthblog/archive/2005/10/08/478037.aspx. Accessed April 22, 2015. 

Exclusion: Wrong study design

Csaba Egyhazy, Raj Mukherji. Interoperability architecture using RM-ODP. Commun ACM. 2004;47(2):93-7. 

Exclusion: No data relevant to a Key Question

Cummins MR, Crouch B, Gesteland P, et al. Inefficiencies and vulnerabilities of telephone-based communication 

between U. S. poison control centers and emergency departments. Clin Toxicol (Phila). 2013;51(5):435-43. PMID: 

23697459.

Exclusion: Not HIE

Cummins MR, Crouch BI, Gesteland P, et al. Electronic information exchange between emergency departments and 

poison control centers: a Delphi study. Clin Toxicol (Phila). 2012;50(6):503-13. PMID: 22612793.

Exclusion: No data relevant to a Key Question

da Silva KR, Costa R, Crevelari ES, et al. Glocal clinical registries: pacemaker registry design and implementation 

for global and local integration--methodology and case study. PLoS ONE. 2013;8(7):e71090. PMID: 23936257.

Exclusion: Not HIE

da Silva ME, Coeli CM, Ventura M, et al. Informed consent for record linkage: a systematic review. J Med Ethics. 

2012;38(10):639-42. PMID: 22403083.

Exclusion: Not HIE

Damberg CL, Raube K, Teleki SS, et al. Taking stock of pay-for-performance: a candid assessment from the front 

lines. Health Aff (Millwood). 2009;28(2):517-25. PMID: 19276011.

Exclusion: Not HIE

D'Amore JD, Mandel JC, Kreda DA, et al. Are Meaningful Use Stage 2 certified EHRs ready for interoperability? 

Findings from the SMART C-CDA Collaborative. J Am Med Inform Assoc. 2014;21(6):1060-8. PMID: 24970839.

Exclusion: Not HIE
D-12

D'Amore JD, Sittig DF, Ness RB. How the continuity of care document can advance medical research and public 

health. Am J Public Health. 2012;102(5):e1-4. PMID: 22420795.

Exclusion: Wrong study design

D'Amore JD, Sittig DF, Wright A, et al. The promise of the CCD: challenges and opportunity for quality 

improvement and population health. AMIA Annu Symp Proc. 2011;2011:285-94. PMID: 22195080.

Exclusion: Not HIE

Daniel GW, Ewen E, Willey VJ, et al. Efficiency and economic benefits associated with the use of a payer-based 

electronic health record in an emergency department among a health insured population. Value Health. 

2009;12(3):A14. 

Exclusion: Not HIE

Darmon D, Sauvant R, Staccini P, et al. Which functionalities are available in the electronic health record systems 

used by French general practitioners? An assessment study of 15 systems. Int J Med Inf. 2014;83(1):37-46. PMID: 

24231269.

Exclusion: Not HIE

Daskalakis S, Katharaki M, Mantas J. The use of data envelopment analysis to measure the efficiency and 

interoperability of information technology in Greek public healthcare organisations. Journal on Information 

Technology in Healthcare. 2008;6(3):188-96. 

Exclusion: Not HIE

Davidson SJ, Zwemer FL, Jr., Nathanson LA, et al. Where's the beef? The promise and the reality of clinical 

documentation. Acad Emerg Med. 2004;11(11):1127-34. PMID: 15528575.

Exclusion: Not HIE

de Brantes F, Emery DW, Overhage JM, et al. The potential of HIEs as infomediaries. J Healthc Inf Manag. 

2007;21(1):69-75. PMID: 17299928.

Exclusion: Wrong study design

de la Torre I, Diaz FJ, Anton M, et al. Performance evaluation of a web-based system to exchange Electronic Health 

Records using Queueing model (M/M/1). J Med Syst. 2012;36(2):915-24. PMID: 20703642.

Exclusion: Not HIE

Deas TM, Jr., Solomon MR. Health information exchange: foundation for better care. Gastrointest Endosc. 

2012;76(1):163-8. PMID: 22726476.

Exclusion: Wrong study design

Delano D. Roadmap of a successful local HIE: The Massachusetts eHealth Collaborative provides an instructive 

success story. Health Manag Technol. 2011;32(9):20-1. PMID: 21961258.

Exclusion: No data relevant to a Key Question

Demski H, Hildebrand C, Brass A, et al. Improvement of cross-sector communication in the integrated health 

environment. Stud Health Technol Inform. 2010;155:95-100. PMID: 20543315.

Exclusion: Wrong study design

Department of Health and Human Services. Doctors and hospitals’ use of health IT more than doubles since 2012. 

Washington, DC. Available at: http://www.hhs.gov/news/press/2013pres/05/20130522a.html. Accessed April 18, 

2014. 

Exclusion: Wrong study design

Detmer D, Bloomrosen M, Raymond B, et al. Integrated personal health records: transformative tools for consumercentric care. BMC Med Inform Decis Mak. 2008;8:45. PMID: 18837999.

Exclusion: Wrong study design
D-13

Detmer DE. Engineering information technology for actionable information and better health - balancing social 

values through desired outcomes, complementary standards and decision-support. Stud Health Technol Inform. 

2010;153:107-18. PMID: 20543241.

Exclusion: Not HIE

Devlies J, De Moor G, De Clercq E, et al. Health data exchange, health data sharing and decentralised clinical data 

collections--recommendations from a Belgian expert group. Stud Health Technol Inform. 2008;141:162-212. PMID: 

18953136.

Exclusion: No data relevant to a Key Question

Devoe JE, Gold R, Spofford M, et al. Developing a network of community health centers with a common electronic 

health record: description of the Safety Net West Practice-based Research Network (SNW-PBRN). J Am Board Fam 

Med. 2011;24(5):597-604. PMID: 21900444.

Exclusion: Not HIE

Devriendt E, Wellens N, Vesentini L, et al. BelRAI software for standardized data exchange between geriatric 

health care organizations. Eur Geriatr Med. 2012;3:S71. 

Exclusion: No data relevant to a Key Question

Dhopeshwarkar RV, Kern LM, O'Donnell HC, et al. Health care consumers' preferences around health information 

exchange. Ann Fam Med. 2012;10(5):428-34. PMID: 22966106.

Exclusion: No data relevant to a Key Question

Dierker L. The state connection. State-level efforts in health information exchange. J AHIMA. 2008;79(5):40-3. 

PMID: 18512425.

Exclusion: Wrong study design

Dimick C. Varying privacy practices that pose barriers to health information exchange are putting HIM concerns in 

the national spotlight. J AHIMA. 2007;78(10):29-33. 

Exclusion: Wrong study design

Dimick C. HISPC privacy and security collaborative hands off three years of work. J AHIMA. 2009;80(5):21-5. 

PMID: 19507777.

Exclusion: Wrong study design

Dimick C. Open for business: private networks create a marketplace for health information exchange. J AHIMA. 

2012;83(5):22-6; quiz 7. PMID: 22670323.

Exclusion: Wrong study design

Dimitropoulos L, Patel V, Scheffler SA, et al. Public attitudes toward health information exchange: perceived 

benefits and concerns. Am J Manag Care. 2011;17(12 Spec No.):SP111-6. PMID: 22216769.

Exclusion: Not HIE

Dimitropoulos L, Rizk S. A state-based approach to privacy and security for interoperable health information 

exchange. Health Aff. 2009;28(2):428-34. PMID: 19275999.

Exclusion: Wrong study design

Disanti W, Rajapakse RO, Korelitz BI, et al. Incidence of neoplasms in patients who develop sustained leukopenia 

during or after treatment with 6-mercaptopurine for inflammatory bowel disease. Clin Gastroenterol Hepatol. 

2006;4(8):1025-9. PMID: 16765651.

Exclusion: Not HIE

Dixon BE. The perceived and real value of health information exchange in public health surveillance. Diss Abstr Int. 

2014;75(5-B(E)).

Exclusion: Wrong study design
D-14

Dixon BE, Grannis SJ, Revere D. Measuring the impact of a health information exchange intervention on providerbased notifiable disease reporting using mixed methods: a study protocol. BMC Med Inform Decis Mak. 

2013;13:121-. PMID: 24171799.

Exclusion: Wrong study design

Dixon BE, Miller T, Overhage JM. Assessing HIE stakeholder readiness for consumer access: lessons learned from 

the NHIN trial implementations. J Healthc Inf Manag. 2009;23(3):20-5. PMID: 19663160.

Exclusion: Not HIE

Dixon BE, Vreeman DJ, Grannis SJ. The long road to semantic interoperability in support of public health: 

Experiences from two states. J Biomed Inform. 2014;49:3-8. PMID: 24680985.

Exclusion: No data relevant to a Key Question

Doarn CR, Nicogossian A. Policy implications of scholarly publications in health information technology. World 

Med Health Policy. 2013;5(2):161-70. 

Exclusion: Wrong study design

DoBias M. RHIOs facing trouble: survey. Few physicians electronically sharing clinical data. Mod Healthc. 

2007;37(50):32. PMID: 18203370.

Exclusion: Wrong study design

Dobrev A, Stroetmann T, Veli N. Sources of financing and policy recommendations to Member States and the 

European Commission on boosting eHealth investment. 2008. Available at: http://www.financingehealth.eu/downloads/documents/feh_d5_3_final_study_report.pdf. Accessed April 22, 2015.

Exclusion: No data relevant to a Key Question

Doebbeling BN, Chou AF, Tierney WM. Priorities and strategies for the implementation of integrated informatics 

and communications technology to improve evidence-based practice. J Gen Intern Med. 2006;21 Suppl 2:S50-7. 

PMID: 16637961.

Exclusion: Not HIE

Dolin RH, Wiesenthal AM. National health information network cost and structure. Ann Intern Med. 

2006;144(2):145; author reply 7. PMID: 16418420.

Exclusion: Wrong study design

Donnelly J, Mussi J, Parisot C, et al. Building an interoperable regional health information network today with IHE 

integration profiles. J Healthc Inf Manag. 2006;20(3):29-38. PMID: 16903659.

Exclusion: Wrong study design

Dorr D, Bonner LM, Cohen AN, et al. Informatics systems to promote improved care for chronic illness: a literature 

review. J Am Med Inform Assoc. 2007;14(2):156-63. PMID: 17213491.

Exclusion: Not HIE

Dowling AF. CHINS-the current state. Information Networks for Community Health. 1997:15-41. 

Exclusion: Wrong study design

Downing GJ, Zuckerman AE, Coon C, et al. Enhancing the quality and efficiency of newborn screening programs 

through the use of health information technology. Semin Perinatol. 2010;34(2):156-62. PMID: 20207265.

Exclusion: No data relevant to a Key Question

Downs SM, van Dyck PC, Rinaldo P, et al. Improving newborn screening laboratory test ordering and result 

reporting using health information exchange. J Am Med Inform Assoc. 2010;17(1):13-8. PMID: 20064796.

Exclusion: Wrong study design

Duftschmid G, Wrba T, Gall W, et al. The strategic approach of managing healthcare data exchange in Austria. 

Methods Inf Med. 2004;43(2):124-32. PMID: 15136861.
D-15

Exclusion: Wrong study design

Dullabh P, Adler-Milstein J, Hovey L, et al. Key Challenges to Enabling Health Information Exchange and How 

States Can Help. Office of the National Coordinator for Health Information Technology. 2014. Available at: 

http://www.healthit.gov/sites/default/files/state_hie_evaluation_stakeholder_discussions.pdf. Accessed April 10, 

2015. 

Exclusion: Wrong study design

Dullabh P, Adler-Milstein J, Nye C, et al. Evaluation of the State Health Information Exchange Cooperative 

Agreement Program: Early Findings from a Review of Twenty-Seven States. Developed by NORC for the Office of 

the National Coordinator for Health IT (ONC). Bethesda, MD: University of Chicago. 2012. Available at: 

http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-coop-program-evaluation.pdf. Accessed 

December 16, 2014. 

Exclusion: More recent data available

Dullabh P, Hovey L, Ubri P, et al. Evaluation of the State Health Information Exchange Cooperative Agreement 

Program: Physician Experiences and Perceptions of Health Information Exchange February 2013 University of 

Chicago: Office of the National Coordinator for Health Information Technology. 2012. Available at: 

http://www.healthit.gov/sites/default/files/providerfocusgroupsynthesis_02_08_13.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

Dullabh P, Hovey L, Ubri P, et al. Evaluation of the State Health Information Exchange Cooperative Agreement 

Program: Case Study Synthesis: Experiences from Five States in Enabling HIE. Bethesda, MD: NORC at the 

University of Chicago 2013. Availible at: http://healthit.gov/sites/default/files/casestudysynthesisdocument_2-8-

13.pdf. Accessed December 16, 2014. 

Exclusion: More recent data available

Dullabh P, Milstein J, Nye C, et al. Evaluation of the State Health Information Exchange Cooperative Agreement 

Program: Early Findings from a Review of Twenty-Seven States: January 2012. University of Chicago: Office of 

the National Coordinator for Health Information Technology 2012. Available at: 

http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-coop-program-evaluation.pdf. Accessed 

April 10, 2015. 

Exclusion: Wrong study design

Dullabh P, Moiduddin A, Nye C, et al. The Evolution of the State Health Information Exchange Cooperative 

Agreement Program: State Plans to Enable Robust HIE: August 2011. NORC at the University of Chicago: Office 

of the National Coordinator for Health Information Technology. 2011. Available at: 

http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-program-evolution.pdf. Accessed April 

10, 2015. 

Exclusion: Wrong study design

Dullabh P, Ubri P, Loganathan S, et al. Evaluation of the State Health Information Exchange Cooperative 

Agreement Program: State Approaches to Enabling HIE: Typology Brief. Office of the National Coordinator for 

Health Information Technology. 2014. Available at: 

http://www.healthit.gov/sites/default/files/statehietypologybrief.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

Dupuits F. The role of community health information networks in disease management. Disease Management &

Health Outcomes. 2000;8(4):185-95. 

Exclusion: Wrong study design

Dykes P, Bakken S. National and regional health information infrastructures: making use of information technology 

to promote access to evidence. Stud Health Technol Inform. 2004;107(Pt 2):1187-91. PMID: 15361000.

Exclusion: Not HIE
D-16

Eason K, Dent M, Waterson P, et al. Bottom-up and middle-out approaches to electronic patient information 

systems: a focus on healthcare pathways. Inform Prim Care. 2012;20(1):51-6. PMID: 23336835.

Exclusion: Not HIE

Edwards A, Hollin I, Barry J, et al. Barriers to cross--institutional health information exchange: a literature review. J 

Healthc Inf Manag. 2010;24(3):22-34. PMID: 20677469.

Exclusion: Systematic review not meeting our requirements 

eHealth Initiative. Report on Health Information Exchange: Sustainable HIE in a Changing Landscape. 2011

Exclusion: More recent data available

eHealth Initiative. 2014 Results from Survey on Health Data Exchange. Available at: 

http://www.ehidc.org/resource-center/publications/view_document/460-webinar-materials-2014-results-fromsurvey-on-health-data-exchange. Accessed January 9, 2014. 

Exclusion: Wrong study design

Einbinder JS, Bates DW. Leveraging information technology to improve quality and safety. Yearb Med Inform. 

2007:22-9. PMID: 17700900.

Exclusion: Not HIE

Ellingsen G, Monteiro E. Big is beautiful: electronic patient records in large Norwegian hospitals 1980s-2001. 

Methods Inf Med. 2003;42(4):366-70. PMID: 14534635.

Exclusion: Not HIE

Ellingsen G, Monteiro E, Roed K. Integration as interdependent workaround. Int J Med Inf. 2013;82(5):e161-9. 

PMID: 23083928.

Exclusion: Not HIE

Elliott E, Bailey JE, Wan JY, et al. Does health information exchange use decrease duplicate imaging in the 

emergency evaluation of back pain? J Gen Intern Med. 2011;26((Elliott E.; Bailey J.E.; Wan J.Y.; Pope R.A.; 

Waters T.M.) Medicine and Preventive Medicine, University of Tennessee, Health Science Center, Memphis, 

United States):S273. 

Exclusion: Wrong study design

Elnahal SM, Joynt KE, Bristol SJ, et al. Electronic health record functions differ between best and worst hospitals. 

Am J Manag Care. 2011;17(4):e121-47. PMID: 21774097.

Exclusion: Not HIE

Ervin NE, Berry MM. Community readiness for a computer-based health information network. J N Y State Nurses 

Assoc. 2006;37(1):5-11. PMID: 16929715.

Exclusion: No data relevant to a Key Question

Evans JM, Guthrie B, Pagliari C, et al. Do general practice characteristics influence uptake of an information 

technology (IT) innovation in primary care? Inform Prim Care. 2008;16(1):3-8. PMID: 18534072.

Exclusion: Not HIE

Exchange IHI. 2010 Annual Report. 2010. Available at: http://mpcms.blob.core.windows.net/bd985247-f489-435fa7b4-49df92ec868e/docs/f42f53db-c797-4620-8dd2-97c9b04ba4f8/ihie-2010-annual-report.pdf. Accessed April 22, 

2015.

Exclusion: No data relevant to a Key Question

Exeter DJ, Rodgers S, Sabel CE. "Whose data is it anyway(alpha)" The implications of putting small area-level 

health and social data online. Health Policy. 2014;114(1):88-96. PMID: 23932285.

Exclusion: Not HIE
D-17

Eysenbach G. Infodemiology and infoveillance tracking online health information and cyberbehavior for public 

health. Am J Prev Med. 2011;40(5 Suppl 2):S154-8. PMID: 21521589.

Exclusion: Not HIE

Fehrenbach SN, Kelly JC, Vu C. Integration of child health information systems: current state and local health 

department efforts. J Public Health Manag Pract. 2004;Suppl:S30-5. PMID: 15643356.

Exclusion: Not HIE

Feldman SS, Horan MTA. Using the Nationwide Health Information Network to Deliver Value to Disability 

Claimants: A case study of social security administration and MedVirginia use of MEGAHIT for disability 

determination. Social Security Administration. Available at: 

http://www.connectopensource.org/sites/connectopensource.org/files/CaseStudy_MedVA_SSA.pdf. Accessed 

December 5, 2014. 

Exclusion: More recent data available

Fernandes L, O'Connor M. Data governance and data stewardship. Critical issues in the move toward EHRs and 

HIE. J AHIMA. 2009;80(5):36-9. PMID: 19507780.

Exclusion: Wrong study design

Fernandez-Aleman JL, Seva-Llor CL, Toval A, et al. Free web-based personal health records: An analysis of 

functionality. J Med Syst. 2013;37(6)PMID: 24221916.

Exclusion: Not HIE

Ferraccioli G, Salaffi F, Lapadula G. RHEUMA-CARD: Involvement of the patient through a secure systems access 

into the treat to target strategy in rheumatology. Ann Rheum Dis. 2013;72(Suppl 3):A1025. 

Exclusion: No data relevant to a Key Question

Ferraris VA, Saha SP, Davenport DL, et al. Thoracic surgery in the real world: does surgical specialty affect 

outcomes in patients having general thoracic operations? Ann Thorac Surg. 2012;93(4):1041-7; discussion 7-8. 

PMID: 22386087.

Exclusion: Not HIE

Ferratt TW, Lederer AL, Hall SR, et al. Surmounting health information network barriers: the greater Dayton area 

experience. Health Care Manage Rev. 1998;23(1):70-6. PMID: 9494823.

Exclusion: Wrong study design

Fidahussein M, Hook J, Kesterson J, et al. Using a regional health information exchange to improve identification of 

post-discharge follow-up providers. J Gen Intern Med. 2011;26:S163-S4. 

Exclusion: No data relevant to a Key Question

Figge HL. Interoperable health information exchange between medication therapy management services and the 

medical home. Am J Health-Syst Pharm. 2010;67(3):190-1. PMID: 20101060.

Exclusion: Not HIE

Finch TL, Mair FS, May CR. Teledermatology in the UK: lessons in service innovation. Br J Dermatol. 

2007;156(3):521-7. PMID: 17300243.

Exclusion: Not HIE

Fincham JE. Significant Potential for Health Information Exchange in Enhancing Quality of Care and Reducing 

Hospital Admissions in the United States. Am Health Drug Benefits. 2012;5(6):340-1. 

Exclusion: Wrong study design

Fine AM, Goldmann DA, Forbes PW, et al. Incorporating vaccine-preventable disease surveillance into the National 

Health Information Network: leveraging children's hospitals. Pediatr. 2006;118(4):1431-8. PMID: 17015533.

Exclusion: No data relevant to a Key Question
D-18

Finn Z, McNeill MH, Cooper LS, et al. Aligning HIE. A model to organize networks on core principles, 

collaborative activities. J AHIMA. 2010;81(8):48-51. PMID: 20795532.

Exclusion: No data relevant to a Key Question

Finnell JT, Overhage JM, Dexter PR, et al. Community clinical data exchange for emergency medicine patients. 

AMIA Annu Symp Proc. 2003:235-8. PMID: 14728169.

Exclusion: Not HIE

Fleischman W, Angiollilo J, Kuperman G, et al. Improving identification of frequent emergency department users 

using a regional health information exchange. Acad Emerg Med. 2012;19:S47-S8. 

Exclusion: Wrong study design

Florence C, Shepherd J, Brennan I, et al. An economic evaluation of anonymised information sharing in a 

partnership between health services, police and local government for preventing violence-related injury. Inj Prev. 

2014;20(2):108-14. 

Exclusion: Not HIE

Flynn D, Gregory P, Makki H, et al. Expectations and experiences of eHealth in primary care: a qualitative practicebased investigation. Int J Med Inform. 2009;78(9):588-604. PMID: 19482542.

Exclusion: Not HIE

Follen M, Castaneda R, Mikelson M, et al. Implementing health information technology to improve the process of 

health care delivery: a case study. Dis Manag. 2007;10(4):208-15. PMID: 17718659.

Exclusion: Not HIE

Fonkych K, Taylor R. The state and pattern of health information technology adoption: Rand Corporation; 2005. 

Exclusion: Not HIE

Ford EW, Menachemi N, Phillips MT. Predicting the adoption of electronic health records by physicians: when will 

health care be paperless? J Am Med Inform Assoc. 2006;13(1):106-12. PMID: 16221936.

Exclusion: Not HIE

Forland L. Evaluating the implementation of an electronic medical record system for a health organization-affiliated 

family practice clinic: ProQuest; 2007. 

Exclusion: Not HIE

Foster-Fishman PG, Salem DA, Allen NA, et al. Facilitating interorganizational collaboration: the contributions of 

interorganizational alliances. Am J Community Psychol. 2001;29(6):875-905. PMID: 11800511.

Exclusion: Not HIE

Foundation of Research and Education AHIMA. State Level Health Information Exchange. Final Report Part I: 

Roles in Ensuring Governance and Advancing Interoperablity. 2008. Available at: 

http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_040348.pdf. Accessed December 5, 2014. 

Exclusion: More recent data available

Foundation of Research and Education AHIMA. State Level Health Information Exchange. Final Report Part II: 

Coordinating Policies That Impact Access, Use, and Control of Health Information. Executive Summary 2008. 

Available at: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_045661.pdf. Accessed 

December 19, 2014. 

Exclusion: Wrong study design

Foundation of Research and Education of American Health Information Management Association. Development of 

State Level Health Information Exchange Initiatives Final Report: Extension Tasks. 2007. Available at: 

http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_033763.pdf. Accessed December 5, 2014. 

Exclusion: More recent data available
D-19

Foundation of Research and Education of American Health Information Management Association. State Level 

Health Information Exchange Initiative Development Workbook: a Guide to Key Issues, Options and Strategies. 

2007. Available at: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_038398.pdf. Accessed 

December 5, 2014. 

Exclusion: More recent data available

Foundation of Research and Education of American Health Information Management Association. State Level 

Health Information Exchange. Final Report Part II: Coordinating Policies That Impact Access, Use, and Control of 

Health Information. 2008. Available at: 

http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_040349.pdf. Accessed December 5, 2014. 

Exclusion: More recent data available

Foxhall K. Stating the case. A new report from AHIMA analyzes state-level health information exchanges. Healthc 

Inform. 2006;23(11):24. PMID: 17144328.

Exclusion: No data relevant to a Key Question

Frady N. Healthcare collaboration for the 21st century: direct project. Tenn Med. 2013;106(8):32. PMID: 24027884.

Exclusion: Wrong study design

Frame A, LaMantia M, Reddy Bynagari BB, et al. Development and Implementation of an Electronic Decision 

Support to Manage the Health of a High-Risk Population: The enhanced Electronic Medical Record Aging Brain 

Care Software (eMR-ABC). EGEMS (Wash DC). 2013;1(1)

Exclusion: Not HIE

Francis LP. The physician-patient relationship and a National Health Information network. J Law Med Ethics. 

2010;38(1):36-49. PMID: 20446982.

Exclusion: Wrong study design

Franklin BD, Reynolds M, Sadler S, et al. The effect of the electronic transmission of prescriptions on dispensing 

errors and prescription enhancements made in English community pharmacies: a naturalistic stepped wedge study. 

BMJ Qual Saf. 2014PMID: 24742778.

Exclusion: Not HIE

Friedman CP, Iakovidis I, Debenedetti L, et al. Across the Atlantic cooperation to address international challenges in 

eHealth and health IT: managing toward a common goal. Int J Med Inform. 2009;78(11):778-84. PMID: 19734085.

Exclusion: No data relevant to a Key Question

Friedmann BE, Shapiro JS, Kannry J, et al. Analyzing workflow in emergency departments to prepare for health 

information exchange. AMIA Annu Symp Proc. 2006:926. PMID: 17238545.

Exclusion: Not HIE

Frisse ME. State and community-based efforts to foster interoperability. Health Aff (Millwood). 2005;24(5):1190-6. 

PMID: 16162562.

Exclusion: Wrong study design

Frisse ME. Health information exchange in Memphis: impact on the physician-patient relationship. J Law Med 

Ethics. 2010;38(1):50-7. PMID: 20446983.

Exclusion: Not HIE

Frisse ME, Holmes RL. Estimated financial savings associated with health information exchange and ambulatory 

care referral. J Biomed Inform. 2007;40(6 Suppl):S27-32. PMID: 17942374.

Exclusion: Wrong study design

Frisse ME, King JK, Rice WB, et al. A regional health information exchange: architecture and implementation. 

AMIA Annu Symp Proc. 2008:212-6. PMID: 18999138.

Exclusion: Wrong study design
D-20

Frohlich J, Karp S, Smith MD, et al. Retrospective: lessons learned from the Santa Barbara project and their 

implications for health information exchange. Health Aff (Millwood). 2007;26(5):w589-91. PMID: 17670777.

Exclusion: Wrong study design

Fu PC, Jr., Rosenthal D, Pevnick JM, et al. The impact of emerging standards adoption on automated quality 

reporting. J Biomed Inform. 2012;45(4):772-81. PMID: 22820003.

Exclusion: Wrong study design

Fuji KT, Gait KA, Siracuse MV, et al. Electronic health record adoption and use by Nebraska pharmacists. Perspect 

Health Inf Manag. 2011;8:1d. PMID: 21796266.

Exclusion: Not HIE

Fung KW, Kayaalp M, Callaghan F, et al. Comparison of electronic pharmacy prescription records with manually 

collected medication histories in an emergency department. Ann Emerg Med. 2013;62(3):205-11. PMID: 23688770.

Exclusion: Not HIE

Gagnon M, Legare F, Labrecque M, et al. Interventions for promoting information and communication technologies 

adoption in healthcare professionals. Cochrane Database Syst Rev. 2009(2). PMID: 19160265.

Exclusion: No data relevant to a Key Question

Ganguly S, Kataria P, Juric R, et al. Sharing information and data across heterogeneous e-health systems. Telemed J 

E Health. 2009;15(5):454-64. PMID: 19548826.

Exclusion: Not HIE

Garets DE. Why RHIOs aren't working: views from an American who can see White Rock, British Columbia, from 

his backyard. Healthc Q. 2008;11(2):102-3. PMID: 18700271.

Exclusion: Wrong study design

Garg N, Kuperman G, Onyile A, et al. Validating health information exchanges data for quality measurement across 

four hospitals. Acad Emerg Med. 2014;21(5):S131. 

Exclusion: Wrong study design

Garrouste-Orgeas M, Timsit JF, Tafflet M, et al. Excess risk of death from intensive care unit-acquired nosocomial 

bloodstream infections: a reappraisal.[Erratum appears in Clin Infect Dis. 2006 Jun 15;42(12):1818]. Clin Infect Dis. 

2006;42(8):1118-26. PMID: 16575729.

Exclusion: Not HIE

Gaynor M, Lenert L, Wilson KD, et al. Why common carrier and network neutrality principles apply to the 

Nationwide Health Information Network (NWHIN). J Am Med Inform Assoc. 2014;21(1):2-7. PMID: 23837992.

Exclusion: Wrong study design

Geissbuhler A. Lessons learned implementing a regional health information exchange in Geneva as a pilot for the 

Swiss national eHealth strategy. Int J Med Inf. 2013;82(5):e118-24. PMID: 23332387.

Exclusion: No data relevant to a Key Question

Genes N, Shapiro JS, Hwang U, et al. GEDI WISE: Notifications about geriatric ED visits via health information 

exchange is feasible and may reduce admissions. Acad Emerg Med. 2014;21(5):S273. 

Exclusion: No comparison group

Georgiou A, Tariq A, Westbrook JI. The temporal landscape of residential aged care facilities--implications for 

context-sensitive health technology. Stud Health Technol Inform. 2013;194:69-74. PMID: 23941933.

Exclusion: Not HIE

Geurts MM, Ivens M, van Gelder E, et al. Development of a web-based pharmaceutical care plan to facilitate 

collaboration between healthcare providers and patients. Inform Prim Care. 2013;21(1):53-9. PMID: 24629657.
D-21

Exclusion: Not HIE

Ghosh T, Marquard J. Development of Regional Health Information Organizations (RHIOs): Knowledge networks 

and collaboration. Int J Public Pol. 2007;2(3-4):298-315. 

Exclusion: Wrong study design

Gichoya J, Gamache RE, Vreeman DJ, et al. An evaluation of the rates of repeat notifiable disease reporting and 

patient crossover using a health information exchange-based automated electronic laboratory reporting system. 

AMIA Annu Symp Proc. 2012;2012:1229-36. PMID: 23304400.

Exclusion: Not HIE

Glaser JP, DeBor G, Stuntz L. The New England Healthcare EDI Network. J Healthc Inf Manag. 2003;17(4):42-50. 

PMID: 14558371.

Exclusion: Wrong study design

Glickman SW, Kit Delgado M, Hirshon JM, et al. Defining and measuring successful emergency care networks: a 

research agenda. Acad Emerg Med. 2010;17(12):1297-305. PMID: 21122011.

Exclusion: Wrong study design

Goedert J. Governance: the HIE differentiator. Health Data Manag. 2009;17(8):26. PMID: 19697558.

Exclusion: Wrong study design

Goedert J. Lesson from the HIE front. Organizations share lessons learned in the effort to develop health 

information exchanges and regional health information organizations. Health Data Manag. 2009;17(2):28-30. 

PMID: 19244811.

Exclusion: Wrong study design

Gold MR, McLaughlin CG, Devers KJ, et al. Obtaining providers' 'buy-in' and establishing effective means of 

information exchange will be critical to HITECH's success. Health Aff. 2012;31(3):514-26. PMID: 22392662.

Exclusion: Wrong study design

Goldberg L, Lide B, Lowry S, et al. Usability and accessibility in consumer health informatics current trends and 

future challenges. Am J Prev Med. 2011;40(5 Suppl 2):S187-97. PMID: 21521594.

Exclusion: Not HIE

Gordon P, Camhi E, Hesse R, et al. Processes and outcomes of developing a continuity of care document for use as a 

personal health record by people living with HIV/AIDS in New York City. Int J Med Inf. 2012;81(10):e63-73. 

PMID: 22841825.

Exclusion: Not HIE

Gore MJ. Gaining links: health information networks arise--with integration challenges. Clin Lab Sci. 1996;9(2):70-

7. PMID: 10163348.

Exclusion: Wrong study design

Goroll AH, Simon SR, Tripathi M, et al. Community-wide implementation of health information technology: the 

Massachusetts eHealth Collaborative experience. J Am Med Inform Assoc. 2009;16(1):132-9. PMID: 18952937.

Exclusion: Wrong study design

Gottlieb LK, Stone EM, Stone D, et al. Regulatory and policy barriers to effective clinical data exchange: lessons 

learned from MedsInfo-ED. Health Aff. 2005;24(5):1197-204. PMID: 16162563.

Exclusion: Wrong study design

Grannis SJ, Biondich PG, Mamlin BW, et al. How disease surveillance systems can serve as practical building 

blocks for a health information infrastructure: the Indiana experience. AMIA Annu Symp Proc. 2005:286-90. PMID: 

16779047.

Exclusion: No data relevant to a Key Question
D-22

Grant RW, Wald JS, Schnipper JL, et al. Practice-linked online personal health records for type 2 diabetes mellitus: 

a randomized controlled trial. Arch Intern Med. 2008;168(16):1776-82. PMID: 18779465.

Exclusion: Not HIE

Grantham D. Confidentiality alternatives for exchanging electronic medical records take shape. Behav Healthc. 

2013;33(3):37-9. PMID: 23821917.

Exclusion: Wrong study design

Graumlich JF, Novotny NL, Stephen Nace G, et al. Patient readmissions, emergency visits, and adverse events after 

software-assisted discharge from hospital: cluster randomized trial. J Hosp Med. 2009;4(7):E11-9. PMID: 

19479782.

Exclusion: Not HIE

Gravely SD, Whaley ES. The next step in health data exchanges: trust and privacy in exchange networks. J Healthc 

Inf Manag. 2009;23(2):33-7. PMID: 19382738.

Exclusion: Wrong study design

Green SD, Thomas JD. Interdisciplinary collaboration and the electronic medical record. Pediatric nursing. 

2008;34(3):225-7, 40. PMID: 18649812.

Exclusion: Not HIE

Grinspan Z, Shapiro JS, Abramson EL, et al. Predicting frequent emergency department users among people with 

epilepsy, VIA health information exchange. 1535-7597. New York, United States: Center for Healthcare 

Informatics and Policy, Weill Cornell Medical Center. 2014. Available at: 

http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L71433373

https://www.aesnet.org/sites/default/files/file_attach/2013%20Abstract%20Supplement-14-1-s1.pdf

http://resolver.lib.washington.edu/resserv?sid=EMBASE&issn=15357597&id=doi:&atitle=Predicting+frequent+em

ergency+department+users+among+people+with+epilepsy%2C+VIA+health+information+exchange&stitle=Epilep

sy+Curr.&title=Epilepsy+Currents&volume=14&issue=&spage=262&epage=&aulast=Grinspan&aufirst=Zachary&

auinit=Z.&aufull=Grinspan+Z.&coden=&isbn=&pages=262-&date=2014&auinit1=Z&auinitm=. Accessed 

December 9, 2014. 

Exclusion: No data relevant to a Key Question

Grinspan ZM, Abramson EL, Banerjee S, et al. Potential value of health information exchange for people with 

epilepsy: crossover patterns and missing clinical data. AMIA Annu Symp Proc. 2013;2013:527-36. PMID: 

24551355.

Exclusion: Not HIE

Grinspan ZM, Abramson EL, Banerjee S, et al. People with epilepsy who use multiple hospitals; Prevalence and 

associated factors assessed via a health information exchange. Epilepsia. 2014;55(5):734-45. PMID: 24598038.

Exclusion: No data relevant to a Key Question

Grinspan ZM, Berg L, Onyile A, et al. Medical information fragmentation for people with epilepsy in new york city 

differs by type of visit. Epilepsy Currents. 2013;13:315. 

Exclusion: Not HIE

Grossman JM, Cross DA, Boukus ER, et al. Transmitting and processing electronic prescriptions: experiences of 

physician practices and pharmacies. J Am Med Inform Assoc. 2012;19(3):353-9. PMID: 22101907.

Exclusion: Not HIE

Guilbert TW, Arndt B, Temte J, et al. The theory and application of UW ehealth-PHINEX, a clinical electronic 

health record-public health information exchange. WMJ. 2012;111(3):124-33. PMID: 22870558.

Exclusion: Not HIE
D-23

Gummadi S, Housri N, Zimmers TA, et al. Electronic medical record: A balancing act of patient safety, privacy and 

health care delivery. Am J Med Sci. 2014;348(3):238-43. PMID: 24879530.

Exclusion: Not HIE

Haarbrandt B, Schwartze J, Gusew N, et al. Primary Care Providers' Acceptance of Health Information Exchange 

Utilizing IHE XDS... International Conference on Informatics, Management, and Technology in Healthcare 

(ICIMTH) Conference, July 5-7th, Athens, Greece. Stud Health Technol Inform. 2013;190:106-8. 

Exclusion: Not HIE

Haarbrandt B, Schwartze J, Gusew N, et al. Primary Care Provider's Acceptance of Health Information Exchange 

Utilizing IHE XDS. Stud Health Technol Inform. 2013;192:998. PMID: 23920772.

Exclusion: Not HIE

HACKL W, HOERBST A, AMMENWERTH E. The Electronic Health Record in Austria: Рhуsicians' Acceptance 

Is Influenced by Negative Emotions. Stud Health Technol Inform. 2009;150:140-4. PMID: 19745284.

Exclusion: Not HIE

Hadjizacharia P, Green DJ, Plurad D, et al. Cocaine use in trauma: effect on injuries and outcomes. J Trauma. 

2009;66(2):491-4. PMID: 19204526.

Exclusion: Not HIE

Haggstrom D, Myers LJ, French DD, et al. Impact of VA health information exchange upon the quality of diabetes 

care. 0884-8734. Indianapolis, United States: VA Health Services Research and Development. 2014. Available at: 

http://download.springer.com/static/pdf/904/art%253A10.1007%252Fs11606-014-2834-

9.pdf?auth66=1417820297_6efc82cb7a7326b3340d48892739e5e2&ext=.pdf. Accessed December 5, 2014. 

Exclusion: Wrong study design

Hagland M. Readying for the RHIO revolution. Behav Healthc. 2006;26(3):47-9. PMID: 16649645.

Exclusion: Wrong study design

Halamka J, Aranow M, Ascenzo C, et al. Health care IT collaboration in Massachusetts: the experience of creating 

regional connectivity. J Am Med Inform Assoc. 2005;12(6):596-601. PMID: 16049225.

Exclusion: Wrong study design

Halamka J, Aranow M, Ascenzo C, et al. E-Prescribing collaboration in Massachusetts: early experiences from 

regional prescribing projects. J Am Med Inform Assoc. 2006;13(3):239-44. PMID: 16501174.

Exclusion: Not HIE

Hall GC, McMahon AD, Dain MP, et al. Primary-care observational database study of the efficacy of GLP-1 

receptor agonists and insulin in the UK. Diabet Med. 2013;30(6):681-6. PMID: 23330649.

Exclusion: Not HIE

Hanmer LA, Roode JD, Isaacs S. Modelling the effect of limited or vulnerable resources on the use of computerised 

hospital information systems (CHISs) in South Africa. Stud Health Technol Inform. 2007;130:299-309. PMID: 

17917203.

Exclusion: Not HIE

Hansagi H, Olsson M, Hussain A, et al. Is information sharing between the emergency department and primary care 

useful to the care of frequent emergency department users? Eur J Emerg Med. 2008;15(1):34-9. PMID: 18180664.

Exclusion: Not HIE

Hargreaves JS. Will electronic personal health records benefit providers and patients in rural America? Telemed J E 

Health. 2010;16(2):167-76. PMID: 20082592.

Exclusion: Not HIE
D-24

Harkavy H. Greater than the sum of the parts. Eighteen New York physician practices gain centralized patient 

information database with ASP-hosted system. Health Manag Technol. 2004;25(7):40-2. PMID: 15283512.

Exclusion: Wrong study design

Harno K, Ruotsalainen P. Sharable EHR systems in Finland. Stud Health Technol Inform. 2006;121:364-70. PMID: 

17095834.

Exclusion: Not HIE

Haux R. Individualization, globalization and health--about sustainable information technologies and the aim of 

medical informatics. Int J Med Inform. 2006;75(12):795-808. PMID: 16846748.

Exclusion: Not HIE

Hayward-Rowse L, Whittle T. A pilot project to design, implement and evaluate an electronic integrated care 

pathway. J Nurs Manag. 2006;14(7):564-71. PMID: 17004967.

Exclusion: Not HIE

Hazard L, Miercort C, Gaffney D, et al. Local-regional radiation therapy after breast reconstruction: what is the 

appropriate target volume? A case-control study of patients treated with electron arc radiotherapy and review of the 

literature. Am J Clin Oncol. 2004;27(6):555-64. PMID: 15577432.

Exclusion: Not HIE

Heads T. Once In A Lifetime. Remain in Light. 1980

Exclusion: Wrong publication type

Hebel E, Middleton B, Shubina M, et al. Bridging the chasm: effect of health information exchange on volume of 

laboratory testing. Arch Intern Med. 2012;172(6):517-9. PMID: 22450942.

Exclusion: Wrong study design

Heimly V. Consent-based access to core EHR information: the SUMO-project. Stud Health Technol Inform. 

2008;136:431-6. PMID: 18487769.

Exclusion: Wrong study design

Heimly V, Berntsen KE. Consent-based access to core EHR information. Collaborative approaches in Norway. 

Methods Inf Med. 2009;48(2):144-8. PMID: 19283311.

Exclusion: Wrong study design

Henderson J, Miller G, Britt H, et al. Effect of computerisation on Australian general practice: does it improve the 

quality of care? Qual Prim Care. 2010;18(1):33-47. PMID: 20359411.

Exclusion: Not HIE

Herbst K, Littlejohns P, Rawlinson J, et al. Evaluating computerized health information systems: hardware, software 

and human ware: experiences from the Northern Province, South Africa. J Public Health Med. 1999;21(3):305-10. 

PMID: 10528958.

Exclusion: Not HIE

Herrin J, da Graca B, Nicewander D, et al. The effectiveness of implementing an electronic health record on 

diabetes care and outcomes. Health Serv Res. 2012;47(4):1522-40. PMID: 22250953.

Exclusion: Not HIE

Hersh W. Health care information technology: progress and barriers. JAMA. 2004;292(18):2273-4. PMID: 

15536117.

Exclusion: Not HIE

Hicken VN, Thornton SN, Rocha RA. Integration challenges of clinical information systems developed without a 

shared data dictionary. Stud Health Technol Inform. 2004;107(Pt 2):1053-7. PMID: 15360973.

Exclusion: Not HIE
D-25

Hincapie A, Warholak T. The impact of health information exchange on health outcomes. Appl Clin Inform. 

2011;2(4):499-507. PMID: 23616891.

Exclusion: Wrong study design

Holman CD, Bass AJ, Rouse IL, et al. Population-based linkage of health records in Western Australia: 

development of a health services research linked database. Aust N Z J Public Health. 1999;23(5):453-9. PMID: 

10575763.

Exclusion: No data relevant to a Key Question

Holmquest DL. Another lesson from Santa Barbara. Health Aff. 2007;26(5):w592-4. PMID: 17670778.

Exclusion: Wrong study design

Hoyle T, Swanson R. Assessing what child health information systems should be integrated: the Michigan 

experience. J Public Health Manag Pract. 2004;Suppl:S66-71. PMID: 15643362.

Exclusion: Wrong study design

Hripcsak G, Sengupta S, Wilcox A, et al. Emergency department access to a longitudinal medical record. J Am Med 

Inform Assoc. 2007;14(2):235-8. PMID: 17213496.

Exclusion: Not HIE

Huang C, Behara RS, Goo J. Optimal information security investment in a Healthcare Information Exchange: An 

economic analysis. Decis Support Syst. 2014;61:1-11. 

Exclusion: No data relevant to a Key Question

Hufstader M, Furukawa M, Hogin E. E-prescribing trends in the United States: 2008-2012. Value Health. 

2012;15(4):A25. 

Exclusion: Not HIE

Hummel J, Gandara BK. Health Information Exchange and Care Coordination of Diabetic Patients Between 

Medicine and Dentistry. Diabetes Spectr. 2011;24(4):205-10. 

Exclusion: No data relevant to a Key Question

Johansen I, Rasmussen M. Electronic interchange of lab test orders and results between laboratories reduces errors 

and gives full traceability. Stud Health Technol Inform. 2010;155:65-8. PMID: 20543311.

Exclusion: No data relevant to a Key Question

Johansson L, Wohed R, Kajbjer K. Medical informatics in a united and healthy Europe. The development of a 

Swedish national information structure... XXIInd International Congress of the European Federation for Medical 

Informatics. Stud Health Technol Inform. 2009;150:53-7. 

Exclusion: No data relevant to a Key Question

Johnson KB, Gadd C. Playing smallball: approaches to evaluating pilot health information exchange systems. J 

Biomed Inform. 2007;40(6 Suppl):S21-6. PMID: 17931981.

Exclusion: Wrong study design

Jones JB, Shah NR, Bruce CA, et al. Meaningful use in practice using patient-specific risk in an electronic health 

record for shared decision making. Am J Prev Med. 2011;40(5 Suppl 2):S179-86. PMID: 21521593.

Exclusion: Not HIE

Jones SS, Rudin RS, Perry T, et al. Health information technology: an updated systematic review with a focus on 

meaningful use. Ann Intern Med. 2014;160(1):48-54. PMID: 24573664.

Exclusion: Systematic review not meeting our requirements 

Joshi JK. Clinical Value-Add for Health Information Exchange (HIE). Internet Journal of Medical Informatics. 

2011;6(1):1. 
D-26

Exclusion: Systematic review not meeting our requirements 

Just BH, Fabian DP, Webb LL, et al. Managing the integrity of patient identity in health information exchange. J 

AHIMA. 2009;80(7):62-9. PMID: 19663149.

Exclusion: Not HIE

Kabachinski J. RHIO: the data saga continues. Biomed Instrum Technol. 2009;43(1):47-51. PMID: 19215168.

Exclusion: Wrong study design

Karmel R, Gibson D. Event-based record linkage in health and aged care services data: a methodological innovation. 

BMC Health Serv Res. 2007;7:154. PMID: 17892601.

Exclusion: Not HIE

Katehakis DG, Sfakianakis S, Tsiknakis M, et al. An infrastructure for Integrated Electronic Health Record services: 

the role of XML (Extensible Markup Language). J Med Internet Res. 2001;3(1):E7. PMID: 11720949.

Exclusion: Wrong study design

Katz SJ, Moyer CA, Cox DT, et al. Effect of a triage-based E-mail system on clinic resource use and patient and 

physician satisfaction in primary care: a randomized controlled trial. J Gen Intern Med. 2003;18(9):736-44. PMID: 

12950483.

Exclusion: Not HIE

Keet G. In or out? HIE patient consent 101: How to populate a successful HIE with the right data, while 

simultaneously maintaining patient privacy and ensuring patient comfort. Health Manag Technol. 2012;33(6):19. 

PMID: 22787948.

Exclusion: Wrong study design

Kemper AR, Uren RL, Clark SJ. Adoption of electronic health records in primary care pediatric practices. Pediatr. 

2006;118(1):e20-4. PMID: 16818534.

Exclusion: Not HIE

Kern L, Barron Y, Dhopeshwarkar R, et al. Health information exchange and quality of care. J Gen Intern Med. 

2011;26((Kern L.; Barron Y.; Dhopeshwarkar R.; Kaushal R.) Weill Cornell Medical College, New York, United 

States):S167. 

Exclusion: Wrong study design

Kern LM, Ancker JS, Abramson E, et al. Evaluating health information technology in community-based settings: 

lessons learned. J Am Med Inform Assoc. 2011;18(6):749-53. PMID: 21807649.

Exclusion: Wrong study design

Kern LM, Barron Y, Blair AJ, 3rd, et al. Electronic result viewing and quality of care in small group practices. J Gen 

Intern Med. 2008;23(4):405-10. PMID: 18373137.

Exclusion: Not HIE

Kern LM, Dhopeshwarkar R, Barron Y, et al. Measuring the effects of health information technology on quality of 

care: a novel set of proposed metrics for electronic quality reporting. Jt Comm J Qual Patient Saf. 2009;35(7):359-

69. PMID: 19634804.

Exclusion: No data relevant to a Key Question

Kern LM, Kaushal R. Health information technology and health information exchange in New York State: new 

initiatives in implementation and evaluation. J Biomed Inform. 2007;40(6 Suppl):S17-20. PMID: 17945542.

Exclusion: Wrong study design

Khan AS, Fleischauer A, Casani J, et al. The next public health revolution: public health information fusion and 

social networks. Am J Public Health. 2010;100(7):1237-42. PMID: 20530760.

Exclusion: Wrong study design
D-27

Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes Information System on inpatient and 

emergency room use: results from a randomized trial. Health Outcomes Res Med. 2010;1(1):e61-e6. PMID: 

20975923.

Exclusion: Not HIE

Khan WA, Hussain M, Afzal M, et al. Personalized-detailed clinical model for data interoperability among clinical 

standards. Telemed J E Health. 2013;19(8):632-42. PMID: 23875730.

Exclusion: Wrong study design

Kho AN, Hynes DM, Goel S, et al. CAPriCORN: Chicago Area Patient-Centered Outcomes Research Network. J 

Am Med Inform Assoc. 2014;21(4):607-11. PMID: 24821736.

Exclusion: Not HIE

Kho AN, Lemmon L, Commiskey M, et al. Use of a regional health information exchange to detect crossover of 

patients with MRSA between urban hospitals. J Am Med Inform Assoc. 2008;15(2):212-6. PMID: 18096903.

Exclusion: Not HIE

Khurshid A, Diana ML, Luce SD. Health information exchange: metrics to address quality of care and return on 

investment. Perspect Health Inf Manag. 2012;9:1e. PMID: 22783153.

Exclusion: No data relevant to a Key Question

Kierkegaard P. eHealth in Denmark: a case study. J Med Syst. 2013;37(6):9991. PMID: 24166019.

Exclusion: Wrong study design

Kilbridge PM, Classen DC. The informatics opportunities at the intersection of patient safety and clinical 

informatics. J Am Med Inform Assoc. 2008;15(4):397-407. PMID: 18436896.

Exclusion: No data relevant to a Key Question

Kimura M, Nakayasu K, Ohshima Y, et al. SS-MIX: a ministry project to promote standardized healthcare 

information exchange. Methods Inf Med. 2011;50(2):131-9. PMID: 21206962.

Exclusion: Not HIE

Kirkby KC. Psychiatric networks in Asia. Int Rev Psychiatry. 2008;20(5):409-12. PMID: 19012124.

Exclusion: Not HIE

Kittler AF, Carlson GL, Harris C, et al. Primary care physician attitudes towards using a secure web-based portal 

designed to facilitate electronic communication with patients. Inform Prim Care. 2004;12(3):129-38. PMID: 

15606985.

Exclusion: Not HIE

Kloss L. Health Information Exchange: State Level Challenges and Opportunities. American Health Informatics 

Management Association. Betheda, MD. Available at: http://www.chita.org/downloads/Kloss.pdf. Accessed April 

22, 2015. 

Exclusion: Wrong study design

Kluge E-HW. Secure e-Health: managing risks to patient health data. Int J Med Inf. 2007;76(5-6):402-6. PMID: 

17084665.

Exclusion: Not HIE

Knaup P, Bott O, Kohl C, et al. Electronic patient records: moving from islands and bridges towards electronic 

health records for continuity of care. Yearb Med Inform. 2007:34-46. PMID: 17700902.

Exclusion: No data relevant to a Key Question

Koff DA. Introducing Integrating the Healthcare Enterprise--Canada. Can Assoc Radiol J. 2005;56(4):225-31. 

PMID: 16419374.
D-28

Exclusion: Not HIE

Kontos EZ, Emmons KM, Puleo E, et al. Communication inequalities and public health implications of adult social 

networking site use in the United States. J Health Commun. 2010;15 Suppl 3:216-35. PMID: 21154095.

Exclusion: Not HIE

Korst LM, Aydin CE, Signer JMK, et al. Hospital readiness for health information exchange: development of 

metrics associated with successful collaboration for quality improvement. Int J Med Inf. 2011;80(8):e178-88. PMID: 

21330191.

Exclusion: Not HIE

Korst LM, Signer JMK, Aydin CE, et al. Identifying organizational capacities and incentives for clinical datasharing: the case of a regional perinatal information system. J Am Med Inform Assoc. 2008;15(2):195-7. PMID: 

18096916.

Exclusion: No data relevant to a Key Question

Korzeniewski SJ, Grigorescu V, Copeland G, et al. Methodological innovations in data gathering: newborn 

screening linkage with live births records, Michigan, 1/2007-3/2008. Matern Child Health J. 2010;14(3):360-4. 

PMID: 19353254.

Exclusion: Not HIE

Kouroubali A, Starren J, Barrows RC, Jr., et al. Practical lessons in remote connectivity. Proc AMIA Annu Fall 

Symp. 1997:335-9. PMID: 9357643.

Exclusion: Wrong study design

Kralewski JE, Zink T, Boyle R. Factors influencing electronic clinical information exchange in small medical group 

practices. J Rural Health. 2012;28(1):28-33. PMID: 22236312.

Exclusion: Not HIE

Kremer T. RHIO stabilizes finances: Rochester RHIO committee develops revenue plan to cover $3 million annual 

operating cost. Health Manag Technol. 2011;32(9):18. PMID: 21961256.

Exclusion: Wrong study design

Kretz JM. National health information network cost and structure. Ann Intern Med. 2006;144(2):145-6; author reply 

7. PMID: 16418418.

Exclusion: Wrong study design

Krist AH, Woolf SH, Frazier CO, et al. An electronic linkage system for health behavior counseling effect on 

delivery of the 5A's. Am J Prev Med. 2008;35(5 Suppl):S350-8. PMID: 18929981.

Exclusion: Not HIE

Krohn R. The business end of HIE. Despite recent developments, exchanges face daunting obstacles to success. J 

Healthc Inf Manag. 2010;24(1):6-7. PMID: 20077916.

Exclusion: Wrong study design

Kruse SC, Regier V, Rheinboldt TK. Barriers Over Time to Full Implementation of Health Information Exchange in 

the United States. JMIR Med Inform. 2014;2(2):e26. 

Exclusion: Systematic review not meeting our requirements 

Kukafka R, Khan SA, Hutchinson C, et al. Digital partnerships for health: steps to develop a community-specific 

health portal aimed at promoting health and well-being. AMIA Annu Symp Proc. 2007:428-32. PMID: 18693872.

Exclusion: Not HIE

Kuo M-H, Kushniruk AW, Borycki EM, et al. National strategies for health data interoperability. Stud Health 

Technol Inform. 2011;164:238-42. PMID: 21335717.

Exclusion: Wrong study design
D-29

Kuperman GJ. Health-information exchange: why are we doing it, and what are we doing? J Am Med Inform Assoc. 

2011;18(5):678-82. PMID: 21676940.

Exclusion: Wrong study design

Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. J Gen Intern Med. 

2013;28(12):1663-6. PMID: 23690236.

Exclusion: Wrong study design

Kurland LT, Molgaard CA. The patient record in epidemiology. Sci Am. 1981;245(4):54-63. PMID: 7027437.

Exclusion: Not HIE

Kussaibi H, Macary F, Kennedy M, et al. HL7 CDA implementation guide for structured anatomic pathology 

reports methodology and tools. Stud Health Technol Inform. 2010;160(Pt 1):289-93. PMID: 20841695.

Exclusion: Not HIE

Laborde DV, Griffin JA, Smalley HK, et al. A framework for assessing patient crossover and health information 

exchange value. J Am Med Inform Assoc. 2011;18(5):698-703. PMID: 21705458.

Exclusion: Wrong study design

Lagoe RJ, Westert GP. Community wide electronic distribution of summary health care utilization data. BMC Med 

Inform Decis Mak. 2006;6:17. PMID: 16549023.

Exclusion: Not HIE

Lammers EJ, Adler-Milstein J, Kocher KE. Effect of health information exchange on repeat imaging in the

emergency department. Acad Emerg Med. 2013;20(5):S15. 

Exclusion: Wrong study design

Landman AB, Lee CH, Sasson C, et al. Prehospital electronic patient care report systems: early experiences from 

emergency medical services agency leaders. PLoS ONE. 2012;7(3):e32692. PMID: 22403698.

Exclusion: Not HIE

Lang RD. Hometown heroes: Small-town Doylestown Hospital has earned distinction for implementing a successful 

HIE. Health Manag Technol. 2012;33(10):16-7. 

Exclusion: Wrong study design

Lapsia V, Lamb K, Yasnoff WA. Where should electronic records for patients be stored? Int J Med Inf. 

2012;81(12):821-7. PMID: 23021932.

Exclusion: No data relevant to a Key Question

Lassila KS, Pemble KR, DuPont LA, et al. Assessing the impact of community health information networks: a 

multisite field study of the Wisconsin Health Information Network. Top Health Inf Manage. 1997;18(2):64-76. 

PMID: 10174731.

Exclusion: Not HIE

Lawrence D. RHIO or not--it works. A pilot program on medication histories in EDs is first for Vermont Health 

Information Exchange. Healthc Inform. 2007;24(9):46. PMID: 17927065.

Exclusion: No data relevant to a Key Question

Leao BF, Bernardes MM, Levin J, et al. The Brazilian National Health Informatics Strategy. Stud Health Technol 

Inform. 2001;84(Pt 1):38-42. PMID: 11604702.

Exclusion: Wrong study design

Lee L, Whitcomb K, Galbreth M, et al. A strong state role in the HIE. Lessons from the South Carolina Health 

Information Exchange. J AHIMA. 2010;81(6):46-50; quiz 1. PMID: 20614703.

Exclusion: Wrong study design
D-30

Lee M, Gatton TM. Wireless health data exchange for home healthcare monitoring systems. Sensors (Basel). 

2010;10(4):3243-60. PMID: 22319296.

Exclusion: Wrong study design

Legg M. Standardisation of test requesting and reporting for the electronic health record. Clin Chim Acta. 

2014;432:148-56. PMID: 24333615.

Exclusion: Wrong study design

Lemay NV, Sullivan T, Jumbe B, et al. Reaching remote health workers in Malawi: Baseline assessment of a pilot 

mHealth intervention. J Health Commun. 2012;17(Suppl 1):105-17. PMID: 22548604.

Exclusion: Not HIE

Lemieux-Charles L, Chambers LW, Cockerill R, et al. Evaluating the effectiveness of community-based dementia 

care networks: the Dementia Care Networks' Study. Gerontologist. 2005;45(4):456-64. PMID: 16051908.

Exclusion: Not HIE

Lester WT, Grant RW, Barnett GO, et al. Randomized controlled trial of an informatics-based intervention to 

increase statin prescription for secondary prevention of coronary disease. J Gen Intern Med. 2006;21(1):22-9. 

PMID: 16423119.

Exclusion: Not HIE

Leventhal R. Health Information Exchange: Moving Forward or Stuck in Neutral? Healthc Inform. 2014;31(3):14-

20. PMID: 24941600.

Exclusion: Wrong study design

Leventhal R. Sutter health goes next-level with data exchange. Healthc Inform. 2014;31(5):36-7. PMID: 25230451.

Exclusion: Wrong study design

Levin-Epstein M. Health information exchanges not ready for prime time. Manag Care. 2014;23(6):31-5. PMID: 

25109045.

Exclusion: Wrong study design

Lewis SH, Holtry RS, Loschen WA, et al. The collaborative experience of creating the National Capital Region 

Disease Surveillance Network. J Public Health Manag Pract. 2011;17(3):248-54. PMID: 21464687.

Exclusion: Wrong study design

Li J-s, Zhou T-s, Chu J, et al. Design and development of an international clinical data exchange system: the 

international layer function of the Dolphin Project. J Am Med Inform Assoc. 2011;18(5):683-9. PMID: 21571747.

Exclusion: Wrong study design

Li YC, Kuo HS, Jian WS, et al. Building a generic architecture for medical information exchange among healthcare 

providers. Int J Med Inf. 2001;61(2-3):241-6. PMID: 11311678.

Exclusion: Wrong study design

Lichtner V, Galliers JR, Wilson S. A pragmatics' view of patient identification. Qual Saf Health Care. 2010;19 

Suppl 3:i13-9. PMID: 20513792.

Exclusion: Not HIE

Lim C, Dokmak S, Cauchy F, et al. Selective policy of no drain after pancreaticoduodenectomy is a valid option in 

patients at low risk of pancreatic fistula: a case-control analysis. World J Surg. 2013;37(5):1021-7. PMID: 

23412469.

Exclusion: Not HIE

Little L. Privacy, trust, and identity issues for ubiquitous computing. Soc Sci Comput Rev. 2008;26(1):3-5. 

Exclusion: Wrong study design
D-31

Liu C-F, Hwang H-G, Chang H-C. E-healthcare maturity in Taiwan. Telemed J E Health. 2011;17(7):569-73. 

PMID: 21718093.

Exclusion: Not HIE

Liu D, Wang X, Pan F, et al. Web-based infectious disease reporting using XML forms. Int J Med Inf. 

2008;77(9):630-40. PMID: 18060833.

Exclusion: Not HIE

Liu GC, Cooper JG, Schoeffler KM, et al. Standards for the electronic health record, emerging from health care's 

Tower of Babel. Proc AMIA Symp. 2001:388-92. PMID: 11825216.

Exclusion: Not HIE

Liu S, Zhou B, Xie G, et al. Beyond regional health information exchange in China: a practical and industrialstrength approach. AMIA Annu Symp Proc. 2011;2011:824-33. PMID: 22195140.

Exclusion: Wrong study design

Lluch M. Healthcare professionals' organisational barriers to health information technologies-a literature review. Int 

J Med Inform. 2011;80(12):849-62. PMID: 22000677.

Exclusion: Systematic review not meeting our requirements 

Lobach DF, Silvey GM, Willis JM, et al. Coupling direct collection of health risk information from patients through 

kiosks with decision support for proactive care management. AMIA Annu Symp Proc. 2008:429-33. PMID: 

18999181.

Exclusion: Not HIE

Lomax A, Grossmann M, Cozzi L, et al. The exchange of radiotherapy data as part of an electronic patient-referral 

system. Int J Radiat Oncol Biol Phys. 2000;47(5):1449-56. PMID: 10889401.

Exclusion: Wrong study design

M. W S. Health information exchange can save money by reducing admissions from the emergency department. 

AHRQ Research Activities. 2012(381):13. 

Exclusion: Wrong study design

Mabry LM, Bailey JE, Wan J, et al. Health information exchange use improves adherence with evidence-based 

guidelines for neuroimaging in the emergency evaluation of headache. J Investig Med. 2011;59(2):533. 

Exclusion: Wrong study design

MacFarlane A, Murphy AW, Clerkin P. Telemedicine services in the Republic of Ireland: an evolving policy 

context. Health Policy. 2006;76(3):245-58. PMID: 16026889.

Exclusion: Not HIE

MacPhail LH, Neuwirth EB, Bellows J. Coordination of diabetes care in four delivery models using an electronic 

health record. Med Care. 2009;47(9):993-9. PMID: 19648836.

Exclusion: Not HIE

Maenpaa T, Suominen T, Asikainen P, et al. The outcomes of regional healthcare information systems in health 

care: a review of the research literature. Int J Med Inf. 2009;78(11):757-71. PMID: 19656719.

Exclusion: Systematic review not meeting our requirements 

Maffei R, Burciago D, Dunn K. Determining business models for financial sustainability in regional health 

information organizations (RHIOs): a review. Popul Health Manag. 2009;12(5):273-8. PMID: 19848569.

Exclusion: More recent data available

Maglogiannis I, Constantinos D, Kazatzopoulos L. Enabling collaborative medical diagnosis over the Internet via 

peer-to-peer distribution of electronic health records. J Med Syst. 2006;30(2):107-16. PMID: 16705995.
D-32

Exclusion: Wrong study design

Magnuson JA, Klockner R, Ladd-Wilson S, et al. Security aspects of electronic data interchange between a state 

health department and a hospital emergency department. J Public Health Manag Pract. 2004;10(1):70-6. PMID: 

15018344.

Exclusion: Wrong study design

Mahon BE, Shea KM, Dougherty NN, et al. Implications for registry-based vaccine effectiveness studies from an 

evaluation of an immunization registry: a cross-sectional study. BMC Public Health. 2008;8:160. PMID: 18479517.

Exclusion: Not HIE

Maiorana A, Steward WT, Koester KA, et al. Trust, confidentiality, and the acceptability of sharing HIV-related 

patient data: lessons learned from a mixed methods study about Health Information Exchanges. Implement Sci. 

2012;7:34. PMID: 22515736.

Exclusion: Not HIE

Mancuso M. Collaborating our way into interoperability. Health Manag Technol. 2014;35(6):24. PMID: 25058982.

Exclusion: Wrong study design

Mandl KD, Mandel JC, Murphy SN, et al. The SMART Platform: early experience enabling substitutable 

applications for electronic health records. J Am Med Inform Assoc. 2012;19(4):597-603. PMID: 22427539.

Exclusion: Wrong study design

Mantzana V, Themistocleous M, Morabito V, et al. Evaluating actors and factors associated with healthcare 

information systems. Evaluating Information Systems: Public and Private Sector2008:179-98.

Exclusion: Not HIE

Manzotti A, Chemello C, Pullen C, et al. Computer-assisted total knee arthroplasty after prior femoral fracture 

without hardware removal. Orthopedics. 2012;35(10 Suppl):34-9. PMID: 23026250.

Exclusion: Not HIE

Marchibroda JM. The impact of health information technology on collaborative chronic care management. J Manage 

Care Pharm. 2008;14(2 Suppl):S3-11. PMID: 18331114.

Exclusion: Wrong study design

Marelli C, Gunnarsson C, Ross S, et al. Statins and risk of cancer: a retrospective cohort analysis of 45,857 matched 

pairs from an electronic medical records database of 11 million adult Americans. J Am Coll Cardiol. 

2011;58(5):530-7. PMID: 21777752.

Exclusion: Not HIE

Marquard J, Brennan PF, Grindrod D, et al. Health information exchange networks: understanding stakeholder 

views. AMIA Annu Symp Proc. 2005:1044. PMID: 16779331.

Exclusion: Wrong study design

Marrs KA, Kahn MG. Extending a clinical repository to include multiple sites. Proc Annu Symp Comput Appl Med 

Care. 1995:387-91. PMID: 8563308.

Exclusion: Wrong study design

Marshall GF, Gillespie W, Fox SJ. Privacy and security in Pennsylvania: ensuring privacy and security of health 

information exchange in Pennsylvania. J Healthc Inf Manag. 2009;23(2):38-44. PMID: 19382739.

Exclusion: Wrong study design

Martiez I, Escayola J, Martinez-Espronceda M, et al. Seamless integration of ISO/IEEE11073 personal health 

devices and ISO/EN13606 electronic health records into an end-to-end interoperable solution. Telemed J E Health. 

2010;16(10):993-1004. PMID: 21087123.

Exclusion: Wrong study design
D-33

Martin Z. Virginia RHIO taking baby steps. Health Data Manag. 2007;15(2):120. PMID: 17375855.

Exclusion: Wrong study design

Masi M, Pugliese R, Tiezzi F. Security analysis of standards-driven communication protocols for healthcare 

scenarios. J Med Syst. 2012;36(6):3695-711. PMID: 22447202.

Exclusion: No data relevant to a Key Question

Mastebroek M, Naaldenberg J, Lagro-Janssen AL, et al. Health information exchange in general practice care for 

people with intellectual disabilities--a qualitative review of the literature. Res Dev Disabil. 2014;35(9):1978-87. 

PMID: 24864050.

Exclusion: Not HIE 

McBride M. Health information exchange will improve quality of patient care, physicians believe. Ophthalmology 

Times. 2012;37(21):89. 

Exclusion: Wrong study design

McCarter D, Lenart D. A "most-wired" hospital targets information sharing. Process improvements include faster 

communication of key patient indicators. Nurs Manage. 2007;Suppl:24, 6, 32. PMID: 18159650.

Exclusion: Not HIE

McCormick D, Bor DH, Woolhandler S, et al. Giving office-based physicians electronic access to patients' prior 

imaging and lab results did not deter ordering of tests. Health Aff (Millwood). 2012;31(3):488-96. PMID: 

22392659.

Exclusion: Not HIE

McCray JC. Delivering health information statewide via the Internet in a collaborative environment: impact on 

individual member institutions. Bull Med Libr Assoc. 1999;87(3):264-9. PMID: 10427425.

Exclusion: Not HIE

McCray JC, Maloney K. Improving access to knowledge-based health sciences information: early results from a 

statewide collaborative effort. Bull Med Libr Assoc. 1997;85(2):136-40. PMID: 9160149.

Exclusion: Not HIE

McCullough JC. The Adoption and Use of Health Information Technologies in Three Settings. 

[Dissertation].University of California, Los Angeles. Los Angeles. Available at: 

https://escholarship.org/uc/item/6br9w3dm#page-1. Accessed April 22, 2015. 

Exclusion: More recent data available

McCullough JC. The adoption and use of health Information Technologies in three settings. Diss Abstr Int. 

2014;75(3-B(E)):No Pagination Specified. 

Exclusion: Wrong study design

McDonald C. Protecting patients in health information exchange: a defense of the HIPAA privacy rule. Health Aff. 

2009;28(2):447-9. PMID: 19276002.

Exclusion: Wrong study design

McDonald C, Overhage J, Barnes M, et al. The Indiana Network for Patient Care: a working local health 

information infrastructure. Health Aff. 2005;24:1214-20. PMID: 16162565.

Exclusion: Wrong study design

McDonald CJ, Schadow G, Barnes M, et al. Open Source software in medical informatics--why, how and what. Int J 

Med Inform. 2003;69(2-3):175-84. PMID: 12810121.

Exclusion: Not HIE
D-34

McGowan J, Evans J, Michl K. Networking a need: a cost-effective approach to statewide health information 

delivery. Proc Annu Symp Comput Appl Med Care. 1995:571-5. PMID: 8563350.

Exclusion: Not HIE

McGowan J, Kuperman G, Olinger L, et al. Strengthening Health Information Exchange: Final Report HIE 

Unintended Consequences Work Group. Rockville, MD: Westat. 2012. Available at: 

http://www.healthit.gov/sites/default/files/hie_uc_workgroup_final_report.pdf. Accessed January 7, 2014. 

Exclusion: More recent data available

McGraw D, Dempsey JX, Harris L, et al. Privacy as an enabler, not an impediment: building trust into health 

information exchange. Health Aff. 2009;28(2):416-27. PMID: 19275998.

Exclusion: Wrong study design

McKenna R. Using information and communications technology to enable the exchange of information between 

New Zealand clinicians and health providers. N Z Med J. 2010;123(1314):92-104. PMID: 20581917.

Exclusion: Not HIE

Meade B, Buckley D, Boland M. What factors affect the use of electronic patient records by Irish GPs? Int J Med 

Inform. 2009;78(8):551-8. PMID: 19375381.

Exclusion: Not HIE

Mears GD, Rosamond WD, Lohmeier C, et al. A link to improve stroke patient care: a successful linkage between a 

statewide emergency medical services data system and a stroke registry. Acad Emerg Med. 2010;17(12):1398-404. 

PMID: 21122025.

Exclusion: Wrong study design

Mehrotra A, Pearson SD, Coltin KL, et al. The response of physician groups to P4P incentives. Am J Manag Care. 

2007;13(5):249-55. PMID: 17488190.

Exclusion: Not HIE

Mendelson DS, Bak PRG, Menschik E, et al. Informatics in radiology: image exchange: IHE and the evolution of 

image sharing. Radiographics. 2008;28(7):1817-33. PMID: 18772272.

Exclusion: Wrong study design

Meter RK. The Synapse health information network. Linking Nebraska and the midwest. Ann N Y Acad Sci. 

1992;670:98-100. PMID: 1309108.

Exclusion: Not HIE

Metsemakers JF, Hoppener P, Knottnerus JA, et al. Computerized health information in The Netherlands: a 

registration network of family practices. Br J Gen Pract. 1992;42(356):102-6. PMID: 1493025.

Exclusion: Not HIE

Metz JM, Coyle C, Hudson C, et al. An Internet-based cancer clinical trials matching resource. J Med Internet Res. 

2005;7(3):e24. PMID: 15998615.

Exclusion: Not HIE

Milburn JA, Driver CP, Youngson GG, et al. The accuracy of clinical data: a comparison between central and local 

data collection. Surgeon. 2007;5(5):275-8. PMID: 17958226.

Exclusion: Not HIE

Miller RH, Sim I. Physicians' use of electronic medical records: barriers and solutions. Health Aff. 2004;23(2):116-

26. PMID: 15046136.

Exclusion: Not HIE
D-35

Miriovsky BJ, Shulman LN, Abernethy AP. Importance of health information technology, electronic health records, 

and continuously aggregating data to comparative effectiveness research and learning health care. J Clin Oncol. 

2012;30(34):4243-8. PMID: 23071233.

Exclusion: Wrong study design

Moehr JR, McDaniel JG. Adoption of security and confidentiality features in an operational community health 

information network: the Comox Valley experience--case example. Int J Med Inf. 1998;49(1):81-7. PMID: 9723805.

Exclusion: Wrong study design

Montori VM, Dinneen SF, Gorman CA, et al. The impact of planned care and a diabetes electronic management 

system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care. 

2002;25(11):1952-7. PMID: 12401738.

Exclusion: Not HIE

Morris G, Afzal S, Bhasker M, et al. Health Information Exchange Driven Subscription and Notification Services: 

Market Assessment and Policy Considerations. Office of the National Coordinator for Health Information 

Technology. 2012. Availible at: Accessed April 10, 2015. 

Exclusion: Wrong study design

Morris G, Afzal S, Finney D. Consumer Engagement in Health Information Exchange. Office of the National 

Coordinator for Health Information Technology. 2012

Exclusion: Wrong study design

Morrissey J. The evolution of a CHIN (community health information network). Mod Healthc. 2000;Suppl:42-3. 

PMID: 11067123.

Exclusion: Wrong study design

Morrissey J. Health information exchange. Hosp Health Netw. 2011;85(2):22-7. PMID: 21485258.

Exclusion: Wrong study design

Mostashari F, Tripathi M, Kendall M. A tale of two large community electronic health record extension projects. 

Health Aff. 2009;28(2):345-56. PMID: 19275989.

Exclusion: Not HIE

Munck LK, Hansen KR, Grethe Molbak A, et al. The use of shared medication record as part of medication 

reconciliation at hospital admission is feasible. Dan Med J. 2014;61(5). PMID: 24814735.

Exclusion: No data relevant to a Key Question

Munoz RT, Fox MD, Gomez MR. Presumed consent models and health information exchanges: hard nudges and 

ambiguous benefits. Am J Bioeth. 2013;13(6):14-5. PMID: 23641837.

Exclusion: Wrong study design

Muscatello DJ, Churches T, Kaldor J, et al. An automated, broad-based, near real-time public health surveillance 

system using presentations to hospital Emergency Departments in New South Wales, Australia. BMC Public Health. 

2005;5:141. PMID: 16372902.

Exclusion: Not HIE

Myers JS, Shannon RP. Chasing high performance: best business practices for using health information technology 

to advance patient safety. Am J Manag Care. 2012;18(4):e121-5. PMID: 22554037.

Exclusion: Wrong study design

Myneni S, Patel VL. Assessment of collaboration and interoperability in an information management system to 

support bioscience research. AMIA Annu Symp Proc. 2009;2009:463-7. PMID: 20351900.

Exclusion: Not HIE
D-36

Nagy PG, Pierce B, Otto M, et al. Quality Control Management and Communication Between Radiologists and 

Technologists. J Am Coll Radiol. 2008;5(6):759-65. PMID: 18514956.

Exclusion: Not HIE

Navas H, Lopez Osornio A, Gambarte L, et al. Implementing rules to improve the quality of concept postcoordination with SNOMED CT. Stud Health Technol Inform. 2010;160(Pt 2):1045-9. PMID: 20841843.

Exclusion: Not HIE

Neame R. Privacy and security issues in a wide area health communications network. Int J Biomed Comput. 

1996;43(1-2):123-7. PMID: 8960932.

Exclusion: Wrong study design

Nesbitt TS, Dharmar M, Katz-Bell J, et al. Telehealth at UC Davis--a 20-year experience. Telemed J E Health. 

2013;19(5):357-62. PMID: 23343257.

Exclusion: Not HIE

Newgard C, Malveau S, Staudenmayer K, et al. Evaluating the use of existing data sources, probabilistic linkage, 

and multiple imputation to build population-based injury databases across phases of trauma care. Acad Emerg Med. 

2012;19(4):469-80. PMID: 22506952.

Exclusion: Not HIE

Newgard CD, Zive D, Malveau S, et al. Developing a statewide emergency medical services database linked to 

hospital outcomes: a feasibility study. Prehosp Emerg Care. 2011;15(3):303-19. PMID: 21612384.

Exclusion: Not HIE

Nguile-Makao M, Zahar J-R, Francais A, et al. Attributable mortality of ventilator-associated pneumonia: respective 

impact of main characteristics at ICU admission and VAP onset using conditional logistic regression and multi-state 

models. Intensive Care Med. 2010;36(5):781-9. PMID: 20232046.

Exclusion: Not HIE

Nicholson C, Jackson C, Tweeddale M, et al. International exchange. Electronic patient records: achieving best 

practice in information transfer between hospital and community providers -- an integration success story. Qual 

Prim Care. 2003;11(3):233-40.

Exclusion: Not HIE

Nirel N, Rosen B, Sharon A, et al. The impact of an integrated hospital-community medical information system on 

quality and service utilization in hospital departments. Int J Med Inform. 2010;79(9):649-57. PMID: 20655276.

Exclusion: Not HIE

Noblin AM. Privacy policy analysis for health information networks and regional health information organizations. 

Health Care Manag (Frederick). 2007;26(4):331-40. PMID: 17992107.

Exclusion: Wrong study design

Nocella KC, Horowitz KJ, Young JJ. Against all odds: designing and implementing a grassroots, communitydesigned RHIO in a rural region. J Healthc Inf Manag. 2008;22(2):34-41. PMID: 19266993.

Exclusion: Not HIE

NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report: 

Experiences from Maine in Enabling Health Information Exchange (HIE). University of Chicago: Office of the 

National Coordinator for Health Information Technology. 2012. Available at: 

http://www.healthit.gov/sites/default/files/me_casestudyreportfinal.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design
D-37

NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report: 

Experiences from Nebraska in Enabling Health Information Exchange (HIE). University of Chicago: Office of the 

National Coordinator for Health Information Technology. 2012. Available at: 

http://www.healthit.gov/sites/default/files/ne_casestudyreport_final.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report: 

Experiences from Texas in Enabling Health Information Exchange (HIE) University of Chicago: Office of the 

National Coordinator for Health Information Technology. 2012. Available at: 

http://www.healthit.gov/sites/default/files/tx_casestudyreport_final.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report: 

Experiences from Washington State in Enabling Health Information Exchange (HIE) University of Chicago: Office 

of the National Coordinator for Health Information Technology 2012. Available at: 

http://www.healthit.gov/sites/default/files/wa_casestudyreport_final.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report: 

Experiences from Wisconsin in Enabling Health Information Exchange (HIE). University of Chicago: Office of the 

National Coordinator for Health Information Technology. 2012. Available at: 

http://www.healthit.gov/sites/default/files/wicasestudyreport_final.pdf. Accessed April 10, 2015. 

Exclusion: Wrong study design

Noss B, Zall RJ. A review of CHIN initiatives: what works and why. J Healthc Inf Manag. 2002;16(2):35-9. PMID: 

11941918.

Exclusion: Wrong study design

Nykanen P, Karimaa E. Evaluation during design of a regional seamless network of social and health care services--

information technology perspective. Stud Health Technol Inform. 2002;90:539-42. PMID: 15460751.

Exclusion: Not HIE

O'Donnell HC, Patel V, Kern LM, et al. Healthcare consumers' attitudes towards physician and personal use of 

health information exchange. J Gen Intern Med. 2011;26(9):1019-26. PMID: 21584839.

Exclusion: Not HIE

Office of the National Coordinator for Health Information Technology DoH, Human S. 2014 Edition Release 2 

Electronic Health Record (EHR) certification criteria and the ONC HIT Certification Program; regulatory 

flexibilities, improvements, and enhanced health information exchange. Final rule. Fed Regist. 2014;79(176):54429-

80. PMID: 25233533.

Exclusion: Wrong study design

Office of the National Coordinator for Health Information Technology DoH, Human S. Medicare and Medicaid 

Programs; Electronic Health Record Incentive Program—Stage 3; 2015 Edition Health Information Technology 

(Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health 

IT Certification Program Modifications; Proposed Rules. March 30, 2015 2015. Available at: 

http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Stage3_Rule.pdf. 

Accessed April 22, 2014. 

Exclusion: Wrong study design

Ogunyemi OI, Meeker D, Kim H-E, et al. Identifying appropriate reference data models for comparative 

effectiveness research (CER) studies based on data from clinical information systems. Med Care. 2013;51(8 Suppl 

3):S45-52. PMID: 23774519.

Exclusion: Wrong study design
D-38

Ohno-Machado L, Agha Z, Bell DS, et al. pSCANNER: patient-centered Scalable National Network for 

Effectiveness Research. J Am Med Inform Assoc. 2014;21(4):621-6. PMID: 24780722.

Exclusion: Not HIE

O'Leary KJ, Liebovitz DM, Feinglass J, et al. Creating a better discharge summary: improvement in quality and 

timeliness using an electronic discharge summary. J Hosp Med. 2009;4(4):219-25. PMID: 19267397.

Exclusion: Not HIE

Oliveira IC, Cunha JPS. Integration services to enable regional shared electronic health records. Stud Health 

Technol Inform. 2011;169:310-4. PMID: 21893763.

Exclusion: Wrong study design

Oliver AL, Montgomery K. A network approach to outpatient service delivery systems: resources flow and system 

influence. Health Serv Res. 1996;30(6):771-89. PMID: 8591929.

Exclusion: Not HIE

Oliver N, Sohrab S. Connecting the disconnected: what FSM is doing? Pac Health Dialog. 2010;16(1):137-40. 

PMID: 20968246.

Exclusion: Wrong study design

Olsen J, Baisch MJ. An integrative review of information systems and terminologies used in local health 

departments. J Am Med Inform Assoc. 2014;21(e1):e20-7. PMID: 24036156.

Exclusion: Not HIE

Olson KL, Grannis SJ, Mandl KD. Privacy protection versus cluster detection in spatial epidemiology. Am J Public 

Health. 2006;96(11):2002-8. PMID: 17018828.

Exclusion: Not HIE

Onyile A, Shapiro JS, Kuperman G. Patient crossover rates vary by disease in a health information exchange. Ann 

Emerg Med. 2011;58(4):S294-S5. 

Exclusion: No data relevant to a Key Question

Orlova AO, Dunnagan M, Finitzo T, et al. Electronic health record - public health (EHR-PH) system prototype for 

interoperability in 21st century healthcare systems. AMIA Annu Symp Proc. 2005:575-9. PMID: 16779105.

Exclusion: Wrong study design

Orphanoudakis S. HYGEIAnet: the integrated regional health information network of Crete. Stud Health Technol 

Inform. 2004;100:66-78. PMID: 15718565.

Exclusion: Wrong study design

Overhage JM, Dexter PR, Perkins SM, et al. A randomized, controlled trial of clinical information shared from 

another institution. Ann Emerg Med. 2002;39(1):14-23. PMID: 11782726.

Exclusion: Not HIE

Overhage JM, Tierney WM, McDonald CJ. Design and implementation of the Indianapolis Network for Patient 

Care and Research. Bull Med Libr Assoc. 1995;83(1):48-56. PMID: 7703939.

Exclusion: No data relevant to a Key Question

Ozkaynak M, Marquard J, Hsieh Y, et al. Are lay people ready for health information exchange? AMIA Annu Symp 

Proc. 2007:1065. PMID: 18694163.

Exclusion: Not HIE

Page D. Health information exchanges hold promise, pose perils. Hosp Health Netw. 2010;84(1):12. PMID: 

20166483.

Exclusion: Wrong study design
D-39

Pagliari C. Implementing the National Programme for IT: what can we learn from the Scottish experience? Inform 

Prim Care. 2005;13(2):105-11. PMID: 15992495.

Exclusion: Wrong study design

Pagliari C, Donnan P, Morrison J, et al. Adoption and perception of electronic clinical communications in Scotland. 

Inform Prim Care. 2005;13(2):97-104. PMID: 15992494.

Exclusion: No data relevant to a Key Question

Pagliari C, Singleton P, Detmer DE. Time for a reality check of NPfIT’s problems. BMJ. 2009;338. PMID: 

19223355.

Exclusion: Wrong study design

Pan E, Cusack CM, Hook JM, et al. Cost of interconnecting health information exchanges to form a national 

network. AMIA Annu Symp Proc. 2007:583-7. PMID: 18693903.

Exclusion: No data relevant to a Key Question

Pare G, Trudel MC. Knowledge barriers to PACS adoption and implementation in hospitals. Int J Med Inform. 

2007;76(1):22-33. PMID: 16478675.

Exclusion: Not HIE

Park SC, Finnell JT. Indianapolis emergency medical service and the Indiana Network for Patient Care: evaluating 

the patient match algorithm. AMIA Annu Symp Proc. 2012;2012:1221-8. PMID: 23304399.

Exclusion: No data relevant to a Key Question

Parrish F, Do N, Bouhaddou O, et al. Implementation of RxNorm as a terminology mediation standard for 

exchanging pharmacy medication between federal agencies. AMIA Annu Symp Proc. 2006:1057. PMID: 17238676.

Exclusion: Wrong study design

Parv L, Saluse J, Aaviksoo A, et al. Economic impact of a nationwide interoperable e-Health system using the 

PENG evaluation tool. Stud Health Technol Inform. 2012;180:876-80. PMID: 22874318.

Exclusion: No data relevant to a Key Question

Patel V, Abramson EL, Edwards A, et al. Physicians' potential use and preferences related to health information 

exchange. Int J Med Inf. 2011;80(3):171-80. PMID: 21156351.

Exclusion: Not HIE

Patel VN, Dhopeshwarkar RV, Edwards A, et al. Low-income, ethnically diverse consumers' perspective on health 

information exchange and personal health records. Inform Health Soc Care. 2011;36(4):233-52. PMID: 21851182.

Exclusion: Not HIE

Patel VN, Dhopeshwarkar RV, Edwards A, et al. Consumer support for health information exchange and personal 

health records: a regional health information organization survey. J Med Syst. 2012;36(3):1043-52. PMID: 

20703633.

Exclusion: No comparison group

Paulus RA, Davis K, Steele GD. Continuous innovation in health care: implications of the Geisinger experience. 

Health Aff (Millwood). 2008;27(5):1235-45. PMID: 18780906.

Exclusion: Not HIE

Payton FC, Brennan PF, Silvers JB. Cost justification of a community health information network: the 

ComputerLink for AD caregivers. Proc Annu Symp Comput Appl Med Care. 1995:566-70. PMID: 8563348.

Exclusion: Not HIE

Pemble KR. Regional health information networks: the Wisconsin Health Information Network, a case study. Proc 

Annu Symp Comput Appl Med Care. 1994:401-5. PMID: 7949958.

Exclusion: Wrong study design
D-40

Pevnick JM, Claver M, Dobalian A, et al. Provider stakeholders' perceived benefit from a nascent health information 

exchange: a qualitative analysis. J Med Syst. 2012;36(2):601-13. PMID: 20703673.

Exclusion: No data relevant to a Key Question

Pfoh E, Abramson E, Edwards A, et al. The Comparative Value of 3 Electronic Sources of Medication Data. Am J 

Manag Care. 10/20/14 ed2014 Available at: 

http://www.ajmc.com/publications/ajpb/2014/ajpb_septemberoctober2014/The-Comparative-Value-of-3-ElectronicSources-of-Medication-Data. Accessed April 22, 2015.

Exclusion: Not HIE

Phillips BO, Welch EE. Challenges for developing RHIOs in rural America: a study in Appalachian Ohio. J Healthc 

Inf Manag. 2007;21(3):37-43. PMID: 19195292.

Exclusion: No data relevant to a Key Question

Pinborough-Zimmerman J, Bilder D, Satterfield R, et al. The impact of surveillance method and record source on 

autism prevalence: collaboration with Utah Maternal and Child Health programs. Matern Child Health J. 

2010;14(3):392-400. PMID: 19475366.

Exclusion: Not HIE

Pirnejad H, Niazkhani Z, van der Sijs H, et al. Impact of a computerized physician order entry system on nursephysician collaboration in the medication process. Int J Med Inf. 2008;77(11):735-44. PMID: 18514020.

Exclusion: Not HIE

Porteous T, Bond C, Robertson R, et al. Electronic transfer of prescription-related information: comparing views of 

patients, general practitioners, and pharmacists... including commentary by Lockyer M. Br J Gen Pract. 

2003;53(488):204-9. PMID: 14694696.

Exclusion: Not HIE

Posner KL, Van Norman GA, Chan V. Adverse cardiac outcomes after noncardiac surgery in patients with prior 

percutaneous transluminal coronary angioplasty. Anesth Analg. 1999;89(3):553-60. PMID: 10475280.

Exclusion: Not HIE

Powell J, Fitton R, Fitton C. Sharing electronic health records: the patient view. Inform Prim Care. 2006;14(1):55-7. 

PMID: 16848967.

Exclusion: Not HIE

Prestigiacomo J. HIE Sustainability Secrets. Healthc Inform. 2011;28(11):24-8. PMID: 22121569.

Exclusion: Wrong study design

Prestigiacomo J. Overcoming interoperability challenges through HIE. Huntington Hospital creates its own 

community information exchange to coordinate care, aid practice viability. Healthc Inform. 2012;29(5):36-7. PMID: 

22655443.

Exclusion: Wrong study design

Prestigiacomo J. Tennessee HIE to begin data exchange. Middle Tennessee eHealth Connect readies its core hospital 

contributors and seeks payer participation. Healthc Inform. 2012;29(5):35. PMID: 22655442.

Exclusion: Wrong study design

Proeschold-Bell RJ, Belden CM, Parnell H, et al. A randomized controlled trial of health information exchange 

between human immunodeficiency virus institutions. J Public Health Manag Pract. 2010;16(6):521-8. PMID: 

20885182.

Exclusion: Not HIE

Protti D. US regional health information organizations and the nationwide health information network: any lessons 

for Canadians? Healthc Q. 2008;11(2):96-101. PMID: 18700270.
D-41

Exclusion: Wrong study design

Protti D. Reflections on international EHR journeys. Healthcare Information Management and Communications.

2009;23(4):6-9. 

Exclusion: Wrong study design

Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New 

Zealand and Denmark, part 1: healthcare system comparisons. Inform Prim Care. 2008;16(3):183-7. PMID: 

19094404.

Exclusion: Not HIE

Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New 

Zealand and Denmark, part 2: historical comparisons. Inform Prim Care. 2008;16(3):189-93. PMID: 19094405.

Exclusion: Not HIE

Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New 

Zealand and Denmark, Part 4: Benefits comparisons. Inform Prim Care. 2008;16(4):291-6. PMID: 19192331.

Exclusion: Not HIE

Protti D, Edworthy S, Johansen I. Adoption of information technology in primary care physician offices in Alberta 

and Denmark, Part 1: Historical, technical and cultural forces. Healthc Q. 2007;10(3):95-102, 4. PMID: 17626551.

Exclusion: Not HIE

Protti D, Johansen I, Perez-Torres F. Comparing the application of Health Information Technology in primary care 

in Denmark and Andalucia, Spain. Int J Med Inform. 2009;78(4):270-83. PMID: 18819836.

Exclusion: Not HIE

Protti D, Nilsson G. Swedish GPs use Electronic Patient Records. Can Med Assoc J. 2005;10

Exclusion: Not HIE

Protti D, Smit C. GP’s have been using EMRs in the Netherlands for over twenty years. Canada Health Infoway. 

2005

Exclusion: Not HIE

Protti D, Treweek S. Scottish physicians are also active users of electronic medical records. Canada Health Infoway. 

2005

Exclusion: Not HIE

Protti D, Wright G, Treweek S, et al. Primary care computing in England and Scotland: a comparison with 

Denmark. Inform Prim Care. 2006;14(2):93-9. PMID: 17059698.

Exclusion: No data relevant to a Key Question

Quinn R. Transaction portal cuts costs. New York payers and providers discover that IT collaboration and the 

sharing of information affords savings that no organization could achieve on its own. Health Manag Technol. 

2003;24(12):40-2. PMID: 14679731.

Exclusion: Wrong study design

Quintana Y, Howard S, Norland M, et al. Pond4Kids - an multi-site online Pediatric Oncology Research Database 

for collaborative protocol research. AMIA Annu Symp Proc. 2005:1090. PMID: 16779377.

Exclusion: Wrong study design

Quintana Y, Patel AN, Arreola M, et al. POND4Kids: A global web-based database for pediatric hematology and 

oncology outcome evaluation and collaboration. Stud Health Technol Inform. 2013;183:251-6. PMID: 23388293.

Exclusion: Not HIE
D-42

Quintana Y, Patel AN, Naidu PE, et al. POND4Kids: A web-based pediatric cancer database for hospital-based 

cancer registration and clinical collaboration. Stud Health Technol Informatics. 2011;164:227-31. PMID: 21335715.

Exclusion: Not HIE

Rajda J, Vreeman DJ, Wei HG. Semantic interoperability of Health Risk Assessments. AMIA Annu Symp Proc. 

2011;2011:1134-43. PMID: 22195174.

Exclusion: Wrong study design

Ralston JD, Silverberg MJ, Grothaus L, et al. Use of web-based shared medical records among patients with HIV. 

Am J Manag Care. 2013;19(4):e114-24. PMID: 23725449.

Exclusion: Not HIE

Raths D. No practice run. Getting large physician practices and IPAs to buy into a RHIO is paramount to its 

survival. Healthc Inform. 2007;24(9):41-2. PMID: 17927063.

Exclusion: Wrong study design

Rawson NSB. Access to linked administrative healthcare utilization data for pharmacoepidemiology and 

pharmacoeconomics research in Canada: anti-viral drugs as an example. Pharmacoepidemiol Drug Saf. 

2009;18(11):1072-9. PMID: 19650154.

Exclusion: Wrong study design

Reed-Fourquet LL, Durand D, Johnson L, et al. CHIME-Net, the Connecticut Health Information Network: a pilot 

study. Proc Annu Symp Comput Appl Med Care. 1995:561-5. PMID: 8563347.

Exclusion: Wrong study design

Reid RJ, Wagner EH. Strengthening primary care with better transfer of information. CMAJ. 2008;179(10):987-8. 

PMID: 18981432.

Exclusion: Wrong study design

Reiss SM, American Pharmacists A. Integrating pharmacogenomics into pharmacy practice via medication therapy 

management. J Am Pharm Assoc (2003). 2011;51(6):e64-74. PMID: 22001957.

Exclusion: Wrong study design

Research K. Health information exchanges: rapid growth in an evolving market. Orem, Utah:2011. Available at: 

www.klasresearch.com. Accessed April 22, 2015.

Exclusion: Wrong study design

Rigby M, Roberts R, Williams J, et al. Integrated record keeping as an essential aspect of a primary care led health 

service. BMJ. 1998;317(7158):579-82. PMID: 9721116.

Exclusion: Wrong study design

Riley L, Smith G. Developing and implementing IS: A case study analysis in social services. J Inform Technol. 

1997;12(4):305-21. 

Exclusion: Not HIE

Rode D. Connecting the dots. Outlining the organizations involved with EHRs and HIE. J AHIMA. 2007;78(4):18-

20. PMID: 17455840.

Exclusion: Wrong study design

Roop ES. Ten elements of a successful HIE. For the Record (Great Valley Publishing Company, Inc). 

2011;23(3):3p. 

Exclusion: No data relevant to a Key Question

Roos NP, Black CD, Frohlich N, et al. A population-based health information system. Med Care. 1995;33(12 

Suppl):DS13-20. PMID: 7500666.

Exclusion: Wrong study design
D-43

Rosen R, Florin D, Hutt R. An anatomy of GP referral decisions. A qualitative study on GPs’ views on their role in 

supporting patient choice. King’s Fund, United Kingdom. Available at: 

http://www.kingsfund.org.uk/publications/anatomy-gp-referral-decisions. Accessed April 22, 2015. 

Exclusion: Not HIE

Rosenfeld S, Bernasek C, Mendelson D. Medicare's next voyage: encouraging physicians to adopt health 

information technology. Health Aff (Millwood). 2005;24(5):1138-46. PMID: 16162556.

Exclusion: Wrong study design

Rosenman M, Szucs K, Finnell SME, et al. Development and Testing of Health Information Exchange Methods for 

Alerting Infection Preventionists About Multi-Drug Resistant Organisms: Making Unstructured Microbiology 

Culture Data Usable. Am J Infect Control. 2014;42:S62-3. 

Exclusion: Wrong study design

Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support 

systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. PMID: 23412440.

Exclusion: Not HIE

Rubin RD. The Community Health Information Movement: Where It’s Been, Where It’s Going. In: O’Carroll P, 

Ripp L, Yasnoff W, Ward ME, Martin E, eds. Public Health Informatics and Information Systems: Springer New 

York; 2003:595-616.

Exclusion: Wrong study design

Rudin RS. Using information technology to exchange health information among healthcare providers: Measuring 

usage and understanding value. Diss Abstr Int. 2012;73(4-B):2158. 

Exclusion: Wrong study design

Rudin RS, Salzberg CA, Szolovits P, et al. Care transitions as opportunities for clinicians to use data exchange 

services: how often do they occur? J Am Med Inform Assoc. 2011;18(6):853-8. PMID: 21531703.

Exclusion: Not HIE

Rudin RS, Schneider EC, Volk LA, et al. Simulation Suggests that medical group mergers won't undermine the 

potential utility of health information exchanges. Health Aff. 2012;31(3):548-59. PMID: 22392665.

Exclusion: Not HIE

Ruotsalainen P. A cross-platform model for secure Electronic Health Record communication. Int J Med Inform. 

2004;73(3):291-5. PMID: 15066561.

Exclusion: Wrong study design

Russler D. Disease registries on the nationwide health information network. J Diabetes Sci Technol. 2011;5(3):535-

42. PMID: 21722569.

Exclusion: Not HIE

Ryan DH. A Scottish record linkage study of risk factors in medical history and dementia outcome in hospital 

patients. Dementia. 1994;5(6):339-47. PMID: 7866488.

Exclusion: Not HIE

Sackett KM, Erdley WS, Jones J. The Western New York regional electronic health record initiative: Healthcare 

informatics use from the registered nurse perspective. Stud Health Technol Inform. 2006;122:248-52. PMID: 

17102258.

Exclusion: Not HIE

Saef SH, Bourne CL, Bush JS, et al. The impact of a health information exchange on resource use and medicareallowable charges at eleven emergency departments operated by four major hospital systems in a midsized 

southeastern city: An observational study using clinician estimates. Ann Emerg Med. 2013;62(4):S97. 
D-44

Exclusion: Wrong study design

Sairamesh J, Griss ML, Weber PA, et al. Innovation in healthcare intelligence: cross-sector convergence beyond 

electronic medical records. Am J Prev Med. 2011;40(5 Suppl 2):S234-7. PMID: 21521599.

Exclusion: Wrong study design

Salzberg CA, Jang Y, Rozenblum R, et al. Policy initiatives for health information technology: a qualitative study of 

U.S. expectations and Canada's experience. Int J Med Inf. 2012;81(10):713-22. PMID: 22902272.

Exclusion: Not HIE

Sands DZ. Help for physicians contemplating use of e-mail with patients. J Am Med Inform Assoc. 2004;11(4):268-

9. PMID: 15252925.

Exclusion: Wrong study design

Schiefelbein EL, Olson JA, Moxham JD. Patterns of Health Care Utilization among Vulnerable Populations in 

Central Texas Using Data from a Regional Health Information Exchange. J Health Care Poor Underserved. 

2014;25(1):37-51. PMID: 24509011.

Exclusion: No data relevant to a Key Question

Schnall R, Bakken S. Testing the Technology Acceptance Model: HIV case managers' intention to use a continuity 

of care record with context-specific links. Inform Health Soc Care. 2011;36(3):161-72. PMID: 21848452.

Exclusion: Not HIE

Schnall R, Cimino JJ, Bakken S. Development of a prototype continuity of care record with context-specific links to 

meet the information needs of case managers for persons living with HIV. Int J Med Inf. 2012;81(8):549-55. PMID: 

22632821.

Exclusion: Wrong study design

Schnall R, Odlum M, Gordon P, et al. Barriers to implementation of a Continuity of Care Record (CCR) in 

HIV/AIDS care. Stud Health Technol Inform. 2009;146:248-52. PMID: 19592843.

Exclusion: Not HIE

Schnall R, Smith AB, Sikka M, et al. Employing the FITT framework to explore HIV case managers' perceptions of 

two electronic clinical data (ECD) summary systems. Int J Med Inf. 2012;81(10):e56-62. PMID: 22841702.

Exclusion: Not HIE

Schneider ME. Interoperability issues limit health-record, data sharing. Caring for the Ages. 2013;14(1):9. 

Exclusion: Wrong study design

Schnipper JL, Hamann C, Ndumele CD, et al. Effect of an electronic medication reconciliation application and 

process redesign on potential adverse drug events: a cluster-randomized trial. Arch Intern Med. 2009;169(8):771-80. 

PMID: 19398689.

Exclusion: Not HIE

Schwartze J, Haarbrandt B, Rochon M, et al. Design and Implementation of an Informed Consent Process for a 

Standardized Health Information Exchange Solution on the Example of the Lower Saxony Bank of Health. Stud 

Health Technol Inform. 2013;192:318-22. PMID: 23920568.

Exclusion: Not HIE

Sek ACH, Cheung NT, Choy KM, et al. A territory-wide electronic health record--from concept to practicality: the 

Hong Kong experience. Stud Health Technol Inform. 2007;129(Pt 1):293-6. PMID: 17911725.

Exclusion: No data relevant to a Key Question

Selenke J. EHR bliss. A small family practice reaps the benefits of a Web-based EHR. Health Manag Technol. 

2007;28(12):38-9. PMID: 18210973.

Exclusion: Not HIE
D-45

Sensmeier J. Laying the foundation for a secure, interoperable, nationwide health information network. Comput 

Inform Nurs. 2009;27(3):195-6. PMID: 19411951.

Exclusion: Not HIE

Shade SB, Chakravarty D, Koester KA, et al. Health information exchange interventions can enhance quality and 

continuity of HIV care. Int J Med Inf. 2012;81(10):e1-9. PMID: 22854158.

Exclusion: No data relevant to a Key Question

Shaikh AR, Prabhu Das I, Vinson CA, et al. Cyberinfrastructure for consumer health. Am J Prev Med. 2011;40(5 

Suppl 2):S91-6. PMID: 21521603.

Exclusion: No data relevant to a Key Question

Shank N. Behavioral health providers' beliefs about health information exchange: a statewide survey. J Am Med 

Inform Assoc. 2012;19(4):562-9. PMID: 22184253.

Exclusion: Not HIE

Shapiro JS. Evaluating public health uses of health information exchange. J Biomed Inform. 2007;40(6 Suppl):S46-

9. PMID: 17919985.

Exclusion: No data relevant to a Key Question

Shapiro JS, Bartley J, Kuperman G. Initial Experience with Opt-in Consent at the New York Clinical Information

Exchange (NYCLIX). AMIA Annu Symp Proc. 2009:1029. 

Exclusion: Wrong study design

Shapiro JS, Genes N, Kuperman G, et al. Health information exchange, biosurveillance efforts, and emergency 

department crowding during the spring 2009 H1N1 outbreak in New York City. Ann Emerg Med. 2010;55(3):274-9. 

PMID: 20079955.

Exclusion: Wrong study design

Shapiro JS, Kannry J, Kushniruk AW, et al. Emergency physicians' perceptions of health information exchange. J 

Am Med Inform Assoc. 2007;14(6):700-5. PMID: 17712079.

Exclusion: Not HIE

Shapiro JS, Kannry J, Lipton M, et al. Approaches to patient health information exchange and their impact on 

emergency medicine. Ann Emerg Med. 2006;48(4):426-32. PMID: 16997679.

Exclusion: Not HIE

Shapiro JS, Onyile A, Genes N, et al. Validating health information exchange data for quality measurement. Ann 

Emerg Med. 2013;62(4):S94. 

Exclusion: More recent data available

Shapiro JS, Onyile A, Patel VR, et al. Enabling 72-hour emergency department returns measurement with regional 

data from a health information exchange. Ann Emerg Med. 2011;58(4):S295. 

Exclusion: Wrong study design

Shapiro JS, Vaidya SR, Kuperman G. Preparing for the evaluation of health information exchange. AMIA Annu 

Symp Proc. 2008:1128. PMID: 18999179.

Exclusion: Wrong study design

Shaw KJ, Gutierrez M, Fridman M, et al. Health care costs associated with changing clinics and "walk-in" 

deliveries: evidence supporting a regionalized health information network. Am J Obstet Gynecol. 

2008;198(6):707.e1-8; discussion .e8. PMID: 18448082.

Exclusion: No data relevant to a Key Question
D-46

Shekelle PG, Morton SC, Keeler EB. Costs and benefits of health information technology. Evid Rep Technol Assess 

(Full Rep). 2006(132):1-71. PMID: 17627328.

Exclusion: No data relevant to a Key Question

Shields AE, Shin P, Leu MG, et al. Adoption of health information technology in community health centers: results 

of a national survey. Health Aff (Millwood). 2007;26(5):1373-83. PMID: 17848448.

Exclusion: Not HIE

Shih FJ, Fan YW, Chiu CM, et al. Needs for providing overseas organ transplant medical function and information 

with eHealth telecare systems-instrument development for health professionals in Taiwan. Transplant Proc. 

2014;46(4):1014-8. PMID: 24815115.

Exclusion: No data relevant to a Key Question

Shih FJ, Shih FJ, Pan YJ, et al. Dilemma of applying telehealth for overseas organ transplantation: comparison on 

perspectives of health professionals and e-health information and communication technologists in Taiwan. 

Transplant Proc. 2014;46(4):1019-21. PMID: 24815116.

Exclusion: Not HIE

Shy BD, Shapiro JS, Shearer PL, et al. A conceptual framework for improved analyses of 72-hour return cases. Am 

J Emerg Med. (0)PMID: 25303847.

Exclusion: Not HIE

Silva PL. Planning for productivity. A Michigan health plan leverages its PM and EMR systems to improve the 

bottom line and speed access to business intelligence. Health Manag Technol. 2008;29(4):32-3, 7. PMID: 18468217.

Exclusion: Not HIE

Simon JS, Rundall TG, Shortell SM. Adoption of order entry with decision support for chronic care by physician 

organizations. J Am Med Inform Assoc. 2007;14(4):432-9. PMID: 17460136.

Exclusion: Not HIE

Simon SR, Evans JS, Benjamin A, et al. Patients' attitudes toward electronic health information exchange: 

qualitative study. J Med Internet Res. 2009;11(3):e30. PMID: 19674960.

Exclusion: Not HIE

Simon SR, Kaushal R, Cleary PD, et al. Physicians and electronic health records: a statewide survey. Arch Intern 

Med. 2007;167(5):507-12. PMID: 17353500.

Exclusion: Not HIE

Simonaitis L, Belsito A, Warvel J, et al. Extensible Stylesheet Language Formatting Objects (XSL-FO): a tool to 

transform patient data into attractive clinical reports. AMIA Annu Symp Proc. 2006:719-23. PMID: 17238435.

Exclusion: Not HIE

Simons WW, Halamka JD, Kohane IS, et al. Integration of the personally controlled electronic medical record into 

regional inter-regional data exchanges: a national demonstration. AMIA Annu Symp Proc. 2006:1099. PMID: 

17238718.

Exclusion: Wrong study design

Slade K, Lambert MJ, Harmon SC, et al. Improving psychotherapy outcome: The use of immediate electronic 

feedback and revised clinical support tools. Clin Psychol Psychother. 2008;15(5):287-303. PMID: 19115449.

Exclusion: Not HIE

Smith E, Kaufman JH. Lowering the barrier to a decentralized NHIN using the open healthcare framework. Stud 

Health Technol Inform. 2006;121:214-20. PMID: 17095820.

Exclusion: Wrong study design
D-47

Smith ME, Newcombe HB. Automated follow-up facilities in Canada for monitoring delayed health effects. Am J 

Public Health. 1980;70(12):1261-8. PMID: 7435743.

Exclusion: Not HIE

Social Security A. Obtaining evidence beyond the current "special arrangement sources." Interim final rule with 

request for comments. Fed Regist. 2014;79(113):33681-3. PMID: 24922983.

Exclusion: No data relevant to a Key Question

Sokolova M, El Emam K, Arbuckle L, et al. P2P watch: personal health information detection in peer-to-peer filesharing networks. J Med Internet Res. 2012;14(4):e95. PMID: 22776692.

Exclusion: No data relevant to a Key Question

Solberg D. 'Pipe dream' HIE proves challenging. Health Manag Technol. 2009;30(7):22. PMID: 19739562.

Exclusion: Wrong study design

Solomon MR. Regional health information organizations: a vehicle for transforming health care delivery? J Med 

Syst. 2007;31(1):35-47. PMID: 17283921.

Exclusion: Wrong study design

Solutions DCfH. Health Information Exchange (HIE) Business Models: The Path to Sustainable Financial Success. 

Available at: http://www.providersedge.com/ehdocs/ehr_articles/Health_Info_Exchange_Business_Models.pdf. 

Accessed April 22, 2014.

Exclusion: Wrong study design

Soti P, Pandey S. Business process optimization for RHIOs. J Healthc Inf Manag. 2007;21(1):40-7. PMID: 

17299924.

Exclusion: No data relevant to a Key Question

Spahni S, Guardia A, Boggini T, et al. Design and Implementation of a Shared Treatment Plan in a Federated Health 

Information Exchange... MEDINFO 2013. Stud Health Technol Inform. 2013;192:1090. PMID: 23920864.

Exclusion: Wrong study design

Spath MB, Grimson J. Applying the archetype approach to the database of a biobank information management 

system. Int J Med Inf. 2011;80(3):205-26. PMID: 21131230.

Exclusion: Not HIE

Spil TA, Schuring RW, Stegwee RA, et al. Towards a better understanding of the e-health user: comparing USE IT 

and Requirements study for an Electronic Patient Record. Available at: http://doc.utwente.nl/55471/. Accessed 

April 22, 2015. 

Exclusion: Not HIE

Sprivulis P, Walker J, Johnston D, et al. The economic benefits of health information exchange interoperability for 

Australia. AMIA Annu Symp Proc. 2005:1119. PMID: 16779406.

Exclusion: Wrong study design

Sprivulis P, Walker J, Johnston D, et al. The economic benefits of health information exchange interoperability for 

Australia. Aust Health Rev. 2007;31(4):531-9. PMID: 17973611.

Exclusion: Wrong study design

Sridhar S, Brennan PF, Wright SJ, et al. Optimizing financial effects of HIE: a multi-party linear programming 

approach. J Am Med Inform Assoc. 2012;19(6):1082-8. PMID: 22733978.

Exclusion: Wrong study design

Stair TO. Reduction of redundant laboratory orders by access to computerized patient records. The Journal of 

emergency medicine. 1998;16(6):895-7. PMID: 9848709.

Exclusion: Not HIE
D-48

Stansfield K, Yetman L, Renwick C. eDoc evaluation - At eighteen months into the challenge. Stud Health Technol 

Informatics. 2009;143:414-8. PMID: 19380970.

Exclusion: Not HIE

Starr P. Smart technology, stunted policy: developing health information networks. Health Aff. 1997;16(3):91-105. 

PMID: 9141326.

Exclusion: Wrong study design

Stiell A, Forster AJ, Stiell IG, et al. Prevalence of information gaps in the emergency department and the effect on 

patient outcomes. CMAJ. 2003;169(10):1023-8. PMID: 14609971.

Exclusion: Not HIE

Stolyar A, Lober WB, Drozd DR, et al. Feasibility of data exchange with a Patient-centered Health Record. AMIA 

Annu Symp Proc. 2005:1123. PMID: 16779410.

Exclusion: Not HIE

Stoves J, Connolly J, Cheung CK, et al. Electronic consultation as an alternative to hospital referral for patients with 

chronic kidney disease: a novel application for networked electronic health records to improve the accessibility and 

efficiency of healthcare. Qual Saf Health Care. 2010;19(5):e54-e. PMID: 20554576.

Exclusion: Wrong study design

Strasberg HR, Hubbs PR, Rindfleisch TC, et al. Analysis of information needs of users of the Stanford Health 

Information Network for Education. Proc AMIA Symp. 1999;Annual Symposium.:965-9. PMID: 10566504.

Exclusion: Not HIE

Stroetmann V, Thiel R, Stroetmann KA, et al. Understanding the role of device level interoperability in promoting 

health - lessons learned from the SmartPersonalHealth Project. Yearb Med Inform. 2011;6(1):87-91. PMID: 

21938330.

Exclusion: Wrong study design

Sucurovic S. Implementing security in a distributed web-based EHCR. Int J Med Inform. 2007;76(5-6):491-6. 

PMID: 17084662.

Exclusion: No data relevant to a Key Question

Suhanic W, Crandall I, Pennefather P. An informatics model for guiding assembly of telemicrobiology workstations 

for malaria collaborative diagnostics using commodity products and open-source software. Malar J. 2009;8:164. 

PMID: 19615074.

Exclusion: Wrong study design

Sullivan FM, McEwan N, Murphy G. Regional repositories, reintermediation and the new GMS contract: 

cardiovascular disease in Tayside. Inform Prim Care. 2003;11(4):215-21. PMID: 14980061.

Exclusion: Not HIE

Szende A. Ontario's province-wide paediatric electronic health record. Stud Health Technol Inform. 2009;143:99-

103. PMID: 19380922.

Exclusion: No data relevant to a Key Question

Takada A, Guo sJ, Tanaka K, et al. Dolphin project--cooperative regional clinical system centered on clinical 

information center. J Med Syst. 2005;29(4):391-400. PMID: 16178336.

Exclusion: No data relevant to a Key Question

Takeda H, Matsumura Y, Kuwata S, et al. An assessment of PKI and networked electronic patient record system: 

lessons learned from real patient data exchange at the platform of OCHIS (Osaka Community Healthcare 

Information System). Int J Med Inform. 2004;73(3):311-6. PMID: 15066564.

Exclusion: Not HIE
D-49

Takeda H, Matsumura Y, Nakagawa K, et al. Healthcare public key infrastructure (HPKI) and non-profit 

organization (NPO): essentials for healthcare data exchange. Stud Health Technol Inform. 2004;107(Pt 2):1273-6. 

PMID: 15361019.

Exclusion: Wrong study design

Tamblyn R, Poissant L, Huang A, et al. Estimating the information gap between emergency department records of 

community medication compared to on-line access to the community-based pharmacy records. J Am Med Inform 

Assoc. 2014;21(3):391-8. PMID: 23956015.

Exclusion: Not HIE

Tambouris E, Williams MH, Makropoulos C. Co-operative health information networks in Europe: experiences 

from Greece and Scotland. J Med Internet Res. 2000;2(2):E11. PMID: 11720930.

Exclusion: Not HIE

Tan SL, Lewis RA. Picture archiving and communication systems: a multicentre survey of users experience and 

satisfaction. Eur J Radiol. 2010;75(3):406-10. PMID: 19523778.

Exclusion: Not HIE

Tchwenko SN, Parnell H, Messer LC. Health outcomes following a health information exchange intervention for 

HIV patients. Am J Epidemiol. 2012;175:S13. 

Exclusion: Wrong study design

Teixeira PA, Gordon P, Camhi E, et al. HIV patients' willingness to share personal health information electronically. 

Patient Educ Couns. 2011;84(2):e9-12. PMID: 20724095.

Exclusion: Not HIE

Tello-Leal E, Chiotti O, Villarreal PD. Process-oriented integration and coordination of healthcare services across 

organizational boundaries. J Med Syst. 2012;36(6):3713-24. PMID: 22434534.

Exclusion: Not HIE

Tennison J, Rajeev D, Woolsey S, et al. The Utah Beacon Experience: Integrating Quality Improvement, Health 

Information Technology, and Practice Facilitation to Improve Diabetes Outcomes in Small Healthcare Facilities. 

EGEMS (Wash DC). 2014;2(3)

Exclusion: Not HIE

Terry K. Electronic exchange of health information dials in new patient consent questions. Med Econ. 

2014;91(13):46-50. PMID: 25174225.

Exclusion: No data relevant to a Key Question

Tham E, Ross SE, Mellis BK, et al. Interest in health information exchange in ambulatory care: a statewide survey. 

Appl Clin Inform. 2010;1(1):1-10. PMID: 23616824.

Exclusion: No data relevant to a Key Question

Thielst CB. Regional health information networks and the emerging organizational structures. J Healthc Manag. 

2007;52(3):146-50. PMID: 17552351.

Exclusion: Wrong study design

Thorn SA. Emergency physicians' perspectives on the usability of health information exchange. Diss Abstr Int. 

2012;72(7-A):2200. 

Exclusion: More recent data available

Thorn SA, Carter MA. The Potential of Health Information Exchange to Assist Emergency Nurses. J Emerg Nurs. 

2013;39(5):e91-6. PMID: 23369772.

Exclusion: Wrong study design
D-50

Thornewill J, Dowling AF, Cox BA, et al. Information infrastructure for consumer health: a health information 

exchange stakeholder study. Am J Prev Med. 2011;40(5 Suppl 2):S123-33. PMID: 21521585.

Exclusion: No data relevant to a Key Question

Tierney WM, Overhage JM, McDonald CJ. Toward electronic medical records that improve care. Ann Intern Med. 

1995;122(9):725-6. PMID: 7702235.

Exclusion: Wrong study design

Ting S, Kwok S, Tsang A, et al. Experiences Sharing of Implementing Template-Based Electronic Medical Record 

System (TEMRS) in a Hong Kong Medical Organization. J Med Syst. 2011;35(6):1605-15. PMID: 20703758.

Exclusion: Not HIE

Tjora A, Tran T, Faxvaag A. Privacy vs usability: a qualitative exploration of patients' experiences with secure 

Internet communication with their general practitioner. J Med Internet Res. 2005;7(2):e15. PMID: 15998606.

Exclusion: Not HIE

Tomines A, Readhead H, Readhead A, et al. Applications of Electronic Health Information in Public Health: Uses, 

Opportunities and Barriers. EGEMS (Wash DC). 2013;1(2)

Exclusion: Wrong study design

Törnvall E, Wilhelmsson S. Nursing documentation for communicating and evaluating care. J Clin Nurs. 

2008;17(16):2116-24. PMID: 18710374.

Exclusion: Not HIE

Toussaint JS, Queram C, Musser JW. Connecting statewide health information technology strategy to payment 

reform. Am J Manag Care. 2011;17(3):e80-8. PMID: 21504263.

Exclusion: Not HIE

Trigg LJ. Social construction of the patient through problems of safety, uninsurance, and unequal treatment. ANS 

Adv Nurs Sci. 2009;32(3):E17-27. PMID: 19707084.

Exclusion: Not HIE

Triska OH, Church J, Wilson D, et al. Physicians' perceptions of integration in three Western Canada Health 

Regions. Healthc Manage Forum. 2005;18(3):18-24. PMID: 16323465.

Exclusion: Not HIE

Tsiknakis M, Brochhausen M, Nabrzyski J, et al. A semantic grid infrastructure enabling integrated access and 

analysis of multilevel biomedical data in support of postgenomic clinical trials on cancer. IEEE Trans Inf Technol 

Biomed. 2008;12(2):205-17. PMID: 18348950.

Exclusion: Not HIE

Tsiknakis M, Kouroubali A. Organizational factors affecting successful adoption of innovative eHealth services: a 

case study employing the FITT framework. Int J Med Inf. 2009;78(1):39-52. PMID: 18723389.

Exclusion: Not HIE

Tufano JT. Information and communication technologies in patient-centered healthcare redesign: Qualitative studies 

of provider experience [Ph.D.]. Ann Arbor, University of Washington; 2009.

Exclusion: Wrong study design

Tuttle MS, Nelson SJ. The role of the UMLS in 'storing' and 'sharing' across systems. Int J Biomed Comput. 

1994;34(1-4):207-37. PMID: 8125633.

Exclusion: Wrong study design

Ullman K. Indiana data network provides one stop for inter-hospital connectivity. How an Indiana-based regional 

health data exchange helps CIOs save time and money. Healthc Inform. 2010;27(8):32. PMID: 20853808.

Exclusion: Wrong study design
D-51

Unertl KM, Weinger M, Johnson K. Variation in use of informatics tools among providers in a diabetes clinic. 

AMIA Annu Symp Proc. 2007:756-60. PMID: 18693938.

Exclusion: Not HIE

Vaidya SR, Shapiro JS, Papa AV, et al. Perceptions of health information exchange in home healthcare. Comput

Inform Nurs. 2012;30(9):503-9. PMID: 22584878.

Exclusion: Not HIE

van der Linden H, Kalra D, Hasman A, et al. Inter-organizational future proof EHR systems. A review of the 

security and privacy related issues. Int J Med Inf. 2009;78(3):141-60. PMID: 18760661.

Exclusion: Wrong study design

van der Linden MW, Plat AW, Erkens JA, et al. Large impact of antidiabetic drug treatment and hospitalizations on 

economic burden of diabetes mellitus in The Netherlands during 2000 to 2004. Value Health. 2009;12(6):909-14. 

PMID: 19508664.

Exclusion: Not HIE

Van der Velde ET, Atsma DE, Foeken H, et al. Remote monitoring of patients with implanted devices: data 

exchange and integration. Eur J Prev Cardiolog. 2013;20(2 Suppl):8-12. PMID: 23702984.

Exclusion: Not HIE

Van Eaton EG, Devlin AB, Devine EB, et al. Achieving and Sustaining Automated Health Data Linkages for 

Learning Systems: Barriers and Solutions. EGEMS (Wash DC). 2014;2(2)

Exclusion: No data relevant to a Key Question

van Walraven C, Taljaard M, Bell CM, et al. Information exchange among physicians caring for the same patient in 

the community. CMAJ. 2008;179(10):1013-8. PMID: 18981442.

Exclusion: Not HIE

van Wingerde FJ, Sun Y, Harary O, et al. Linking multiple heterogeneous data sources to practice guidelines. Proc 

AMIA Symp. 1998;Annual Symposium.:391-5. PMID: 9929248.

Exclusion: Wrong study design

Velamuri S. QRDA--technology overview and lessons learned. J Healthc Inf Manag. 2010;24(3):41-8. PMID: 

20677471.

Exclusion: No data relevant to a Key Question

Vest JR, Gamm LD. Health information exchange: persistent challenges and new strategies. J Am Med Inform 

Assoc. 2010;17(3):288-94. PMID: 20442146.

Exclusion: Wrong study design

Vest JR, Jasperson J. What should we measure? Conceptualizing usage in health information exchange. J Am Med 

Inform Assoc. 2010;17(3):302-7. PMID: 20442148.

Exclusion: Not HIE

Vest JR, Menachemi N, Ford EW. Governance's role in local health departments' information system and 

technology usage. J Public Health Manag Pract. 2012;18(2):160-8. PMID: 22286285.

Exclusion: Not HIE

Virga PH, Jin B, Thomas J, et al. Electronic health information technology as a tool for improving quality of care 

and health outcomes for HIV/AIDS patients. Int J Med Inf. 2012;81(10):e39-45. PMID: 22890224.

Exclusion: Not HIE
D-52

Viswanathan KP, Bass R, Wijetunge G, et al. Rural mass casualty preparedness and response: the Institute of 

Medicine's Forum on Medical and Public Health Preparedness for Catastrophic Events. Disaster Med Public Health 

Prep. 2012;6(3):297-302. PMID: 23077273.

Exclusion: Wrong study design

Voigt C, Torzewski S. Direct results. An HIE tests simple information exchange using the direct project. J AHIMA. 

2011;82(5):38-41. PMID: 21667863.

Exclusion: No data relevant to a Key Question

Wagner PJ, Dias J, Howard S, et al. Personal health records and hypertension control: a randomized trial. J Am Med 

Inform Assoc. 2012;19(4):626-34. PMID: 22234404.

Exclusion: Not HIE

Walker J, Pan E, Johnston D, et al. The value of health care information exchange and interoperability. Health Aff 

(Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8. PMID: 15659453.

Exclusion: Wrong study design

Walker JM, Carayon P. From tasks to processes: the case for changing health information technology to improve 

health care. Health Aff (Millwood). 2009;28(2):467-77. PMID: 19276006.

Exclusion: Not HIE

Walker R, Blacker V, Pandita L, et al. Learning from the implementation of inter-organisational web-based care 

planning and coordination. Aust J Prim Health. 2013;19(4):297-302. PMID: 23866768.

Exclusion: Not HIE

Walsh MN, Albert NM, Curtis AB, et al. Lack of association between electronic health record systems and 

improvement in use of evidence-based heart failure therapies in outpatient cardiology practices. Clin Cardiol. 

2012;35(3):187-96. PMID: 22328100.

Exclusion: Not HIE

Warnekar PP, Bouhaddou O, Parrish F, et al. Use of RxNorm to exchange codified drug allergy information 

between Department of Veterans Affairs (VA) and Department of Defense (DoD). AMIA Annu Symp Proc. 

2007:781-5. PMID: 18693943.

Exclusion: Not HIE

Weber GM. Federated queries of clinical data repositories: the sum of the parts does not equal the whole. J Am Med 

Inform Assoc. 2013;20(e1):e155-61. PMID: 23349080.

Exclusion: Not HIE

Weber SC, Lowe H, Das A, et al. A simple heuristic for blindfolded record linkage. J Am Med Inform Assoc. 

2012;19(e1):e157-61. PMID: 22298567.

Exclusion: Not HIE

Weber SC, Seto T, Olson C, et al. Oncoshare: lessons learned from building an integrated multi-institutional 

database for comparative effectiveness research. AMIA Annu Symp Proc. 2012;2012:970-8. PMID: 23304372.

Exclusion: Wrong study design

Webster PC. Infoway tacks towards "networked" patients. CMAJ. 2011;183(4):E223-4. PMID: 21324865.

Exclusion: Wrong study design

Weitzman ER, Kelemen S, Kaci L, et al. Willingness to share personal health record data for care improvement and 

public health: a survey of experienced personal health record users. BMC Med Inform Decis Mak. 2012;12:39. 

PMID: 22616619.

Exclusion: Not HIE
D-53

Wells S, Hill-Smith I. Bridging the communication gap in diabetes care. Practical Diabetes International. 

1996;13(6):174-6. 

Exclusion: Not HIE

Wen K-Y, Kreps G, Zhu F, et al. Consumers' perceptions about and use of the internet for personal health records 

and health information exchange: analysis of the 2007 Health Information National Trends Survey. J Med Internet 

Res. 2010;12(4):e73. PMID: 21169163.

Exclusion: Not HIE

Were MC, Meeks-Johnson J, Overhage JM. Enhanced laboratory reports: using health information exchange data to 

provide contextual information to laboratory results for practices without electronic records. AMIA Annu Symp 

Proc. 2008:1174. PMID: 18999174.

Exclusion: Not HIE

Westbrook JI, Braithwaite J, Georgiou A, et al. Multimethod evaluation of information and communication 

technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assoc. 

2007;14(6):746-55. PMID: 17712083.

Exclusion: Not HIE

Westbrook JI, Braithwaite J, Iedema R, et al. Evaluating the impact of information communication technologies on 

complex organizational systems: a multi-disciplinary, multi-method framework. Stud Health Technol Inform. 

2004;107(Pt 2):1323-7. PMID: 15361029.

Exclusion: Not HIE

Wilcox AB, Shen S, Dorr DA, et al. Improving access to longitudinal patient health information within an 

emergency department. Appl Clin Inform. 2012;3(3):290-300. PMID: 23646076.

Exclusion: Not HIE

Wiljer D, Urowitz S, Apatu E, et al. Patient accessible electronic health records: exploring recommendations for 

successful implementation strategies. J Med Internet Res. 2008;10(4):e34. PMID: 18974036.

Exclusion: Not HIE

Willis E. Engagement in online health communities: Expressed attitudes and self-efficacy of arthritis selfmanagement behaviors. Diss Abstr Int. 2011;74(4-A(E)).

Exclusion: Wrong study design

Wilt DH, Muthig BA. Crossing barriers: EMR implementation across a nationwide continuum of care. J Healthc Inf 

Manag. 2008;22(2):23-6. PMID: 19266991.

Exclusion: No data relevant to a Key Question

Winthereik B, Vikkelsø S. ICT and Integrated Care: Some Dilemmas of Standardising Inter-Organisational 

Communication. Comput Support Coop Work. 2005;14(1):43-67. 

Exclusion: Not HIE

Wong HJ, Caesar M, Bandali S, et al. Electronic inpatient whiteboards: improving multidisciplinary communication 

and coordination of care. Int J Med Inform. 2009;78(4):239-47. PMID: 18786851.

Exclusion: Not HIE

Woods SE, Coggan JM. Developing a medical informatics education program to support a statewide health 

information network. Bull Med Libr Assoc. 1994;82(2):147-52. PMID: 8004015.

Exclusion: Not HIE

Woodside JM. EDI and ERP: a real-time framework for healthcare data exchange. J Med Syst. 2007;31(3):178-84. 

PMID: 17622020.

Exclusion: Wrong study design
D-54

Wright A, Soran C, Jenter CA, et al. Physician attitudes toward health information exchange: results of a statewide 

survey. J Am Med Inform Assoc. 2010;17(1):66-70. PMID: 20064804.

Exclusion: No data relevant to a Key Question

Wu M, Rhyner P. Design of an integrated system for Milwaukee children with developmental disabilities. AMIA 

Annu Symp Proc. 2005:1156. PMID: 16779442.

Exclusion: No data relevant to a Key Question

Wynn A, Wise M, Wright MJ, et al. Accuracy of administrative and trauma registry databases. J Trauma. 

2001;51(3):464-8. PMID: 11535892.

Exclusion: Not HIE

Yang W-H, Hu J-S, Chou Y-Y. Analysis of network type exchange in the health care system: a stakeholder 

approach. J Med Syst. 2012;36(3):1569-81. PMID: 21046205.

Exclusion: Not HIE

Yaraghi N, Du AY, Sharman R, et al. Professional and geographical network effects on healthcare information 

exchange growth: does proximity really matter? J Am Med Inform Assoc. 2014;21(4):671-8. PMID: 24287171.

Exclusion: Wrong study design

Yasnoff WA, Humphreys BL, Overhage JM, et al. A consensus action agenda for achieving the national health 

information infrastructure. J Am Med Inform Assoc. 2004;11(4):332-8. PMID: 15187075.

Exclusion: Not HIE

Yee KC, Miils E, Airey C. Perfect match? Generation Y as change agents for information communication 

technology implementation in healthcare. Stud Health Technol Inform. 2008;136:496-501. PMID: 18487780.

Exclusion: Not HIE

Zafar A, Dixon BE. Pulling back the covers: technical lessons of a real-world health information exchange. Stud 

Health Technol Inform. 2007;129(Pt 1):488-92. PMID: 17911765.

Exclusion: Wrong study design

Zhao J, Zhang Z, Guo H, et al. E-health in China: challenges, initial directions, and experience. Telemed J E Health. 

2010;16(3):344-9. PMID: 20406121.

Exclusion: Wrong study design

Zimmerman CR, Chaffee BW, Lazarou J, et al. Maintaining the enterprisewide continuity and interoperability of 

patient allergy data. Am J Health-Syst Pharm. 2009;66(7):671-9. PMID: 19299376.

Exclusion: Not HIE

Zulman DM, Nazi KM, Turvey CL, et al. Patient interest in sharing personal health record information: a web-based 

survey. Ann Intern Med. 2011;155(12):805-10. PMID: 22184687.

Exclusion: Not HIE

Zulman DM, Piette JD, Jenchura EC, et al. Facilitating out-of-home caregiving through health information 

technology: survey of informal caregivers' current practices, interests, and perceived barriers. J Med Internet Res. 

2013;15(7):e123. PMID: 23841987.

Exclusion: Not HIE

Zvarova J, Lhotska L, Seidl L, et al. Health data collecting and sharing: case studies of Czech e-health applications. 

Stud Health Technol Inform. 2012;180:672-6. PMID: 22874276.

Exclusion: Not HIE
E-1

Appendix E. Study Design Terminology

The studies included in this review are described in terms of their design, data source and 

analysis approach. Study designs are included in summary tables, while all three characteristics 

may be discussed in the text.

1) Study design:

Randomized controlled trial

Cohort (prospective or retrospective)

Case Control (be sure it actually is)

Cross-sectional

Time Series

Multiple site case studies

Case series

2) Data Source:

Database (administrative data, clinical data)

Survey, questionnaire, focus group

Audit logs

Observations

Documents

3) Analysis:

Quantitative

Descriptive statistics

Regression/Other multivariable analysis

Qualitative

Content or Thematic Analysis

Mixed methods

Includes quantitative and qualitative

Narrative description

Ethnographic
F-1

Appendix F. Evidence Table

Table F1. Evidence Table

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Abramson, et al.,

201276

Cross-sectional Measure EHR and HIE adoption

in New York State hospitals

New York State Hospital Survey of hospitals May-December 2009

Abramson, et al.,

201477

Cross-sectional Measure EHR and HIE adoption

in New York State nursing homes

New York State Nursing homes Survey of nursing homes November 2011-

March 2012

Abramson, et al.,

201496

Cross-sectional To determine rates of

participation in HIE

New York Hospitals Survey responses November 2012 -

February 2013
F-2

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Abramson, et al.,

201276

Various HIEs around New York

State

Type of data exchanged NR NR All 205 hospitals in New York

State

Abramson, et al.,

201477

Nursing homes around New York

State

Exchange of data (NR) with pharmacies, lab, hospitals,

physician offices, and RHIO

NR All 632 nursing homes in New

York State

Abramson, et al.,

201496

NA NA NA Surveyed Hospital IT directors

or chief information officer
F-3

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Abramson, et al.,

201276

Various HIEs All hospitals in New York State NA None

Abramson, et al.,

201477

Various HIEs All nursing homes in New York

State

NA None

Abramson, et al.,

201496

Contacted: 210 Hospitals

Respondents: 129 (61.4%)

Nonrespondents: 81 (38.6%)

All hospitals in New York state NA Results compared
F-4

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Abramson, et al.,

201276

Participation in HIE Participate in HIE (exchange of data) NA Quantitative

Descriptive statistics

Abramson, et al.,

201477

Participation in HIE Participate in HIE (exchange of data) NA Quantitative

Descriptive statistics

Abramson, et al.,

201496

Use of HIE, if information is sent and/or

received by the institution, type of

institution information is shared with,

barriers to implementation

NA NA Descriptive statistics
F-5

Author, Year Results

Risk of

Bias

Abramson, et al.,

201276

23% of respondent hospitals participate and exchange data vs. 37% participate but do not exchange data vs. 40% do not participate Low

Abramson, et al.,

201477

54.4% participate in HIE,

OR of participating in HIE: 2.26 more likely when have EHR

Exchange with providers when EHR

59.7% within system vs. 31.3% outside system

HIE highest usage

Pharmacies: 41.8%

Labs: 38.5%

Hospitals: 38 5%

Low

Abramson, et al.,

201496

-79.1% (n=102) of respondents reported actively exchanging any electronic patient-level clinical data with an entity outside their institution

in 2012 vs. 60% in 2009

Type of institution respondents exchanged data with:

Hospitals outside your system: 70.6% (n=72) 

Ambulatory providers outside your system: 68.6% (n=70)

Long term care facilities: 45.1% (n=46)

Home health agencies: 38.2% (n=39)

The most commonly exchanged data were radiology reports, followed by laboratory results. Only 45 respondents (44.1%) exchanged

medication lists and clinical history with hospitals outside their system.

Respondents reporting participation in a regional arrangement for HIE:

Any data exchange: 89.9% (n=116)

Actively sending and receiving data: 50.9% (n=59)

Sending data only: 25.9% (n=30)

Receiving data only: 16.4% (n=19)

Barriers to HIE participation reported by responding hospitals:

Privacy concerns: 54.7% (n=70)

Security concerns: 52.3% (n=67)

Lack of IT staff to support HIE: 38.2% (n=49)

Lack of architecture to support HIE: 35.9% (n-46)

No differences in barriers among hospitals engaging in HIE and those not engaging in HIE were found. When hospitals engaged in

sending and receiving data were compared with hospitals only sending or only receiving data hospitals only engaged in one activity were

more likely to identify lack of architecture p=0.05 and cost of participating p=0.03 as barriers to HIE

Moderate
F-6

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Adjerid and

Padman, 2011140

Cross-sectional -Analyze data from compilation of

privacy laws and Adler-Milstein

2009 analysis of RHIOs

-Examine association of state

"consent prior to disclosure" laws

with number of operational HIEs

U.S. Any Survey

Data from compilation of privacy

laws and Adler-Milstein 2009

analysis of RHIOs

2009-2010

Adler-Milstein and

Jha, 2014108

Cross-sectional -Analyze data from annual AHA

survey of hospital IT

-Measure HIE usage among U.S.

hospitals

U.S. Any Survey

Hospital survey database,

augmented with market and other

characteristic data

Late 2012

Adler-Milstein,

Bates, and Jha,

201179

Cross-sectional Measure number of RHIOs,

participation in them by

ambulatory practices and

hospitals, and number financially

viable

U.S. Any Survey of RHIOs June 2008-December

2009

Adler-Milstein,

Bates, and Jha,

201325

Cross-sectional Measurement of types of data

exchanged, organizations

involved, and sources of financial

support

U.S. Any Survey of HIE organizations August-November

2012

Adler-Milstein,

DesRoches, and

Jha, 2011107

Cross-sectional Measurement of participation in a

regional HIO and exchange of

data with hospitals or ambulatory

providers of a different system

U.S. Hospital Hospital survey database AHA survey from

spring-summer 2009
F-7

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Adjerid and

Padman, 2011140

All in U.S. All types NA 313 HIE initiatives from 2004-

2009

Adler-Milstein and

Jha, 2014108

All in U.S. All types NA 2,849 U.S. hospitals that

responded to AHA IT survey

Adler-Milstein,

Bates, and Jha,

201179

All in U.S. All types provided by a RHIO NA 197 organizations meeting

definition of RHIO

Adler-Milstein,

Bates, and Jha,

201325

All in U.S. All types NA 221 organizations facilitating

HIE

Adler-Milstein,

DesRoches, and

Jha, 2011107

All in U.S. All types NA 3,101 acute-care, nonfederal

hospitals that were U.S. based

members of AHA
F-8

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Adjerid and

Padman, 2011140

All 313 HIE initiatives HIE status; state health

disclosure law status

None None

Adler-Milstein and

Jha, 2014108

All of population All hospitals responding to

survey

None None

Adler-Milstein,

Bates, and Jha,

201179

165 RHIOs All RHIOs Not meeting definition of

RHIO

None

Adler-Milstein,

Bates, and Jha,

201325

NA All organizations facilitating HIE Organizations only

participating in HIE

None

Adler-Milstein,

DesRoches, and

Jha, 2011107

Various HIEs All acute-care, nonfederal

hospitals that were U.S. based

members of AHA

Hospitals that were federal

or nonacute or were not

members of AHA

None
F-9

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Adjerid and

Padman, 2011140

Total, operational, and failed HIE -Health disclosure law

-Population

-Per capita GDP

HIE size not accounted for Quantitative

Econometric models

Adler-Milstein and

Jha, 2014108

Participating in HIE -Ownership

-Market position

-Size

-Teaching status

-Cardiac ICU

-System affiliation

-Medicaid admissions

-EHR system

NA Quantitative Multivariate

Analysis

OR of likelihood of participation

Adler-Milstein,

Bates, and Jha,

201179

Operational RHIOs, supporting stage 1

meaningful use, ambulatory practices and

hospitals participating in RHIOs, and

number of financially viable

Operational RHIOs, supporting stage 1

meaningful use, ambulatory practices

and hospitals participating in RHIOs,

and number of financially viable

NA Quantitative

Adler-Milstein,

Bates, and Jha,

201325

Operational exchange or data, types of

data exchanged, barriers to exchange

Operational exchange or data, types of

data exchanged, barriers to exchange

NA Quantitative

Descriptive statistics; compared

with previous reports

Adler-Milstein,

DesRoches, and

Jha, 2011107

Participation in HIE and market

characteristics

-Hospital profit status

-Market share

-Teaching status

-Size

-Cardiac ICU

-System affiliation

-Medicaid admissions

-EHR system

NA Quantitative

Analysis of database

Logistic regression models
F-10

Author, Year Results

Risk of

Bias

Adjerid and

Padman, 2011140

States with stronger privacy laws have more operational HIEs, fewer failed HIEs, and take less time to reach operational status. NA

Adler-Milstein and

Jha, 2014108

-30% of hospitals engage in HIE, varying widely by state

-For-profit hospitals less likely to engage than nonprofit hospitals. Hospitals with larger market share or in less competitive markets more

likely to exchange

Low

Adler-Milstein,

Bates, and Jha,

201179

-75 operational RHIOs, covering 14% of U.S. hospitals and 3% of ambulatory practices

-13 supporting meaningful use, covering 3% of hospitals, 0.9% of ambulatory practices; 67% not meeting criteria for financial viability

Low

Adler-Milstein,

Bates, and Jha,

201325

Predominant organization nonprofit;

Sources of support

Grants and contracts: 52%; participant fees: 28%; operating costs not covered by revenue: 57%

Barriers to development

Sustainability: 74%; lack of funding: 57%; privacy: 60%; mandates: 55%; technical barriers: 61%; competition: 56%; linking; 54%

Low

Adler-Milstein,

DesRoches, and

Jha, 2011107

10.7% participation in regional HIO; statistically significantly higher for private/nonprofit status, greater market bed share, teaching status,

large size, cardiac ICU presence, and had EHR system

Low
F-11

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Adler-Milstein, et 

al., 200881

Cross-sectional Measurement of activities and

financing of functioning RHIOs

U.S. Any Survey of RHIOs July 2006-March

2007

Adler-Milstein,

Bates, and Jha,

200978

Cross-sectional Measurement of types of data

exchanged, organizations

involved, and sources of financial

support

U.S. Any Survey of operational RHIOs 2008, following up of

survey from 2007

Adler-Milstein,

Landefeld, and 

Jha, 201080

Cross-sectional Measure factors associated with

becoming operational and

achieving financial viability

U.S. Any Survey of RHIOs Mid-2008

Afilalo, et al., 

200766

RCT Impact of sending family

physicians electronic vs. mailed

reports of ED visits for their

patients

Montreal, Canada ED and family

physician

practices

Survey

Survey of family physician

satisfaction

Not stated but likely

same as Lang, 2006
F-12

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Adler-Milstein, et 

al., 200881

All in U.S. All types provided by a RHIO NA 138 organizations meeting

definition of RHIO

Adler-Milstein,

Bates, and Jha,

200978

All in U.S. All types NA 207 organizations defined as

RHIOs

Adler-Milstein,

Landefeld, and 

Jha, 201080

All in U.S. All types provided by a RHIO NA 131 organizations meeting

definition of RHIO

Afilalo, et al., 

200766

Adult university teaching hospital in

Montreal

Report of ED visit sent to family physicians NR Patients visiting ED during

0800-2200
F-13

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Adler-Milstein, et 

al., 200881

32 RHIOs actively exchanging data 20 RHIOs actively exchanging

clinical data for 5000+ patients

Not actively exchanging

data

None

Adler-Milstein,

Bates, and Jha,

200978

All 44 operational RHIOs exchanging data for ≥5,000

patients

All RHIOs exchanging data for

≥5,000 patients

RHIOs not exchanging data

or doing so for <5,000

patients

None

Adler-Milstein,

Landefeld, and 

Jha, 201080

81 RHIOs currently or planning to exchange data for

5000+ patients

81 RHIOs currently or planning

to exchange data for 5000+

patients

Not meeting definition of

RHIO

None

Afilalo, et al., 

200766

2,022 (out of 3,168) patients visiting ED Patients visiting ED Patients in altered mental

state (129), state of

agitation (21), or with

language barrier (29)

ED visit summary provided

electronically vs. on paper sent

by mail
F-14

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Adler-Milstein, et 

al., 200881

Proportion of RHIOs sending and receiving

data to different entities and proportion 

exchanging specific types of data

-Entity sending data

-Entity receiving data

-Type of data exchanged

NA Quantitative

Descriptive statistics

Adler-Milstein,

Bates, and Jha,

200978

RHIO exchanging data for ≥5,000 patients -Types of data

-Entities exchanging data

-Sources of financial support

NA Quantitative

Descriptive statistics

Adler-Milstein,

Landefeld, and 

Jha, 201080

Factors associated with becoming

operational and achieving partial or full

financial viability

-Participation

-Types of data exchanged, focused on

a specific population, history of

collaborating, and sources of revenue

NA Quantitative

Multivariate logistic regression for

predictors

Afilalo, et al., 

200766

Physician attitudes on aspects of continuity

of care for patients

Survey Physicians already are sent carbon

copies of first page of ED note; selfreport of followup data

Quantitative
F-15

Author, Year Results

Risk of

Bias

Adler-Milstein, et 

al., 200881

Entities providing data

Hospitals: 83%; ambulatory settings: 67%; labs: 60%; imaging results: 56%

Entities receiving data

Ambulatory settings: 95%; hospitals: 83%; public health departments: 50%; payers: 44%

Type of data exchanged

Test results: 90%; inpatient data test results: 90%; inpatient data: 70%; medication history: 70%; outpatient data: 60%

Low

Adler-Milstein,

Bates, and Jha,

200978

Source of funding

Time or in-kind resources: 64%; recurring fee: 55%; grant: 48%

Types of data exchanged

Test results: 84%; inpatient data: 70%; medication history: 66%; outpatient data: 64%

28% of operational RHIOs expected to eventually cover operating costs

Barriers

Lack of funding, concerns about privacy/security, legal/regulatory changes, costs higher than expected, technical/infrastructure challenges

Low

Adler-Milstein,

Landefeld, and 

Jha, 201080

Likelihood of being operational associated with exchanging narrow set of data and involving broad group of stakeholders, likelihood of

financial viability associated with involvement of hospitals and ambulatory physicians and early funding from participants. Financial viability

diminished with early grant funding.

Low

Afilalo, et al., 

200766

ED visits followed up by electronic reports led to family physicians having OR of higher rate of information receipt, more useful information,

better knowledge of ED visits, better patient management, and more actions initiated by physicians. There was not perception of higher

rate of followup in family practice offices.

Moderate
F-16

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

AHRQ, 2006166 Multiple Case

Studies

To describe current state HIE

environment and analyze state

HIE activities and initiatives.

National scan, in

depth case

studies of 8

States: Arizona,

Florida, Hawaii,

New York, North

Carolina, Rhode

Island,

Tennessee, Utah

Multiple Multiple Sources

Literature reviews, web-based

research, reports, interviews,

2005-2006

Altman, et al.,

201257

Cross-sectional To assess clinicians’ impressions

of an hourly notification of ED

visit, hospital admission or

hospital discharge with respect to

the notifications effect on the

continuity and coordination of

patient care

New York Family practice

clinics

Survey

Interviews

July 2011-October

2011

Anand, et al., 

201292

Cross-sectional Is real-time alerting useful and

does it lead physicians to take

action?

Indiana Primary care

physician offices

Databases, questionnaire

Survey of value for real-time

alerting for patient ED visit

anywhere in state

June-November 2012

Audet, Squires

and Doty, 2014109

Cross-sectional Measurement of physician

exchange of data outside of

practice or to receive hospital

discharge reports

U.S. Physician offices Surveys March-July, 2012 (as

well as comparison

from data with 2009

survey, specific dates

not provided)
F-17

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

AHRQ, 2006166 Varies Varies 2003 to 2005 All HIE projects in US in 2055-

2006

Altman, et al.,

201257

New York Clinical Information

Exchange (NYCLIX)

Hourly electronic notifications sent to family practice clinicians

when any of 3 patient events occur at a participating hospital:

(1) a new ED visit, (2) a hospital admission, or (3) a hospital

discharge.

November 2010 Family practice clinicians in

single health system receiving

HIE notifications

86% MDs

50% male

Anand, et al., 

201292

Indiana HIE (IHIE) Patient data concerning ED visit 1994 Known physicians (538) of

patients (1,275) seen in an ED

for asthma

Audet, Squires

and Doty, 2014109

All in U.S. Physician exchange of data outside of practice or to receive

hospital discharge reports

NA 1,012 primary care physicians

in 2012
F-18

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

AHRQ, 2006166 101 HIE projects in 35 states for which information was

available.

8 States for in depth case studies

HIE projects that included State

and/or Medicaid involvement,

targeted patients statewide or in

large portions of the state,

involve a RHIO or RHIO like

organization

HIE projects within a single

hospital or health system or

that focused on

administrative exchange or

reducing fraud

Comparison of HIE project

characteristics across states

Altman, et al.,

201257

14 of 20 total Clinicians receiving notifications None Changes in practice as

perceived by interviewee

Anand, et al., 

201292

79 physicians (10%) receiving 126 (15%) notifications Physicians who had ≥1 patient

seen in ED and faxed notification

letter back to HIE

NA Information helpful, resulted

followup action

Audet, Squires,

and Doty, 2014109

Various HIEs Primary care physicians in U.S. NA None
F-19

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

AHRQ, 2006166 Number of HIE projects

Similarities and differences among projects

NA NA Qualitative

Altman, et al.,

201257

Usage logs of number of notifications sent

to each clinician over a period of several

months, questionnaires

NA NA Thematic analysis

Themes of clinician perceptions

identified and compared with

recorded usage logs

Anand, et al., 

201292

Rates of information helpful, resulted in

followup action

Survey None Quantitative

Descriptive statistics

Audet, Squires,

and Doty, 2014109

Proportion of physicians exchanging data

outside of practice or receiving hospital

discharge reports

Proportion of physicians exchanging

data outside of practice or receiving

hospital discharge reports

NA Quantitative

Descriptive statistics and logistic

regression
F-20

Author, Year Results

Risk of

Bias

AHRQ, 2006166 States have multiple HIE projects

Project have similar goals but vary widely across other characteristics, particularly infrastructure which makes sharing lessons learned

challenging

Most projects are in early stages and have overly optimistic timelines

Funding varies widely

Sustainability is a long term goal but has not yet been realized. Most have not identified long term sources of funding

While state are critical stakeholders many do not plan to play primary leadership roles indefinitely.

NA

Altman, et al.,

201257

Notifications from an HIE system can enhance clinicians’ awareness of their patients’ interactions in the medical system. Clinicians

perceived improvements in communication and followup scheduling as a result of notifications. Increase in clinician workload and change

in responsibility may be unintended effects of notifications Workflow issues should be carefully considered. Timely notifications may further 

improve clinician-to-clinician communication

Moderate

Anand, et al., 

201292

-35% found information helpful vs. 20% not helpful

-24% made followup call to patient vs. 4% sent attached letter

NA

Audet, Squires,

and Doty, 2014109

32% use of HIE, with higher proportion for formal IT support, part of integrated system, receiving financial incentives, larger practice Low
F-21

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Bailey, et al.,

201339

Retrospective

cohort

To determine

whether HIE by ED personnel in

the evaluation of patients

with headache reduces use of

neuroimaging, increases

adherence with guideline

Memphis,

Tennessee

ED Log file

Diagnostic neuroimaging,

evidence-based guideline 

adherence

August 2007-July

2009

Bailey, et al.,

201340

Retrospective

cohort

To determine whether HIE

reduces repeated diagnostic

imaging and costs in ED back

pain evaluation

Memphis,

Tennessee

ED Log file

Administrative data for imaging

log in patient record for HIE

access

August 2007-July

2009

Ben-Assuli,

Shabtai, and

Leshno, 201572

Retrospective

cohort

Probability of single-day

admission and 7-day readmission

when HIE viewed

Israel ED Log file 2004-2007
F-22

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Bailey, et al.,

201339

MidSouth e-Health Alliance

(MSeHA).

MSeHA HIE connects 15 major adult hospitals and 2 regional

clinic systems in 4 counties of the Memphis Metropolitan

Statistical Area. Patient demographic, diagnosis, all hospital

radiologic and laboratory reports, most procedure reports, 

and discharge summaries are exchanged. ED providers have

read-only access to data.

2007 Patients presenting to

participating EDs with principle

diagnosis of headache

Bailey, et al.,

201340

MidSouth e-Health Alliance

(MSeHA), 15 major hospitals and

2 regional clinic systems in the 4

most populous counties of the

Memphis Metropolitan Statistical

Area. Decentralized, query-based

exchange.

Consent was ‘opt-out.

Secure, password-protected, read-only access to clinical

information from participating hospitals and clinics through a

Web portal separate from each facility’s electronic health

record system.

MSeHA HIE connects 15 major adult hospitals and 2 regional

clinic systems in 4 counties of the Memphis Metropolitan

Statistical Area. Patient demographic, diagnosis, all hospital

radiologic and laboratory reports, most procedure reports, 

and discharge summaries are exchanged. ED providers have

read-only access to data.

2007 All patients with an ED visit for

back pain in the Alliance

hospitals

Ben-Assuli,

Shabtai, and

Leshno, 201572

Clalit HMO, Israel Query 2004 All ED referrals
F-23

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Bailey, et al.,

201339

2,101 2nd or subsequent visits for 1,252 patients ≥18years, a second or

subsequent ED visit to a MSeHA

participating general hospital’s

ED between August 1, 2007 and

July 31, 2009 with a primary

discharge diagnosis of primary

headache disorder (ICD-9-CM

codes 346.0, 346.1, 346.9 and

784.0); and no discharge

diagnosis of stroke (ICD-9-CM

430–438), brain cancer (ICD-9-

CM 191.x, 225.0 and V10.85),

traumatic injury, motor vehicle

accident, poisoning, or fall.

Primary diagnosis (ICD-9

codes) of variants of

migraine (346.2),

hemiplegic migraine

(346.3), chronic migraine

(346.7), other forms of

migraine (346.8), and

tension headache (307.81,

339.1)

1st visit for headache

None

Bailey, et al.,

201340

Patients: 478

Visits: 800

≥18 years, >1 visit to system ED

for back pain, index (previous

visit) with imaging

Discharge diagnosis of

trauma or cancer.

Repeat visits in which HIE was

accessed vs. repeat visits in

which HIE was not used

Ben-Assuli,

Shabtai, and

Leshno, 201572

340,804 admitted and 474,310 non-admitted patients Referred to ED and had a

creatinine test

None Access HIE information
F-24

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Bailey, et al.,

201339

Use of diagnostic neuroimaging (CT, CT

angiography, MRI or MRI angiography),

evidence-based guideline adherence and

economic

-Any HIE use

-HIE use by physician or nurse

practitioner

-HIE use by administrative/nursing staff

nonuse of HIE Quantitative

Modeling using the generalized

estimating equation method to

adjust for repeated measures

(since some subjects had >1

visit) and for clustering of 

subjects within hospital system

Bailey, et al.,

201340

-Use of repeated lumbar or thoracic

imaging

-% cases HIE used

-Cost

-HIE accessed by any ED staff during

repeat ED visit (Yes/No)

-Type of staff accessing HIE (MD or

Nurse Practitioner vs. admin or

nursing)

-Patient age, sex and race

-Comorbidity

-Hospital

-Number of previous ED visits

Quantitative

Chi2

Multivariate: generalized

estimating equation

Ben-Assuli,

Shabtai, and

Leshno, 201572

Same-day admission and 7-day

readmission

Access HIE information None Quantitative

Same-day admission and 7-day

readmission via logistic

regression
F-25

Author, Year Results

Risk of

Bias

Bailey, et al.,

201339

OR (95% CI ) of any HIE use

Neuroimaging: 0.38 (0.29 to 0.50)

Adherence to guideline: 1.33 (1.02 to 1.73)

-Increased odds of neuroimaging by subjects of older age, black race,

and higher comorbidity

-Prior visits lower the odds of imaging 7%, but the effect was reduced to 2% with use of HIE

- No significant change in costs

Secondary analyses

-Administrative/nursing staff neuroimaging: OR 0.25 (95% CI, 0.18 to 0.34)

-Physician/Nurse Practitioner HIE use and interaction terms for previous visits were not significantly associated

-No secondary analyses were significant for guideline adherence

Low

Bailey, et al.,

201340

Repeated imaging for any HIE: OR 0.36 (95% CI, 0.18 to 0.71), p<0.05

Visits with repeated imaging: 22.4% (179/800)

HIE used: 12.5%

-Physician or Nurse practitioner use of HIE lowered OR for repeat imaging OR 0.47 (95% CI, 0.23 to 0.96)

- No cost savings associated with HIE use because of increased CT imaging when health care providers used HIE

Low

Ben-Assuli,

Shabtai, and

Leshno, 201572

When external information viewed, probability of single-day admission decreased 9.5% and of 7-day readmission decreased 6.5% Low
F-26

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Ben-Assuli,

Shabtai, and

Leshno, 201341

Retrospective

Cohort

To determine whether HIE use

was associated with reduced

readmissions and "avoidable"

admissions

Main Israeli HMO

network

7 acute care

hospitals EDs

belonging to 

largest Israeli

HMO

Log file 2004-2007

Bouhaddou, et al.,

201182

Multiple site case

studies with

focus on

identification of

patients eligible, 

matching, and

consent; usage

Across 3 large integrated delivery

systems, how many patients can

and will participate; how much

used

San Diego,

California

Integrated delivery

system

Database and survey

Patient identifier and

demographic data

NR
F-27

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Ben-Assuli,

Shabtai, and

Leshno, 201341

Largest Israeli HMO network 3.8

million patients, operates 7

hospitals

Clinical and administrative data from all HMO hospitals,

community clinics and thousands of labs, imaging centers etc.

Demographics, prescriptions, allergies, lab, imaging, past

medical history, procedures.

2004 Adult patients presenting to

Israeli ED with 1 of 5 main

diagnosis; gastroenteritis,

abdominal pain, chest pain,

pneumonia organism, urinary

tract infection

Bouhaddou, et al.,

201182

Veterans Lifetime Electronic

Record (VLER)

Query-based, transfer of records between integrated delivery

systems

NR Patients of 3 large IDSs who

opted in to HIE
F-28

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Ben-Assuli,

Shabtai, and

Leshno, 201341

115,719 ED Visits NR NR HIE vs. local EMR and no EMR

HIE vs. local EMR use

Bouhaddou, et al.,

201182

1,144 patients shared between VA and KP

Nationwide Health Information Network allows users to

pull in data from other organizations. The VA and DoD

used the VLER systems for eHealth exchange with

private sector. Federated pull (query-based) model

Transfer of records between integrated delivery

systems; National query-based. Patient consent: Optin.

Patients identified as getting

care in VA and KP

None None
F-29

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Ben-Assuli,

Shabtai, and

Leshno, 201341

-OR for 7-day readmission for

gastroenteritis, abdominal pain, chest pain,

pneumonia organism or urinary tract

infection

-OR for 1-day admission for gastroenteritis,

abdominal pain, chest pain, pneumonia

organism, or urinary tract infection

-Economic

-MD Viewed EMR

-MD Viewed local EMR

-MD viewed external information (HIE)

-HMO to which patient belonged

-Differential Diagnosis

-ED sub department (Int. med or

surgical)

-Specific Hospital

-Age

-Gender

-Authors list all these variables as

independent but some are more

confounding per se

-Age

-Gender

-HMO

-ED

-Hospital

Quantitative

-t test for continuous variables

-Chi2 for dichotomous

-Multi-variate regression analysis

-P<0.05, no adjustment for

multiple hypothesis testing

Bouhaddou, et al.,

201182

Patients who opted in and provided valid

authorization, with subsequent measure of

records exchanged between KP and VA 2-

3 per week

-Patients correlated across KP and VA

-Actual records exchanged

NA Quantitative

Survey, descriptive statistics
F-30

Author, Year Results

Risk of

Bias

Ben-Assuli,

Shabtai, and

Leshno, 201341

OR for all 5 differential diagnosis as composite

Readmission within 7 days: 0.52 for HIE vs. local EMR and no EMR, p<0.001

1-day admission: 0.76, p<0.001

Readmission within 7 days: 1.272, p=0.05 for local EMR vs. HIE

1-day admission: 1.13, p=0.005 for local EMR vs. HIE

-Decrease in readmissions within 7 days when HIE used 56.1%

-Decrease in single-day readmissions when HIE used 29.0%

-Viewing external medical history more highly correlated with lower single-day admissions and 7-day readmissions than local medical

history

Low

Bouhaddou, et al.,

201182

Of 363 patients who opted in and provided valid authorization, 264 could be correlated; exchange of records between KP and VA 2-3 per

week. Older patients were more likely to consent for HIE.

NA
F-31

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Byrne, et al.,

2014116

Multiple site case

studies

Describe key findings, lessons,

implications from VLER pilot

project

12 sites across

U.S.

Unrestricted Audit logs, database, survey,

interviews, documents from

meetings

Veterans authorization

preferences, system dashboard,

VA provider (11/12 site) and

veteran interviews. 73 provider

interviews, 50 veteran interviews

December 2009-

October 2012
F-32

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Byrne, et al.,

2014116

Veterans Lifetime Electronic

Record (VLER)

Query-based HIE between VA, DOD, nonfederal care

organizations. The Nationwide Health Information Network.

The VA and DoD used the VLER systems for eHealth

exchange with private sector. Federated pull model transfer of

records between integrated delivery systems; 12 total sites, 4

did 3 way exchange, 8 did 2 way between VA and private

sector. Federated pull model via eHealth Exchange

December 2009 Veterans
F-33

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Byrne, et al.,

2014116

12 pilot sites

N=73 provider and 50 veteran interview

12 VLER pilot sites. Veterans

included were any who opted in.

None NA
F-34

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Byrne, et al.,

2014116

-Veterans accept

-Veteran concerns about participation

-Veterans perceived benefit

-Veteran awareness of VLER use during

their care

-Veterans preference of signed

authorizations

-Metrics of exchanged data

NA NA Mixed Methods

Quantitative, descriptive analysis

on usage; qualitative, thematic

analysis
F-35

Author, Year Results

Risk of

Bias

Byrne, et al.,

2014116

-64,237 veterans provided authorization and opted in

-Opted in then out: <0.01%

-Veterans matched with exchange partner: 31,080 (48%), range: 12-88%

-Highest matching rates with exchange partners using social security number in their algorithm

-Inbound discloser's to VA from exchange partners 5,524

-Outbound disclosure to exchange partner 13,913

-Inbound disclosures to VA from exchanged partners per matched patients 18/100

-Unique VA patient with exchange partner data retrieved: 2,724

-Unique VA providers retrieving exchange partner data: 1,764

- Percent of matched veterans for whom there was ≥1 disclosure to VA from exchange partner: 9%

-75% of providers trusted VLER data, 90% trusted privacy and security

-Most frequently cited provider benefits, more data for medical decision making, improved quality of care, reduced repeat testing, timelier

and faster access to information

-23/73 interviewed providers reported using VLER, 79% of users reporting overall satisfaction

-43% reported challenges with system response time, 29% with identifying patients who might have data

-Identified minimizing provider steps in information retrieval, one site Indiana HIE had an automated query resulting in push into their

system to allow providers pushed access anytime a patient was admitted discharged or transferred

-Providers at outside organizations did not having additional sign ones

-Workflow improvements suggested by outside users was to have data pushed in their EMR

-Sustaining HIE requires ongoing resources and oversight, often unanticipated technical issues arose

-Requires national policies and central coordination

-None of the veterans interviewed were aware if their providers were using HIE, the user-interfaces at the sites face the provider not the

patient

-Providers increased usage after training on VLER system

-Providers noted barriers of missing data, additional sign-on and need for better integration with workflow

NA
F-36

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Campion, et al.,

201397

Cross-sectional Determine the extent to which

automated HIE queries supported

patient encounters.

Binghamton, New

York

Hospital/clinic HIE log data 2010 until 23 months

following

Campion, et al.,

201258

Cross-sectional What is usage and satisfaction of

push and pull HIE

Buffalo and

Rochester, New

York

Health systems,

health 

departments,

practice

associations,

RHIO

Survey

Online survey responses from

112/584 invited physicians (19%

response rate)

July-December 2010
F-37

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Campion, et al.,

201397

Southern Tier HealthLink RHIO in

Binghamton, New York part of

SHIN-NY. Automated queries

occurred evening prior to

ambulatory patient appointments to

generate CCRs and for the

hospitals during ED visits, at

inpatient admission, inpatient unit

transfer and provided CCD doc to

providers. Providers could also log

in manually. Auto queries started

month 1 for clinics and month 17

for hospitals.

Lawson Cloverleaf HIE, centralized data repository with MPI. 5

hospitals, one imaging center and 30 ambulatory care

practices affiliated with single integrated delivery system.

2005 ≥18 years, with positive

consent to participate in HIE

Campion, et al.,

201258

HealtheLINK (Buffalo) and

Rochester RHIO

Direct exchange (push) of local lab and radiology results; querybased (pull) searching for lab and radiology results across

greater Buffalo and Rochester area.

Robust RHIOs using HIE platform from Axolotl Corporation

(San Jose, California)

2007-2009 Physicians
F-38

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Campion, et al.,

201397

202,365 auto queries ≥18 years, who had automated

HIE query generated, which

occurred when a care transition

occurred

Lack of known provider or

lack of known facility in

auto-queries from HIE

NA

Campion, et al.,

201258

112/584 invited physicians (19% response rate). Only

99 completed. 75% were primary care providers. Most

practices had 2-19 providers.

Physicians who completed

survey and rated overall

outcome of satisfaction with HIE

Respondents who did not

rate satisfaction with HIE

Compared various attributes of

HIE for push vs. pull
F-39

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Campion, et al.,

201397

Generation of automated HIE queries NA NA Quantitative

Descriptive statistics

Campion, et al.,

201258

Use of push vs. pull HIE. Satisfaction with

types of HIE.

Type of HIE: push or pull NR Quantitative

Descriptive statistics
F-40

Author, Year Results

Risk of

Bias

Campion, et al.,

201397

-202,365 automated HIE queries: 54% to hospitals, 46% to clinics

-After exclusions, duplicates removed: 145,668 unique patient encounters

-81,687 unique patients provided consent for query based HIE during study period, 41% had ≥1 supported encounter

-For the 33,219 patient with ≥1 clinic encounter: median IQR 3

-98% of patients had between 1 and 20 encounters, 71% had ≥2

-530 patients with ≥20 encounters

-52% occurred in hospital, 48% in clinics

Care Transitions

-28% of the 145,668 unique encounters occurred as care transitions

-53% were patients from a clinic to hospital, 36% in reverse, 11% clinic to clinic

NA

Campion, et al.,

201258

-80% used push HIE and 53% used pull HIE

-A greater proportion of MDs reported using push HIE always or most of the time (68%) vs. pull HIE (19%), p=0.001

-MDs more satisfied with push HIE vs. pull HIE, p<.0.05

-112 physician respondents (19% response), 13 then excluded for 99 participants

->50% of physicians felt HIE improved 8 domains; access to timely, completeness, accurate information, admin efficiency, communication

with colleagues, and quality

-Only 30% felt it improved reducing test redundancy and security of PHI

-Physicians who used push and pull vs. only single type had higher rates of perceived effects of HIE in same 8 domains, (3of 8 domains

p<0.05)

Moderate
F-41

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Campion, et al.,

201398

Cross-sectional

survey

Measure usage patterns of query

based HIE with respect to

practice sites, users, patients,

and data

3 separate RHIOs

encompassing 1

community each

(~1 million patient

population) in 

New York state

(from HEAL-NY)

Unclear, inpatient/ 

outpatient

System log data

Demographics of patient,

provider character (i.e. role,

location etc.)

A, B: January 2009-

March 2011

C: September 2010-

May 2011

Caffrey and ParkLee, 201393

Cross-sectional To determine use of EHR and

HIE by residential care

communities.

U.S. Residential care

communities

Survey

2010 National Survey of

Residential Care Facilities

2010

Carr, et al., 201470 Case series Does HIE reduce unneeded test

ordering and costs, admissions

Charleston, 

South Carolina

ED Questionnaire

User-initiated survey, with costs

calculated for self-reported

testing not performed

August-December

2011
F-42

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Campion, et al.,

201398

NY State HIE consists of 12 RHIOs

(HEAL NY)

Axolotl Virtual Health Record-commercial product. Web based

secure stand alone portal. Federated architecture with MPI,

RLS and user directory.

2007, 2007, 2010;

A, B and C,

respectively.

All patients

Caffrey and ParkLee, 201393

NR NR NR Residential care communities

Carr, et al., 201470 Carolina eHealth Alliance Access to EHRs and ED from all hospitals in region NR Physicians, Nurse 

Practitioners, Physician

Assistants, and students
F-43

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Campion, et al.,

201398

Combined 2.9 million total patients in 3 RHIO

communities

All patients None NA

Caffrey and ParkLee, 201393

Sampled: 3,605

Interviewed: 2,302

Residential care communities

that have been licensed,

registered, listed, certified or

otherwise regulated by the states

with >4 beds, >1 resident

currently living in the community,

and provide room and board with

at least 2 meals a day, around

the clock onsite supervision, and

help with personal care such as

bathing and dressing or

health=related services such as

medication management.

Communities licensed to

serve severely mentally ill

or intellectually or

developmentally disabled

populations exclusively.

Nursing homes were also

excluded unless they had a

unit or wing meeting

inclusion criteria where

residents could be

enumerated separately.

NA

Carr, et al., 201470 18,529 patient encounters, with 998 logons (5.39%) by

60 clinicians. 138 (13.8%) surveys completed. 105

(10.5%) of patients had data in HIE.

All survey responses from HIE

users

NA None
F-44

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Campion, et al.,

201398

-% practice sites accessing data

-Type of practice accessing HIE

-Number of roles and primary practice of

users accessing HIE

-Characteristics of patients whose data was

accessed

-Consenting of patients related to access

NA NA Quantitative

Descriptive statistics

Caffrey and ParkLee, 201393% of residential care communities that

used EHR with computerized support for

HIE

NA NA Quantitative

Regression

Carr, et al., 201470 -Services, costs, and admissions avoided

-Perceived time saved

Tests, costs, and admissions avoided NA Quantitative

Self-reported tests and

admissions avoided, calculation

of costs saved based on local

data.
F-45

Author, Year Results

Risk of

Bias

Campion, et al.,

201398

A vs. B vs. C

-Of sites registered to use system: 18% vs. 30% vs. 82% accessed in first 9 months

-After 27 months 60% vs. 59% vs. NR of sites had accessed

-In each community majority of practice sites from which access occurred were out patient

-In A and B majority of sessions were from outpatient sites, C was inpatient

-Registered users in community: 368 vs. 3461 vs. 118

-More than 1/2 users accessing system in A and B were nurses + staff, in C 2/3 were MDs + physician extenders

-Majority of all users practiced in ambulatory setting

-Patients whose data was accessed were older than those whose was not and then the entire population

-For community A&B majority had data accessed on same day as consent

-Majority of patients in A and B had their data accessed in community setting, C was inpatient

-% of patient whose data was accessed from ≥2 sites in first 9 months: 0.1% vs. 1.8% vs. 0.01%; after 27 months: 0.1% vs. 11.6% vs. NR

-System access occurred from 60% to 82% of practice sites registered to use system, depending on community

-Proportions of patients whose data were accessed varied between 5%-60%

-Most frequently accessed data were patient summaries, followed by lab and radiology data

NA

Caffrey and ParkLee, 201393

17% of residential care communities reported using EHR

% of residential care communities using EHR with computerized system to support HIE by provider type:

Any provider: 40

Pharmacy: 23

Other health or long-term care provider: 20

Physician: 17

Corporate office: 17

Other: 17

Low

Carr, et al., 201470 -Reported avoiding: 30.5% lab/micro tests ($462), 47.6% radiology tests ($161,000), 19% consultations ($4,000), 11.4% admissions

($118,000)

-86.7% reported improved quality of care

-81% reported time savings, averaging 120.8 minutes

Moderate
F-46

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Chang, et al.,

201051

Cross-sectional Development and evaluation of

enhanced reporting of lab data

based on data available to HIE

Indiana Physician office,

outpatient

Survey

Survey of physicians who were

potential users of reporting

interface

2 week period in 2007

Codagnone and

LupiañezVillanueva, 201394

Cross-sectional To measure and explain levels of

availability and use (adoption) of

eHealth applications and services

31 countries:

EU27 countries

plus Croatia,

Iceland, Norway

and Turkey

Varies as this was

an international 

survey

Survey, interviews, focus

groups

October 25, 2012 to

March 6, 2013

Dixon, Miller, and

Overhage, 2013141

Cross-sectional What are barriers to participation

in a mature state HIE?

Indiana Small hospitals,

small physician

practices, and

large physician 

practices

Survey and interviews

Initial mixed methods interviews

with most physician groups given

online survey

August 2009-March

2010

Dixon, Jones, and

Grannis, 201383

Cross-sectional Awareness and engagement of

infection preventionists in HIE for

public health surveillance

6 states with HIE -

3 funded by CDC

for explicit HIEbased reporting

and three with

mature HIEs

Case reporting for 

public health 

reporting of 

notifiable 

conditions

Survey

Online survey of 63 infection

preventionists

NR
F-47

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Chang, et al.,

201051

Indiana Network for Patient Care Collection of all lab data with enhancements (prior results,

other historical lab results, prescriptions, encounters),

pharmacy data, and patient encounter data

Not stated, but in

1990s

Primary care physicians who

were users of HIE

Codagnone and

LupiañezVillanueva, 201394

Varies as this was an international

survey

Varies as this was an international survey Varies as this was an

international survey

Random sample of general

practitioners who use a

computer

Dixon, Miller, and

Overhage, 2013141

Indiana HIE (IHIE) Full medical record in HIE 1994 Small hospitals, small

physician practices, and large

physician practices in Indiana

who were not participating in

HIE

Dixon, Jones, and

Grannis, 201383

6 states with mature HIEs but

details not explicitly provided

6 states with HIE — 3 funded by CDC for explicit HIE-based

public health surveillance reporting for infections, versus three

with mature HIEs, but without active surveillance reporting.

63 preventionists.

Not specific, would

be variable by state

Infection preventionists
F-48

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Chang, et al.,

201051

NA Convenience sample of primary

care physicians

NA None

Codagnone and

LupiañezVillanueva, 201394

9,196 general practitioners General practitioners who use a

computer

General practitioners who

don't use a computer

Comparison of HIE use by

country to prior survey in 2007

Dixon, Miller, and

Overhage, 2013141

12 small hospitals, 20 small physician practices, and

11 large physician practices who were not participating

in HIE

Small hospitals, small physician

practices, and large physician

practices in Indiana who were

not participating in HIE

Small hospitals, small

physician practices, and

large physician practices in

Indiana who were

participating in HIE

Barriers of cost, lack of

sufficient technical or human

resources, or lack of

awareness regarding value

proposition

Dixon, Jones, and

Grannis, 201383

NA Infection preventionists in public

health departments in 6 states

NA Comparisons in states with

active public health

surveillance vs. those without
F-49

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Chang, et al.,

201051

Evaluation of developed report Various factors related to usefulness

and completeness

NA Quantitative

Satisfaction survey

Codagnone and

LupiañezVillanueva, 201394

Use of 15 functions of HIE and 4 functions

of telehealth. Comparison with previous

survey in 2007.

Country, Types of HIE use Addressed thoroughly in multiple

analyses of use and adoption.

Quantitative multivariate

analysis

Factor analysis to create 1 overall

composite indicator, and 4

smaller composite indicators

(EHR, HIE, telehealth, PHR).

Comparison with 2007 results.

Dixon, Miller, and

Overhage, 2013141

Barriers of cost, lack of sufficient technical

or human resources, or lack of awareness

regarding value proposition

Survey None Mixed methods

Qualitative content analysis of

interviews and quantitative

tabulation of surveys

Dixon, Jones, and

Grannis, 201383

-EHR use

-EHR involvement in implementation

-Involvement in HIE

-Method for notifiable case reporting

-Organizations with EHR

-Involved in implementation of EHR

-Engaged in HIE

-Reporting methods for notifiable cases

NA Quantitative

Descriptive Statistics
F-50

Author, Year Results

Risk of

Bias

Chang, et al.,

201051

-9 physicians sampled

-Average 5 point Likert scales reported showed perception was generally favorable. ELRs well organized (4.2±0.97) and easy to interpret

(4.3±0.50). Additional data elements were valuable: relevant test (4.2±0.97), contextual drugs (4±0.89), visit histories (3.25±0.71) and

computer generated clinical reminders (3.25±0.71). Compared with traditional lab results ELRs generally saved time (3.78±0.67), reduce

the need to search for information (3.67±0.71) and improve quality of care (3.78±0.67). Physicians asked whether they would prefer to

use ELRs instead of traditional reports (3.78±0.67).

Moderate

Codagnone and

LupiañezVillanueva, 201394

Substantial increases in HIE use between 2007 and 2013. Qualitative results on barriers to adoption and use.

Countries with National Health Systems have high HIE use that countries with social insurance or transition systems.

Barriers to implementation included lack of interoperability, issues with system resilience, and security concerns. Systems that focused on

administrative rather than clinical applications were used less.

Low

Dixon, Miller, and

Overhage, 2013141

Barriers (small hospitals, small physician practices, large physician practices)

Cost: 100%, 50%, 55%

Lack of sufficient technical or human resources: 42%, 45%, 36%

Lack of awareness regarding value proposition: 33%, 15%, 36%

Moderate

Dixon, Jones, and

Grannis, 201383

-72% in organizations with EHR; 20% involved in implementation of EHR; 10% engaged in HIE; 49% unaware of organizational

involvement in HIE

-<5% reporting via secure email, web-based entry, through EHR, or through HIE each

Moderate
F-51

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Dixon, McGowan,

and Grannis, 

201142

Retrospective

cohort

To determine completeness and

quality of data for public health 

electronic laboratory reporting in

an HIE

Indiana Public health Log file

-7.5 lab results reported in HIE

-Statutory public health reporting

records

November 14, 2010-

December 15, 2010

Dobalian, et al.,

2012142

Cross-sectional Describe lessons learned from

one Nationwide Health

Information Network

implementation

Long Beach, 

California

3 hospitals, 2

ambulatory

practice groups

Interviews

Test data

2008

Dullabh and

Hovey, 2013158

Multiple case

studies

1) Assess the experience of

states in establishing governance

structures, technical services to

enable health information

exchange, and privacy and

security frameworks; 2) Assess

stakeholder priorities, current

use, and anticipated need for

information exchange; 3) Identify

common enablers, barriers, and

challenges; and 4) Collect and

characterize lessons learned.

Maine, Nebraska,

Texas,

Washington, 

Wisconsin

Health Systems,

provider

association, state

health IT 

coordinators, state

public health 

agencies

Site visits, interviews, focus

groups

Not clearly stated but suggests:

lab exchange, e-prescribing and

exchanging clinical care

documents.

November 29, 2011 -

March 21, 2012

Fairbrother, et al.,

2014143

Cross-sectional Describe the Beacon community

program experience

Greater

Cincinnati area, 

Ohio

Primary care,

hospitals, federally

qualified health 

centers and

community

centers insurance

partners

Interviews

Alerts for diabetic and pediatric

asthma patients in ED or

admitted sent to primary care.

Fall 2012
F-52

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Dixon, McGowan,

and Grannis, 

201142

Indiana HIE (IHIE)- includes lab

reports

Reporting of all lab data NR, but in 1990s All patients having lab tests

Dobalian, et al.,

2012142

One site in Nationwide Health

Information Network, another used

First Gateways exchange

(HealthView). This specific HIE

was called Long Beach Network

for Health

Make inpatient and outpatient data available to ED. Were not

yet able to exchange data about patient care.

2008 ED patients

Dullabh and

Hovey, 2013158

Not described per state States had two models of HIE: “thin layer” model with services

based on light infrastructure (Texas, Washington and

Wisconsin), or a heavy infrastructure model (Nebraska and

Maine) with features such as a central repository"

NR NR

Fairbrother, et al.,

2014143

87 primary care, 18 hospital, 7

federally qualified health centers

and community centers, 3

insurance partners

Data exchange, registries, alerts to PC practices when patient

in ED or admitted to hospital.

September 1, 2010 -

March 31, 2013

Adult diabetics, pediatric

asthma patients
F-53

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Dixon, McGowan,

and Grannis, 

201142

7.6 million lab reports from 168 hospitals and lab

information systems, of which 16,365 from 49 hospitals

and lab information systems were enhanced by a

Notifiable Condition Reporter

All laboratory values NA Proportion of fields in lab

reports that were complete

Dobalian, et al.,

2012142

N=18 to sample NR NR Participants in LBNH vs. not in

LBNH

Dullabh and

Hovey, 2013158

N=105 to sample; no response rate reported. NR NR Comparison of 5 states

Fairbrother, et al.,

2014143

N=38 interviews to sample Adult diabetics, pediatric asthma

patients

NR NA
F-54

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Dixon, McGowan,

and Grannis, 

201142

Comparison of completeness of lab test

results for regular and enhanced systems

19 data elements NA Quantitative

Completeness of data fields

Dobalian, et al.,

2012142

Descriptive narrative only NA NA Qualitative

Dullabh and

Hovey, 2013158

Descriptive narrative only NA NA Qualitative

Fairbrother, et al.,

2014143

Descriptive narrative only NA NA Qualitative
F-55

Author, Year Results

Risk of

Bias

Dixon, McGowan,

and Grannis, 

201142

-Patient identifiers and test, name, and results were nearly 100% complete for both; most but not all measures more complete for

enhanced system

-15 of 18 record fields showed improved completeness with enhanced system. Units of measure, normal range and abnormal flag fields

all showed reduced completeness with enhanced system. No tests of statistical significance performed.

Low

Dobalian, et al.,

2012142

"Despite a limited concentration on ED care, virtually all respondents noted concerns regarding the sustainability, or business case, for the

exchange of health information."

NA

Dullabh and

Hovey, 2013158

"Results show the last 2 years have seen unprecedented growth in HIE infrastructure. Key factors such as maturity of HIE at baseline and

healthcare market characteristics have shaped governance models and technical infrastructures." "Given the significant concerns about

sustainability and who will pay for state-offered services in the long term, it may also prove beneficial to ensure that states have

assistance, either from state or national informational resources, in developing both sustainability plans and contingency plans."

NA

Fairbrother, et al.,

2014143

Despite some setbacks and delays, the basic technology infrastructure was built, the alert system was implemented, 19 practices focusing

on diabetes improvement were recognized as patient-centered medical homes, and many participants agreed that the program had helped

transform care.

High
F-56

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Feldman and

Horan, 201143

Retrospective

cohort

To determine challenges and

successes of HIE for Social

Security disability determination

Virginia SSA, MedVirginia

HIE, and Bon

Secours Health

System

Database, interviews, audit

logs

Semi-structured interviews of 43

individuals from the 3

participating organizations

June-November 2009

Dullabh, 2014159 Multiple Case

Studies

To understand the effects of the

State HIE Program on HIE

progress

Six US States

Iowa, Mississippi,

New Hampshire,

Utah, Vermont

and Wyoming

Multiple Site visits, interviews, meetings 2012-2014

Feldman,

Schooley, and

Bhavsar, 2014144

Cross-sectional Obtain insights into technical,

organizational, and governance

issues of a large private health

system participating in a state

HIE

Virginia Integrated delivery

system

Interviews, observations,

documents

Direct observation, informal

information gathering, document

analysis, and semi-structured

interviews

August 2012-June

2013

Finnell and 

Overhage, 2010133

Cross-sectional To describe the underlying

technology, the utilization

statistics, and the survey results

from the medics who used an

integrated emergency medical

service point-of-care system and

RHIE system

Indianapolis,

Indiana

EMS providers

using tablets

Survey, database July 1, 2009-

December 31, 2009
F-57

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Feldman and

Horan, 201143

Medical Evidence Gathering

Through Health IT (MEGAHIT)

Data for Social Security disability determination transmitted

from health system through HIE to SSA via NHIN, push of

background, lab, and medication data in a CCD from health

system to SSA

February 2008 Patients being evaluated for

Social Security disability

determination; interviewed

included personnel from the 3

participating organizations

Dullabh, 2014159 Multiple Most projects enabled both directed and query-based HIE.

While services varied they included care summary exchange,

lab results, public health reporting, and transmission of

admission/discharge/transfer messages.

Varies State HIE programs supported

by the Office of the National

Coordinator (U.S. Federal

Government).

Feldman,

Schooley, and

Bhavsar, 2014144

ConnectVirginia EXCHANGE Query of Continuity of Care Documents August 2012 All patients in Invoa IDS

Finnell and 

Overhage, 2010133

30 hospitals, 5 health systems,

Marian County Health Department

and various physician practices.

EMS providers use a button that links to the Indiana Network

for Patient Care (INPC). Data are stored in a secured,

password protected, centralized database. Medics receive a

data abstract (pdf) of patient demographics, lab, ED, inpatient,

chief complaint, coded diagnoses and procedures.

Started in 1994 Number of patients who were

seen by EMS.
F-58

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Feldman and

Horan, 201143

203 Members of 3 organizations NA None

Dullabh, 2014159 Programs In 6 states States not included in prior

rounds of case studies.

States were selected for

variation in program factors,

state contextual factors, state

HIE progress,

States included in prior

case studies of this

program

Programs were compared

across states in terms of

leadership models and other

characteristics.

Feldman,

Schooley, and

Bhavsar, 2014144

10 individuals from IDS, HIE, and vendors Members of all organizations None None

Finnell and 

Overhage, 2010133

26,754 patient contacts by medics. Also survey of 58

medics on use of INPC

Invited all 180 medics. 58/180

responded

NR Comparison of use over time of

study.
F-59

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Feldman and

Horan, 201143

Technical, organizational, and governance

attributes

Mean Social Security disability case

processing time 59 days (vs. average

of 84)

NA Quantitative, Mixed Methods

Development of Collaborative

Enactment Model

Dullabh, 2014159 Provider participation

Critical mass of data exchange

Technical model

Leadership model

Variety and type of stakeholders

NA Qualitative

Feldman,

Schooley, and

Bhavsar, 2014144

Technical, organizational, and

governmental attributes

NA NA Qualitative

Themes extracted from data

Finnell and 

Overhage, 2010133

Number of unique medic users over 6

months, number of INPC requests.

HIE use, barriers to use NR Quantitative

Multivariable analysis
F-60

Author, Year Results

Risk of

Bias

Feldman and

Horan, 201143

-Technical challenges of HIE can be overcome but organizational and governance factors are also important

30% decrease in mean case processing time from 84 to 59 days from the usual method to HIE supported method, respectively.

Moderate

Dullabh, 2014159 Local stakeholder needs in the long and short term influenced decisions

Other factors were cost, privacy and security

Tangible intermediate goals supported implementation.

Providing value and meeting Stage 2 meaningful use criteria were related to estimates of sustainability.

Most programs were planning to use subscription fees for long term financial support.

NA

Feldman,

Schooley, and

Bhavsar, 2014144

Some technical challenges required workarounds, leadership and adequate resources essential, and appropriate decision making

authority required

NA

Finnell and 

Overhage, 2010133

Over a six month study period, requests for patient data via HIE increased from 15% to 26% per patient contact. The majority of medics

surveyed felt the HIE information was an important for delivering quality patient care.

Moderate
F-61

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Foldy, 200784 Cross-sectional Description of projects, stages,

users, organizational home,

governance, scope, standards,

drivers, challenges,

recommendations

Wisconsin Any Survey

Unable to access due to broken

URL link

2006

Fontaine, et al.,

201085

Cross-sectional Examine factors that motivate or

prevent small primary care

practices from participating in

EHR and HIE use as mandated

by Minnesota e-Health Law from

2007

Minnesota Primary care

practices with <20

providers in 1 of

the 3 described

HIE regions

Survey and Interviews November 10, 2008-

February 20, 2009

Frisse, et al.,

201244

Retrospective

cohort

To examine the financial impact

of HIE in EDs

Memphis,

Tennessee

ED Log file

Tennessee Hospital Association

billing database of all ED visit

records

January 2007-

December 2008
F-62

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Foldy, 200784 NA HIE defined as projects in which multiple independent

organizations routinely send or receive electronic clinical

information about patients for purposes other than billing or

claims payment

NA eHealth board, staff,

consultants, workgroup

members and survey

respondents all nominated the

survey recipients

Fontaine, et al.,

201085

Various HIEs 9 primary care practices in Minnesota

3 HIE initiatives in Minnesota 1) a 10 year old HIO that

promotes HIE and coordinates immunization registry, 2)

network of independent metropolitan community clinics that

received MN e-health grant funding to implement EHRs, 3)

initiative to develop PHR with congestive heart failure patients

NR 39 participants in discussions

Frisse, et al.,

201244

MidSouth e-Health Alliance

(MSeHA)

11 of 12 hospitals accessed information through a dedicated

secure web portal. 1 hospital printed encounter summaries as

part of triage for the first 10 months of the study.

Patient demographic, diagnosis, all hospital radiologic and

laboratory reports, most procedure reports, and discharge

summaries are exchanged.

2005 All ED visits
F-63

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Foldy, 200784 30 Organizations contacted, 27 (90%) responded eHealth board, staff, consultants,

workgroup members and survey

respondents all nominated the

survey recipients

NR NA

Fontaine, et al.,

201085

Unclear NA NA NA

Frisse, et al.,

201244

15,798 visits in which HIE was accessed; matched

comparison group of 15,798 cases

ED visit to 1 of the participating

hospitals. Visit only in HIE or no

HIE subset.

Patients in both the HIE

and no HIE subset (932)

HIE accessed in non ED

setting (3,555)

Encounters with vs. without

HIE
F-64

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Foldy, 200784 -Status of projects operation vs. planned

-Stage of development

-Description of information users

-Organization, funding, governance

-Scope

-Standards

-Drivers

-Challenges

-Recommendations

NA NA Quantitative

Descriptive Stats

Fontaine, et al.,

201085

-Use of EHR

-What data elements are being

sent/received

NA NA Qualitative

Descriptive statistics

Frisse, et al.,

201244

-Financial consequences based on EDoriginated hospital admissions

-Admissions for observation, lab tests,

head or body CT, ankle or chest

radiographs, echocardiograms

HIE accessed during ED visit -Admission type

-Length of stay

-Charlson comorbidity index

-Patients matched on age, gender,

race, site of ED, diagnosis and

payer

Quantitative Multivariate

Analysis

Generalized estimating equation

logistic regression
F-65

Author, Year Results

Risk of

Bias

Foldy, 200784 -27 responded, 21 judged to be HIE organizations, 21 respondents had 16 operational projects, 11 planned projects

-Rating of most advanced HIE project had 40% of respondents in implementation and 40% in operational

-44% deliver data only to central registries, 50% deliver to providers and registries and only 1 to providers only

-62.5% are based in government organizations

-73% started with only public funds, 20% exclusively private, 75 used both

-For continued operations 57% rely entirely on public funds, 21% only on private and 21% a combo

-Governance all have multiple stakeholders

-14 are statewide, 7 southeast Wisconsin, 2 south, central and north and west.

-Standards 46% of projects have specific vocabulary or data standards

Moderate

Fontaine, et al.,

201085

-8/9 practices uses EHR

-Only 1 practice was able to transmit/receive patient health records

-All 9 practices shared information with department of health immunization registry though not through any of the EHRs in the practices

-Labs were next most common Several practices were receiving data directly into EHRs

-None were sharing data with nonaffiliated practices

-HIE motivations themes: External - government mandates, payer mandates, quality reporting; Internal - cost savings, quality/patient

safety, efficiency

-HIE barriers: lack of interoperability, lack of buy-in, competition, security, costs, creating business model, limited success and large time

investment, limited technical support

-No practice was fully involved in a regional HIE; HIE was not part of most practices’ short-term strategic plans.

Moderate

Frisse, et al.,

201244

HIE accessed: 6.8% of ED visits (in 12 EDs)

Admissions when HIE used

Adjusted OR 0.27; 95% CI, 0.210 to 0.351, p<0.0001

191 fewer admissions with HIE vs. without HIE

-In 11 EDs directly accessing HIE data only through a secure Web browser, access was associated with a decrease in hospital

admissions (adjusted OR 0.27; p<0001)

-In 12th ED relying on print summaries, HIE access was associated with a decrease in hospital admissions (OR 0.48; p<0001) and

statistically significant decreases in head CT use, body CT use, and laboratory test ordering

-HIE access associated with annual cost savings of

$1.9 million, with hospital admission reductions accounting for 97.6% of total cost reductions

Moderate
F-66

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Furukawa, et al.,

2013111

Time Series Describe extent of HIE in U.S.

hospitals

All 50 states and

the District of

Columbia

Hospital Survey

Health IT supplements to the

American Hospital Association

Annual survey of hospitals, 2008-

2012. 63% response rates.

2,805 hospitals in 2008, 2,836

hospitals in 2012. nonfederal

acute care hospitals

2008-2012

Furukawa, et al.,

2014110

Cross-sectional NAMCS Survey, How have rates

of EHR changed since HITECH?

What % of MDs are engaged in

HIE in 2013? What % are using

PHR in 2013? How did these

things vary by physician and

practice characteristics?

U.S. U.S. ambulatory

providers

Surveys 2009-2013

Gadd, et al.,

201186

Cross-sectional To assess the usability of an HIE

in a densely populated

metropolitan region

3 counties around

Memphis,

Tennessee

ED and outpatient

clinics

Survey

Email survey responses from

165/ 237 health care

professionals (70% response

rate)

June-November 2009

Genes, et al.,

2011145

Cross-sectional What are perceptions of ED

users of HIE?

New York City ED Interviews

Semi-structured interviews of

users and nonusers

NR
F-67

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Furukawa, et al.,

2013111

NA NA NA U.S. acute care nonfederal

hospitals

Furukawa, et al.,

2014110

NA NA NA Ambulatory physicians not 

radiologists, pathology, or

anesthesia

Gadd, et al.,

201186

MidSouth e-Health Alliance

(MSeHA)

A rapid deployment HIE that

consolidated data from several

sources

Consolidated data from multiple hospital EDs and communitybased ambulatory clinics. Decentralized, query-based

exchange.

Consent was opt-out.

2004 in 3 counties Medical staff (Physicians,

Nurse Practitioners,

Physicians assistants, nurses,

and other) at organizations

participating in the HIE

Genes, et al.,

2011145

New York Clinical Information

Exchange (NYCLIX)

All data from 10 academic medical centers 2009 ED physicians
F-68

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Furukawa, et al.,

2013111

2,805 hospitals in 2008 and 2,836 in 2012

Various HIEs

NA NA NA

Furukawa, et al.,

2014110

NR NA NA NA

Gadd, et al.,

201186

162 responses analyzed

Details on sample: 345 people identified; 269 valid

contacts; 237 surveys distributed; 165 responses

(69.6%); 3 excluded for missing responses on

satisfaction items.

NR other than list of roles

included

People who were no longer

employed by the system

were not contacted

The impact of usability on use

of HIE

Genes, et al.,

2011145

18 users of NYCLIX ED pilot All users NA -For users, was HIE data

useful?

-For nonusers, why not using?
F-69

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Furukawa, et al.,

2013111

Any exchange activity with outside

providers outside the organizations

NA -Provider type

-Organizational affiliation

-Type of clinical information

-Hospital characteristics

-Area characteristics

Qualitative

Descriptive statistics

Furukawa, et al.,

2014110

Descriptive statistics NA NA Quantitative

Descriptive statistics and logistic

regression

Gadd, et al.,

201186

-Use

-Questionnaire for User Interaction

Satisfaction (QUIS 7.0)

-Trust

None None Quantitative, multivariable

analysis

-Wilcoxon rank sum test

-Descriptive statistics

-Ordinal logistic regression

Genes, et al.,

2011145

-For users, was HIE data useful?

-For nonusers, why not using?

Semi-structured interviews None Qualitative
F-70

Author, Year Results

Risk of

Bias

Furukawa, et al.,

2013111

-58% of hospitals exchanging in 2012, 41% increase of 2008, p<0.01

-2012 51% hospitals exchanged with unaffiliated ambulatory providers, 36% with other hospitals outside their organization

-2012 52%, 53%, 35% and 33% exchanging radiology reports, labs, care summaries and prescription lists with outside providers,

respectively. That is a 39%, 51%, 40%, 55% increase, respectively.

-After adjusting for hospital and area characteristics hospitals with basic EHR and participation in Health information organizations had

highest rates of exchange activity in 2012, 80% of hospital with EHR and HIO were exchanging, 71% with HIO but no EHR were

exchanging 60% of hospitals with EHR but no HIO were exchanging, all consistent across different providers types and clinical information

types

-Hospital characteristics associated with lower exchange rates, rural, for-profit, locations with greater Medicare part A spending

Low

Furukawa, et al.,

2014110

-Broad HIE definition (39% of office-based physicians reported having an HIE with other providers or hospitals). Increased odds of HIE

both within and outside of their organization with larger practice, health-system owned practice and multispecialty practice. Very few

characteristics associated with HIE outside of the practice, significantly lower outside HIE with community health centers and practice

outside of metropolitan statistical centers

-35 % HIE inside, and 13% HIE outside

Low

Gadd, et al.,

201186

151 users (93%), 11 non users

Average usage per week

<1 hour: 65 (43%)

1 hour to <4 hours: 58 (39%)

≥4 hours: 27 (18%)

Mean usability scale: 6.5 SD 1.4 (>5 is favorable, out of 9)

Association of Scales with higher use (ORs)

Overall reactions: 1.50, p<0.01

Learning: 1.32, p<0.05

System functionality: 1.34, p<0.01

Trust not predictive of usage. Users commented that HIE needs more tech support and could use more types of data

Low

Genes, et al.,

2011145

-Half of users reported usage affecting patient care on ≥1 occasion

-nonusers reporting forgotten login credentials

Low
F-71

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Goldwater, et al.,

2014146

Cross-sectional Evaluate the progress of the HIE,

how many providers and

hospitals were participating in the

program, and what benefits were

being realized through the use of

the HIE.

Washington, 

District of

Columbia

6 acute care

hospitals

Interviews, focus groups,

survey

Demographic, inpatient,

encounter notifications, lab

testing, electronic prescribing

services, integration with public

health and Medicaid providers.

July 1, 2013-January

6, 2014. Survey of

148 individuals and

stakeholders released

October 1, 2013 and

closed November 4,

2013.

Greenhalgh, et 

al., 2010121

Mixed-method;

multi-level case

study of

England's

Summary Care

Record (SCR)

1) What is usability, use,

functionality, and impact of SCR;

2) What explains variation in its

adoption and use;

3) How has the programme been

constrained by influences at the

macro, meso, micro level;

4) What are the transferable

lessons for practice and policy?

3 districts within

the English 

National Health

Service

ED and

unscheduled care

Qualitative data:

140 interviews of policy makers,

managers, clinicians, software

suppliers;

2,000 pages of ethnographic field 

notes;

Observation of 214 clinical

consultations;

3,000 pages of documents.

Quantitative Data:

416,325 encounters in 3

participating clinics

2009-2010?

Not quite clear

Grossman,

Kushner, and 

November,

2008160

Multiple case

studies

Compare differences in success

and barriers for HIEs

Indiana, 

Cincinnati, 

Northeast 

Tennessee, 

Tampa Bay

Any Interviews of stakeholders February-August

2007
F-72

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Goldwater, et al.,

2014146

The 6 acute care hospitals chose

the Chesapeake Regional

Information System for our Patients

Demographic, inpatient, encounter notifications, lab testing,

electronic prescribing services, Integration with public health

and Medicaid providers.

Launched February

2012

Survey sent to 148, 30

completed 20% response rate

Greenhalgh, et 

al., 2010121

SCR, which was comprised of 3

data fields - medications, allergies

and adverse reactions

Not specified 2007-2010 2007-two early adopter clinics;

2010 - 113 of 152 primary care

trusts in England had

committed to participating; by

2010, 16 had begun to create

SCRs;

By 2010, 1.5 million records

had been created.

Grossman,

Kushner, and 

November,

2008160

IHIE, HealthBridge, CareSpark,

Tampa Bay RHIO

All types Varying Stakeholders in 4 HIEs
F-73

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Goldwater, et al.,

2014146

NR NR NR NA

Greenhalgh, et 

al., 2010121

1.5 million records in 2010 3 districts who were

implementing SCRs

Not specified None

Grossman,

Kushner, and 

November,

2008160

2 mature and 2 newer NA None None
F-74

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Goldwater, et al.,

2014146

Descriptive narrative only NA NA Mixed Methods

Greenhalgh, et 

al., 2010121

What is usability, use, functionality and

impact of the SCR;

What explains variation in adoption and

use;

How does context play in;

What are the lessons to practice and policy

None None Qualitative

Interpreted and themed

Quantitative

Descriptive statistics and logistic

regression

Grossman,

Kushner, and 

November,

2008160

Success, barriers, sustainability NA NA Qualitative
F-75

Author, Year Results

Risk of

Bias

Goldwater, et al.,

2014146

"HIE is used to electronically capture and report immunization data; and in requiring electronic lab reporting and results as part of the

Meaningful Use Requirement—which can assist in detecting HIV/AIDS and providing better care for the district’s high population of

individuals with HIV/AIDS. Electronic lab reporting and electronic prescribing within the HIE can assist the Department of Health and

providers in identifying specific diseases, such as tuberculosis and viral hepatitis, before they affect a significant part of the population. '

Moderate

Greenhalgh, et 

al., 2010121

Adoption was complex, technically challenging, labour intensive;

Went more slowly than planned;

SCR accessed in 4% of all encounters;

SCR accessed in 21% of encounters where an SCR was available;

Main determinant of success was clinician characteristics (which were not specified);

When available, clinicians accessed SCR 0% to 84% of time;

SCR supported better quality care and increased clinician confidence;

No direct evidence of improved safety;

SCR not associated with shorter clinical consultations;

Successful implementation hinged on successful interactions among multiple stakeholders (clinical, technical, political)

Low

Grossman,

Kushner, and 

November,

2008160

Stakeholder buy-in essential for success, offering hospitals value to reduce costs important, hospitals concerned about controlling access

to data, employers and health plans not buying in

NA
F-76

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Gutteridge, et al.,

2014112

Cross-sectional To describe the development and

use of a CEN system based on

an HIE.

New York

metropolitan are

ED, hospital, and

outpatient

Subscription lists and reports

generated

March 11, 2013-

March 2, 2014

Hamann and 

Bezboruah,

2013113

Secondary 

analysis of crosssectional survey

To examine ownership

differences (for-profit; nonprofit)

in the use of technology in long

term care facilities

U.S. Nursing homes

and residential

care

Surveys

2004 National Nursing Home

Survey; 2010 National Survey of

Residential Care Facilities

Nursing home:

August 2004-January

2005

Residential care:

2010

Herwehe, et al.,

2012124

Cross-sectional To conduct a formative evaluation

of an HIE for HIV that integrates

public health and clinical

information

Louisiana Health

department,

hospital,

outpatient

Interviews, focus groups, log

data

February 1, 2009 and

January 31, 2011
F-77

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Gutteridge, et al.,

2014112

Healthix A federated architecture for data sharing. Log in is via a

standalone web portal

-Healthix included a total of 107 organizations with 383

facilities, 9.2 million patients, and >6,500 users performing

>10,000 patient searches per month as of January 2014

2004 was initial

funding

CEN system March

2013

Geriatric patients seen in ED

and admitted to hospitals

Hamann and 

Bezboruah,

2013113

Varies, NR Varies, NR Varies Long term care Facilities

Nursing home is U.S.

Residential Care (aka Assisted

Living in U.S.)

Herwehe, et al.,

2012124

The Louisiana Public Health

Information Exchange

(LaPHIE)

A secure bi-directional public health informatics application (an

HIE in a broad sense, as defined by Dixon et al.), linking

statewide public health surveillance data with patient-level EMR

data.

Started February

2009 and in all 

participating hospitals

by September 2009

Patients with HIV seen for non

HIV services at 7 Louisiana

Hospitals; 442 clinicians (206

physicians and 236 nurses)

trained on system to serve as

peer trainers
F-78

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Gutteridge, et al.,

2014112

These patient who are enrolled in the system NA NA None

Hamann and 

Bezboruah,

2013113

Nursing home Sample: 1,174 response rate 81%

Residential care Sample: 2,302 response rate 81%

Various HIEs

NR NR Nonprofit vs. for profit use of

health IT including HIE

Herwehe, et al.,

2012124

16 focus groups n=149; and 23 key informant

interviews with patients

NA NA NA
F-79

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Gutteridge, et al.,

2014112

-Enrollment of patients

-Number of notifications sent

NA NA Counts

Hamann and 

Bezboruah,

2013113

Whether facility shares information

electronically with other care partners and

the extent of HIE defined as the number of

entities with which the facility shares

information

Nonprofit or for-profit ownership -Chain ownership

-Size of facility and type of

residents

-Use of volunteers

-% revenue from Medicaid and

Medicare

Quantitative

-Chi2

-Ordered Logit regression

Herwehe, et al.,

2012124

Patients identified and matched providers

responses to alerts

NA NA Mixed methods

-Description

-Counts of alerts and responses
F-80

Author, Year Results

Risk of

Bias

Gutteridge, et al.,

2014112

-5,722 patients enrolled (612 notifications sent)

-Without duplications 497 event notifications about 206 unique patients

-Notifications originated from 23 separate institutions, ED visits comprised 44% (219 of the 497 notifications), 98 notifications were for

inpatient admissions

-121 of 497 (55%) during normal business hours

-Hospital admissions resulted from 45% of ED visits; 17.8% of these lasted <48 hours, suggesting they were avoidable

-70% of notifications were received within 1 hour of the event, during the study year; in following year 71% were received within 15

minutes

NA

Hamann and 

Bezboruah,

2013113

For Profit/Nonprofit (corrected F)

% Residential care using HIE: 0.14/0.21 (10.29), p=0.00

Number of partners in HIE: 0.32/0.42 (2.56), p=0.02

Regression results: for profits less likely to participate in HIE OR 0.663, p<0.001

Supports hypothesis and proposed framework for why nonprofits are more likely to use health IT

NOTE: NH survey did not have HIE question

Low

Herwehe, et al.,

2012124

In the 2 year period 2/1/2009 to 1/31/2011:

-488 registrations of patient (345 unique patients) with HIV identified

-Clinicians responded to 73% of alerts and documented actions on note that was shared with public health

-Results include statement that 'no negative feedback has been received from providers' with no detail

-Summary of patient interviews found general acceptance of data sharing as long as there was patient benefit and a preference for care in

the healthcare verses the public health system

-Challenges: concerns about data ownership and ethics and disparate data systems, but these are reported as challenges they were able

to address

NA
F-81

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Hessler, et al.,

200987

Cross-sectional To understand assessment of

HIE by RHIO and state and local

public health department

representatives

U.S. RHIOs and State

and Local Health

Departments

Survey

Online survey created by

researchers

late February 2007-

March 25, 2007

Hincapie, et al.,

2011132

Cross-sectional Assess perceptions of physicians

users of HIE

Arizona All physician use Focus group meetings of 29

physicians on HIE quality of care,

workflow and cost

NR

Hyppönen, et al.,

2014133

Cross-sectional To compare usability of different

regional health information

exchange system (RHIE) types

as well as the factors related to

the experienced level of success

Finland Varies as this

includes sites with

RHIE

Survey 2010

Jha, et al.,

2008117

Cross-sectional,

mixed modes

To assess health IT, including

HIE adoption in 7 countries

U.S., U.K.,

Canada,

Germany,

Netherlands, 

Australia, New

Zealand

Physicians and

hospitals

Literature review, available

surveys, (Medline and Google)

and interviews with governmental

and nongovernmental experts

Literature review:

2000 -2006
F-82

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Hessler, et al.,

200987

Varies, NR Varies, NR Varies 164 RHIOs

540 health agencies

Hincapie, et al.,

2011132

Arizona Medical Information

Exchange (AMIE)

Medication history, lab test results, and discharge summaries October 2008 Physicians who agreed to

participate in focus groups

Hyppönen, et al.,

2014133

Regional Health Information

Exchange

Varies depending on type of RHIE system. Type 1: master

patient index required separate login to centralized database.

Type 2: web distribution model. Limited group of referring

physicians could see hospital info. Type 3: regional virtual

model. If patient grants permission, clinician uses integrated

system that includes all inpatient and outpatient information.

Before 2010 Inpatients and outpatients of 

physicians working in public

sector in 13 regions of Finland

where RHIE systems were in

use.

Jha, et al.,

2008117

Varies, NR Varies, NR Varies Developed countries
F-83

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Hessler, et al.,

200987

N=44 RHIOs (27% response); 20 non-governmental

N=138 Health agencies (26% response); 41 state and

97 local public health agencies

RHIOs: listed in 1 of 7 sources

Public Health: on list from

national associations

Missing or invalid email

addresses or an exchange

specific to 1 disease

RHIOs vs. state vs. local health

officials

Hincapie, et al.,

2011132

29 physicians Physicians who agreed to use

system and participate in focus

groups

None None

Hyppönen, et al.,

2014133

1,693 physician respondents aged less than 65 years.

1,079 specialize care; 614 primary care

Physicians working in public

sector in 13 regions of Finland

where RHIE systems were in

use.

Physicians in the private

sector or in regions where

RHIE not in use whole

region or was unavailable

Comparison of HIE usability by

type of RHIE and EHR

Jha, et al.,

2008117

7 selected for data availability NA NA HIE use across countries
F-84

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Hessler, et al.,

200987

-Sharing of data

-Challenges

-Unique resources

-Minimal requirements

Type of respondent Characteristic reported but not

used in analysis

Mixed Methods

-Descriptive statistics, no

significance tests

-Qualitative assessment of openended responses

Hincapie, et al.,

2011132

Benefits and disadvantages of HIE Transcripts NA Qualitative

Thematic analysis from

transcripts

Hyppönen, et al.,

2014133

Levels of agreement to 11 statements

about HIE success

RHIE type used, local EHR system

used, working sector and primary

means of HIE

Managed multi-collinearity Quantitative, multivariable

analysis

Models to predict successful HIE,

stratified by type of clinician user

(specialized or primary care).

Results were broken out by

function of HIE.

Jha, et al.,

2008117

-HIE existence

-Use

-Policies promoting development

Country NR Descriptive, qualitative
F-85

Author, Year Results

Risk of

Bias

Hessler, et al.,

200987

Public Health: 50 (36%) no RHIO in jurisdiction; 16 (12%) no relationship with RHIO; 26 (40% responding to item) are exchanging

information

RHIOs: 12 (60%) are exchanging info; 7 (35% with public health); lab data shared most frequently (86% of the time)

Challenges (RHIO/Local/State % endorsing)

Lack of standards: 33/12/15

Limited resources: 17/67/45

Unique resources Public Health brings

Perspective: 41/45/30

Data: 35/16/39

Minimum Public Health must bring

Commitment: 50/31/23

Funding/sweat equity: 33/43/47

More dialogue about needs and expectations could increase HIE; early successes with lab data could encourage future use.

High

Hincapie, et al.,

2011132

Benefits included identification of "doctor shopping", avoiding duplicate testing, and increased efficacy for gathering information;

disadvantage was limited availability of data

Moderate

Hyppönen, et al.,

2014133

Users of three local EHR systems preferred electronic HIE to paper to a larger extend than users of other EHR systems. Experiences with

an integrated RHIE system (type 3) were more positive than those with other types or RHIE systems.

Low

Jha, et al.,

2008117

Australia: early pilots, but no major investment. Lack of unified patient identification an issue

Canada: province-wide efforts, particularly Alberta; national--early development of Health Infoway but little info exchanged

Germany: most computers with records not connected; Germans have smart cards, but only admin data now

The Netherlands: National SwithPoint pilot with 20% of population, plan full implementation in 2008

New Zealand: planning stage, have unified patient Id, focus of discharge, lab and path reports to GPs

U.K.: National Program, but mostly small amount of data exchanged in more minor programs

U.S.: RHIOs, but <12% of organizations exchanging data and <1% of population involved

High
F-86

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Johnson, et al.,

200899

Multiple site case

studies

To assess first year of MidSouth

eHealth Alliance

Memphis,

Tennessee

EDs Audit logs, database

(administrative), comments by

users

Implied 1 year after

May 2006; but data

on use in January

2008

Johnson, et al.,

2011118

Multiple site case

studies

To explore characteristics of use

and uses of a regional HIE

Memphis,

Tennessee

EDs, ambulatory

groups

Audit logs, database

administrative data,

observations, comment cards,

feedback in system, interviews,

observations

Interviews 1 month, 1

year after system in

use in all sites

Audit data and ED

visits January 2008-

June 2008

Jones, Friedberg,

and Schneider, et 

al., 201168

Cross-sectional To evaluate the association

between hospitals’ HIE and

health IT use and 30-day risk

adjusted readmission

U.S. Hospitals Database

2007 AHA Survey

2009 September Hospital 

Compare

June 2005-June 2008

for Hospital Compare

Kaelber, et al.,

2013120

Cross-sectional What is use and perceived value

of HIE?

Northeast Ohio Public healthcare

system

Usage logs, survey of users November 2010-

December 2011
F-87

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Johnson, et al.,

200899

MidSouth eHealth Alliance

(MSeHA)

Multiple hospital emergency departments and communitybased ambulatory clinics. Decentralized, query-based

exchange.

Data Exchanged: demographics, ICD-9 discharge

codes, lab results, encounter data, and dictated

reports.

These are in a vault controlled by the hospital, but accessed

when a query is made, unless patient opts out.

May 2006 ED staff in 5 participating sites

Johnson, et al.,

2011118

MidSouth eHealth Alliance

(MSeHA)

Data Exchanged: demographics, ICD-9 discharge

codes, lab results, encounter data, and dictated

reports.

Multiple hospital emergency departments and communitybased ambulatory clinics. Decentralized.

These are in a vault controlled by the hospital, but accessed

when a query is made, unless patient opts out.

May 2006 in EDs

later in clinics (NR)

6 ED sites and 9 clinics for

interviews

All visits records and usage

logs

Jones, Friedberg,

and Schneider, et 

al., 201168

Varied. As defined by hospital Varied. As defined by hospital Varied. As defined

by hospital

Hospitals in U.S.

Kaelber, et al.,

2013120

HIE in Northeast Ohio 10 hospitals and affiliated practices using Care Everywhere November 2010 Not stated for patient

population, 412 physician 

users
F-88

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Johnson, et al.,

200899

5 sites; number of users varies by site NR NR HIE use across sites and

overall

Johnson, et al.,

2011118

Number of people interviewed NR

369 comments (12% of all visits)

NA NA NA

Jones, Friedberg,

and Schneider, et 

al., 201168

2,406 hospitals (58% of eligible hospitals responded to

AHA survey)

General acute care non federally

owned U.S. hospitals

Not specified. Specialty

and federal implied by

inclusion criteria

Hospitals that self report

exchanging any information

with ambulatory providers

outside their system vs.,

hospitals who say they do not

participate in this type of HIE

Kaelber, et al.,

2013120

74 (18%) of physicians who replied to survey All users NA -Measurement of usage

-Perceptions of users
F-89

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Johnson, et al.,

200899

-% of ED visits with HIE use

-% of users who logged in

-Theme from comments: perception that

HIE reduces redundant testing was most

common

NA Role (Nurse, MD, registrar, unit

clerk)

Quantitative, descriptive

statistics

Counts and percentages

Johnson, et al.,

2011118

-HIE Access

-Type of data accessed

-Provider log on rates

NA -Profession (Doctors or

nurse/clerk)

-Type of visit

Mixed Methods

-quantitative, descriptive data

-qualitative analysis

-Counts and percentages

Jones, Friedberg,

and Schneider, et 

al., 201168

All- cause 30-day risk-standardized

readmission rates for patients initially

admitted with acute myocardial infarction,

heart failure, or pneumonia.

HIE Participation

(also use of health IT)

Hospital characteristics

(ownership, critical access status,

trauma status, number of beds,

teaching status, system

membership, core-based statistical

area type, U.S. census division,

long term care unit, critical care

unit)

Quantitative

-Unadjusted mean differences

-Propensity score matching

-Linear regression

Kaelber, et al.,

2013120

-Measurement of usage

-Perceptions of users

-Usage of HIE

-Survey of users

None Quantitative

Descriptive and Multivariate
F-90

Author, Year Results

Risk of

Bias

Johnson, et al.,

200899

HIE viewed in 2.6% of all visits and 9.5% of visits where patient had visit to other site in past 30 days.

% of total users who logged on ranged from 0 in one site where the high was 12% to 75% by unit clerks in a site that had high use by other

professions

-MSeHA was used for 3% of all visits

-The site with the highest usage had registrars looking up HIE data when patient arrived at the ED

-The site that mostly serves pediatric patients used MSeHA the least vs. other sites

NA

Johnson, et al.,

2011118

HIE access

Patient encounters increased over 24 months: 4% to 6.5% (range: 1 to 16 % across sites)

14.6% for return ED visits and 18.7% for return clinic visits (p<0.001)

Higher where nurses and clerks involved and lowest where MD only access

Patient opt out rates: 1% to 3%

Primary user reported consequence of HIE: provided additional history (29%), prevented repeat test or procedure (19.8%)

NA

Jones, Friedberg,

and Schneider, et 

al., 201168

Unadjusted readmission rates (no HIE vs. HIE)

Acute myocardial infarction: 20.0 vs. 19.8, p=0.14

Heart failure: 24.6 vs. 24.3, p=0.003

Pneumonia: 18.2 vs. 18.1, p=0.68

Hospitals did not participate in HIE: 58.7%

Adjusted readmission rates (no HIE vs. HIE)

Acute myocardial infarction: 19.9 vs. 19.8, p=0.18

Heart failure: 24.4 vs. 24.2, p=0.11

Pneumonia: 18.2 vs. 18.1, p=0.68

Low

Kaelber, et al.,

2013120

Usage of HIE

ED: 31% to 35%

Primary care: 18% to 22%

Specialty care: 9% to 11%

-Usage highest among patients who were older, with more comorbid illness, Medicare/Medicaid insured, and black

-Self-reported impact was more efficient care (93%), time savings (85%), prevented admissions (15%), decreased tests ordered (84%),

decreased imaging ordered (74%), and improved care in other ways (82%)

NA
F-91

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Kaushal, et al.,

201060

Cross-sectional To assess users experiences with

an HIE project that provided

medications information to EDs.

Massachusetts 5 Massachusetts 

Emergency

Rooms

Survey

Semi-structured interview

covering need for intervention,

history, personal use, induction,

current us, completeness and

accuracy, value added, rollout to

other hospitals and evaluation

Pharmacy benefit claims data

December 2005

Kern, et al.,

2011171

Same as Kern, et 

al., 2009173

Prospective

cohort

To determine predictors of

sustainability among communitybased organizations

implementing health IT including

HIE in a state with significant

funding of such organizations.

New York Varies (setting

was part of

analysis)

Survey and administrative data

Baseline assessment and New

York State Department of Health

information on awarded grants

Phone Interviews

January-February

2007 (same as

baseline for Kern,

2009).

New York State

Department of Health

data: March 2008

Kern, et al.,

2009173

Time series To identify lessons for statebased initiatives that can be

learned from HEAL NY

New York NR Organizational assessment

Baseline and followup

assessments

Baseline: JanuaryFebruary 2007 

Followup: July-August

2008

Kern, et al.,

201245

Retrospective

cohort

To determine the effect of HIE on

ambulatory quality

Hudson Valley

region, New York

Physician small

group practices

Log file

From Portal for usage, MVP

Health Care Quality Reports

including HEDIS measures and

satisfaction

January 2005-June

2006 (split into 3 6-

month periods)
F-92

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Kaushal, et al.,

201060

MedsInfo-ED, a project

Massachusetts Health Data

Consortium (MHDC)

Claims data from pharmacy benefit managers (PBMs) were

made available at the point of care to clinicians in the EDs

2004 Staff at participating sites

Kern, et al.,

2011171

Same as Kern, et 

al., 2009173

Varies NR Varies HEAL 1 Grantees given

awarded funds for health IT

Kern, et al.,

2009173

Varies NR Varies HEAL Grantees given

awarded funds for health IT

Kern, et al.,

201245

MedAllies Portal

covers 2 counties, 5 hospitals, and

2 labs

Internet-based with secure log-in from any computer.

Providers can view tests and results order by themselves or

others.

2001 Taconic Independent Practice

Association MDs
F-93

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Kaushal, et al.,

201060

N=12 interviewed of 15 contacted 3 EDs that were pilot sites; 2

more added in expansion.

Agreement to participate from

MassHealth and 5 health plans.

Patients not covered by

participating plans

Comparisons across the 3

initial pilot sites

Kern, et al.,

2011171

Same as Kern, et 

al., 2009173

26 Phase I grantees (100%) HEAL 1 Grantee NA Organizations that received

further funding vs. those that

did not

Kern, et al.,

2009173

26 HEAL grantees NA NA NA

Kern, et al.,

201245

138 MDs with quality information (out of 168, 82%) 79

nonusers and 59 users of the HIE portal

≥150 patients with MVP Health

Care

No quality of care data Physicians who used portal vs.

those who did not
F-94

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Kaushal, et al.,

201060

Descriptive narrative only NA NA Thematic analysis

Coding of interview transcripts by

tow investigators

Kern, et al.,

2011171

Same as Kern, et 

al., 2009173

Receipt of HEAL 5 funds -Responses to 26 questions covering 9

areas

-Type of organization that was the lead

application (health care or health

information)

NA Quantitative multivariate

analysis

-Bivariate and multivariate logistic

regression

-Backward stepwise elimination

Kern, et al.,

2009173

-Grantee still in operation

-Exchanging data or implementing other IT

-Met definition of RHIO

NA None reported Quantitative

-Counts and proportions

-McNemar 2-sample test for

binomial proportions for matchedpair data for comparison between

baseline and followup

Kern, et al.,

201245

-Rate of portal use

-Quality of care

Any portal use -Physician characteristics

-Case mix

Quantitative

-Chi2

-t-tests

-Fischer exact tests

-Generalized estimating equation

regression
F-95

Author, Year Results

Risk of

Bias

Kaushal, et al.,

201060

Need: respondents believed gaps in medical information are an important problem and this system could help

Information was perceived as accurate, range of estimate of patients with information 15% to 80%

Perception: system improved knowledge but did not decrease time and did not improve care enough to justify hospital paying for system

Barriers: need for patient consent, difficulty matching patients

Suggestions: increasing the types of information included (e.g., psychiatric, HIV, and mail order medications) and improving the format of

the output

High

Kern, et al.,

2011171

Same as Kern, et 

al., 2009173

Predictors of funding from bivariate (OR, 95%CI)

Lead by health information organization: 11.4, 1.7 to 78.4, p=0.01

Performed community-based needs assessment: 5.1, 0.8 to 32.3, p=0.08

Targeting long term care settings: 0.14, 0.02 to 0.79, p=0.03

Predictors of funding from multivariate (OR, 95%CI)

Lead by health information organization: 6.4, 0.8 to 52.6, p=-.08

High

Kern, et al.,

2009173

-All grantees still existed at followup

-Half decreased number of planned projects (3 possible: HIE EHR, electronic prescriptions)

-HIE all grantees planning at baseline, 85% at followup (22 of 26)

-9 (35%) had users ranging from 5 to 1600. HIE was most common project.

-13 baseline/20 followup met definition of RHIO

-Expected interventions (not just HIE) to save money: 65% baseline, 35% followup p=0.02

-Concern about financial and technical barriers increased by followup

Moderate

Kern, et al.,

201245

-% of MDs using portal: 33% months 1-6 vs. 42% months 7-12 vs. 43% months 13-18

-Mean days logged in per month by MD: 8 (SD 6)

-Quality score at followup: 49 for nonusers vs. 64 for users, p<0.0001

-OR for higher quality use of portal: 1.42 (95% CI, 1.04 to 1.95)

-Average ambulatory quality of care for composite of 15 measures, stratified by time and use of HIE showed difference between non-users

vs. users (49% vs. 64%, p<0.0001) at followup and among users between baseline vs. follow-up (57% vs. 64%, p<0.001)

Low
F-96

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Kern, et al.,

2012147

Cross-sectional To understand which components

of EHRs and HIE are most likely

to drive financial savings in the

ambulatory, inpatient, and ED

settings.

NA Ambulatory,

inpatient, and ED

settings.

Literature and expert

consensus

Literature search results, input of

28 national experts, analysis of

Stage 1 of Meaningful Use

April 2007 (expert

review)

Kho, et al., 201388 Prospective

cohort

To describe the use of an HIE for

tracking patients with

antimicrobial resistance

Indianapolis,

Indiana

Hospital and 

associated clinics

Survey, log data June 2007-June 2010

Kierkegaard, 

Kaushal, and 

Vest, 2014127

Multiple site case

studies

To investigate how HIE can better

meet the needs of care

practitioners

3 communities

(RHIOs) in New

York State

ED and

outpatients

Observations, interviews

2 day site visits, onsite and

telephone interviews with HIE

users and nonusers,

observations of workflow

May-June 2013

Lammers, AdlerMilstein, and

Kocher, 201469

Cross-sectional To evaluate whether HIE is

associated with decreases in

repeat imaging in EDs

California and

Florida

EDs Database

State ED databases, Health

Information Management

Systems Society data, AHA

annual survey

2007-2010
F-97

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Kern, et al.,

2012147

NA NA NA HIE functions by settings

Kho, et al., 201388 Indiana Network for Patient Care

(INPC)

5 hospital systems (17 hospitals) May 2007 for this

tracking function

Infection preventionists at all

hospitals; patients with MRSA

or VRE

Kierkegaard, 

Kaushal, and 

Vest, 2014127

NA 2 federated model, 1 centralized model.

All required login to standalone web portal

2 provided automated delivery of imaging and lab results

1 included patient portal and iPhone app

1 included secure messaging and event notification. 

Query- based but also provided direct exchange of CCD

NR 11 RHIOs in NY and users

and non users of HIE

Lammers, AdlerMilstein, and

Kocher, 201469

Varies, not a single HIE Varies Varies ED visits in California and

Florida
F-98

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Kern, et al.,

2012147

Top 10 functions based on researcher ratings In top 10 for function based on:

1) probability of achieving a

benefit, 2) time to benefit, 3)

probability of measuring a

benefit for initial framework.

Experts added 3 additional

criteria

4) complexity, 5) likelihood of

usage, and 6) expected

magnitude of impact

Rating below top 10 High rated functions across

setting and between HIE and

EHRs

Kho, et al., 201388 NR NA NA NA

Kierkegaard, 

Kaushal, and 

Vest, 2014127

N= 38 interviews

3 sites (13, 15, 10)

3 EDs, 7 outpatient

3 types of respondents: MDs, other clinical users,

administrative users

Received HEAL NY funding and

been in existence for ≥7 years,

and distinct.

NA Themes across sites

Lammers, AdlerMilstein, and

Kocher, 201469

Patients at HIE adopters: 33,084 (11%)

Patients at non adopters: 274,640

ED visits with data in State and

HIMSS, patient had another ED

visit in prior 30 days in different

EDs, or selected imaging in

index visit

ED visits that resulted in

admissions

37 EDs that participated in HIE

vs. 410 that did not
F-99

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Kern, et al.,

2012147

Rating of function Setting type (HIE, EHRs) NA Quantitative

ANOVA for scores across

settings

t-tests for HIE, EHRs

comparisons

Kho, et al., 201388 -Number of alerts generated

-Number of patients admitted to multiple

hospitals

-User satisfaction/ burden

-Coordinated antibiotic-resistant infection

tracking, alerting and prevention

NA NA Counts

Kierkegaard, 

Kaushal, and 

Vest, 2014127

Themes related to use of HIE Site and type of setting NA Qualitative

-Thematic analysis from

transcripts

-Dual coding of interviews

-Iterative coding, grouping of

themes in categories continued

until saturation

Lammers, AdlerMilstein, and

Kocher, 201469

Repeat CT, ultrasound or chest x-ray in

same body region within 30 days at

unaffiliated EDs

HIE participation in each year -Patient demographics

-Number of days between ED visits

-comorbidities

-Total annual ED discharges

-ED characteristics

Quantitative

Regression with fixed effects and

trends
F-100

Author, Year Results

Risk of

Bias

Kern, et al.,

2012147

-73 setting-HIE function pairs were identified

-Mean function score (range 6 to 18): 13.0 EHR vs. 11.3 HIE, p<0.0001

-No difference in scores across setting (p=0.33)

-High scoring HIE functions: transferring imaging reports (all settings), receiving lab results (outpatient and ED), enabling structured

medication reconciliation

-HIE functions were considered more difficult to implement (complexity and time) vs. EHRs

-HIE is most likely to generate a positive financial effect through its ability to coordinate care among providers. Based on assessment for

EHRs adding decision support to HIE could potentially yield even greater financial returns

Kho, et al., 201388 Over 3 years

-12,748 email alerts on 6,270 unique patients

-23% (MSRA) and 22% (VRE) had previous history identified at a different hospital system

10 Infection Preventionists surveyed

-All reported email alerts were useful

-Estimated receiving 5 alerts per day; half already known; alerts used to identify patients requiring intervention

-3 said system added time, 1 saved time, 6 neutral

-Most comment recommendation was to add automate capture of lab data

Low

Kierkegaard, 

Kaushal, and 

Vest, 2014127

Availability of information varied based on patient consent (required in New York State) and healthcare organization participation.

USE

-MDs had low tolerance for search failures.

-Practice staff are important to obtaining patient consent. Where clerks were not trained or supported, fewer patients consented.

-Patients saw providers covered by other exchanges, suggesting need for larger areas

-Physician use HIE less than other clinical users; MDs often delegate the task.

USABILTY

-Login process perceived as a burden

-Slow system response times

Moderate

Lammers, AdlerMilstein, and

Kocher, 201469

Probability of repeat ED imaging (percentage points [95% CI]), relative reduction

CT: -8.7 (-14.7 to -2.7), 59%

Ultrasound: -9.1 (-17.2 to -1.1), 44%

Chest x-ray: -13.0 (-18.3 to -7.7), 67%

-Repeat tests more likely in large EDs

Low
F-101

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Lang, et al.,

200665

RCT Impact of sending family

physicians electronic vs. mailed

reports of ED visits for their

patients

Montreal, Canada ED and family

physician 

practices

Database

Surveys and determination of

patient outcomes

June 2001-April 2002

Lee, et al., 201289 Pre-post

implementation

survey

To understand MD perception

prior to HIE implementation and

post implementation use and

evaluation

South Korea Hospital and 

ambulatory clinics

Survey, audit logs June 2008 Week 1

and 2 (pre survey)

Post: NR

Lobach, et al.,

2007100

Cross-sectional To describe use of an HIE for

population health management

Durham County,

North Carolina

Outpatient Audit logs September 2006-

February 2007
F-102

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Lang, et al.,

200665

Adult university teaching hospital in

Montreal

Report of ED visit sent to family physicians NR Patients visiting ED during

0800-2200

Lee, et al., 201289 Seoul National University Bundag

Hospital and 35 clinics

Federated architecture model with ebXML RS and ebSML RIM

standards

Included demographics, diagnoses, medications, lab results,

imaging, treatment, care plans, vital signs, history and

summaries.

June 2008 with

updates October 

2009

MDs in hospital (50) and

clinics (147) for pre; MDs

using the HIE for post

Lobach, et al.,

2007100

Northern Piedmont Community

Care Network set up a system

called COACH (CommunityOriented Approach to Coordinated

Healthcare) includes 32 private

practices, 3 federally qualified

health centers, 4 community

hospitals, 9 government agencies

(county health departments and

departments of social services), 1

academic medical center, and 2

care management teams: Durham

County, North Carolina, Medicaid

The 4 types of data collected by the system include*: 1)

administrative (demographics and identifiers, services used,

provider associations, audit trails); 2) care management (care

management encounters, health risk and environment

assessment, socio-economic data, special needs, and care

management plans); 3) clinical (encounters,

problems/procedures, missed appointments, medications,

allergies, laboratory results, disease-specific care plans); and

4) communication (messages and alerts, referrals, notices of

new information).

2001 Patients in program
F-103

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Lang, et al.,

200665

2,022 (out of 3,168) patients visiting ED Patients visiting ED Patients in altered mental

state (129), state of

agitation (21), or with

language barrier (29)

ED visit summary provided

electronically vs. on paper sent

by mail

Lee, et al., 201289 23 from hospital and 48 from 20 clinics (46% and 33%

response) for pre; 15 from hospital and 25 from clinics

for post out of all MDs using the system

MD at pilot site <50% of items completed Hospital vs. clinic based MDs

Lobach, et al.,

2007100

11,899 patients in Durham County in Medicaid NA NA NA
F-104

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Lang, et al.,

200665

-Physician satisfaction

-Return visits at 14 and 28 days

-Duplication of requests for diagnostic tests

-Duplication of specialty consult requests

- Economic

-Physician satisfaction

-Return visits at 14 and 28 days

-Duplication of requests for diagnostic

tests

-Duplication of specialty consult

requests

Physicians already are sent carbon

copies of first page of ED note; selfreport of followup data

Quantitative

Survey, analysis of followup care

Lee, et al., 201289 -Pre: Perceptions

-Post: Information transmission rate

Information utilization rate

Setting (hospital vs. clinic based) -Gender

-Age

-Specialty

Quantitative

Fischer exact tests

Lobach, et al.,

2007100

Sentinel events: resource utilization by

patients (events of commission) that were

considered excessive (e.g., 3 ED visits in

90 days) or potentially avoidable (e.g., ED

visit for asthma) and that could potentially

be modified by the involvement of care

managers and other providers

None None Quantitative

Counts, observation
F-105

Author, Year Results

Risk of

Bias

Lang, et al.,

200665

-Reports found to be received, especially in timely manner, and were more likely to be legible, comprehensive, and useful.

-No difference in return visits within 14 and 28 days, although near significance for fewer visits for patients >65 years within 28 days.

-No difference in duplicate test ordering but greater subspecialty consult requests in intervention group.

Moderate

Lee, et al., 201289 Pre HIE

-Mean Likert scale that HIE is needed (5 strongly agree): 4.2, p=0.8888 for all and by setting. Similar responses about the need for HIE

for specific items (e.g., lab reports) and perceived benefits of HIE.

-Hospital based MDs had higher levels of agreement about concerns related to HIE than clinic based MDs

Post HIE

Most commonly transmitted information differed by setting

From hospital was working diagnosis: 99.5% vs . 70.5% for clinic, p<0.0001

From clinic it was clinical findings: 79.8%, but this did not differ from hospital

The most useful was lab or imaging in both settings but it was more frequently rated as useful by hospitals (88.2% and 72.9% of cased

p<0.0001)

High

Lobach, et al.,

2007100

In an analysis of 11,899 continuously enrolled patients from a single county over a six-month period 19.3% (2,285 unique patients) had

7,226 sentinel health events

Frequency of types of events

Hospital admit asthma: 43

Hospital admit diabetes: 76

Low-severity ED: 2, 546

≥2 missed appointments in 60 days: 1,728

Implementation lessons

-Political issues are more challenging than technical issues

-Perceived value of notices was dependent on timeliness and completeness of underlying HIE dataset.

-Difficult to determine who should be notified of these events, how many notices should be resent and how to prioritize them.

Low
F-106

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Maass, et al.,

200861

Cross-sectional Ascertain benefits of HIE when

they occurred

Finland Regional

information

system for 

exchange of

clinical data

between hospital

and primary care

offices

Survey

Time-motion study of diabetic

patients in a health center

NR

Machan,

Ammenwerth, and

Schabetsberger,

200662

Cross-sectional Assess value of different aspects

of regional network of hospitals

and physician practices

Tyrol region of

Austria

Regional

information

system for 

exchange of

clinical data

between hospital

and primary care

offices

Survey, interviews

Initial qualitative development of

survey followed by quantitative

evaluation of responses

May-August 2004

Mäenpää, et al.,

201146

Retrospective

cohort

What is impact of a regional

health information system on test

ordering and referrals?

Tampere, Finland Hospital district

that includes 1

hospital district

and its community

health system.

Outpatient

Log file

Usage of HIE and ordering of

laboratory and radiology tests as

well as specialty referrals

Data collected 2004-

2008
F-107

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Maass, et al.,

200861

Regional information system in

Finland

Transmission of patient data into physician EHR NR Physicians in health centers in

Finland

Machan,

Ammenwerth, and

Schabetsberger,

200662

Tiroler Landeskrankenanstaleten

(TILAK)

Transmission of discharge letters and clinical findings from

hospitals to general practitioners. Direct exchange via

email.

June 2003 General practitioners in Tyrol,

Austria

Mäenpää, et al.,

201146

Regional information system in

Finland

Full medical record in regional information system 2004 About 234,000 inhabitants in

hospital district and associated

clinics
F-108

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Maass, et al.,

200861

20 visits by patients with diabetes NR NR Use of information system and

description of benefits

Machan,

Ammenwerth, and

Schabetsberger,

200662

4 providers followed by cross-sectional survey of 104

of 242 (43%) providers.

All general practitioners in Tyrol None None

Mäenpää, et al.,

201146

NR NA NA Appointments, ED visits,

laboratory and radiology tests

for primary and specialty care
F-109

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Maass, et al.,

200861

Use of information system and description

of benefits

System used and benefits described NA Thematic analysis

Time-motion study

Machan,

Ammenwerth, and

Schabetsberger,

200662

-Measurement of overall satisfaction

-Desirability for receiving reports

electronically

-Reduced work for filing and archiving

-Leading to improved quality of care

Survey NA Mixed methods

-Quantitative, descriptive data

-Qualitative, content analysis

Mäenpää, et al.,

201146

-Rates of laboratory and radiology test

ordering

-ED visits and primary care referrals

None Use of HIE not correlated

specifically with outcomes

Quantitative

Log analysis
F-110

Author, Year Results

Risk of

Bias

Maass, et al.,

200861

20 visits, 4 involved use of information system, with 1 allowing faster treatment decision and 3 providing access to latest test results High

Machan,

Ammenwerth, and

Schabetsberger,

200662

Satisfaction with HIE

Positive: 66.4%

Agreeing desirable for receiving all reports electronically: 83.7%

Reporting less work for filing and archiving: 82.7%

Agreeing it led to improved quality of care: 78.8%

Low

Mäenpää, et al.,

201146

Change in rates of ordering over time (primary vs. specialty care)

Laboratory tests per appointment: 19.0% vs. 7.0%

Laboratory tests per inhabitant: 19.0%, 17.9%

Clinical chemistry ordering per appointment: 6.6% overall

Clinical chemistry ordering per inhabitant: 17.5% overall

Radiology exams per appointment: -16.4% vs. -11.0%

Radiology exams per inhabitant: -18.9% vs. -1.9%

ED visits: -1%, -16.2%

Primary care referral to specialist per appointment: 43.6%

Primary care referral to specialist per inhabitant: 35.2%

Low
F-111

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Mäenpää, et al.,

2012115

Retrospective

cohort

What is usage of a regional

health information system for

different amounts of test ordering

and referrals?

Tampere, Finland Hospital district

that includes 1

hospital district

and its community

health system

Audit logs

Usage of HIE and ordering of

laboratory and radiology tests as

well as specialty referrals

Data collected 2004-

2008

Magnus, et al.,

201247; Herwehe,

et al., 2012124

Retrospective

cohort

To describe patients identified by

the LaPHIE system and HIVrelated outcomes associated with

LaPHIE over 2 years.

Louisiana HIV specialty,

inpatient and 

outpatient care

within Louisiana

State University

Health Care

Division system.

Includes 7 safety

net hospitals

Log file

Alerts for HIV patients that

continue to appear until patients

receive CD4 or VL testing;

actions taken by the provider are

documented within the structured

EMR

February 1, 2009-July

31, 2011
F-112

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Mäenpää, et al.,

2012115

Regional information system in

Finland

Full medical record in regional information system 2004 10 municipalities;

About 234,000 inhabitants in

hospital district and associated

clinics

Magnus, et al.,

201247; Herwehe,

et al., 2012124

Seven safety-net hospitals; LaPHIE is a secure bi-directional public health informatics

application linking statewide public health surveillance data

with patient-level EMR data. The exchange functions in realtime throughout the integrated data networks emergency

departments, primary care and specialty ambulatory clinics,

and inpatient units.

February-September

2009 (Herewhe,

2012)

HIV patients coming to

Louisiana State University

Health Care Services division

clinics or ED.
F-113

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Mäenpää, et al.,

2012115

NR NA NA Usage of HIE by physicians,

nurses, and department

secretaries, and number of

appointments, ED visits, and

laboratory and radiology tests

Magnus, et al.,

201247; Herwehe,

et al., 2012124

419 patients in 60 clinics; alerts to 223 clinicians HIV persons identified by

LaPHIE with no CD4 or VL

monitoring in >1 year, were

followed in 6-month intervals for

retention in HIV specialty care, 

inpatient and outpatient

healthcare utilization

HIV patients who had been

seen within past year and

had no break in care of >1

year since diagnosis

Time-matched random sample

of HIV-infected persons who

had been seen for HIV care

within the Louisiana State

University Health Care

Services Division integrated

data network ≥1 within the past

5 years at the time of

comparison.
F-114

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Mäenpää, et al.,

2012115

-Rates of laboratory and radiology test

ordering

-ED visits and primary care referrals

Usage of HIE Use of HIE not correlated

specifically with outcomes

Quantitative

Descriptive statistics and

negative binomial regression

Magnus, et al.,

201247; Herwehe,

et al., 2012124

-CD4 <200 cells/mm3

-VL >10,000 RNA copies/mL

-Having been prescribed antiretroviral

treatment during each 6-month interval

Use of LaPHIE Adjusted for demographic and

clinical characteristics

and timing of entry into the cohort

Quantitative

-Chi2 tests, unadjusted logistic

regression, and adjusted logistic

regression

-Generalized estimating

equations using an

exchangeable correlation matrix
F-115

Author, Year Results

Risk of

Bias

Mäenpää, et al.,

2012115

Usage of HIE (views per year)

Physicians: 1,333

Nurses: 758

Department secretaries: 497

-No associations detected between use of HIE and test ordering outcomes

References (means one view of the HIE) viewed in primary health care in 2004–2008:

By physicians from n=486 to n=3581

By nurses from n=59 to n=2,3535

By department secretaries from n=26 to n=13,542

References viewed in special care in 2004–2008:

By physicians from n=1,496 to n=25,051

By nurses from n=284 to n=20,587

By department secretaries from n=1,156 to n=6,958

-The HIE utilization rates increased annually in all 10 federations of municipalities, and the viewing of reference information increased

steadily in each professional group over the 5-year study period. In these federations, a significant connection was found to the number of

laboratory tests and radiology examinations, with a statistically significant increase in the number of viewed references and use of HIE.

The higher the numbers of emergency visits and appointments, the higher the numbers of emergency referrals to specialized care, viewed

references, and HIE usage among the groups of different health care professionals.

NA

Magnus, et al.,

201247; Herwehe,

et al., 2012124

"After adjustment for demographic and clinical characteristics and timing of entry into the cohort, the LaPHIE-identified group remained

significantly more likely to be immunocompromised (CD4 < 200 cells/mm3) than their counterparts (OR 3.22, 95% CI 1.72 to 6.04,

p<0.001). However, there was improvement over time, with a decrease in odds of having a CD4 < 200 cells/mm3 at each successive sixmonth interval (OR 0.91, 95% CI 0.83 to 0.99, p<0.05). VL proved more responsive to changes in treatment and care; LaPHIE-identified

persons rapidly became similar to their in-care counterparts, with no significant differences between VL, and again, decreased odds of

having a VL > 10,000 copies/mL at each successive interval (OR 0.83, 95% CI 0.73 to 0.93, p<0.01)."

24% of those identified had not had a CD4 count or VL since initial diagnosis. Of remaining 76% who had been in care previously, 55%

had been out of care for ≥18 months. Following LaPHIE identification, 42% had CD4 counts < 200 cells/mm3 and 62% had VL >10,000

RNA copies/mL. Of 344 patients with at least 6 months of followup, 85% had ≥1 CD4 and/or VL after being identified.

Low
F-116

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Massy-Westropp,

et al., 2005134

Cross-sectional Pilot the effectiveness of

electronic data linking tools to

assist in the transfer of

information between an acute

care hospital and the main

regional provider of home-based

care.

Adelaide, South

Australia

Link patient health 

information

between the

hospital and

community

services sector

Survey, focus group

Email alert to community; remote

access to hospital reports; flag

community patients; web access

to community reports.

Piloted over 6 months

2002-2003

McCarthy, et al.,

2014161

Multiple case

studies

Factors influencing technical

architecture, clinical outcomes,

and challenges for Beaconfunded HIEs

Regions within

Maine, Indiana,

Ohio,

Washington, 

Pennsylvania,

Oklahoma, New

York

Any Interviews

Written and telephone interviews

of implementers of 7 HIEs

NR

McCullough, et 

al., 2014135

Cross-sectional To assess barriers and benefits

to HIE participation in 2

underserved settings

San Gabriel

Valley, California

and Minneapolis

St. Paul,

Minnesota

Outpatient small

practices

(California) and

federally qualified 

health centers

(Minnesota)

Interviews of clinicians,

administrators and office staff

users

NR

McGowan, et al.,

2007148

Cross-sectional To ascertain lessons learned in

the development of Vermont's

RHIO

Vermont NR Interviews and documents and

presentations about the

development of VTMEDNET

NR
F-117

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Massy-Westropp,

et al., 2005134

Public teaching hospital, ED and

aged home-based care community

services organization.

Email alert to community; remote access to hospital reports;

flag community patients; web access to community reports.

Piloted over 6 months

2002-2003

Medical, nursing, and alliedhealth staff across the

organizations

McCarthy, et al.,

2014161

Beacon Communities within Maine,

Indiana, Ohio, Washington,

Pennsylvania, Oklahoma, New

York

Varied from hybrid-federated to centralized 1994-2009,

depending on HIE

Operational, technical, and

clinical leaders of each HIE

McCullough, et 

al., 2014135

Citrus Valley Health Partners

Federally Qualified Health Center

Urban Health Network (FUHN)

California: Collaborate system. a web-based tool enabling all

providers to view data exchanged from 3 hospitals, an

anticipated 90 providers, and laboratories in the community

and to securely message other providers.

Data are available to be viewed by all participating providers,

regardless of whether a physician is contributing data to the

system.

Minnesota: CentraHealth aimed at enabling electronic

exchange between FQHCs and the hospitals serving

their Accountable Care Organization patients. This system was

in implementation at time of study

NR Independent practices serving

predominately Hispanic

patients and federally qualified 

health centers developing an

accountable care organization

McGowan, et al.,

2007148

VTMEDNET (early HIE) and more

recent statewide RHIO

Federally funded (NLM and AHRQ) initiated by hospitals, but

developed by a coalition. No other detail provided

NR NA
F-118

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Massy-Westropp,

et al., 2005134

82 medical, nursing and allied-health staff. HIE

included up to 4,000 patients.

Satisfaction survey responses from 55 or 132 nurses,

clinicians and allied health staff.

NR NR 82 respondents of HIE project

vs. 50 care providers outside of

the HIE project

McCarthy, et al.,

2014161

7 HIEs funded by Beacon Community grants NA None Compared various factors

across hybrid-federated vs.

centralized HIEs

McCullough, et 

al., 2014135

N=24 providers, administrators, and office staff in 16

sites

Individuals who would be

involved in adoption decisions

and integration of HIE into

workflows at each organization

None None

McGowan, et al.,

2007148

5 interviews: 2 CIO of hospitals and 3 key leaders NA NA Description of 2 efforts. Some

limited comparison of the 2
F-119

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Massy-Westropp,

et al., 2005134

Satisfaction with electronic data linking NA NA Mixed methods

-Quantitative, descriptive

statistics

-Qualitative, content analysis

McCarthy, et al.,

2014161

-Trust

-EHR context

-Clinical transformation

-Clinical research

Qualitative NA Qualitative

Interviews

McCullough, et 

al., 2014135

Benefits and barriers to HIE use NA NA Qualitative

Thematic analysis from

transcripts

McGowan, et al.,

2007148

Facilitators and barriers to creation and

implementation

NA NA Qualitative

Simple summary of interviews
F-120

Author, Year Results

Risk of

Bias

Massy-Westropp,

et al., 2005134

Provided bar graphs (figures 2 and 3) but not specific quantitative results except for a statement about use and satisfaction. Those who

had embraced the use of the Integration tools were significantly more likely to rate integration higher than those who were not using it as

often (p<0.001). In the discussion they estimated a 20% savings in staff time.

High

McCarthy, et al.,

2014161

Hybrid-federated models maintain autonomy, accommodate disparate EHRs, and build incrementally, while centralized models require

trust fabric, leverage common EHRs, and while providing long-run cost-efficiency may require larger upfront investment. Hybrid-federated

models provide most functionality at individual organization level while centralized models leverage value of communitywide data and

usage.

Moderate

McCullough, et 

al., 2014135

Barriers

-Lack of well-functioning area-level exchange

-Market characteristics

-Relationships or previous experiences with exchange partners

-Challenge achieving a critical mass of users

-Health IT used

-Data ownership and provider liability concerns

Benefits

-Improved productivity at initial visit

-Improved completeness of records

-Avoidance of duplicative services/patient financial risk

-Improved nonvisit consults

Low

McGowan, et al.,

2007148

Major facilitators for success

-Public awareness

-Provider buy-in

-Benefits understood in terms of patient safety and quality of care

Barriers

-Perceived public perception of privacy issues

-Providers lack working knowledge of HIE concepts

-Need for a sustainable business model is recognized but not solved

-Need for health information to cross state lines

High
F-121

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Merrill, et al.,

2013174

Time series Evaluate the complex dynamics

involved in implementing

electronic HIE for public health

reporting at a state health

department, and to identify policy

implications to inform similar

implementations

New York State health

department, 3

RHIOs

Interviews, documents

Lab results and other information

for rapid and efficient

identification, monitoring,

investigation, and treatment of

communicable and emerging

diseases

2010-2011

Messer, et al.,

2012138

Before-after (1) Assess and

enhance organizational readiness

to adopt information technology,

(2) develop a RHIO to share

electronic data between

medical and ancillary care

providers, (3) implement the

RHIO

and begin active information

exchange and (4) evaluate the

effect of the intervention on

provider-related attitudes and

satisfaction with information

exchange

North Carolina Ambulatory HIV

providers and

ancillary care

providers

Interviews

-Pre-post survey

-HIV patient data and lab results

2010
F-122

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Merrill, et al.,

2013174

3 RHIOs and New York State

Department of Health.

Lab results and other information for rapid and efficient

identification, monitoring, investigation, and treatment of

communicable and emerging diseases

August 2007-August 

2011

Not described but patients

who would be reported to the

health department for risk and

disease.

Messer, et al.,

2012138

Carolina HIV information

cooperative regional health

information organization (CHIC

RHIO)

1 large academic medical center and 5 AIDS service

organizations. Used CAREWare from HRSA. Federated,

query-based exchange.

2008 organization

begun

HIV care providers and

ancillary service providers
F-123

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Merrill, et al.,

2013174

NR NR NR NA

Messer, et al.,

2012138

1 large academic medical center and 5 AIDS service

organizations mostly providing case management.

Interviews and assessment with 39 stakeholders; pre

and post survey of 29 providers' satisfaction with 

HIE, relationships with other providers, barriers.

Leaders of the individual

organizations, HIV providers

NA NA
F-124

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Merrill, et al.,

2013174

Descriptive narrative only NA NA Qualitative

Messer, et al.,

2012138

-Organization readiness for Charge

measure

-Qualitative process summary

-Provider surveys of effectiveness

NA NA Mixed Methods

-Quantitative, descriptive data

-Qualitative, theme analysis from

transcripts.
F-125

Author, Year Results

Risk of

Bias

Merrill, et al.,

2013174

Three casual loop diagrams captured well recognized system dynamics: Sliding Goals, Project Rework, and Maturity of Resources. The

findings were associated with specific policies that address funding, leadership, ensuring expertise, planning for rework, communication,

and timeline management.

Low

Messer, et al.,

2012138

-Organizational readiness assessment found organizations were well prepared to adopt new technology, in the 4 domains (motivation,

adequacy of resources, staff attributes, and org climate) only motivation was slightly below nationally determined levels. Results were

consistent by agency type and respondent type

-Largely positive response to quality process. Improved sense of mission, more contact with other agencies, better awareness of other

agency roles.

-Providers found increased case manager knowledge of medical care

-Concerns: Initial concerns about confidentiality dismissed over time as trust was built; Respondents noted it is important to manage

expectations upfront; Clinic staff must use 2 systems the EHR and CAREWare which takes effort and increases errors; There was an

unmet need for training for report generation

-Quantitative provider survey: AIDS service organizations and medical providers generally both felt increased ease of data exchanged and

that patient care improved. For AIDS service organizations 7/8 satisfaction related questions improved statistically from pre-post, in clinic

survey 4/8 improved statistically

Moderate
F-126

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Miller, 2012162 Multiple case

studies

Assessed how well 5 diverse

California health care entities’

HIE capabilities, policies, and

procedures satisfied the patient

and consumer principles as of

early 2011.

California A captivated

integrated delivery

system (Kaiser); a

physician 

management

service 

organization

(Nautilus); a large 

public hospital; a

large Medicaid

HMO; a regional

HIE organization

Interviews

EHR, Patient portal, HIE,

administrative, inpatient, 

outpatient. Patients’ 

medications, allergies, chronic

disease diagnoses, history, and

lab results. Providers could also

view hospital radiology reports.

August 2010-April

2011

Miller and Tucker

2014149

Cross-sectional How does size of user (hospital

health system or network) affect

HIE usage?

U.S. Health systems

and networks

Survey

Hospital Electronic Health

Record Adoption Database (AHA,

funded by ONC and is intended

to be the most comprehensive

and representative survey of the

state of healthcare IT)

2007-2009
F-127

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Miller, 2012162 1 capitated integrated delivery

system (Kaiser); a physician 

management service organization

(Nautilus); a large public hospital; a

large Medicaid HMO; a regional

health information exchange

organization

Each of the 5 systems had their own HIE. Some used EPIC,

Next Gen, Siemen's NetAccess, Axoloti's Elysium HIE software

NR NR

Miller and Tucker

2014149

Various Various, within-system and out-of-system HIE Various U.S.
F-128

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Miller, 2012162 N=5 organizations; 23 interviews with 18 people NR NR They compared against 9

principles e.g., important

benefits for individual health;

important benefits for

population health; inclusivity 

and equality; etc.

Miller and Tucker

2014149

430 hospital systems, 4,060 hospitals; average system

contains 6 hospitals and operates in just under 4

regional markets

NR None NA
F-129

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Miller, 2012162 Discussed each principle and how well it

was met

NA NA Qualitative

Descriptive

Miller and Tucker

2014149

Self reported internal or external exchange

of data by hospitals

System's size, defined as the number

of hospitals owned, leased, sponsored

or contract-managed by a central

organization

Patient flow, insurance status

(Medicaid, Medicare fractions) per

capita payroll, physician

relationship (independent practice

association, group practice,

integrated salary model);

profit/nonprofit status; specialty vs.

general; IT vendor (HIE capability),

EMR age

Quantitative multivariate

analysis

Unit of analysis is hospital,

logistic regression p (exchange) =

system size, etc.
F-130

Author, Year Results

Risk of

Bias

Miller, 2012162 Discussed each principle. Also discussed challenges and barriers. Moderate

Miller and Tucker

2014149

68% do internal exchange: HIE increases with system size; each additional hospital in system increases likelihood by 2 percentage

points; increase if nonprofits, decrease w/ more Medicaid, Medicare, unaffected by location in U.S., age of technology, vendor

17% do external exchange: larger hospital systems are less likely to exchange information externally. Each additional hospital in a

system lowers the chance of external data exchange from hospitals in that system by 0.7 percentage points. Not affected by relative

number of outside hospitals; more sharing with number of beds, number of doctors, % Medicare, per capita payroll; regardless of age of

system or size of vendor

-Robust to type of data (demographic or clinical);

-No relation to HMO, PPO, etc.;

-Same effects stronger with higher per capita salaries, suggesting some strategic benefit

Moderate
F-131

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Moore, et al.,

2012106

Cross-sectional To describe the status and

lessons learned from the

development and establishment

of an HIE based system to alert

ambulatory providers when their

patients are admitted or

discharged from the hospital or

ED.

New York City Hospital, ED, and

out patient

System logs November 1, 2010-

April 30, 2011 (6

months)

Myers, et al.,

2012128

Multiple site case

studies

Describe how members of HIV

patients’ care teams perceived

usefulness and ease of use of

newly implemented, innovative

HIEs in diverse HIV treatment

settings.

Urban settings

and 1 suburban

setting in New

York, New

Jersey, 

Louisiana,

California, North

Carolina

Hospital specialty

clinics, support

services, primary

care clinics,

testing sites, ED,

outpatient and 

inpatient clinics,

Office of Public

Health, insurers,

laboratory and 

pharmacy services

Survey and interviews during site

visits.

Laboratory, diagnostic, medical,

and service utilization; referrals;

and ancillary care support, such

as case management, counseling

and testing, transportation, and

substance use and mental health

services

July 2008-December

2010
F-132

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Moore, et al.,

2012106

New York Clinical Information

Exchange (NYCLIX)

-An event detection and notification system based on a RHIO

including major medical centers, primary care

physicians, a home health care agency, long-term care

facilities and a Medicaid managed care plan

-NYCLIX uses a federated architecture in which the clinical

repository is spread over a collection of “edge servers” that 

reside in each of the members’ data centers.

-Alerts are considered 1-to-1 communication between

providers and are limited to name, date and location of service,

so patient consent was not required

November 2009 63,305 patients enrolled from 

3 hospitals

Myers, et al.,

2012128

5† HIEs that were part of the

Information Technology Networks

of Care Initiative that included

Bronx-Lebanon Hospital Center,

Duke university; hospitals, the city

of Paterson, Louisiana State

University Health Care Services

Division, NY Presbyterian Hospital,

St. Mary Medical Center

Foundation. Query-based

5 HIEs, each site designed, tailored, and implemented

enhancements to existing HIEs according to local needs

NR Members of HIV patient care

teams
F-133

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Moore, et al.,

2012106

NR NA NA NA

Myers, et al.,

2012128

60 case workers, medical providers, nonclinical staff.

62 of 102 responded (62%)

Medical providers, case

managers and nonclinical

members of the participating HIE

organizations

NR Comparison by type of

responder
F-134

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Moore, et al.,

2012106

Number of events detected overall and per

patient

NA NA Quantitative

Descriptive statistics

Myers, et al.,

2012128

-10-item perceived ease of use

-10-item perceived usefulness

Role NR Mixed methods

Quantitative: Descriptive

statistics stratified by role and

analysis of variance comparison

by role

Qualitative: Thematic analysis

of the qualitative data interviews

were organized
F-135

Author, Year Results

Risk of

Bias

Moore, et al.,

2012106

-42,818 events detected, on average 238 per day

-≥1 event: 6,913 patients

-1 event: 1,879 patients

-≥10 events: 623 patients

-Mean events of inpatients who had an event: 7.7 events

-Mean events of all patients: 0.7 events

Moderate

Myers, et al.,

2012128

Quantitative: vs. medical providers (57%) and case managers (39%) nonclinical staff members (12%) were significantly less likely to report

that they provided input into the design of the HIE (p <0.008). Mean composite for ease of use was high (3.9/5.0) and no difference by role. 

Mean composite for usefulness was also high (4.0/5.0) and no differences by role.

Qualitative: adoption of the HIEs and perceptions of its use and usefulness varied by occupational role of the patient-care team. Also

noticed that case workers outside the clinic used the HIE routinely. Those within clinics used HIE sporadically.

Low
F-136

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Nagykaldi, et al.,

201448

Retrospective

cohort

Describe a pilot study on a more

sophisticated architecture that

may provide a preliminary

roadmap for building HIE with

intelligence.

Central

Oklahoma

30 primary care

practices, several

specialty

practices, and the

Norman Physician

Hospital 

Organization

including an

academic hospital

and 11 other 

major hospitals.

Log file

Specialty referrals, hospital

admissions, prescriptions,

laboratory imaging results, and

emergency care

March 2010-June

2012

Morris, et al.,

2012163

Multiple Case

Studies

To understand the lessons

learned from HIE organizations

and projects that have succeed

and those that have failed.

U.S. States Multiple Interviews and Surveys Not reported
F-137

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Nagykaldi, et al.,

201448

exHUB

SMRTnet is a statewide network

that includes 120 healthcare

organizations.

Comprehensive patient registry and clinical decision support

tool and reminder system for preventive care and chronic

disease management. Preventive Services Reminder System

NR 346 patients from 6 primary

practices. Average age 66.3

years, 67.1% female, 20%

ethnic minority

Morris, et al.,

2012163

Closed HIOs include CareSpark.

Consolidated HIOs include

Minnesota HIE (MN HIE) and

Galveston County HIE. Additional

HIOs were studied but declined to

be included in the public report.

Successful HIOs include:

Chesapeake Regional Information

System for Our Patients (CRISP),

Delaware Health Information

Network (DHIN), HealthInfoNet,

Indiana Health Information

Exchange (IHIE), Michiana Health

Information Exchange, and

Rochester RHIO.

All query based Varies Query based HIE project in

U.S.
F-138

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Nagykaldi, et al.,

201448

346 patients NR NR Before and after HIE

Morris, et al.,

2012163

9 HIEs provided data that they permitted to be reported

publicly.

HIE organizations that ceased

operations, merged or continued

to operate at the time of the

study

HIE organizations that

refused to have their

information made public

Successful to failed HIE

organizations
F-139

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Nagykaldi, et al.,

201448

-Time-motion studies

-Complete documentation on preventive

screenings and flu vaccinations

-Medication reconciliation

Before and after SMARTnet employed NR Quantitative

Descriptive

Morris, et al.,

2012163

Whether the HIE organization continued to

operate

Ability to make changes to technology

Ambulatory practices participation

Payers participation

Months to deployment

Months to live data

Months to live clinical data

NA Qualitative
F-140

Author, Year Results

Risk of

Bias

Nagykaldi, et al.,

201448

All increased significantly (p<0.001 from pre to post)

Completed mammograms: 22.1% to 57.1%

Recommended colonoscopies: 31.7% to 53.8%

Pneumococcal immunization: 39.1% to 50.6%

Influenza immunization: 22.7% to 41.7%

Medication reconciliation (defined as the ratio of matching practice records and patient reports before and after the HIE implementation):

35.3% (370 of 1047) to 44.9% (468 of 1043)

Barriers included: delays and difficulties in collaborating with commercial technology vendors who gave innovation a low priority

Facilitators included: strategic planning, shared goals, and establishing communication methods

Moderate

Morris, et al.,

2012163

Facilitators:

Key to successful implementation is abilities to move beyond pilot to have volume and breadth of data: id early adopters who find value

and get to a high number of queries, records returned.

Successful HIE projects seem to be those that have some level of control over the technology they use.

Sustainability is related to the ability of HIE organizations to innovate and react quickly to changes in markets. This requires a

combination of leadership and technology.

NA
F-141

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Nøhr, et al.,

2001139

Before-after Compare expectations with

experiences after HIE launched

Denmark Hospitals and 

primary care

Survey, interviews 1999

Nykänen and

Karimaa, 2006150

Cross-sectional Factors of success and failure for

a regional IS network of hospital

and physician offices

Finland Regional

information

system for 

exchange of

clinical data

between hospital

and primary care

offices

Interviews and documents 

Study of HIE documents and

processes; interviews of users in

pilot phase

NR

Onyile, et al.,

2013125

Cross-sectional Determine the geographic 

distribution of patients using the

New York metro RHIO

New York Multiple settings Database and Audit logs

Ambulatory physician groups,

long-term care facilities, a

Medicaid managed care plan, the

nation’s largest home healthcare provider and academic

medical centers that serve as

major referral centers with a total

of 7,503 inpatient beds, 341,065

annual inpatient discharge and

540,854 annual ED visits

Cumulative: 2009-

2011 (patients 

entered by time of

study, 2011)

Overhage, Evans,

and Marchibroda,

2005151

Cross-sectional Community readiness for HIE. U.S. Various Survey

Web based survey for

Connecting Communities for

Better Health

2004
F-142

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Nøhr, et al.,

2001139

Varies as this was a national effort

in Denmark

Four types were described:

Common database

EDI: copies of data are transferred between systems

Middle ware: software between application and database

Internet technology: data communicated via browser

1998 to 1999 Not reported

Nykänen and

Karimaa, 2006150

Regional information system in

Finland

Not well-described NR Pilot users of system

Onyile, et al.,

2013125

New York Clinical Information

Exchange (NYCLIX) - Manhattan

based RHIO

NYCLIX - Manhattan based RHIO, ambulatory groups, long

term care, home health care, academic health centers,

Medicaid managed care plan

March 2009 Patients who visited a NYCLIX

facility

Overhage, Evans,

and Marchibroda,

2005151

Various Various NA Organizations and individuals

who might be interested: 839

(national associations: 110,

government agencies: 57,

individuals: 117, national

organizations: 354, statefocused organizations: 201)
F-143

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Nøhr, et al.,

2001139

Survey respondents: Expected benefits in 1998

(n=102); Experiences in benefits in 1999 (n=57);

Expected barriers in 1998 (n=101); Experiences in

barriers in 99 (n=99)

Seven persons involved in each

HIE project.

NR Expectation vs. Experience.

Also comparison to paper

systems at times.

Nykänen and

Karimaa, 2006150

Unspecified number NA None None

Onyile, et al.,

2013125

3,980,016 patients (after excluding 26,589 with invalid

zip code)

In RHIO master patient index Invalid zip code NA

Overhage, Evans,

and Marchibroda,

2005151

134 NR NR NA
F-144

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Nøhr, et al.,

2001139

Expected benefits and barriers.

Experienced benefits and barriers.

NA NA Mixed Methods

-Quantitative, descriptive data

-Qualitative analysis

Nykänen and

Karimaa, 2006150

Perform work tasks and how the HIE

changes them

Qualitative NA Qualitative

Interviews, observations,

usability, and analysis

Onyile, et al.,

2013125

Visited RHIO facility (in master patient

index)

Calculated distance from Times

Square

NR Quantitative

Mapped the most current zip

code for each unique patient to

the appropriate U.S. county,

calculated the distance from each

zip code to Times Square,

mapped with EpiInfo v3.5.3,

spatial regressions with SatScan

v9.1.1 and RR of visit by spatial

cluster

Overhage, Evans,

and Marchibroda,

2005151

None NA NA Quantitative

Descriptive - provide only

percentages
F-145

Author, Year Results

Risk of

Bias

Nøhr, et al.,

2001139

"What was expected, but not found, was resistance to EPR, as a result of changes in skills and power. The most obvious benefits are

increased data accessibility and improved decision making. The most considerable disadvantage is an enormous growth in discontent with

the systems performance and the fact, that all the projects are delayed. Many different types of integration solutions are chosen, because

of a lack of a common model for integration. Generally the projects find, that EPJ yields increased security, but logistical problems arise in

having the systems running 24 hours 7 days a week"

Moderate

Nykänen and

Karimaa, 2006150

Quality of design process deemed a success factor. General statement that users experienced better planning of patient care and access

to data, but no details given.

Moderate

Onyile, et al.,

2013125

NYCLIX has representation in all 50 U.S. states, 4 U.S. territories and 57 International standards organization countries. 12.1 visits/ 100

within 30 miles; 0.4 visits/ 100 at 100 miles; 87.7% live within 30 miles of Times Square; "inflection point" where visits are less than 1 per

100 is 80 miles from Times Square; for cluster counties, RR for visit is 14.4; 77.7% of entire U.S. counties represented; more patients from

outer boroughs than from Manhattan

Low

Overhage, Evans,

and Marchibroda,

2005151

-22% in beta stage, 28% in pilot, 28% operational, 22% conceptual; of 64 self-reported operational, only 9 could be verified

-5% no organizational structure; 28% "loose affiliation"; 29% had corporate structure; of these 23% hospitals, 16% provider organizations,

10% academic medical centers, 9% dedicated community HIE, 2% public health

-Long lists of organizations to be involved, without actual details of roles; clinicians heavily involved in all, leading the way in 24%;

architectures 2% PHR, 20% peer to peer, 3% federated, 54% centralized database; 18% not decided; most planned centralized; broad

functionality and data inclusion proposed by participants, without specifics about implementation

-Standards proposed: 82% ICD-9, 73% CPT4, 38% LOINC, 41% SNOMED, 48% NDC

-One third had identified funding; planned funding over 60% external, 45% subscribers

NA
F-146

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Overhage,

Grannis, and 

McDonald, 200849

Retrospective

cohort

Compare the completeness and

timeliness of laboratory reporting

for public health in manual and

electronic systems

Marion County,

Indiana

Marion County,

Indiana (public 

health system)

Log file

Indiana Network for Patient Care:

9 of 13 hospitals in county,

physician practices, laboratories,

radiology centers, public health

departments

First quarter of 2001

Ozkaynak and

Brennan, 2013129

Multiple site case

studies

To describe sociotechnical

system in terms of social

structure determination of

technical forms: "how social

systems define technology and its

usefulness."

Madison,

Wisconsin

3 EDs in different

systems in same

metropolitan area

Observations, interviews

210 hours direct observations,

varied across shifts, in 5 rounds,

by 1 or 2 observers (industrial/

systems engineers, nurses,), with

informal conversations to enquire

and followup, plus 13 open

ended HIE interviews

2008-2010
F-147

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Overhage,

Grannis, and 

McDonald, 200849

Indiana Network for Patient Care

(INPC) automated public health

reporting based on LOINC codes

Indiana Network for Patient Care: 24 hospitals, physician

practices, laboratories, radiology centers, public health

departments in Indiana

NR County wide public health

Ozkaynak and

Brennan, 2013129

NR Clinicians choose when to use HIE, which is always available NR ED clinicians
F-148

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Overhage,

Grannis, and 

McDonald, 200849

Marion county population Notifiable condition in eHIE

system or in manual system(s)

No match of identifiers Manual public health reporting

by physician offices,

laboratories (in and out of 

Indiana) to state and local

public health departments,

case finding

Ozkaynak and

Brennan, 2013129

184 patient care episodes NR NR NA
F-149

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Overhage,

Grannis, and 

McDonald, 200849

-Completeness

-Timeliness of public health laboratory 

reporting

Electronic or manual reporting system NR Quantitative

Number identified in eHIE vs.

number identified by manual

reporting, time to reporting

Ozkaynak and

Brennan, 2013129

-Use of HIE

-Views of clinician-users

NA NA Mixed methods

-Quantitative descriptive

-Qualitative analysis

-Inductive iterative analysis,

systems engineers, nurses,

physician
F-150

Author, Year Results

Risk of

Bias

Overhage,

Grannis, and 

McDonald, 200849

Overwhelming positive effect: 4,635 found by eHIE, 944 by manual; for 818 identified by both, eHIE reported 7.9 days earlier on average,

across 53 conditions, eHIE found more for all but 3 conditions; 5/18 data items more often present in manual, 10/18 more often present in

eHIE; but false matches (4 Ebola); nondisease positives (rubella screen); repeat testing known positives; delayed report till confirmed or

typed (Shigella)

Low

Ozkaynak and

Brennan, 2013129

-184 patient care episodes (10 use the HIE system, about 5%)

-2 unexpected uses of the HIE: (1) The HIE was being used mostly for patients only with specific characteristics. (2) The information from

the HIE could be used to confront with the patients.

-System used mainly for patients with chronic pain to check previous visits (and prescribing); workflow issues interfered; extra time and

effort expended when needed,

-When the observers asked the reason of use of the system, the reason mentioned by the majority of the interviewed clinicians was to

detect drug-seeking behavior

Moderate
F-151

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Pagliari, Gilmour,

and Sullivan,

2004122

Multiple case

studies

To explore the processes and

outcomes of implementation,

barriers and facilitators to system

adoption and benefits and 

drawbacks for professional users.

Scotland Primary and 

Secondary Care

Survey responses from users

and project managers,

interviews, and document 

review

November 2001 - May

2003;

(August 2002-May

2003 for minimum

dataset)
F-152

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Pagliari, Gilmour,

and Sullivan,

2004122

Electronic Clinical Communication

Implementation Program (ECCI)

The ECCI is a program initiated as part of the Scottish National

health Service Information Management and Technology

strategy. It targets six electronic deliverables relating to direct

hospital outpatient appointment booking from primary care,

referral from primary to secondary care, results reporting from

secondary care laboratories to primary care, transfer of hospital

discharge and clinic letters to primary care and clinical email.

2000 16 Scottish Health Board

areas included in minimum

dataset;

Survey - in-depth studies of 7

regional sites, chosen to

represent the others in terms

of geographic and 

demographic spread and initial

IM & T maturity.
F-153

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Pagliari, Gilmour,

and Sullivan,

2004122

16 Scottish Health Board areas included in minimum

dataset;

Survey - in-depth studies of 7 regional sites, chosen to

represent the others in terms of geographic and

demographic spread and initial IM & T maturity;

64% survey response rate for primary care; 34% for

specialty care.

Survey sample represents 17% of Scottish practices;

therefore respondents represent 11%.

Minimum dataset: all 16 areas;

surveys - limited to 7 regions

see inclusion criteria None
F-154

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Pagliari, Gilmour,

and Sullivan,

2004122

6 electronic deliverables:

1) direct hospital outpatient appointment

booking from primary care;

2) referral from primary to secondary care;

3) results reporting from secondary care

labs to primary care;

4) transfer of hospital discharge and clinic

letters to primary care;

clinical email (second opinion

correspondence)

NA NA Qualitative

Minimum dataset: descriptive

statistics

Surveys: mailed or email
F-155

Author, Year Results

Risk of

Bias

Pagliari, Gilmour,

and Sullivan,

2004122

From the minimum dataset:

GP practices with access to e-results reporting software: 37%

GP practices using e-RR: 36%;

GP practices with access to e-OP appointment booking system: 3%;

GP practices using e-OP system: 2%;

GP practices with access to e-referral system: 47%;

Referral letters e-transmitted: 18%;

GP practices using clinical email: 9%;

Consultant led departments using clinical email: 5%;

Hospital wards able to send e-discharges: 10%; 

Wards generating and sending e-discharges: 7%;

Specialties able to generate e-clinic letters: 11%;

Specialties generating and sending e-clinic letters: 3%.

Surveys - of responding practices:

93% used e-Lab results;

58% e-referrals;

42% e-discharges;

16% e-OP booking;

Percent reporting daily or weekly use:

90% e-results; 96% e-discharges; 92% e-referrals; 28% e-OP booking.

Clinicians most common users of e-reporting/e-referrals; admin/clerical staff most common users of e-discharge/e-OP booking.

Implementation was facilitated by successful engagement of stakeholders that focused on proactive methods. Other facilitators were ease

of use, good training, communication and commitment from staff. Barriers included differences in IT and system bugs or problems and

slow system development.

Moderate
F-156

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Park, et al.,

201363

Cross-sectional To assess patients’ perception of

an HIE which includes patients’

preferences regarding information

exchange operations,

endorsement of the technology,

and expected and perceived

benefits and concerns about the

technology, and to examine the

influence of demographic

characteristics and HIE

experience on patients’

perceptions.

South Korea Tertiary care and

affiliated clinics

Survey

interview pre-, telephone post2008-2009

Patel, et al.,

201391

Cross-sectional To provide national estimates of

physician capability to

electronically share clinical

information with other providers

and to describe variation in

exchange capability across

states and EHR vendor.

U.S. Out patient Survey

-2011 National Ambulatory

Medical Care Survey

-Electronic medical record

supplement

2011

Phillips, et al.,

2014164

Multiple case

studies

Study 3 RHIOs implementing a

public health use case

New York Any, but this study

focused on public

health reporting 

and querying

Interviews and documents 

Semi-structured interviews and

review of documentation of RHIO

NR
F-157

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Park, et al.,

201363

Korean HIE pilot Federated architecture, stores and transfers HL7 CDAs

CDA exchanges between referring providers and SUNBH

June 2008 All patients visiting tertiary

hospital and affiliated clinics

Patel, et al.,

201391

Several Varies Varies Nonfederal office–based

physicians

who provide direct patient care

Phillips, et al.,

2014164

3 RHIOs in New York state All types Varying Interviews with leaders of the

3 HIEs
F-158

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Park, et al.,

201363

Pre: 322 hospital + 408 clinic; Post: 306 of 536 HIE

participants, 180 offline information exchange, 208

referral letter only

Not explicitly stated (visited

hospital or clinic)

Not explicitly stated 1) paper based, offline (USB

stick) and online (HIE); 2)

participants and non

participants,3) before and after

implementation

Patel, et al.,

201391

4,326 respondents (61% weighted response rate) Out patient MDs Federal physicians NA

Phillips, et al.,

2014164

NA NA None None
F-159

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Park, et al.,

201363

-Need for HIE

-Experience with HIE

-Preferences

-Endorsement

-Perceived benefits and concerns

-Satisfaction

HIE exposure status (pre, post, offline,

letter)

Demographics Quantitative

Descriptive, MANOVA

Patel, et al.,

201391

Reported capacity for exchange of

pharmacy, lab and clinical summary

information

-State

-Physician demographics

-Physician use of EHR

-Practice characteristics

-EHR vendor

NA Quantitative

'-t-tests

-Profit regression models

Phillips, et al.,

2014164

Certification and becoming operational for

public health use case

Qualitative NA Qualitative
F-160

Author, Year Results

Risk of

Bias

Park, et al.,

201363

-Group A (offline 'HIE') older, more likely to have operation, inpatient care; 14% used USB, etc., 10% paper HIE; only 23% concerned MD

do not know about prior care; all preferred consent based HIE, 80% in HIE, 55-59 in non-HIE;

-Post: satisfied, would recommend: 92% of HIE, 88% of non HIE; HIE and offline 'HIE' equally cited convenience, expedited care; all

endorsed HIE, HIE group most strongly; all cited convenience, expedited care, HIE group most strongly; HIE group less concerned about

privacy, complexity, inconvenience

- A higher percentage of HIE patients (80%) compared with A(55%) & B(59%) reported their preferred method of information exchange

was HIE

-In general those who experienced HIE had statistically higher rates of agreement with survey questions regarding need for HIE

Low

Patel, et al.,

201391

Overall: 31% could share clinical summaries, of these 76% could both send and receive, 64% of these exchanges were through an EHR

vendor and 28% through a hospital-based system. 55% could e- prescribe, 67% could view lab results, 42% could incorporate lab results

into EHR.

State differences: the capacity to electronically exchange clinical summaries with patients varied from 55% (Minnesota) to 18%

(Louisiana). The proportion of physicians who exchange clinical summaries with other providers varied from 61% (Wisconsin) to 15%

(Alabama).

-Adoption of EHR is strongest practice characteristic associated with exchange capacity, p<.001

-EHR vendors have a wide range of capacities for exchange: 24% to 77% of MDs report exchange capacity by vendor

-Primary care providers were more likely to exchange vs. specialists, age of MD was NS

Low

Phillips, et al.,

2014164

2 common factors influenced risk management and implementation success: leadership capable of agile decision-making and

commitment to a strong organizational vision

Low
F-161

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Pirnejad, Bal,

and Berg,

2008152

Cross-sectional How are data integration and

data integrity attained in a 

communication network?

Almere, the 

Netherlands

Community -

hospital interface

Interviews, observations,

documents

Interviews (pharmacist focus);

documents, observations of

pharmacist work after

implementation

2005-2006

Poulidi, 1999165 Multiple Case

Study

To review the lessons learned in

the context of HIE related to

collaboration among stakeholders

United Kingdom National Health

Care system wide

In depth interviews used to

create a stakeholder analysis;

comparison to an analysis

complete in the U.S.

Post 1996, but not 

reported
F-162

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Pirnejad, Bal,

and Berg,

2008152

Trans-mural exchange of

medication data in Almere (TUMA)

Medication information exchange community GP/pharmacist

with hospital pharmacy; same vendor, different systems,

shared server

2005 Hospitalized people in Almere,

Netherlands

Poulidi, 1999165 NHSnet Wide area networking was set up to facilitate the exchange of

administrative, purchasing and clinical data.

1993 UK, sub areas not specified
F-163

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Pirnejad, Bal, 

and Berg,

2008152

0 of 115 GPs, 2 of 17 community pharmacists, 4

hospital pharmacists in 1 hospital pharmacy; project

lead and 2 managers

None given None given Pre-post

Poulidi, 1999165 NR NR NR Greater Dayton Area

Community Patient health

Information Network in the U.S.
F-164

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Pirnejad, Bal,

and Berg,

2008152

Second stage: changes in work,

improvement, problems; after network

tested, reasons for problems in test results

First stage: study context, medication

data communication, information gaps

NA Qualitative

-Grounded theory

-Semi quantitative, formative

Poulidi, 1999165 Stakeholder perceptions and attitudes NA NA Qualitative
F-165

Author, Year Results

Risk of

Bias

Pirnejad, Bal,

and Berg,

2008152

-Pitfalls and information gaps in the old medication data communication: missing medication information on admission, delay in

information at discharge, dependence on patients for prescription information

-TUMA effect on bridging the information gaps and improving the communication, focusing on the test results and their analysis.

-Important unforeseen problems: (a) technical challenges in system interface (though same vendor); (b) data integrity problems (59 errors

in 32/100 records before fix, 55 items in 14/100 records after fix); (c) problems with coding system and its application, with software and its

application, (d) and conflicts related to the articulation work and responsibility distribution between the involved parties - e.g. coding

differences by GPs and pharmacists

-Aim was to replace patient as weakest link - learned that instead "contribution of patients in saving the integrity of data and in integrating

medication data is valuable"

Moderate

Poulidi, 1999165 Confidentiality was a major concern for physicians and a barrier that slowed implementation.

The NHS case is more complex than the regional US case in that more types of stakeholders are involved, more settings are involved in

the NHS implementation and the scope of the data exchanged is greater.

NA
F-166

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Ross, et al.,

2010167

Multiple case

studies

Elucidate perspectives of clinical

and administrative leaders in 

smaller ambulatory practices

regarding desired HIE functions,

key motivators, barriers to and

potential incentives for adoption.

Colorado SNOCAP-USA

Practice-based

Research

Networks;

small to mediumsized practices

(<20 providers) in

primary care

practices

Interviews

-Topic guide created based on

literature

-Telephone and on-site guided

discussions

November 2008-April

2009

Ross, et al.,

201350

Retrospective

cohort

Does HIE affect laboratory and

radiology test ordering

Mesa County,

Colorado

Physician offices -

outpatient

Log file

Claims data

April 2005-December

2010

Rudin, et al.,

2009153

Cross-sectional What are providers' decisionmaking processes in

implementing HIE?

Massachusetts Physician offices Interviews

Semi-structured interviews

Summer-Fall 2007
F-167

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Ross, et al.,

2010167

1) Community-wide HIE - currently

exchanged information, but could

use paper or electronic medical

records;

2) Paper charts only - No use of

community-wide HIE;

3) EMR only - No use of communitywide HIE.

2 types of community-HIE: 1) traditional RHIO that provides

limited EMR functionality that includes storage and retrieval of

tests, dictations, meds, allergies, e-prescribing (2 urban (1

indigent clinic; 1 private clinic), 1 rural site (private clinic); 22

providers total).

2) nontraditional HIE-one EMR across multiples sites in an

independent practice association (still met investigators

definition of HIE); (1 suburban site; private; 16 providers).

Patterns included: 1) bulk of info exchanged was related to

ordering tests and studies and receiving results from hospitals

and independent labs; 2) vital to exchange info with hospitals

and specialty practices (consultation reports and discharge

summaries).

NR Family practice sites

participating in SNOCAP-USA

practice based research

network

Ross, et al.,

201350

Quality Health Network Query-based and directed 2005 Claims for 34,818 patients 

served by 306 providers in 69

practices who had access to

the HIE

Rudin, et al.,

2009153

Massachusetts eHealth

Collaborative (MAeHC)

Hybrid HIE NR Members of MAeHC

collaborative and physician

users
F-168

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Ross, et al.,

2010167

Purposeful sampling Family practice sites

participating in SNOCAP-USA

practice based research network

None listed Paper chart only practices and

EMR only practices vs.

community HIE practices

Ross, et al.,

201350

Claims for 34,818 patients All having access to HIE None Rates of laboratory and

radiology testing for primary

care and specialist care

physicians

Rudin, et al.,

2009153

14 key informants All interviewed NA Technical HIE architecture

chosen
F-169

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Ross, et al.,

2010167

-Desired HIE functions

-Key motivators

-Barriers to and potential incentives for

adoption

Practice group None listed Qualitative

Qualitative analysis was iterative,

allowing for investigator

corroboration, triangulation, and

checking; then coding and

theming, creation of briefing

sheet, then use of modified

Delphi method to finalize

analysis. Sites also reviewed and

corrected reports prior to final

report creation.

Ross, et al.,

201350

-Rates of laboratory and radiology testing

-Economic

Rates of laboratory and radiology

testing

None Quantitative

Mixed effects regression model

Rudin, et al.,

2009153

Technical HIE architecture chosen NA None Qualitative
F-170

Author, Year Results

Risk of

Bias

Ross, et al.,

2010167

Desired functions of HIE: Universally valued was improved ability to receive and review clinical info from outside the practice; this much

more so than improved ability to send or make available info from inside the practice. Paper- and EMR-only anticipated little value in

sharing their data with others, but HIE practices realized the value of having their data available anytime/from anywhere. There was

consensus that community hospitals and independent lab info would be essential. Also highly desirable to include exchange with

specialists. Test results considered most important; followed by discharge summaries.

Mean ranking of potential HIE functions (1=highest; 5=lowest rank): looking up info 1.9; delivering results 2.2; e-prescribing 2.5 (lack of

computers in exam rooms was a barrier for this one); placing nonprescription orders 3.8; creating reports 4.7; secure email was a lower

priority.

Essential attributes of HIE: solid reliability and responsive service; live and direct technical support; comprehensive policies and systems

for privacy, security and data use

Motivations for adopting HIE: motivated to gain uniformity in workflow; improved efficiency (even though did not anticipate monetary

benefit; improved quality of care through better coordination and information;

Barriers and facilitators:

1) Barrier: technical-need to interface with existing systems

2) Barrier: workflow issues-most sites did not want to re-engineer workflow

3) Best facilitator: technical assistance for implementation & maintenance; and training

4) Barrier: financial issues; secondary, but important; capital costs were barrier; not concerned with loss of revenue

5) Facilitators: solidarity & trust were important (easier in smaller cities); wanted involvement by practice leaders, NOT health plans;

neutral about government, foundations

6) Practices thought they could education patients to have trust

Moderate

Ross, et al.,

201350

For PCPs, rate of laboratory testing increased over the time span (baseline 1041 tests/1000 patients/quarter, increasing by 13.9 each

quarter) and shifted downward with HIE adoption (downward shift of 83, p<0.01). For specialist providers (baseline 718 tests/1000

patients/quarter, increasing by 19.1 each quarter, with HIE adoption associated with a downward shift of 119, p<0.01). Imputed charges for

laboratory tests did not shift downward significantly in either provider group. For radiology testing, HIE adoption was not associated with

significant changes in rates or imputed charges in either provider group.

Low

Rudin, et al.,

2009153

To become established, HIE efforts must foster trust, appeal to strategic interests of the medical community as a whole, and meet

stakeholder expectations of benefits from quality measurements and population health interventions.

Moderate
F-171

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Rudin, et al.,

2011136

Cross-sectional What affects clinician use of HIE Massachusetts Hospitals and 

physician offices

Interviews of clinician users

and HIE staff

October 2009-

February 2010

Saff, et al.,

2010154

Cross-sectional Description of motivation,

implementation and use of San

Francisco Bay Area HIE

San Francisco

Bay Area

5 health 

organizations;

2,800 MDs;

900,000 patients;

numerous labs;

several IT vendors

Database

Varying types of clinical and

administrative data - varies by

site

Each medical center

joined the HIE at a

different time, dating

from 2002

Schabetsberger,

et al., 2006172

Prospective

cohort

Describe evolution and use of

system, problems.

Tyrol, Austria Tiroler

Landeskrankenan

stalten, 6 hospital,

6,000 staff, 1,000

physician, 300,000

outpatient, 70,000

inpatient, 400 

medical student

health system

Audit logs June 2003 and

October 2004
F-172

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Rudin, et al.,

2011136

Massachusetts eHealth

Collaborative (MAeHC)

All nontext portions of medical record. Could link directly from

the EHR to existing HIE. Query-based exchange. Consent was

'opt-in'.

Mid-2007 Clinician users and staff who

implemented HIE

Saff, et al.,

2010154

NR Each medical center valued the HIE for different reasons;

descriptions are provided

NR 900,000 patients in the San

Francisco and the East Bay

Schabetsberger,

et al., 2006172

Various (1) Discharge summaries push to GP EHRs as text documents,

92+% electronically

(2) Standalone web-based archive of hospital documents for

nonaffiliated physician access

May 2002-October

2004

Tyrol, Austria physicians
F-173

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Rudin, et al.,

2011136

15 clinicians and 2 HIE staff and 3 administrators NA None None

Saff, et al.,

2010154

900,000 patients in San Francisco and the East Bay None specifically stated; all

patients included

None specifically stated;

all patients included

None

Schabetsberger,

et al., 2006172

NR NR NR NA
F-174

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Rudin, et al.,

2011136

Motivators and moderators of use Qualitative NA Qualitative

Content analysis

Saff, et al.,

2010154

Lessons learned Characteristics of each health system;

this is a descriptive case study

NA Quantitative

Descriptive

Schabetsberger,

et al., 2006172

System use NA NA Quantitative

Descriptive
F-175

Author, Year Results

Risk of

Bias

Rudin, et al.,

2011136

-Motivators were belief in improved quality of care, time savings, and reduced need to answer questions. Cost of care was not listed as a

motivator.

-Motivation was moderated by missing data, workflow issues, and usability issues (too many clicks required to get to information).

-Missing data was attributed contributing providers not "locking their notes" on their EHR.

-Patient-related moderators were those who had trouble communicating, multiple comorbid illnesses, and who received care at multiple

sites within but not outside HIE.

-Clinician-related moderators varied by specialty, use of paper and fax, and integration into workflow.

-HIE-related moderators were gaps in data from local nonparticipants, poor usability, and downtimes.

-Clinicians varied in how quickly they "locked" data for transfer into HIE.

Low

Saff, et al.,

2010154

Lessons learned

-Moved from a competitive to collaborative model

-EMR/PHR integration

-Extensive testing required to ensure quality of data fit for use

-Physician education and engagement required/important

High

Schabetsberger,

et al., 2006172

-6% to 8% of approximately 40,200 discharge letters were sent out electronically

-Problems: corrupt data in physician database; differing implementations of standards (EDIFACT standard); independent, nonfederated

patient index; 4 GPs and the psych ward had security concerns

NA
F-176

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Schoen, et al.,

201295

Cross-sectional To explore the experiences of

physicians in primary care with

health reform policies.

Australia, 

Canada, France,

Germany the 

Netherlands, New

Zealand, Norway,

Switzerland, The

United Kingdom

and the U.S.

Primary Care

Practices

Survey responses March - July 2012

Shapiro, et al.,

201351

Retrospective

cohort

Measure incremental increase in

number of frequent ED users

identified when data from all EDs

(using HIE) were compared with

use of site-specific data only

New York City 10 hospitals that

participated in

NYCLIX

Log file

NYCLIX data (which also

included data from site-specific

EMRs)

June 1, 2010-May 31,

2011
F-177

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Schoen, et al.,

201295

NR Electronic exchange of patient summaries and test results with

doctors outside their practice.

NR General practice and family

practice physicians in all

countries, as well as general

internists and pediatricians in

Germany and the U.S.

Shapiro, et al.,

201351

10 hospitals that participated in

New York Clinical Information

Exchange (NYCLIX); NYCLIX is a

RHIO in NY City; data sent to

NYCLIX by each participant

organizations; master patient index

links each patient across sites;

NYCLIX staff was 'honest broker'

and provided data.

New York Clinical Information Exchange (NYCLIX) NR All patients with ≥1 instance of

≥4 ED visits within 30 days

during study period
F-178

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Schoen, et al.,

201295

Primary Care Physicians Surveyed

Australia: 500

Canada: 2,124

France: 501

Germany: 909

The Netherlands: 522

New Zealand: 500

Norway: 869

Switzerland: 1,025

United Kingdom: 500

U.S.: 1,012

Overall: 8,462

Practicing physicians were

randomly selected from public

and private lists typically used in

each country

NR NR

Shapiro, et al.,

201351

924,675 ED visits by 591,632;

920,507 ED visits by 591,632 patients

All patients with ≥1 instance of

≥4 ED visits within 30 days

during study period

4,168 visits because they

occurred within 6 hours of

a previous ED visit, which

investigators decided a

priori might represent

clerical errors

EMR use without accessing

HIE
F-179

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Schoen, et al.,

201295

Ability to electronically exchange patient

summaries and test results with doctors

outside their practice

NA NA Quantitative

Survey, Chi2 tests

Shapiro, et al.,

201351

-Number ED visits

-Number of patients experiencing these

visits

-Average number ED visits per patient

during 12 months

-Number patients frequent ED users (per

definition)

-Number of ED visits accounted for by

frequent users

-Average number visits per frequent user

-Increase in number of frequent users

when estimated across HIE (vs. within

each site)

-Gender

-Age

Cross-over visits (different EDs) Quantitative

-Chi2

-Wilcoxon sign rank test
F-180

Author, Year Results

Risk of

Bias

Schoen, et al.,

201295

% of primary care physicians reporting HIE capabilities:

Australia: 27

Canada: 14

France: 39

Germany: 22

The Netherlands: 49

New Zealand: 55

Norway: 45

Switzerland: 49

United Kingdom: 38

U.S.: 31

In the U.S. capacity for electronic exchange of patient information was concentrated in larger practices and those in integrated health

systems (50% of physicians reported HIE vs. 23% of physicians not part of integrated practices p<0.05)

High

Shapiro, et al.,

201351

Total visits: 924,675 (591,632 unique patients)

After exclusion: 920,507 visits by 591,632 patients

Mean ED visits/year: 1.6

When used only site-specific data only: 4,786 patients met criteria of frequent user (represented 0.8% of all users)

Number of ED visits: 45,771

Mean visits/years: 9.6 (accounted for 5% of ED visits)

HIE-wide results

5,756 frequent ED users

20% increase in number of frequent user events identified

53,031 visits (6% of all ED visits)

Thus HIE data produced 16% increase in number ED visits that could be identified

Frequent users more likely to be male: 51% vs. 45%, p<0.0001

Mean age higher: 40.7 vs. 37.9 years, p<0.0001

More had cross-over visits: 28.8% vs. 3%, p<0.0001

Moderate
F-181

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Sicotte and Paré,

2010168

Multiple case

studies

Describe the implementation and

deployment of 2 large HIE

projects.

Quebec, Canada Case 1: 3 pediatric 

hospitals.

Case 2: Primary 

care network

linking a public

hospital to 10

private clinics.

Interviews, observations,

documents

52 interviews (27 for Case 1, 25

for Case 2); all documents from

the HIE project team, HIE

organizations and vendors; and

observations at HIE project

meetings

January 2001 + 42

months (Case 1); May

2001 + 32 months

(Case 2)

Silvester and Carr,

2009114

Before and after Description of implementation -

use of system.

Brisbane & 

Northern

Territories of

Australia

239 GPs from 66

practices,

2 major public

hospitals, 3 large

private hospitals,

11 allied health/ 

community based

partners

Database

Registration, communication, and

clinical database.

Clinical database contains

socioeconomic status,

medications, diagnosis, allergies,

medical history, diagnostic

results, care team members,

unstructured documents

April 30, 2007-July

2008

Soderberg,

Laventure, and

Minnesota, 201390

Time Series To monitor progress toward

meeting the legislative

requirement that all health care

providers have an interoperable

EHR by 1/2015.

Minnesota Clinics Survey

72 survey questions

February 15-March

15, 2013
F-182

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Sicotte and Paré,

2010168

Case 1: 3 pediatric hospitals.

Case 2: Primary care network

linking a public hospital to 10

private clinics.

Case 1: large pediatric hospital, 2 community pediatric hospital,

4 pediatric clinics.

Case 2: public hospital, over 100 physicians at 10 private

clinics.

Access to laboratory and imaging results.

Specific date unclear Key informants description

limited to HIE project staff and

HIE users

Silvester and Carr,

2009114

Name NR

239 GPs from 66 practices,

2 major public hospitals,

3 large private hospitals,

11 allied health/community based

partners

Software developed by HealthConnect; web services, HL-7

messaging, extracts data from clinician's software package,

interfaces seamlessly with clinician's software, uses Medicare

Australia's public key infrastructure security certificates for

authentication; patients 'opt-in'.

Prior to April 30, 

2008; implemented

iteratively to ensure

success

Registered patients with

chronic conditions, cared for at

these sites

Soderberg,

Laventure, and

Minnesota, 201390

Varies Varies Varies 1,623 ambulatory

clinics
F-183

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Sicotte and Paré,

2010168

52 interviews (27 for Case 1, 25 for Case 2) NR NR NA

Silvester and Carr,

2009114

1,108 patients in population None, other than stated in

population and sample

None, other than stated in

population and sample

Before implementation

Soderberg,

Laventure, and

Minnesota, 201390

The response rate was 79%, with 1,286 clinics

responding

Any location where primary or 

specialty care ambulatory

services are provided for a fee

by ≥1 physician

NR None
F-184

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Sicotte and Paré,

2010168

Descriptive narrative only NA NA Qualitative

Empirical observations were

organized into narrative using a

risk analysis framework

Silvester and Carr,

2009114

-Frequency of use (number of events

uploaded per patient)

-User access logs and patient registration

growth rates and connection metrics

-User surveys

-Patient case studies

None None Mixed methods

-Descriptive summaries

-Qualitative analysis

Soderberg,

Laventure, and

Minnesota, 201390

Exchanges with affiliated and unaffiliated

hospitals

NA NA Quantitative

Descriptive statistics
F-185

Author, Year Results

Risk of

Bias

Sicotte and Paré,

2010168

Case 1: 4 stages described: project planning with small part-time team; technical system with risks evolving; testing requiring de-scoping;

piloting with user and technical challenges. Overall deliverable not reached, users discouraged and usage was low.

Case 2: 4 stages described: project planning with full-time staff, system integrator consultant and clinical champions; solicitation of user

views and realistic understanding of context, participant contracts signed; system customization and testing, leveraging super-users;

piloting, troubleshooting system performance issues. Overall view was successful with high usage.

Low

Silvester and Carr,

2009114

-Mean events uploaded for each patient record during 12 months: 9.7

-Increased HIE use by nurses

-Number of patients registered increased: 474 (July 2007) to 1,320 (June 2008)

-Increased commitment to use

-Case studies demonstrated use prevented unplanned inpatient admissions

-Interest to adopt by others

Improved staff perceptions in answers to 3 pre-post questions on 5-point Likert scale

Improved understanding of system: 2 to 3

Improved sharing of information: 2 to 2.3

Impact on care delivery: 3 to 3.6

-2 patient-specific case studies showed improved use, communication, satisfaction

-Lessons learned included connectivity, interoperability, change management, clinical leadership, targeted patient involvement,

information at point-of-care, and governance

High

Soderberg,

Laventure, and

Minnesota, 201390

-54% exchange data with affiliated hospitals

-36% with unaffiliated hospitals

-Common challenges for HIE: limited capacity of others to exchange, lack of technical support or expertise, competing priorities, cost and

privacy concerns

Moderate
F-186

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Steward, et al.,

2012169

Multiple case

studies

Understand the dynamic

capabilities that enabled the 6

demonstration projects of the

Information Technology Networks

of Care Initiative to implement

HIE.

New York, New

Jersey, 

California,

Louisiana, New

York

Hospital specialty

clinics, support

services, primary

care clinics,

testing sites, ED,

outpatient and 

inpatient clinics,

Office of Public

Health, insurers,

laboratory and 

pharmacy services

Interviews

Laboratory, diagnostic, medical,

and service utilization; referrals;

and ancillary care support, such

as case management, counseling

and testing, transportation, and

substance use and mental health

services.

NR explicitly but at 2

points in time: as the

HIE were being

developed and 1-2

years after the HIE

became operational.

Swain, et al.,

201526

ONC Data Brief Summarize trends in HIE use in

non-federal acute care hospitals

from 2008-2014

NA NA Data are from the American

Hospital Association (AHA)

Information Technology (IT)

Supplement to the AHA Annual

Survey. Since 2008, ONC has

partnered with the AHA to

measure the adoption and use of

health IT in U.S. hospitals.

2014 update
F-187

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Steward, et al.,

2012169

6 HIEs that were part of the

Information Technology Networks

of Care Initiative that included

Bronx-Lebanon Hospital Center,

Duke university; hospitals, the city

of Paterson, Louisiana State

University Health Care Services

Division, NY Presbyterian Hospital,

St. Mary Medical Center

Foundation

Each of 6 projects implemented a different HIE. NR 111 project staff and IT

specialists; staff from

community-based

organizations and public

health organizations; users of

HIE.

Swain, et al.,

201526

Varies, as these data are from the

AHA survey

Varies, as these data are from the AHA survey NA The survey was administered

to 4,451 non-federal acute

care hospitals, with a

response rate of 60%.
F-188

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Steward, et al.,

2012169

NR NR NR Cross-site evaluation

Swain, et al.,

201526

The survey was administered to 4,451 non-federal

acute care hospitals, with a response rate of 60%.

The survey was administered to

4,451 non-federal acute care

hospitals, with a response rate of

60%.

Federal and non-acute

care hospitals

prior years
F-189

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Steward, et al.,

2012169

Implementation outcomes NA NA Qualitative

-Developed 16 coding topics

-Convergent and divergent

perspectives examined within and

across sites

Swain, et al.,

201526

HIE use between hospitals and hospitals;

HIE use between hospitals and outside

providers;

Types of data exchanged (Labs, radiology,

meds, clinical care summaries)

NA A logistic regression model was

used to predict the propensity of

survey response as a function of

hospital characteristics, including

size, ownership, teaching status,

system membership, availability of

a cardiac intensive care unit, urban

status, and region. Hospital-level

weights were derived by the

inverse of the predicted propensity.

Estimates considered unreliable

had a relative standard error

adjusted for finite populations

greater than 0.49. Responses

with missing values were

assigned zero values. Significant

differences were tested using p <

0.05 as the threshold.
F-190

Author, Year Results

Risk of

Bias

Steward, et al.,

2012169

Found evidence for importance of 3 dynamic capabilities: information systems, reconfiguration capacity, and organization size and human

resources. Reconfiguration capacity was most important.

Moderate

Swain, et al.,

201526

Hospitals’ electronic health information exchange with hospitals or ambulatory care providers outside their organization increased by 85%

from 2008 to 2014, and increased by 23% since last year (2013).

In 2014, 47 states and the District of Columbia had at least 60% or more of their hospitals electronically exchange key clinical data with

outside providers. In contrast, in 2010, 10 states had 60% or more of their hospitals electronically exchange key clinical data with outside

providers.

In 2014, state rates of hospitals’ electronic exchange of key clinical data with outside providers ranged from 42% to 100%; whereas in

2010, hospitals’ health information exchange with outside providers ranged from 24% to 67%

Approximately two-thirds of hospitals electronically exchanged laboratory results (69%), radiology reports (65%) and clinical care

summaries (64%) with outside providers in 2014.

Close to six in ten (58%) hospitals exchanged medication history with outside providers. This is an increase of 176% since 2008 and an

increase of 57% since 2013.

Summary:

More than three-quarters (76%) of non-federal acute care hospitals electronically exchanged laboratory results, radiology reports, clinical

care summaries, and/or medication lists with any outside providers. This represents an 85% increase since 2008 and a 23% increase

since last year. Close to seven in ten hospitals (69%) electronically exchanged health information with ambulatory providers outside of

their organization, representing a 92% increase since 2008 and a 21% increase since 2013.

NA
F-191

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Thorn, Carter, and

Bailey, 2014130

Multiple site case

studies

To explore views of emergency

physicians having access to HIE,

about their access of and use of

HIE data

NR ED in 4 hospitals,

private and public

settings

Interviews

Individual unstructured

interviews, audio recorded and

transcribed

NR

Tripathi, et al.,

2009123

Multiple case

studies

Description of initiative,

collaborative design and lessons

learned;

also includes opt in data by

consumer

Massachusetts 3 communities

chosen to pilot

HIE, Brockton

(diverse

community),

Newburyport

(affluent), North

Adams (rural)

Focus groups, documents

Community steering committees,

MAeHC, stakeholders; consumer

focus groups

Began in 2005

Duration not clear

Tzeel, Lawnicki,

and Pemble, 

201152

Retrospective

cohort

Assess the association of HIE

use on health care costs

S.E. Wisconsin

(Milwaukee

County)

EDs in 5 health

systems in a

county

Log file

WHIE data - health plan member

with ED encounter when HIE

access occurred.

Humana claims data - costs and

utilization of ED encounter.

December 2008-

March 2010
F-192

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Thorn, Carter, and

Bailey, 2014130

HIE name NR but may be MSeHA

Regional HIE operational for 4

years, linking over 450 providers in

15 clinics and 9 major hospitals

serving a population of 1 million

Data in HIE NR

Decentralized, query-based exchange. Consent was 'opt-out'

NR ED physicians in 3 urban

settings

Tripathi, et al.,

2009123

Massachusetts eHealth

Collaborative (MAeHC)

NR NR Number of participants in

committees and stakeholders

involved not stated

Tzeel, Lawnicki,

and Pemble, 

201152

Wisconsin Health Information

Exchange (WHIE)

Links 5 health systems in the county. Access to patient

demographics, chief complaint, allergy, primary care provider,

diagnosis, meds, procedures, encounter date & location.

December 2008 Commercial, fully insured

members of Humana health

plan (denominator); members

in the WHIE database having

≥2 ED visits
F-193

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Thorn, Carter, and

Bailey, 2014130

N=15 physicians from 4 urban hospital systems having

<10% usage of HIE. Cross section of public and private

hospitals. 1 Level I Trauma center. 2 of 4 settings had

not implemented EHRs

Full or part-time physicians

working regularly scheduled ED

shifts. Purposeful selection of 2

because of a 4-year history of

HIE use. Rest recruited with

"theoretical sampling"

NR NA

Tripathi, et al.,

2009123

NA NA NA NA

Tzeel, Lawnicki,

and Pemble, 

201152

Test group: 428 members with ED visits having an HIE

query

Control group: 1,054 members with ED visits with no

HIE query.

Propensity score matching for test group (N=326) with

HIE database query in all ED visits vs. control group

(N=325) with HIE database not queried in any ED visit.

≥1 year continuous insurance

coverage with health plan

<6 months coverage before

program started or <3

months after start of

program

Pairs matched for age, gender,

and costs for net care per

participant per month

prescriptions, inpatient,

outpatient, ED, and physician.
F-194

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Thorn, Carter, and

Bailey, 2014130

Descriptive narrative only NA NA Qualitative

Thematic, constant comparative

analysis of narrative

Tripathi, et al.,

2009123

-Descriptive narrative only

-Type of patient consent

-Type of data to share

NA NA Qualitative

Tzeel, Lawnicki,

and Pemble, 

201152

-Comparison of net costs and ED costs per

participant

-Comparison of top 5 ED procedures in test

group vs. matched control 1 year before

and 1 year after the first ED visit

Pairs matched for age, gender, and

costs for net care per participant per

month prescriptions, inpatient,

outpatient, ED, and physician

NR Quantitative

Matched pairs t-tests
F-195

Author, Year Results

Risk of

Bias

Thorn, Carter, and

Bailey, 2014130

Themes

-Users varied in their HIE use. Stated influencers including trouble accessing system, acuity of patient or history not available, team

members' ability to access.

-HIE use affected decisions sometimes, for specific cases (e.g. drug seekers); often HIE use did not affect decisions

-Use was negatively affected by access challenges, separate login, variability in data being pertinent, absence of data types or data on

specific patients, user design flaws, and lack of technical support.

-Benefits with usage included reducing redundant testing, more accurate history, reducing faxing, knowledge of primary care provider

name

-Barriers to usage included continued practice of defensive medicine, desire for autonomy, changing the culture, belief HIE does not alter

decisions, health system competition, and reduced revenue, workflow disruption.

Low

Tripathi, et al.,

2009123

Discussion of experience/lessons learned

-Decision on consent: opt in chosen due to state law stricter than federal HIPAA law; use of a centralized data repository; and consumer

feedback.

-Data shared: 3 communities agreed on what to share - all EHR except text notes, consult letters and scanned reports.

-Consumer focus groups identified themes to drive HIE/opt in: promote convenience and costs, promote with providers, say benefits up

front, confront risks, use professional marketing

-Consumer opt In across 2 smaller communities: 88% and 92%

NA

Tzeel, Lawnicki,

and Pemble, 

201152

Unadjusted: ED costs in test group changed $1,068 to $999 from 1st to subsequent visit vs. control group changed $1,043 to $1,157

Adjusted for propensity matching: Net costs (per participant per month) in test patients with higher net costs overall in and

subcategories

ED costs: $29 less in test patients from first visit vs. subsequent visits.

Top ED procedures: 4 of 5 were reduced in test group (lab, radiology, CT, EKG)

Low
F-196

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Tzeel, Lawnicki,

and Pemble, 

201253

Retrospective

cohort

Assess the association of HIE

use on hospital admissions

S.E. Wisconsin

(Milwaukee

County)

EDs in 5 health

systems in a

county

Log file

WHIE data - health plan member

with ED encounter when HIE

access occurred.

Humana claims data - costs and

utilization of ED encounter.

December 2008-

March 2010

Unertl, et al.,

2013170

Multiple case

studies

To investigate how technology

and health system coevolve to

reduce information fragmentation

and improve care coordination

(Extension of Unertl 2012 study)

Memphis,

Tennessee 

region

6 EDs and 8

ambulatory clinics

Interviews, observations

Direct observation at 14 sites,

informal interviews at sites, 9

semi structured telephone

interviews

January-August 2009

Unertl, Johnson,

and Lorenzi,

2012119

Multiple site case

studies

To understand the interaction

between HIE and workflow.

How have sites integrated HIE

into existing approaches?

Are there common HIE workflow

patterns across sites?

How do providers incorporate HIE

into clinical practice?

Memphis,

Tennessee 

region

6 EDs and 8

ambulatory clinics

Observations, interviews

Direct observation (180 hours) at

14 sites, informal interviews at

sites, 9 semi structured

telephone interviews with

physicians, nurses and IT

management

January-August 2009
F-197

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Tzeel, Lawnicki,

and Pemble, 

201253

Wisconsin Health Information

Exchange (WHIE)

Links 5 health systems in the county. Access to patient

demographics, chief complaint, allergy, primary care provider,

diagnosis, meds, procedures, encounter date & location.

December 2008 Commercial, fully insured

members of Humana health

plan (denominator);

Members in the WHIE

database having at least 2

Emergency Dept. (numerator)

was the study population.

Unertl, et al.,

2013170

MidSouth eHealth Alliance

(MSeHA), regional HIE around

Memphis includes majority of large

hospitals and 2 safety net clinic

systems.

HIE structure from Vanderbilt University. Data on >1 million

patients includes test results, imaging, discharge summaries,

diagnosis codes and claims data. Opt out model.

2004 NR

Unertl, Johnson,

and Lorenzi,

2012119

MidSouth eHealth Alliance

(MSeHA), regional HIE around

Memphis includes majority of large

hospitals and 2 safety net clinic

systems.

HIE structure from Vanderbilt University. Consolidated data

from multiple hospital emergency departments and communitybased ambulatory clinics. Decentralized, query-based

exchange. Data on >1 million patients includes test results,

imaging, discharge summaries, diagnosis codes and claims

data. Opt out model.

2004 NR
F-198

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Tzeel, Lawnicki,

and Pemble, 

201253

Test group: 428 members with ED visits having an HIE

query

Control group: 1,054 members with ED visits with no

HIE query

Matched pairs: 325

≥1 year continuous insurance

coverage with health plan

<6 months coverage before

program started or <3

months after start of

program

Pairs matched for age, gender,

and costs for net care per

patient per month,

prescriptions, inpatient,

outpatient, ED, and physician.

Unertl, et al.,

2013170

NA NR NR NA

Unertl, Johnson,

and Lorenzi,

2012119

NA NR NR NA
F-199

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Tzeel, Lawnicki,

and Pemble, 

201253

-Admissions per 1,000 members, at time of

ED visit (1st, 2nd visit)

-Conditional probability of admission at ED

visit (1st, 2nd)

-Bed days per 1,000 members

-Average length of stay

Pairs matched for age, gender, and

costs for net care per patient per

month, prescriptions, inpatient,

outpatient, ED, and physician

NR Quantitative

Chi2

Unertl, et al.,

2013170

Descriptive narrative only NA NA Qualitative

Open-ended grounded theory

analysis, followed by the

application of the Information

Ecology Framework to structure

additional analysis

Unertl, Johnson,

and Lorenzi,

2012119

Descriptive narrative only NA NA Qualitative

Grounded method using open

coding, and framework-focused

axial coding.
F-200

Author, Year Results

Risk of

Bias

Tzeel, Lawnicki,

and Pemble, 

201253

Adjusted for propensity matching

Admission/1,000 members (1st to 2nd ED visit): 269 to 664 for test group vs. 321 to 555 for control group

Probability of admission higher at 1st ED visit in control group, and higher at 2nd ED visit in test group

Test group had 771 fewer bed days/1,000 members and lower length of stay than control group

Post–propensity matching analysis showed that test group had 199 more admissions per 1000 members than control group, these

admissions might have been more appropriate. Test group admissions resulted in less time spent as inpatients and by average length of

stay (4.27 days per admission for all admissions and 0.95 days per admission when catastrophic cases removed).

Low

Unertl, et al.,

2013170

-All sites had coexisting use of HIE and manual processes to access information

-Observations were used to map 5 Info Ecology Framework components to a newly developed "Regional Health Information Ecology": 1.

system - HIE to reduce information silos; 2. locality - sites had distinct local context; 3. diversity - staff had varied roles with varied HIE

processes; 4. keystone species - info consumers, who used data for varied reasons; info reservoirs, people who played formal and

informal roles; exchange facilitators, who assisted others and bridged gap between consumers and reservoirs.

-Paradox observed: providers describe HIE useful, regardless of use frequency ('when we use it, it's great"); but, provider belief that HIE

not being used to full potential.

-Examples of impact were identified using their model: a. reduce fragmentation of information; b. reduce time to obtain information; c.

increase provider awareness of patient-health system interactions (e.g., drug seeking)

Low

Unertl, Johnson,

and Lorenzi,

2012119

Cross organizational patterns; 2 models identified

1. Nurse workflow: prompted by patient reporting recent hospitalization event during intake, HIE access by nurse or assistant, printed

discharge summary, added to chart

2. Physician workflow: HIE accessed by provider (doctor or nurse practitioner) for greater reasons beyond hospitalization; HIE access

occurred at various points of care; HIE review of more information including history

-Other observations: clerks tracked biopsy results; workflow patterns evolved over time, due to factors such as access policies or staffing

changes; residents logged into other EMR due to lack of HIE access

-Reasons to access HIE: visit to another hospital; issues of patient trust; communication challenges; referrals

Moderate
F-201

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Vest and 

Jasperson,

2012103

Case control How does HIE access vary by job

type and organization in an

indigent care HIE in central

Texas?

Austin, Texas Indigent patients 

and facilities that

care for them

Log files from clinical data

repository (Indigent Care

Collaboration of Austin, Texas

safety network providers founded

1997); 18 hospitals, public and

private clinics, government

agencies (federally qualified

health centers)

January 2006-June

2009

Vest, 200954 Retrospective

cohort

Test the hypotheses that HIE

information access reduced ED

visits and inpatient

hospitalizations for ambulatory

care sensitive conditions among

medically indigent adults.

Central Texas 18 members in

HIE (I-Care):

hospital systems,

public and private

clinics, and

governmental

agencies

operating federally 

qualified health 

centers

Log file

Demographic, clinical

information, diagnoses,

medication orders, prior visits,

payer sources for uninsured

patients.

January 1, 2005-

June 30, 2007
F-202

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Vest and 

Jasperson,

2012103

Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin,

Texas safety network providers founded 1997); 18 hospitals,

clinics, government agencies (federally qualified health

centers)

HIE 1997; I-Care

database 2002, 3.1

million encounters,

600,000 individuals

Indigent people, not Medicare

Vest, 200954 18 members in HIE: hospital

systems, public and private clinics,

and governmental agencies

operating federally qualified health 

centers

Each site contributes patient electronic data to I-Care through

secure electronic interfaces. In turn, each location may access

data from I-Care at a secured website.

HIE 1997; I-Care

database 2002, 3.1

million encounters,

600 thousand

individuals

Uninsured 18 to 64 years old

and excluded encounters at

the public mental health

provider and Planned

Parenthood
F-203

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Vest and 

Jasperson,

2012103

105,705 unique user sessions User session as all system

viewing activity (i.e., screens

accessed) by a given user for a

given patient on a given date.

Could not classify 35 user

sessions (0.03%) and

excluded them as too few

for meaningful analysis.

None

Vest, 200954 3463 HIE access, 2651 No access; 6,114 included out

of 600,000 individuals, 3.1 million encounters

Uninsured 18 to 64 years old Encounters at the public 

mental health provider and

Planned Parenthood. Also

excluded encounters

related to accidents,

pregnancy, labor and

delivery.

Persons with no information

accessed in the HIE vs. those

with accessed information
F-204

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Vest and 

Jasperson,

2012103

Administrative vs. clinical vs. repetitive vs.

mixed use

-User types and unique job titles

-Workplaces

Same day, within a week, within a

month, within a year, longer than a

year, or no encounter

Cross tabulation to compare

usage categories with A) job

categories, B) workplace

categories, and C) timing of

usage categories. Associations

evaluated between types of

usage and these variables using

the Pearson chi2 test of

independence

Vest, 200954 -ED visits and inpatient hospitalizations

due to ambulatory care sensitive

hospitalizations

-Logs document the user’s location, the

patient viewed, the date accessed, and

information screen viewed

-Predictors of HIE use (e.g.,

demographics, number of chronic

conditions, prior ED visits or

hospitalizations)

-HIE for predicting ED and

hospitalizations

-Clinical, demographic,

comorbidity, service measures

-Created a chronic condition index

by summing chronic conditions

(diabetes, hypertension, asthma,

ischemic heart disease,

hypercholesterolemia and stroke)

Quantitative

-Frequencies and percent

-Multiple logistic regression

adjusting for confounders
F-205

Author, Year Results

Risk of

Bias

Vest and 

Jasperson,

2012103

->6/10 sessions users accessed the system in a minimal fashion

-Average pattern length: 2.89 screens

-Shortest pattern length included only 1 screen and the longest pattern involved 83 screens

-65.7% of all user sessions had a pattern length of only 2 screens

-Use was overwhelmingly (93.9%) administrative, roughly evenly distributed across workplaces but for dominance of hospital accesses

(37.6%) and about half same day, a fifth first week, a fifth over the year, 1/10 unassociated with encounter; usage type associated with job

category: admin, nurse, pharmacy, physician, public/mental health, social services; most clinical access in ED, and public/mental health

-297 users, 113 unique job titles, collapsed into administration (59% of users), nurse (~6% of users), pharmacy (~1% of users), physician

(~12% of users), public health (~6% of users), and social services (~15% of users)

-Workplaces: ambulatory care (~9% of users), ED (~18% of users), children’s ED (3% of users), hospital (53% of users), public health

agency (8% of users), or mental health agency (8% of users).

-In more than 6 out of 10 sessions, users accessed the system in a minimal fashion.

-Average pattern length was 2.89 screens (range 1-83 screens); 66% of all user sessions had a pattern length of only two screens.

Low

Vest, 200954 Adjusted OR of HIE information access

Increasing age: 1.03; number of chronic conditions: 1.13; ≥1 prior year clinic visit: 1.63; a prior year ED visit: 1.96; and being hospitalized

in 2004: 2.02

All levels of HIE information access were associated with increased expected ED visits and ambulatory care sensitive hospitalizations vs.

no information access

-HIE was used more for those that used the system more, or were sicker.

-HIE was not accessed for 43% of individuals

-Ultimately, these results imply that HIE information access did not transform care in the ways many would expect. Expectations in

utilization reductions, however logical, may have to be reevaluated or postponed.

-Patients with HIE information accessed one time had an 83% higher expected count of ED visits.

Low
F-206

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Vest, 2010155 Cross-sectional Which nontechnological and

technological factors may still

hamper the existence of effective

HIE even in light of the

substantial financial incentives

offered via the HITECH Act?

U.S. U.S. Hospitals Surveys

2008-2009 HIMSS Analytic

Database; AHA Annual Survey

2007

After 2009

Vest and Miller

201164

Retrospective

cohort

Do hospitals using HIE have

higher reported communication

among health professionals

and/or higher patient 

satisfaction?

U.S. Hospitals Log file

-2008-2009 HIMSS Analytic

Database

-AHA Annual Survey 2007

-Review of all HIE facilitating

efforts in U.S., linked to HCAHPS

survey

After 2009

Vest, et al.,

2011105

Case control Do hospitalizations, ED visits,

and other factors predict HIE use

for indigent adults?

Austin, Texas Indigent patients 

and facilities that

care for them

Log files from clinical data

repository (Indigent Care

Collaboration of Austin, Texas

safety network providers founded

1997); 18 hospitals, clinics,

government agencies (federally

qualified health centers)

January 2006-June

2009
F-207

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Vest, 2010155 Various Various Various U.S.

Vest and Miller

201164

Various Various Various U.S

Vest, et al.,

2011105

Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin,

Texas safety network providers founded 1997); 18 hospitals,

public and private clinics, government agencies (federally

qualified health centers)

HIE 1997; I-Care

database 2002, 3.1

million encounters,

600,000 individuals

Indigent people, not Medicare
F-208

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Vest, 2010155 4,830 hospitals in AHA and HIMSS-AD In AHA or HIMSS survey NR Operational vs. adopted not

operational vs. not adopted

Vest and Miller

201164

3,278 hospitals, 340 adopted, 351 implemented HIE Participated in AHA or HIMSS

survey

Too few observations

(HCAHPS survey

responses <100)

Adopted vs. implemented vs.

none

Vest, et al.,

2011105

271,305 encounters (111,482 unique patients) from 10

facilities; (Vest 2009 was 3,463 HIE access, 2,651 no

access; 6,114 included out of 600,000 individuals, 3.1

million encounters)

All ED encounters among

patients ages 18 to 64 that 

occurred between January 1,

2006 and June 30, 2009

Excluded any ED

encounters occurring at

facilities before the hospital

had an authorized user of

the I-Care system.

None
F-209

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Vest, 2010155 HIE adoption (operational, implementing,

nonadapter)

Technological readiness (number of

live applications, CCHIT EMR), vertical

integration, horizontal integration,

high/low information needs, inpatient

admissions, market competition,

uncompensated care burden, primary

care rate, health system/network size

-Classic markers of innovation

adoption considered covariates

-Total number of beds (size)

-Average days cash on hand from

all sources

-Nonmetropolitan location

-General innovativeness was

measured both as academic

affiliation and specialization, the 

standardized total number of

professional job categories

Quantitative multivariate

analysis

-Begins with, or assumes, TOE

framework: technological,

organizational, and

environmental; missing values

imputed from earlier versions of

AHA Guide and HIMSS-AD

-Logistic regression on adoption,

logistic regression on operational

Vest and Miller

201164

-Percentage of patients who reported their

doctors and their nurses always

communicated well

-Percentage of patients who would

definitely recommend the hospital

-Percentage of patients who gave the

hospital a high global rating (≥9 on a 10-

point scale)

Level of HIE participation: implemented

(active sharing); adopted (participating

but not yet sharing); or none

Organizational variables

associated with HCAHPS

outcomes; other AHA

organizational characteristics,

overall level of automation in

hospital, external factors such as

state regulations

Quantitative

-Least squares regression

-Propensity score adjustment

Vest, et al.,

2011105

No usage vs. basic usage vs. novel usage

(more screens)

-Familiarity

-Complexity

-Mental/substance use

-Frequency of prior utilization

elsewhere

-Time constraints

Assessed with multivariate

analysis, otherwise NR

Quantitative multivariate

Logistic regression with

adjustment for by-patient

clustering
F-210

Author, Year Results

Risk of

Bias

Vest, 2010155 -59 operational and 123 nonoperational exchanges

-453 hospitals operational HIE, 446 adopted HIE, and 3,931 had not adopted HIE; sample includes more general service type and fewer

for-profit hospitals than the more nationally representative AHA survey

-Overall, 81.4% of hospitals had not adopted or implemented HIE

-Adjusted regression OR of adoption for not for profit: 8.57; public: 9.53; number operational application: 1.02; physician portals: 1.38;

network membership: 1.33; ED visit: 1.01' primary care MD in HRR: 1.03

-Adjusted regression OR of implementation: network membership: 1.96; hi competition: 0.15; primary care MD: NS

Low

Vest and Miller

201164

-10.4% had adopted

-10.7% had implemented HIE

-Implemented hospitals, but not adopted hospitals, had higher nurse communication (0.75 increase [95% CI, 0.13 to 1.38]), global

satisfaction (0.82 [95% CI, 0.01 to 1.64]), and would recommend scores (1.34 [95% CI, 0.41 to 2.27]), and a trend toward higher doctor

communication scores (NS after controlling for confounders); results attenuated in propensity score analysis

-Communication: higher for smaller hospitals, rural hospitals, fewer Medicaid patients, higher nurse/patient ratios

-Satisfaction: higher for nonprofit, smaller, Midwest or south, fewer Medicaid patients, higher nursing ratios

Low

Vest, et al.,

2011105

-No access of system for 97.7% of encounters

-Users accessed the I-Care system for 2.3% of the 271,305 encounters

-Basic usage (2,527) 41.1% of instances

-Sample was predominately Hispanic, younger, and a higher proportion of charity care recipients

-Adjusted OR of access for African American and Hispanic: 0.76 to 0.89; higher for unknown or charity care; but mainly for unknown

payer: 4.7 vs. 2.6; access higher for more ED visits; hospitalizations: ~1.25-1.5 (from graph)

-Access lower for alcohol use, injury, poisoning, unfamiliar patient, busier than average day

Low
F-211

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Vest, et al.,

2011104

Case control Do hospitalizations, ED visits,

and other factors predict HIE use

for indigent children?

Austin, Texas Indigent patients 

and facilities that

care for them

Log files from clinical data

repository (Indigent Care

Collaboration of Austin, Texas

safety net providers founded

1997); 18 hospitals, clinics,

government agencies (federally

qualified health centers)

January 2006-June

2009

Vest, et al.,

2012103

Case control Use of HIE in 2 ambulatory

indigent clinics without EHRs,

and patient factors associated

with this use.

Austin, Texas 2 ambulatory

clinics serving

indigent people, 

part of nonprofit

hospital system,

10,550-12,250

encounters/year

Log files from clinical data

repository (Indigent Care

Collaboration of Austin, Texas

safety network providers founded

1997); 18 hospitals, public and

private clinics, government

agencies (federally qualified

health centers)

January 2006-June

2009

Vest, Campion Jr.,

and Kaushal, 

2013156

Cross-sectional Identify the strengths and

weaknesses of organizational

models to achieve exchange, and

what can be done to ensure the

sustainability and effectiveness o

New York State HEAL-NY (HIE

promotion

legislation), HITEC

(academic

collaborative

performs

evaluations)

Interviews

Semi structured interviews with

selected experts

March - June 2010
F-212

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Vest, et al.,

2011104

Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin,

Texas safety network providers founded 1997); 18 hospitals,

clinics, government agencies (federally qualified health

centers)

HIE 1997; I-Care

database 2002, 3.1

million encounters,

600,000 individuals

Indigent people, not Medicare

Vest, et al.,

2012103

Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin,

Texas safety network providers founded 1997); 18 hospitals,

clinics, government agencies (federally qualified health

centers)

HIE 1997; I-Care

database 2002, 3.1

million encounters,

600,000 individuals

Indigent people, not Medicare

Vest, Campion Jr.,

and Kaushal, 

2013156

Various Various Various New York State
F-213

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Vest, et al.,

2011104

179,445 encounters All ED encounters among

patients <18 years occurred

between January 1, 2006 and

June 30, 2009 and had parental

consent

Excluded any ED

encounters occurring at

facilities before the hospital

had an authorized user of

the I-Care system.

None

Vest, et al.,

2012103

39,447 encounters 6,393 patients Age 19-64 years Austin metro

area, consent to inclusion

Children (different

utilization) or ≥65 years

(Medicare)

None

Vest, Campion Jr.,

and Kaushal, 

2013156

17 of 21 invited HIE experts Selected to represent public,

private, leaders, participators,

policymakers

None stated NA
F-214

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Vest, et al.,

2011104

No usage vs. basic usage vs. novel usage

(more screens)

3 factors as indicative of uncertainty

that creates an information need:

comorbidity, prior utilization, and

unfamiliarity with the patient

NR Quantitative multivariate

Logistic regression with

adjustment for by-patient

clustering

Vest, et al.,

2012103

Encounter level or retrospective usage -Age

-Gender

-Race

-ED visits over 3 months

-Hospitalization over 12 months

-Fragmentation (N of clinics -1)

-Payer (Medicaid or not)

-Charlson comorbidity

-Independent mental health/substance

abuse comorbidity

-AHRQ chronic conditions indicator

definitions

Assessed with multivariate

analysis, otherwise NR

Quantitative multivariate

Primary care encounter: unit of

analysis; multinomial regression,

clustered to account of unit of

analysis, adjusted for

confounders

Vest, Campion Jr.,

and Kaushal, 

2013156

NA NA NA Qualitative

Semistructured interview

exploring issues from literature,

open independent coding and

comparison by 2 investigators,

consensus; [no triangulation of

data or analysis, no member

check]
F-215

Author, Year Results

Risk of

Bias

Vest, et al.,

2011104

-System accessed: 15,586 of 179,445 encounters (8.7%)

-OR of basic HIE access for >1 year old vs. ≤1 year old: ~1.5 (from graph); lower for race unknown; higher for payer unknown; PC visits

within 12 months: ~1.5 (from graph); ED visits within 12 months: 1.5-2 (from graph); hospitalized: 1.3; number of diagnoses: 1.05;

unfamiliar: 0.46; busier than average: 0.65

-OR of novel HIE access for >1 year old vs. ≤1 year old: ~1.3; NS for race unknown; higher for payer unknown; PC visits within 12 months:

~2 (from graph); NS for ED visits within 12 months; hospitalized: 1.15; number of diagnoses: 1.05; unfamiliar: 0.19; NS busier than

average

Low

Vest, et al.,

2012103

-Access for 21% of encounters

-7,101 encounter based, 1,227 retrospective

-Adjusted OR for association with access for female: 1.12; >40 years: 1.16; chronic disease: 1.19; ED visit last 3 months: 1.13;

-Retrospective access, same 4 factors plus hospitalized last 4 months OR 1.33 and fragmentation OR 1.52

Low

Vest, Campion Jr.,

and Kaushal, 

2013156

Themes: (A) HIE is a public good; (B) challenges (1) financial challenges include upfront costs, discordance between investors and

beneficiaries of technology "how to make that savings accrue to us and not to the payers.”; opportunity cost of lost revenue and lack of

ROI "from a business perspective, HIE is kind of a bad idea. Why would we send out patient information elsewhere? We want to do it, we

think it’s necessary for better care of the patient, but we’ll lose money by doing it.”" (2) governance because "Federal, state, and private

representatives were fairly unanimous in their opinions that the functioning of RHIOs was not a technical issue" and the necessity of trust;

(3) mismatch of geographical model with reality of large integrated multistate delivery systems; (C) alternatives include Direct (lightweight,

treatment focused, lower organizational overhead; enterprise RHIOs, e.g.. "he hospital systems, they are the RHIO and they don’t want to

play with anybody else because they basically have quasi monopolies and cartels.” and don't need outside connection or support; Vendor

models likely but suboptimal; any of these not c/w state intent; (D) Sustainability quixotic, aims are contrary to market, contradiction of "

tension between providing a public good with little market incentives and operating like a private business"; alternatives: grow exchange

effort, specify a focus, evolve as an organization

NA
F-216

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Vest, et al.,

2013102

Case control Display and analyze the pattern

of radiology report requests

among organizations participating

in an HIE, and identify the patient

and provider factors associated

with use of a HIE system to

access radiology report

Western New

York State

Nonprofit RHIO

working with

Hospital systems,

reference

laboratories,

radiology groups,

insurance

providers, and

county offices

Log files, RHIO information about

job title, job type, and location,

and claims data.

The log file was

limited to patients 18

years and older and

reflected patient

encounters from

January 2009-March

2011

Vest, et al.,

201456

Retrospective

cohort

Examines the hypothesis that

usage of an HIE system reduces

the odds that a patient in the ED

will be hospitalized.

Rochester, New

York

HEAL NY

legislation,

statewide HIE

initiatives

Log file

Claims files from 2 health plans 

that insure more than 60% of the

area population, log files of 

usage, RHIO roster of users

2009-2010
F-217

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Vest, et al.,

2013102

Rochester RHIO Commercial query-based web portal product, which includes

patients’ discharge summaries, prior diagnoses, radiology

reports, medication history, and payer information. Both

radiology reports and images are accessible within the HIE

system and are typically available in near-real time after signoff. 

Imaging studies are accessible only if the user first views the

radiology report. Our analysis is limited to the viewing of reports

only.

NR Patients in health system in

western New York

Vest, et al.,

201456

Rochester RHIO >70 organizations in 13 county regions of western New York.

Web-based portal that includes discharge summaries,

diagnoses, radiology reports and images, medication history,

and payer information

Fully operational in 

March 2009

1,318 users accessed patient

records in 156 different

outpatient, emergency, 

inpatient, long-term care, and

specialty care settings via a

web portal. 7 EDs were

included; 800,000 patients

(>70% of the area's adult

population)
F-218

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Vest, et al.,

2013102

29,528 radiology documents originating at 17 different

source organizations, including hospitals and radiology

practices. A total of 126 different practice locations

viewed these documents.

Claims data only covers 60% of

population, included consenting

patients with ≥1 encounter in 6

months after consent

<18 years, not in health

system (included 60% of

pop, not the other 40%), 

had claims (64%, not the

other 36%)

NA

Vest, et al.,

201456

1,5645 Claims files for 65% of patients

≥18 years with valid consent

dates (n=198,067) who had ≥1

encounter with a provider

registered to use the HIE system

in the 6 months following their

consent date.

None reported HIE access vs. no HIE access

(from log files)
F-219

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Vest, et al.,

2013102

Radiology report access -Demographics

-Encounter history

-User characteristics

-Insurance type

-AHRQ CCS ICD-9 codes

-Use of services in 30 days prior to

access

-Claims for imaging procedures

-Health professional encounters

NR Quantitative multivariate

Using network/graph analysis

assessed the difference between

the average number of

connections among sources vs.

user practice locations, as well as

the average number of radiology

documents exchanged by data

sources vs. data users. Then (2)

mixed effects logistic regression

on 134,127 sessions, 64% linked

to claims files, with some

accounting for clustering by

patient, user, workplace - report

results without control for

confounders, multiple

comparisons problem

Vest, et al.,

201456

Hospital admission via the ED

Economic

HIE system use at the time of the ED

visit, measured in a yes/no fashion

-Gender

-Age

-Payer

-Disease severity in the 12-month

period

-Any primary care, specialty care,

or ED visits in the 30 days after the

index hospitalization (or up until

the date of readmission)

Quantitative

Logistic regression models. The

full model adjusts for all

independent variables with

patient age, the count of major

aggregated diagnostic groups,

and the number of prior

hospitalizations treated as

continuous variables, 4 sensitivity

analyses to explore the

robustness including physician

effects and patient subgroup

(sickest) effects
F-220

Author, Year Results

Risk of

Bias

Vest, et al.,

2013102

Network: each source organization sent on average 971 (range: 6-8,002) documents to 49 (3-106) other organizations. User

organizations accessed on average 49 (1-8,444) documents from 6 (1-17) source organizations. Algorithm suggests 11/17 source

organizations represent a core set of data providers, including 8 hospitals and 3 stand-alone radiology sites. Thus the overall number of

radiology reports retrieved in the outpatient setting was 16.9 times greater than the number of reports retrieved in the ED and inpatient

settings combined (23,201 outpatients vs. 1,333 ED and 313 inpatients).

Factors: 86,152 user sessions with associated claims files represented the activity of 1,119 different users representing 145 different

workplace locations. 86.4% were staff; physicians represented only about 4% of all sessions; overall 11.2% of sessions included access

of radiology reports.

Low

Vest, et al.,

201456

-ED visit within 6 months of consent: 15,645

-Of ED visits, HIE accessed: 2.4% (n=374)

-16/229 MDs used system

-OR of admission for Medicare: 2.02; Medicaid: 0.61; male: 1.47

-Adjusted OR of HIE access: 0.7; HIE access on same day as ED visit: 0.83 (95% CI, 0.55 to 1.25)

-Odds of an admission were 30% lower when the system was accessed after controlling for confounding (OR 0.70; 95% CI, 0.52 to 0.95)

-Annual savings in the sample was $357,000

Low
F-221

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Vest and Issel,

2014157

Cross-Sectional To examine factors related to

public health organizations data

exchange capabilities

United States State and local

health 

departments

Surveys 2007-2008

Vest, et al.,

201455

Retrospective

cohort

To determine the association

between usage of an HIE system

post- discharge and 30-day samecause hospital readmissions.

Rochester, New

York

HEAL NY

legislation,

statewide HIE

initiatives.

Outpatient

Log file

Claims files from 2 health plans 

that insure more than 60% of the

area population, log files of 

usage, RHIO roster of users

2009-2010

Willis, et al.,

201367

RCT To evaluate 2 decision support

interventions: patient adherence

reports to providers and reports

to providers and emails to care

managers by comparing to usual

care.

North Carolina Outpatient Database

EHR and claims as well as logs

of contacts and cost/revenue

data

-December 7, 2009-

December 6, 2010

was intervention

period

-Followup for

outcomes ended

August 30, 2011
F-222

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Vest and Issel,

2014157

Varies, any system that would allow

data sharing

Varies Varies U.S. states

Vest, et al.,

201455

Rochester RHIO Web based portal that includes discharge summaries,

diagnoses, radiology reports and images, medication history,

and payer information, 38 healthcare organizations in 11

counties

Fully operational in 

March 2009

800 000 patients (>70% of the 

area's adult population)

Willis, et al.,

201367

Northern Piedmont Community

Care Network. Set up a system

called COACH (CommunityOriented Approach to Coordinated

Healthcare)

-Included 9 clinics and 5 hospitals

-Data collected by the system include: 1) administrative data

2) care management data; 3) claims/billing data ; 4)

scheduling data; 5) clinical data; 6) data on communications

NR Network Medicaid

beneficiaries
F-223

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Vest and Issel,

2014157

44 states with representatives who responded to both

surveys

Executive officer of local health

department and state health

officials

States missing data on

either survey

Public health organizations that

don't have the capacity to

exchange data

Vest, et al.,

201455

196,314 patients, 11 hospitals (2/3 of sample) ≥18 years, consented during

2009-2010, continuously

enrolled in health plan, ≥1

encounter in 6 months following

consent, (196,314 patients met

these requirements). Only the

patient's first hospital admission

within the first 5 months after

consent. Each patient appears in

the dataset only once and each

discharge could be followed for

≥30 days.

<30 observations in the

dataset (n=11)

HIE access vs. no HIE access

(from log files)

Willis, et al.,

201367

N=2219

739 to usual care

744 clinic reports

735 clinic reports and care manager notices

Patients with ≥1 of 6 targeted

IOM priority conditions

Not continuously enrolled

during the intervention

period

Provider report vs. provider

report and case manager event

vs. usual care in which neither

type of alert was delivered
F-224

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Vest and Issel,

2014157

Bidirectional data sharing for childhood

immunizations, vital records and reportable

conditions

Organizational characteristics including

size, structure, processes and IT

readiness

None reported Quantitative Multivariate Analysis

Vest, et al.,

201455

Readmission within 30 days of discharge

for the same cause as the index

hospitalization

HIE system usage -Gender

-Age

-Payer

-Disease severity in the 12-month

period any primary care, specialty

care, or ED visits in the 30 days

after the index hospitalization (or

up until the date of readmission)

-Described the index

hospitalization site: hospital bed

size, teaching status, affiliation with

a multi-hospital healthcare system,

and critical access hospital

classification, case mix index

derived from the relative values of

diagnosis-related groups seen at

the hospital

Quantitative

Random effects logistic

regression models, a series of

models adjusting for patient

characteristics, then adding postdischarge utilization measures,

and lastly including hospital-level

characteristics. Controlled for

potential hospital-level clustering

using the index admission

hospital as a random intercept.

Then 2 sensitivity analyses.

Willis, et al.,

201367

-Clinical outcomes including: medical

adherence, outpatient, ED visits, and

hospitalizations

-Care coordination

costs/revenues

-Clinician satisfaction

Group assignment None reported Quantitative

Generalized estimating equation

models that accounted for

clustering by family
F-225

Author, Year Results

Risk of

Bias

Vest and Issel,

2014157

Data sharing capacity varied by activity. 66% had capacity for Immunizations

30.2% for vital records and

18.9% for reportable conditions

Moderate

Vest, et al.,

201455

-Readmitted within 30 days: 9.8% (668/6,807); 29.6% at a different facility; 394 had HIE access within 30 days after discharge, 20 (5.8%)

readmitted; p=0.00113

-ED visits within 30 days post discharge: NS

-HIE access associated with lower readmissions: OR 0.43 (95% CI, 0.27 to 0.70)

-Primary care or specialty care associated with lower readmissions rates: ORs 0.48 and 0.67 in final model

-ED visits associated with higher rates: OR 9.3 in final model

-Accessing patient information in the HIE in the 30 days after discharge associated with a 57% lower adjusted odds of readmission (OR

0.43; 95% CI 0.27 to 0.70). Estimated annual savings in the sample from averted readmissions associated with HIE usage was $605,000.

Low

Willis, et al.,

201367

Control vs. reports vs. reports and email

% medication adherence: 41.3% vs. 41.2% vs. 42.9%, p=NS; no differences between groups at 6 months

Encounter rates of outpatient: 46.0 vs. 46.6 vs. 44.5, p=NS

Encounter rates of ED: 0.87 vs. 0.84 vs. 0.89, p=NS

Encounter rates of hospitalizations: 0.19 vs. 0.21 vs. 0.21, p=NS

-15% to 50% of reports were not available to providers at time of patient encounter

-Even when they had reports, clinicians did not always discussion medication adherence with patients

Moderate
F-226

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

Winden, et al.,

201471

Case series To determine value of Epic Care

Everywhere in an ED

Minneapolis,

Minnesota

ED Observations

Chart review, focus groups,

survey

January-November,

2012

Yeager, et al.,

2014137

Cross-sectional To examine the barriers and

facilitators affecting the decision

to participate in an HIE and,

separately, which factors are

affecting the use of HIE.

Louisiana NR in this paper Interview March to April 2013
F-227

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

Winden, et al.,

201471

Allina Health and local

organizations using Epic

Directed transfer of Epic records to Allina ED August, 2010 All patients for whom CE used;

focus groups of clinician users

Yeager, et al.,

2014137

Louisiana HIE (LaHIE), statewide.

Number of centers/settings not

presented in this paper.

Louisiana HIE (LaHIE). LaHIE functions as a hybrid centralized

and federated model, web-based platform for providers to share

patient care continuity documents (commonly referred to as

CCDs), laboratory results, and electrocardiogram results.

NR Patients in Louisiana
F-228

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

Winden, et al.,

201471

Focus groups: 49 clinicians in 4 hospitals; Survey: 118

of 408 ED staff; review of 1,488 notes where CE used

Focus groups: clinicians; Survey:

ancillary staff; Notes: use of CE

Notes: CE not used Focus group and survey: value

for care; Chart review: tests

avoided

Yeager, et al.,

2014137

16 Healthcare representatives from organizations

interested in joining LaHIE but not yet enrolled (n=4),

not interested in joining (n=4), or already enrolled (n=8)

NR NR NA
F-229

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

Winden, et al.,

201471

Focus groups: provided value for patient

care, especially for avoiding duplicate

testing and detecting drug-seeking

behavior; Survey: provided value in patient

care; Chart review: procedures avoided

Focus groups and survey: value for

patient care; Chart review: procedures

avoided

None Quantitative

Survey, chart review

Yeager, et al.,

2014137

Barriers to implementation of LaHIE as

identified by interviews with health care

representatives

NA NA Qualitative, content analysis
F-230

Author, Year Results

Risk of

Bias

Winden, et al.,

201471

Focus groups: provided value for patient care, especially for avoiding duplicate testing and detecting drug-seeking behavior; Survey: 74%

agreed provided value in patient care; Chart review: 560 procedures avoided in 237 notes out of 1,488 assessed

Moderate

Yeager, et al.,

2014137

"Findings suggest that Meaningful Use requirements are a critical factor influencing the decision to participate in the HIE, specifically the

mandate that hospitals be able to electronically transfer summary of care documents. Creating buy-in within a few large hospital networks

legitimized the HIE and hastened interest in those markets. Fees charged by electronic health record (EHR) vendors to develop HIE

interfaces have been prohibitive. Funding from the federal incentive program is intended to offset the costs associated with EHR

implementation and increase the likelihood that HIEs can provide value to the population; however, costs and time delays of EHR interface

development may be key barriers to fully integrated HIEs. State HIEs may benefit from targeted involvement of state health care leaders

who can champion the potential value of the HIE"

Moderate
F-231

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

eHealth Initiative

2013 Report73

Cross-sectional To assess the status of data

exchange in the U.S.

Nationwide Any Survey responses 2013; comparison to

2011
F-232

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

eHealth Initiative

2013 Report73

Various 199 of 315 completed the survey; these were a mix of

community data exchanges, statewide efforts, & healthcare

delivery organizations.

Varies 315 data exchange initiatives

were identified
F-233

Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

eHealth Initiative

2013 Report73

-199 of 315 completed the survey; these were a mix of

community data exchanges, statewide efforts, &

healthcare (HC) delivery organizations.

-90 organizations self-identified as community-based

HIEs; 45 as state; 50 as health care delivery

organizations.

-There is no single dominant model for HIE; 125

organizations used a query model, 124 used secure

electronic messaging; 111 used end-to-end integration;

84 used a combination of models.

-'Direct' is a standards-based protocol for securely

exchanging data; 90 organizations use M117'Direct',

mostly in transitions of care.

-Patient consent for data exchange generally remains

an 'all-or-nothing' proposition, with 'opt-out' the most

common consent model.

NR NR NA
F-234

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

eHealth Initiative

2013 Report73

-Number of initiatives reaching 'advanced

stages of operation, sustainability or

innovation (as defined by eHI's

developmental framework)

-Number of years to become operational

-Trends in use since 2011

-Number of organizations self-identifying as

community, state-, or HC delivery system

-Types of professionals most commonly

providing and using data

-Types of data most commonly

provided/viewed

-Number having hired personnel from

ONC's Workforce Development Program

(WDP)

-Protocol used for securely exchanging

information

-Key Findings

-Issues for the future

NR NR NR
F-235

Author, Year Results

Risk of

Bias

eHealth Initiative

2013 Report73

84 organizations had reached an 'advanced' stage of operation, sustainability, or innovation.

Most took 2 years to become operational.

Among organizations responding in 2011 and 2013, 27 more had reached stages 5, 6, or 7 in 2013.

90 organizations self-identified as community-based HIEs; 45 as state-; 50 as HC delivery organizations.

Hospitals and Am Care providers are stakeholders most commonly providing/viewing data. Labs also commonly provide data. Community

public health clinics commonly view data.

24 reported they had hired staff from the ONC's WDP, compared to only 3 in 2011.

'Direct' is a standards-based protocol for securely exchanging data; 90 organizations use 'Direct', mostly in transitions of care.

There is no single dominant model for HIE; 125 organizations used a query model, 124 used secure electronic messaging; 111 used endto-end integration; 84 used a combination of models.

Key Findings:

1) Achieving interoperability with disparate information systems is a major concern; 68 initiatives have had to connect with more than 10

different systems;

2) To overcome interoperability challenges, exchanges would like to see standardized pricing and integration solutions from vendors;

3) Many exchanges are not sharing data with competing organizations;

4) Exchanges are focusing on functionalities to support health reform and advance analytics;

5) Patient engagement remains low amongst organizations exchanging data;

6) Patient consent for data exchange generally remains an 'all-or nothing' proposition, with 'opt-out' the most common consent model;

7) Since 2011, more initiatives have become more financially viable. However, hospitals and payers are still expected to fund most

exchange activity; of the 51 that were NOT sustainable, 31 (of 51) receive more than 50% of their funding from the federal government

and 22 report they are a state-HIE.

Overall, in 2011, 16 reported they were sustainable; in 2013, 35 reported they were sustainable. Organizations realize the precariousness

of government funding and are trying to offer valuable services for a fee.

Issues for the future:

1) Interoperability concerns need to be addressed;

2) Health reform provides exchanges an opportunity to show value;

3) Patient engagement remains poor.

NA
F-236

Author, Year Study Design

Study Purpose/Research

Question

Geographic

Location Setting

Data Source(s)/ Evaluation

Data

Time Period of Data

Collection

eHealth Initiative

2014 Report74

Cross-sectional To assess the status of data

exchange in the US.

Nationwide Any Survey responses 2013; comparison to

2011
F-237

Author, Year Name of HIE (Intervention) Description of HIE (this will become Types)

Date HIE

Implemented Population

eHealth Initiative

2014 Report74

Various 199 of 315 completed the survey; these were a mix of

community data exchanges, statewide efforts, & healthcare

delivery organizations.

Varies 315 data exchange initiatives

were identified
F-238

Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison

eHealth Initiative

2014 Report74

-199 of 315 completed the survey; these were a mix of

community data exchanges, statewide efforts, &

healthcare (HC) delivery organizations.

-90 organizations self-identified as community-based

HIEs; 45 as state; 50 as health care delivery

organizations.

-There is no single dominant model for HIE; 125

organizations used a query model, 124 used secure

electronic messaging; 111 used end-to-end integration;

84 used a combination of models.

-'Direct' is a standards-based protocol for securely

exchanging data; 90 organizations use M117'Direct',

mostly in transitions of care.

-Patient consent for data exchange generally remains

an 'all-or-nothing' proposition, with 'opt-out' the most

common consent model.

NR NR NA
F-239

Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods

eHealth Initiative

2014 Report74

-Number of initiatives reaching 'advanced

stages of operation, sustainability or

innovation (as defined by eHI's

developmental framework)

-Number of years to become operational

-Trends in use since 2011

-Number of organizations self-identifying as

community, state-, or HC delivery system

-Types of professionals most commonly

providing and using data

-Types of data most commonly

provided/viewed

-Number having hired personnel from

ONC's Workforce Development Program

(WDP)

-Protocol used for securely exchanging

information

-Key Findings

-Issues for the future

NR NR NR
F-240

Author, Year Results

Risk of

Bias

eHealth Initiative

2014 Report74

Who provides data: 112 hospitals, 100 Am Care providers, 56 labs, 52 community/public health clinics.

Who accesses data: 111 Am Care providers, 104 hospitals, 75 community/public health clinics, 65 behavioral or mental health providers.

Key Barriers: 1) Cost and technical challenges are key barriers to interoperability; 2) Regulatory policies appear to have prompted

increased use of core HIE services such as 'Direct', care summary exchange, and transitions of care; 3) Advanced initiatives are

supporting new payment and advanced delivery models; 4) Sustainable organizations have replaced federal funding with revenue from

fees and membership dues.

Key finding 1: Interoperability Challenges include costs of building interfaces, getting consistent and timely response from EMR vendors

and interface developers, and technical difficulty of building interfaces. 112 organizations have had to construct multiple interfaces and 18

have had to construct more than 25 interfaces.

Suggestions for overcoming interoperability challenges include: 1) standardized pricing and integration solutions from vendors; 2) 'plug

and play' platform; 3) federally mandated standards; 4) cultural changes in willingness to share data; 5) greater use among providers of

consensus-based standards.

Key finding 2: Regulatory Policies prompt use of core HIE Services:

101 incorporate secure messaging into their models; 78 offer a 'Direct' address directory; more respondents are using 'Direct' for all given

use cases (when compared to last year). 74 have met at least one Stage 2 Meaningful Use criteria. 7 stages of Development are

delineated (see slide in report for detail);

Key finding 3: Advanced initiatives are supporting new payment & delivery models: 106 reported they have reached stage 6 (operating) or

higher on the eHI's HIE maturity scale (an increase of 11% over 2013).

64 support an ACO; 52 support a PCMH; 21 support a State Innovation Model; 12 support a bundled payment initiative.

Key finding 4: Sustainable groups replace fed funding with fees and membership dues: 45 use fees to completely cover operational

expenses; 38 use fees but need additional funding. 41 report that dues or fees are greatest revenue source; 89 believe dues or fees will

eventually be their primary revenue stream.

Looking to the future:

1) Data exchange is reaching a point of stability and acceptance.

2) Organizations are settling on a set of core service offerings and a standard approach to sustainability (sub-bullet: despite expiration of

large funding sources, radical changes in overall landscape are not evident);

3) As organizations mature, they will offer new and innovative services (public health has already leverages HIE; alert notification services

may help ACOs to track patients);

4) Organizations are encouraged to work collaboratively to overcome remaining challenges (especially work with regional/community

partners to avoid creating 'pockets' of exchange).

NA
F-241

* this is from billing data, not EHR

†one site dropped that didn't have comparable qualitative data.

A1c= glycated hemoglobin; AHA= American Hospital Association; AHRQ= Agency for Healthcare Research and Quality; aka= also

known as; AMIE= Arizona medical information exchange; ANOVA= analysis of variance; BHIX= Brooklyn Health Information

Exchange; CCD= continuity of care document; CCHIT= Certification Commission for Healthcare Information Technology; CCR= 

community care record; CCS= clinical classification software; CD4= HIV helper cell count; CDA = clinical document architecture;

CDC= Centers for Disease Control and Prevention; CE= Care Everywhere; CEN= clinical event notification; CHIC RHIO= Carolina

HIV information cooperative regional health information organization; CI= confidence interval; CIO= chief information officer;

COACH= Community Oriented Approach to Coordinated Healthcare; CPT4= Current procedure Terminology; CT= computed

axial tomography scan; DOD= Department of Defense; e= electronic; e.g.= for example; ebSML RIM= electronic business using

extensible markup language registry information model; ebXML RS= electronic business using extensible markup language; ECCI= 

Electronic Clinical Communication Implementation Program; ED= emergency department; EDI= electronic data interchange;

EDIFACT= electronic data interchange for administration, commerce and transport; eHIE= electronic health information exchange;

EHR= electronic health records; EKG= electrocardiogram; ELRs = enhanced laboratory reports; EMR= electronic medical records;

EMS= emergency medical services; e-OP= electronic outpatient appointment booking; EPIC= electronic privacy information center;

et al.= and others; etc.= etcetera; EPR= electronic patient records; e-RR= electronic results reporting; EU27= 27 nations in the

European Union; FITT= fit between individuals tasks and technologies; FUHN= Federally Qualified Health Center Urban Health

network; FQHCs= federally qualified health centers; GDP= gross domestic product; GP= general practitioner; HC= Health Care;

HCAHPS= Hospital Consumer Assessment of Healthcare Providers and Systems; HEAL = Health Care Efficiency and Affordability

Law; HEAL NY= Health Care Efficiency and Affordability Law for New York; HEDIS= health care effectiveness data and

information set; HIE= health information exchange; HIMSS= healthcare information and management systems society; HIMSSAD= healthcare information and management systems society analytical database; HIO= Health Insuring Organization; HIPAA= 

Health Insurance Portability and Accountability Act; HITECH= Health Information Technology for Economic & Clinical Health

Act; HL-7= Health Level 7; HL7; HMO= health maintenance organization; HRR= unadjusted hazard ratio; HRSA= `Health

Resources and Services Administration; Id = Identifier; i.e.= that is; ICC= integrated care collaboration; ICD-9= Ninth Revision of

the International Classification of Diseases; ICD-9-CM= International Classifications of Diseases, Clinical Modification; ICU= 

intensive care unit; IDS= integrated delivery system; I-EMS= Indianapolis Emergency Medical Services; IHIE= Indiana Health

Information Exchange; IM & T=information management & technology; INPC= Indiana Network fro Patient Care; IOM= Institute

of Medicine's; IQR= interquartile range; IS = information system; IT= information technology; KP= Kaiser Permanente?;
F-242

LaHIE=Louisiana HIE; LaPHIE= Louisiana Public Health Information Exchange; LBNH= Long Beach Network for Health;

LOINC= Logical Observation Identifiers Names and Codes; MAeHC= Massachusetts eHealth Collaborative; MANOVA= 

multivariate analysis of variance; MD= Doctor of Medicine; MEGAHIT= Medical Evidence Gathering Through Health IT;

MHDC= Massachusetts Health Data Consortium; mL= milliliter; mm= millimeter; MN= Minnesota; MPI= master patient index;

MRI= magnetic resonance imaging; MRSA= Methicillin Resistant Staphylococcus Aureus; MSeHA= MidSouth e-Health Alliance;

N= sample size; NA= not applicable; NAMCS= National Ambulatory Medical Care Survey; NDC= National Drug Code; NE= 

northeast; NHIN= Nationwide Health Information Network; NLM= National Library of Medicine; NR= not relevant; NS= not 

significant; NY= New York; NYCLIX= New York Clinical Information Exchange; OLS= ordinary least squares; ONC= Office of

the National Coordinator for Health Information Technology; OR= odds ratio; PBMs= pharmacy benefit managers; PC= primary

care; PCP = primary care provider; PDF= portable document format; PHI= personal health information; PHR= personal health

record; PPO= preferred provider organization; QUIS= Questionnaire for User Interaction Satisfaction; RCT= randomized,

controlled trial; RHIE = regional health information exchange; RHIO= regional health information organization; RLS= record

locator service; RNA= ribonucleic acid; RR= relative risk; SCR= summary care record; SD= standard deviation; S.E.= southeast;

SF-12= Short Form-12 item survey; SHIN-NY= Statewide Health Information Network for New York; SMRTnet= Secure Medical

Records Transfer Network; SNOCAP-USA= State Networks of Colorado Ambulatory Practices & Partners United States of

America; SNOMED= Systemized Nomenclature of Medicine; SSA= Social Security Administration; SUNBH = Seoul National

University Bundang Hospital; TILAK= Tiroler Landeskrankenanstaleten ; TOE= technological, organizational and environmental;

TUMA= Trans-mural exchange of medication data in Almere; U.K.= United Kingdom; U.S.= United States; URL= uniform

resource locator; USB= universal serial bus; VA= U.S. Department of Veterans Affairs; VL= viral load; VLER= Veterans Lifetime

Electronic Record; VRE= Vancomycin resistant enterococci; vs.= versus; WHIE= Wisconsin Health Information Exchange; XML=

extensible markup language.
G-1

Appendix G. Risk of Bias Assessment Criteria

Our assessment of risk of bias was based on the recommendations in the Agency for 

Healthcare Research and Quality Methods Guide for Effectiveness and Comparative 

Effectiveness Reviews.

1 Included studies were classified according to type of design (see 

Appendix E) as part of the data abstraction phase, and each major type of study was assessed for 

bias according to relevant criteria. This criteria included questions that assessed selection bias, 

performance bias, detection bias, attrition bias, and reporting bias (i.e., those about adequacy 

of randomization, similarity of groups at baseline, appropriateness of the comparators, 

consideration of concurrent interventions or unintended exposures, quantity of missing data, 

methods of handling missing data, identification and assessment of important confounding 

variables, use of intention-to-treat analysis, reliability and validity of outcome measures, and 

reporting of pre specified outcomes).

Criteria for Randomized Controlled Trials

Selection bias

• Was randomization adequate?

• Was allocation concealment adequate?

• Were groups similar at baseline?

• Did the study maintain comparable groups throughout the study?

• Was the eligibility criteria specified?

Detection bias

• Was the study adequately blinded (outcome assessor, care provider, and patient)?

Attrition bias

• Was the loss to followup not differential or high?

Reporting bias

• Did the study report attrition, crossovers, adherence, and contamination?

• Was an intention-to-treat analysis used?

• Were outcomes prespecied?

Criteria for Cohort, Case-Control, and Other Observational 

Studies

Selection bias

• Are the comparison groups or time periods appropriate?

• Were the inclusion and exclusion criteria specified and applied equally to each group?

• Did the design and analyses account for important potential confounding and modifying 

variables appropriately?

• Were valid and reliable measures used (inclusion/exclusion, confounding, outcomes)?

Detection bias

• Were non-biased and valid ascertainment methods used (inclusion/exclusion, 

confounding, outcomes)?
G-2

• Was the timing and/or time period for the measurement of the intervention and outcomes 

appropriate?

Attrition bias

• Was there NO missing data? If missing data, was it handled appropriately?

Reporting bias

• Were outcomes prespecified and were prespecified outcomes reported?

Definition of ratings based on above criteria:

Low risk of bias:

Studies rated “low risk of bias” were considered to have the least risk of bias, and their 

results are considered valid. Low risk of bias studies include clear descriptions of the 

population, setting, interventions, and comparison groups clear reporting of missing data; 

appropriate means for preventing bias; and appropriate measurement of outcomes. 

Moderate risk of bias:

Studies rated “moderate risk of bias” were susceptible to some bias, though not enough to 

necessarily invalidate the results. These studies may not meet all the criteria for a rating 

of low risk of bias, but do not have flaws likely to cause major bias. The study may be 

missing information, making it difficult to assess limitations and potential problems. The 

moderate risk of bias category is broad, and studies with this rating will vary in their 

strengths and weaknesses. The results of some moderate risk of bias studies are likely to 

be valid, while others may be only possibly valid.

High risk of bias:

Studies rated “high risk of bias” have significant flaws that imply biases of various types 

that may invalidate the results. They will have a serious or “fatal” flaw in design, 

analysis, or reporting; large amounts of missing information; or discrepancies in 

reporting. The results of these studies will be least as likely to reflect flaws in the study 

design as the true difference between the compared interventions. We did not exclude 

studies rated as being high risk of bias a priori, but high risk of bias studies were 

considered to be less reliable than lower risk of bias studies when synthesizing the 

evidence, particularly if discrepancies between studies were present. 

Criteria for Surveys, Focus Groups, and Interview Studies

Selection bias

1. Is the sampling strategy or selection criteria reported and appropriate? 

2. Are the response or participation rates reported and are they acceptable given the type of 

study?

3. Are characteristics (e.g., demographics) of respondents/participants reported?

Detection bias

4. Is how the questions were developed/selected reported and is it appropriate?

5. Were confounders considered (could be in analysis or presentation, such as stratifying 

results)?

Other

6. Is analysis appropriate (given the type of data)?
G-3

Reference for Appendix G

1. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ 

Publication Number 10(14)-EHC062-EF. Rockville, MD: Agency for Healthcare 

Research and Quality. January 2014. Available at: www.effectivehealthcare.ahrq.gov. 

Accessed April 18, 2014. PMID: 21433403.
H-1

Appendix H. Strength of Evidence Criteria1

The set of five required domains comprises the main constructs that Evidence-based Practice 

Centers (EPCs) should use for all major outcomes and comparisons of interest. As briefly 

defined below in Table H1, these domains represent related but separate concepts and each is 

scored independently. The concepts are explained in more detail below. 

Table H1. Required domains and their definitions

Domain Definition and Elements Score and Application 

Study 

Limitations 

Study limitations is the degree to which the included 

studies for a given outcome have a high likelihood of 

adequate protection against bias (i.e., good internal 

validity), assessed through two main elements: 

• Study design: Whether RCTs or other designs such as 

nonexperimental or observational studies. 

• Study conduct. Aggregation of ratings of risk of bias of 

the individual studies under consideration. 

Score as one of three levels, separately 

by type of study design: 

• Low level of study limitations 

• Medium level of study limitations 

• High level of study limitations 

Directness Directness relates to (a) whether evidence links 

interventions directly to a health outcome of specific 

importance for the review, and (b) for comparative 

studies, whether the comparisons are based on headto-head studies. The EPC should specify the 

comparison and outcome for which the SOE grade 

applies. 

Evidence may be indirect in several situations such as: 

• The outcome being graded is considered intermediate 

(such as laboratory tests) in a review that is focused on 

clinical health outcomes (such as morbidity, mortality). 

• Data do not come from head-to-head comparisons but 

rather from two or more bodies of evidence to compare 

interventions A and B—e.g., studies of A vs. placebo 

and B vs. placebo, or studies of A vs. C and B vs. C but 

not direct comparisons of A vs. B. 

• Data are available only for proxy respondents (e.g., 

obtained from family members or nurses) instead of 

directly from patients for situations in which patients are 

capable of self-reporting and self-report is more reliable. 

Indirectness always implies that more than one body of 

evidence is required to link interventions to the most 

important health outcome. 

Score as one of two levels: 

• Direct 

• Indirect 

If the domain score is indirect, EPCs 

should specify what type of indirectness 

accounts for the rating. 

Consistency Consistency is the degree to which included studies find 

either the same direction or similar magnitude of effect. 

EPCs can assess this through two main elements: 

• Direction of effect: Effect sizes have the same sign 

(that is, are on the same side of no effect or a MID) 

• Magnitude of effect: The range of effect sizes is 

similar. EPCs may consider the overlap of CIs when 

making this evaluation. 

The importance of direction vs. magnitude of effect will 

depend on the key question and EPC judgments. 

Score as one of three levels: 

• Consistent 

• Inconsistent 

• Unknown (e.g., single study) 

Single-study evidence bases (including 

mega-trials) cannot be judged with 

respect to consistency. In that instance, 

use “Consistency unknown (single 

study).” 
H-2

Domain Definition and Elements Score and Application 

Precision Precision is the degree of certainty surrounding an 

effect estimate with respect to a given outcome, based 

on the sufficiency of sample size and number of events. 

• A body of evidence will generally be imprecise if the 

OIS is not met. OIS refers to the minimum number of 

patients (and events when assessing dichotomous 

outcomes) needed for an evidence base to be 

considered adequately powered. 

• If EPCs performed a meta-analysis, then EPCs may 

also consider whether the CI crossed a threshold for an 

MID. 

• If a meta-analysis is infeasible or inappropriate, EPCs 

may consider the narrowness of the range of CIs or the 

significance level of p-values in the individual studies in 

the evidence base. 

Score as one of two levels: 

• Precise 

• Imprecise 

A precise estimate is one that would 

allow users to reach a clinically useful 

conclusion (e.g., treatment A is more 

effective than treatment B). 

Reporting Bias Reporting bias results from selectively publishing or 

reporting research findings based on the favorability of 

direction or magnitude of effect. It includes: 

• Study publication bias, i.e., nonreporting of the full 

study. 

• Selective outcome reporting bias, i.e., nonreporting (or 

incomplete reporting) of planned outcomes or reporting 

of unplanned outcomes. 

• Selective analysis reporting bias, i.e., reporting of one 

or more favorable analyses for a given outcome while 

not reporting other, less favorable analyses. 

Assessment of reporting bias for individual studies 

depends on many factors–e.g. availability of study 

protocols, unpublished study documents, and patientlevel data. Detecting such bias is likely with access to 

all relevant documentation and data pertaining to a 

journal publication, but such access is rarely available. 

Because methods to detect reporting bias in 

observational studies are less certain, this guidance 

does not require EPCs to assess it for such studies. 

Score as one of two levels: 

• Suspected 

• Undetected 

Reporting bias is suspected when: 

• Testing for funnel plot asymmetry 

demonstrates a substantial likelihood of 

bias, 

And/or 

• A qualitative assessment suggests the 

likelihood of missing studies, analyses, 

or outcomes data that may alter the 

conclusions from the reported evidence. 

Undetected reporting bias includes all 

alternative scenarios. 

CI = confidence internal; EPC = Evidence-based Practice Center; MID = minimally important difference; OIS = optimal 

information size; RCT = randomized controlled trial[ SOE = strength of evidence

Study Limitations Domain Definition 

Scoring the study limitations domain is the essential starting place for grading strength of the 

body of evidence. It refers to the judgment that the findings from included studies of a treatment 

(or treatment comparison) for a given outcome are adequately protected against bias (i.e., have 

good internal validity), based on the design and conduct of those studies. That is, EPCs assess 

the ability of the evidence to yield an accurate estimate of the true effect without bias 

(nonrandom error).

Directness Domain Definition

Directness of evidence expresses how closely available evidence measures an outcome of 

interest. Assessing directness has two parts: directness of outcomes and directness of 

comparisons. Applicability of evidence (external validity) is considered explicitly but separately 

from strength of evidence.
H-3

Consistency Domain Definition

Consistency refers to the degree of similarity in the direction of effects or the degree of 

similarity in the effect sizes (magnitudes of effect) across individual studies within an evidence 

base. EPCs may choose which of these two notions of consistency (direction or magnitude) they 

are scoring; they should be explicit about this choice.

Precision Domain Definition

Precision is the degree of certainty surrounding an estimate of effect with respect to an 

outcome. It is based on the potential for random error evaluated through the sufficiency of 

sample size and, in the case of dichotomous outcomes, the number of events. A precise body of 

evidence should enable decisionmakers to draw conclusions about whether one treatment is 

inferior, equivalent, or superior to another.

Reporting Bias Definition

Reporting bias occurs when authors, journals, or both decide to publish or report research 

findings based on their direction or magnitude of effect. Table 2 defines the three main types of 

reporting bias that either authors or journals can introduce: publication bias and outcome and 

analysis reporting bias.

Four Strength of Evidence Levels 

The four levels of grades are intended to communicate to decisionmakers EPCs’ confidence 

in a body of evidence for a single outcome of a single treatment comparison. Although assigning 

a grade requires judgment, having a common understanding of the interpretation will be useful 

for helping EPCs as they conduct their own global assessment and for improving consistency 

across reviewers and EPCs. 

Table H2 summarizes the four levels of grades that EPCs use for the overall assessment of 

the body of evidence. Grades are denoted high, moderate, low, and insufficient. They are not 

designated by Roman numerals or other symbols. EPCs should apply discrete grades and should 

not use designations such as “low to moderate” strength of evidence. 

Table H2. Strength of evidence grades and definitions

Grade Definition 

High We are very confident that the estimate of effect lies close to the true effect for this 

outcome. The body of evidence has few or no deficiencies. We believe that the findings are 

stable, i.e., another study would not change the conclusions. 

Moderate We are moderately confident that the estimate of effect lies close to the true effect for 

this outcome. The body of evidence has some deficiencies. We believe that the findings are 

likely to be stable, but some doubt remains. 

Low We have limited confidence that the estimate of effect lies close to the true effect for this 

outcome. The body of evidence has major or numerous deficiencies (or both). We believe that 

additional evidence is needed before concluding either that the findings are stable or that the 

estimate of effect is close to the true effect. 

Insufficient We have no evidence, we are unable to estimate an effect, or we have no confidence in 

the estimate of effect for this outcome. No evidence is available or the body of evidence has 

unacceptable deficiencies, precluding reaching a conclusion. 
H-4

Each level has two components. The first, principal definition concerns the level of 

confidence that EPCs place in the estimate of effect (direction or magnitude of effect) for the 

benefit or harm; this equates to their judgment as to how much the evidence reflects a true effect. 

The second, subsidiary definition involves an assessment of the level of deficiencies in the body 

of evidence and belief in the stability of the findings, based on domain scores and a more 

holistic, summary appreciation of the possibly complex interaction among the individual 

domains.

Assigning a grade of high, moderate, or low implies that an evidence base is available from 

which to estimate an effect for either the benefit or the harm. The designations of high, moderate, 

and low should convey how confident EPCs would be about decisions based on evidence of 

differing grades, which can be based on either quantitative or qualitative assessment.

For comparative effectiveness questions, the comparison is typically a choice of either 

direction (A>B, A=B, A<B) or magnitude (difference between A and B). In some instances 

assigning different grades regarding the direction and the magnitude of an effect may be 

appropriate. An example of this situation is when studies consistently find that an intervention 

improves an outcome (e.g., apnea-hypopnea index is reduced by a statistically significant amount 

or beyond a minimally important difference), but the degree of heterogeneity about the estimate 

is high (e.g., range -10 to -46 events/minute; I2 = 86%).

The importance of the distinctions among high, moderate, and low levels (and the distinction 

with insufficient strength of evidence) can vary by the type of outcome, comparison, and 

decisionmaker. EPCs understand that some stakeholders may want to take action only when 

evidence is of high or moderate strength, whereas others may want to understand clearly the 

implications of low versus insufficient evidence. Even when strength of evidence is low or 

insufficient, consumers, clinicians, and policymakers may find themselves in the position of 

having to make choices and decisions, and they may consider factors other than the evidence 

from a specific systematic review, such as patient values and preferences, costs, or resources.

Reference for Appendix H

1. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ 

Publication No. 10(13)-EHC063-EF. Rockville (MD) :Agency for Healthcare Research 

and Quality. January 2014. Availible at: www.effectivehealthcare.ahrq.gov. 
I-1

Appendix I. Quality Assessment Tables

Table I-1. Quality assessments of randomized controlled trials

Author, Year Randomization 

adequate?

Allocation

concealment 

adequate?

Groups similar at

baseline?

Maintain

Comparable

Groups?

Eligibility

criteria

specified?

Outcome

assessors

masked?

Care provider

masked? Patient masked?

Afilalo, et al.,

200766

Lang, et al.,

200665

Yes Yes Yes Yes Yes Yes No No

Willis, et al.,

201367

Yes Not Reported Unclear Unclear Yes Unclear No No
I-2

Author, Year

Reporting of

attrition, crossovers,

adherence, and

contamination

No Loss to

followup:

differential/

high

Intention-to-treat

analysis No Postrandomization 

exclusions

Outcomes

Prespecified

Funding source Risk of bias

Afilalo, et al.,

200766

Lang, et al.,

200665

Unclear No No No Yes Yes Moderate

Willis, et al.,

201367

Yes Yes Yes Yes Yes Agency for 

Healthcare 

Research and

Quality

Moderate
I-3

Table I-2. Quality assessments of cohort, case-control, and other observational studies

Author, Year

Are the

comparison

groups or time

periods

appropriate?

Were the inclusion and

exclusion criteria

specified and applied 

equally to each group?

Did the design and

analyses account

for important

potential 

confounding and 

modifying variables

appropriately?

Were valid and reliable

measures used?

(inclusion/exclusion, 

confounding, 

outcomes)

Were non-biased and

valid ascertainment

methods used? 

(inclusion/exclusion, 

confounding, 

outcomes)

Was the timing 

and/or time period

for the 

measurement of the

intervention and

outcomes 

appropriate?

Bailey, et al., 201340 Yes Yes Yes Yes NR Yes

Bailey, et al., 201239 Yes Yes Yes Yes NR Yes

Ben-Assuli, Shabtai,

and Leshno, 201341

Yes NR Yes Yes NR Yes

Ben-Assuli, Shabtai,

and Leshno, 201572

Yes Yes Yes Yes NR Yes

Carr, et al., 201470 No No Unclear NR Yes Yes

Dixon, McGowan,

and Grannis, 201142

Yes NA NA Yes Yes NA

Feldman and Horan

201143

Yes No No Yes Yes Yes

Frisse, et al., 201244 Yes Yes Yes Yes NR Yes

Jones, Friedberg

and Schneider, 

201168

Yes Yes Yes Yes Yes Yes

Kern et al., 201245 Yes Yes Yes Yes Yes Yes

Kho et al., 201388 NA Yes No Yes Yes Yes

Lammers, AdlerMilstein, and

Kocher, 201469

Yes Yes Yes Yes Yes Yes

Lobach, et al.,

2007100

NA NA NA Yes NA Yes

Magnus, et al.,

201247

Yes Yes Yes Yes Yes Yes
I-4

Author, Year

Was there no

missing data? If 

missing data, was it 

handled 

appropriately?

Were outcomes

prespecified and

were prespecified

outcomes 

reported? Risk of bias

Bailey, et al., 201340 Yes Yes Low

Bailey, et al., 201239 NR Yes Low

Ben-Assuli, Shabtai,

and Leshno, 201341

Yes Yes Low

Ben-Assuli, Shabtai,

and Leshno, 201572

Yes Yes Low

Carr, et al., 201470 Yes Yes Moderate

Dixon, McGowan,

and Grannis, 201142

NA Yes Low

Feldman and Horan

201143

No Yes Moderate

Frisse, et al., 201244 Yes Yes Moderate

Jones, Friedberg

and Schneider, 

201168

Yes Yes Low

Kern et al., 201245 No Yes Low

Kho et al., 201388 Yes NA Low

Lammers, AdlerMilstein, and

Kocher, 201469

Yes Yes Low

Lobach, et al.,

2007100

Unclear NA Low

Magnus, et al.,

201247

NR Yes Low
I-5

Author, Year

Are the

comparison

groups or time

periods

appropriate?

Were the inclusion and

exclusion criteria

specified and applied 

equally to each group?

Did the design and

analyses account

for important

potential 

confounding and 

modifying variables

appropriately?

Were valid and reliable

measures used?

(inclusion/exclusion, 

confounding, 

outcomes)

Were non-biased and

valid ascertainment

methods used? 

(inclusion/exclusion, 

confounding, 

outcomes)

Was the timing 

and/or time period

for the 

measurement of the

intervention and

outcomes 

appropriate?

Mäenpää, et al.,

2011115

Yes Yes Yes Yes Yes Yes

McCarthy, et al.,

2014161

Unclear Yes No No No Yes

McGowan, et al.,

2007148

No No No No No No

Miller and Tucker,

2014149

Yes Yes No Yes No Yes

Moore, et al.,

2012106

No comparison

group

Yes No Yes No Yes

Nagykaldi, et al.,

201448

Yes Unclear Unclear Unclear Unclear Yes

Onyile, et al.,

2013125

Yes Yes NA Yes Yes Yes

Overhage, Grannis,

and McDonald,

200849

Yes Yes Yes Yes Yes Yes

Ross, et al., 201350 Yes Yes Yes Yes Yes Yes

Saff, et al., 2010154 NA NA No Uncertain Unclear Yes

Shapiro, et al.,

201351

Yes Yes Yes Yes Yes Yes

Silvester and Carr,

2009114

Yes Yes No Yes Yes Yes
I-6

Author, Year

Was there no

missing data? If 

missing data, was it 

handled 

appropriately?

Were outcomes

prespecified and

were prespecified

outcomes 

reported? Risk of bias

Mäenpää, et al.,

2011115

Yes Yes Low

McCarthy, et al.,

2014161

NA No Moderate

McGowan, et al.,

2007148

Unclear No High

Miller and Tucker,

2014149

Potentially missing

data handled to best

of their ability

Yes Moderate

Moore, et al.,

2012106

Yes Yes Moderate

Nagykaldi, et al.,

201448

NR Yes Moderate

Onyile, et al.,

2013125

Not clear NA Low

Overhage, Grannis,

and McDonald,

200849

Yes Yes Low

Ross, et al., 201350 Unclear Yes Low

Saff, et al., 2010154 Unclear No High

Shapiro, et al.,

201351

Yes Yes Moderate

Silvester and Carr,

2009114

Yes Yes High
I-7

Author, Year

Are the

comparison

groups or time

periods

appropriate?

Were the inclusion and

exclusion criteria

specified and applied 

equally to each group?

Did the design and

analyses account

for important

potential 

confounding and 

modifying variables

appropriately?

Were valid and reliable

measures used?

(inclusion/exclusion, 

confounding, 

outcomes)

Were non-biased and

valid ascertainment

methods used? 

(inclusion/exclusion, 

confounding, 

outcomes)

Was the timing 

and/or time period

for the 

measurement of the

intervention and

outcomes 

appropriate?

Tzeel, Lawnicki, and

Pemble, 201253

Yes Yes Yes Yes Unclear Yes

Tzeel, Lawnicki, and

Pemble, 201152

Yes Yes Yes Yes Unclear Yes

Vest, 200954 Yes Yes Yes Yes Yes Yes

Vest, 2010155 Yes Yes Yes Yes Yes Yes

Vest, et al., 2011104 Yes Yes Yes Yes Yes Yes

Vest, et al., 2011105 Yes Yes Yes Yes Yes Yes

Vest and Miller,

201164

Yes Yes Yes Yes (no information on

survey reporting)

Yes (Data are from

multiple surveys)

Yes

Vest, et al., 2012101 Yes Yes Yes Yes Yes Yes

Vest and Jasperson,

2012103

No comparison

group; time period

appropriate

Yes Yes Yes Yes Yes

Vest, et al., 2013102 Unclear Yes No Yes No No

Vest, et al., 201455 Yes Yes Yes Yes Yes Yes

Vest, et al., 201456 Yes Yes No Yes Yes Yes

Winden, 201471 No Yes No Unclear Unclear Yes
I-8

Author, Year

Was there no

missing data? If 

missing data, was it 

handled 

appropriately?

Were outcomes

prespecified and

were prespecified

outcomes 

reported? Risk of bias

Tzeel, Lawnicki, and

Pemble, 201253

Yes Yes Low

Tzeel, Lawnicki, and

Pemble, 201152

Yes Yes Low

Vest, 200954 Yes Yes Low

Vest, 2010155 Yes Yes Low

Vest, et al., 2011104 Not clear Yes Low

Vest, et al., 2011105 Not clear Yes Low

Vest and Miller,

201164

Unclear Yes Low

Vest, et al., 2012101 Unclear Yes Low

Vest and Jasperson,

2012103

Yes Yes Low

Vest, et al., 2013102 Not clear Yes Low

Vest, et al., 201455 Not clear Yes Low

Vest, et al., 201456 Not clear Yes Low

Winden, 201471 No Yes Moderate

NA= not applicable; NR = not relevant.
I-9

Table I-3. Quality assessment of surveys, focus groups, and interview studies

Author, Year

1. Is the sampling 

strategy or

selection criteria

reported and

appropriate?

2. Are the response or participation

rates reported and are they

acceptable given the type of study?

3. Are characteristics (e.g.,

demographics) of

respondents/participants

reported?

4. Is how the questions were

developed/selected reported and is it

appropriate?

Abramson, et al.,

201276

Yes Yes, 72% Yes, hospitals in New York

State

Yes

Abramson, et al.,

201477

Yes 59.3% (375/632) response rate Yes, nursing homes in New

York State

Yes

Abramson, et al.,

201496

Yes Yes Yes Yes

Adler-Milstein, et al.,

200881

Yes Yes, 60% Yes

Adler-Milstein, Bates, 

and Jha, 200978

Yes Yes, 78% Yes, operational RHIOs Yes, pilot testing

Adler-Milstein,

Landefeld, and Jha, 

201080

Yes Yes, 83% Yes, operational RHIOs Yes

Adler-Milstein, Bates, 

and Jha, 201179

Yes Yes, 84% Yes, operational RHIOs Yes

Adler-Milstein,

DesRoches, and Jha, 

2011107

Yes Yes - 69% Yes Yes

Adler-Milstein, Bates, 

and Jha, 201325

Yes Yes, 78% Yes, operational RHIOs Yes, pilot testing

Adler-Milstein and Jha, 

2014108

Yes Yes Yes Yes

Altman, et al.,

201257

Unclear; 

convenience sample

Yes, 70% (14/20) Yes Yes

Audet, Squires, and

Doty, 2014109

Yes Yes, 35% Yes Yes

Caffrey and Park- Lee

201393

Yes Yes Yes Yes
I-10

Author, Year

5. Were confounders

considered? (could be in 

analysis or presentation,

such as stratifying

results)

6. Is analysis appropriate?

(given the type of data)

Risk of bias

Abramson, et al.,

201276

Unclear Yes Low

Abramson, et al.,

201477

Unclear Yes Low

Abramson, et al.,

201496

Yes Yes Moderate

Adler-Milstein, et al.,

200881

Unclear Yes Low

Adler-Milstein, Bates, and

Jha, 200978

Unclear Yes Low

Adler-Milstein, Landefeld, 

and Jha, 201080

Unclear Yes Low

Adler-Milstein, Bates, and

Jha, 201179

Unclear Yes Low

Adler-Milstein, DesRoches, 

and Jha, 2011107

Yes Yes Low

Adler-Milstein, Bates, and

Jha, 201325

Unclear Yes Low

Adler-Milstein and Jha, 

2014108

Unclear Yes Low

Altman, et al.,

201257

NA, descriptive

interviews

Mostly descriptive results

presented

Moderate

Audet, Squires, and Doty,

2014109

Unclear Yes Low

Caffrey and Park- Lee

201393

Yes Yes Low
I-11

Author, Year

1. Is the sampling 

strategy or selection

criteria reported and

appropriate?

2. Are the response or

participation rates reported

and are they acceptable

given the type of study?

3. Are characteristics

(e.g., demographics) of

respondents/participan

ts reported?

4. Is how the questions were

developed/selected reported and is it

appropriate?

Campion, et al.,

201258

Yes Yes (19%) Yes Yes

Codagnone,

LupiañezVillanueva 201394

Yes Yes Yes Yes

Chang, et al.,

201059

No No, 9 primary care

physicians selected for 

Yes yes

Dixon, Miller, and

Overhage, 2013141

Yes Yes Yes Yes

Dixon, Jones, and

Grannis, 201383

Yes Yes, 69% (44/63) "Infection preventionists" Yes, pilot administration with

modification of survey

Fairbrother, et al.,

2014143

Yes NR but these were interviews Yes NR

Finnell and 

Overhage,

2010131

Yes Yes, 32% response rate Yes Unclear. Survey not well described.

Foldy, 200784 Unclear-basically

asked experts whom to

ask

Yes No NR - survey URL broken

Fontaine, et al.,

201085

Yes NR NR Yes

Furukawa,

2 014110

Yes Yes Yes Unclear

Furukawa,

2013111

Yes Yes No Yes

Gadd, et al.,

201186

Yes Yes, email survey responses

from with 70% response rate

from health care

professionals (165/237).

Yes Yes

Genes, et al.,

2011145

Yes Yes, 18/22 participated in

interviews

Yes Yes

Goldwater, et al.,

2014146

Yes Yes for interviews. 20%

response to emailed survey.

No NR
I-12

Author, Year

5. Were confounders

considered? (could 

be in analysis or

presentation, such as

stratifying results)

6. Is analysis

appropriate? (given

the type of data) Risk of bias

Campion, et al.,

201258

Yes Yes Moderate

Codagnone,

Lupiañez- Villanueva

201394

Yes Yes Low

Chang, et al.,

201059

No, descriptive only Yes Moderate

Dixon, Miller, and

Overhage, 2013141

No Yes Moderate

Dixon, Jones, and

Grannis, 201383

Unclear Yes Moderate

Fairbrother, et al.,

2014143

NA NR High

Finnell and 

Overhage, 2010131

NA Yes, descriptive only Moderate

Foldy, 200784 No Yes Moderate

Fontaine, et al.,

201085

No Yes Moderate

Furukawa, 2014110 Yes Yes Low

Furukawa, 2013111 Yes Yes Low

Gadd, et al., 201186 Yes Yes Low

Genes, et al., 2011145 NA Yes Low

Goldwater, et al., 2014146 NA Yes, descriptive only Moderate
I-13

Author, Year

1. Is the sampling 

strategy or selection

criteria reported and

appropriate?

2. Are the response or

participation rates reported and

are they acceptable given the type

of study?

3. Are characteristics (e.g.,

demographics) of

respondents/participants

reported?

4. Is how the questions were

developed/selected reported and is it

appropriate?

Greenhalgh, et

al., 2010121

Yes No No Yes

Hamann and 

Bezboruah,

2013113

Yes Yes NA Yes

Hessler, et al.,

200987

Yes No Yes Yes

Hincapie, et al.,

2011132

Yes Yes No, no table of participants.

Types of providers were

mentioned with qualitative

themes.

Yes

Hyppönen, et al.,

2014133

Yes Yes Yes Yes

Jha, et al.,

2008117

Yes No No No

Kaushal, et al.,

201060

No Yes No Yes

Kern, et al.,

2009173

Yes Yes Yes Yes

Kern, et al.,

2011171

No Yes No Yes

Kierkegaard, 

Kaushal, and 

Vest, 2014127

Yes NA Yes, characteristics of sites

reported and types of HIE

users are described but not

quantified.

NR

Lee, et al.,

201289

Unclear (post given to

all, for pre this is 

unclear)

No (rate given but low; only collected

for 2 weeks)

Yes Yes

Machan,

Ammenwerth,

and 

Schabetsberger,

200662

Yes, questionnaire

sent to all practitioners

registered in HIE

project.

Yes, 43% (104/242) practitioners

responded.

Yes, physician users of HIE. Yes, development process for

interviews guide and questionnaire

described thoroughly. No

psychometrics presented.
I-14

Author, Year

5. Were confounders

considered? (could be

in analysis or

presentation, such as

stratifying results)

6. Is analysis

appropriate? (given the

type of data) Risk of bias

Greenhalgh, et

al., 2010121

Yes Yes Low

Hamann and 

Bezboruah,

2013113

Yes Yes Low

Hessler, et al.,

200987

No No High

Hincapie, et al.,

2011132

NA Yes Moderate

Hyppönen, et al.,

2014133

Yes Yes Low

Jha, et al.,

2008117

No Unclear High

Kaushal, et al.,

201060

No Yes High

Kern, et al.,

2009173

No Yes Moderate

Kern, et al.,

2011171

No No High

Kierkegaard, 

Kaushal, and 

Vest, 2014127

NA Yes, coded interviews

with Nvivo

Moderate

Lee, et al.,

201289

Yes No High

Machan,

Ammenwerth,

and 

Schabetsberger,

200662

No, only descriptive

analysis

Yes, descriptive analysis

only.

Low
I-15

Author, Year

1. Is the sampling 

strategy or selection

criteria reported and

appropriate?

2. Are the response or

participation rates reported and

are they acceptable given the type

of study?

3. Are characteristics (e.g.,

demographics) of

respondents/participants

reported?

4. Is how the questions were

developed/selected reported and is it

appropriate?

MassyWestropp, et al.,

2005134

Yes, convenience

sample of 82 users of

HIE and then

additional sample of

50 providers not in HIE

program as controls.

Reported as 42% (55/80) but this

doesn’t account for 50 controls so

the response rate is 24% (55/132).

No No

Maass, et al.,

200861

Yes, only 1 person

interviewed

NR NR NR

McCullough, et

al., 2014135

Yes, used purposive

sample strategy

Yes, reported recruitment rate of

practices.

Yes Yes

Merrill, et al.,

2013174

Yes Yes Yes Yes

Messer, et al.,

2012138

Yes, interviews and

assessment with

39 stakeholders; pre

and post survey of 29

providers' satisfaction

with HIE, relationships

with other providers,

barriers.

NR, it is not clear how many surveys

were sent out to compute a response

rate.

No Yes

Miller, 2012162 Yes NR, but these were interviews NR Yes, questions developed jointly by the

University of California, San Francisco,

and Consumers Union

Myers, et al.,

2012128

Yes, used purposive

sample strategy

Yes, 62/102 emailed invitations to

survey

Yes for key respondents. No

for survey.

Yes, developed after literature review.

Reported Chronbach alphas of .57-.97

for scaled items.
I-16

Author, Year

5. Were confounders

considered? (could be

in analysis or

presentation, such as

stratifying results)

6. Is analysis

appropriate? (given the

type of data) Risk of bias

MassyWestropp, et al.,

2005134

No NA High

Maass, et al.,

200861

No Yes High

McCullough, et

al., 2014135

NA Yes Low

Merrill, et al.,

2013174

Yes Yes Low

Messer, et al.,

2012138

NA Yes, for qualitative and

quantitative.

Moderate

Miller, 2012162 NA Yes Moderate

Myers, et al.,

2012128

Stratified by role Yes Low
I-17

Author, Year

1. Is the sampling 

strategy or selection

criteria reported and

appropriate?

2. Are the response or participation

rates reported and are they

acceptable given the type of study?

3. Are characteristics (e.g.,

demographics) of

respondents/participants

reported?

4. Is how the questions were

developed/selected reported

and is it appropriate?

Nøhr, et al., 2001139 Yes Yes Yes NR/Yes

Nykänen and Karimaa, 

2006150

Yes Yes No Yes

Ozkaynak and Brennan,

2013129

Yes NA Yes NR

Pagliari, Gilmour, and

Sullivan, 2004122

Yes Yes No No

Patel, et al., 201363 Yes Yes Yes Yes

Park, et al., 201363

Yes Yes Yes Yes

Phillips, et al.,

2014164

Yes NA No Yes

Pirnejad, Bal, and

Berg, 2008152

Yes for RN surveys; No

for interviews

Yes Yes Yes for surveys - published

surveys used to identify

questions; No for interviews

Ross, et al.,

2010167

Yes Yes No Yes

Rudin, et al.,

2009153

Yes Not reported No Yes

Rudin, et al.,

2011136

Yes NR Yes Yes/Yes

Schoen, et al.,

201295

Yes Yes No No

Sicotte and Paré, 2010168 Yes Yes NR Yes

Steward, et al.,

2012169

Yes NR but these were interviews NR Partnered with UCSF

qualitative experts to
I-18

Author, Year

5. Were confounders

considered? (could be in 

analysis or presentation,

such as stratifying

results)

6. Is analysis

appropriate? (given

the type of data) Risk of bias

Nøhr, et al., 2001139 NA Yes Moderate

Nykänen and

Karimaa, 2006150

No Yes Moderate

Ozkaynak and

Brennan, 2013129

NA Yes Moderate

Pagliari, Gilmour, and

Sullivan, 2004122

No Yes Moderate

Patel, et al., 201363 Yes Yes Low

Park, et al., 201363

No Yes Low

Phillips, et al., 

2014164

Yes Yes Low

Pirnejad, Bal, and

Berg, 2008152

No Yes Moderate

Ross, et al.,

2010167

No Yes Moderate

Rudin, et al.,

2009153

No Yes Moderate

Rudin, et al.,

2011136

NA Yes Low

Schoen, et al., 201295 No Yes High

Sicotte and Paré,

2010168

Yes Yes Low

Steward, et al.,

2012169

NA Yes Moderate
I-19

Author, Year

1. Is the sampling 

strategy or selection

criteria reported and

appropriate?

2. Are the response or

participation rates reported and

are they acceptable given the type

of study?

3. Are characteristics (e.g.,

demographics) of

respondents/participants

reported?

4. Is how the questions were

developed/selected reported and is it

appropriate?

Soderberg and

Laventure,

201390

Yes Yes No Unclear

Thorn, Carter,

and Bailey,

2014130

Yes, used purposive

sample strategy

Yes, mentioned all physicians agreed

to participate and no one dropped

out.

Yes Types of questions mentioned but no

mention of interview guide.

Unertl, et al.,

2013170

Yes Yes NR Yes

Unertl, Johnson,

and Lorenzi,

2012119

Yes NA Yes. Characteristics of sites

and interviewees described.

NR in main text but mentioned online

appendix but no link to access it.

Yeager, et al.,

2014137

Yes Yes Yes No/Yes. Types of questions mentioned

but no mention of interview guide.
I-20

Author, Year

5. Were confounders

considered? (could be

in analysis or

presentation, such as

stratifying results)

6. Is analysis

appropriate? (given the

type of data) Risk of bias

Soderberg and

Laventure,

201390

Yes Yes Moderate

Thorn, Carter,

and Bailey,

2014130

NA Yes Low

Unertl, et al.,

2013170

Yes Yes Low

Unertl, Johnson,

and Lorenzi,

2012119

NA Yes. Coded interviews

with Nvivo

Moderate

Yeager, et al.,

2014137

NA Yes Moderate

HIE= health information exchange; NA= not applicable; NR= not relevant; RHIO= regional health information organization;

RN= registered nurse; UCSF= University of California, San Francisco; URL= uniform resource locator.