Evaluation of the Flexibility Critical Access Hospital Health 

Information Technology Network Implementation Program 

Final Report

December 2010

U.S. Department of Health and Human Services

Health Resources and Services Administration
Prepared for the U.S. Department of Health and Human Services, Health Resources and Services 

Administration, Office of Rural Health Policy under contract with the Altarum Institute. 
Evaluation of the Flexibility Critical Access Hospital Health 

Information Technology Network Implementation Program 

Final Report

December 2010
FLEX CAH HIT Evaluation- Final Report i

Table of Contents

Section Page

Table of Contents................................................................................................................................i

I. Executive Summary.........................................................................................................................1

II. Introduction...................................................................................................................................6

III. Methods.................................................................................................................................9

IV. Analysis and Results..............................................................................................................15

Case Study 1: Flex CAH HIT Network Implementation Grant, University of North Dakota Center for 

Rural Health.....................................................................................................................................35

Case Study 2: Northeastern Oklahoma CAH EHR Network, Oklahoma State University Center for 

Health Sciences................................................................................................................................49

Case Study 3: Lakelands Rural Health Network HIE, South Carolina Office of Rural Health..................70

Case Study 4: Rural Wisconsin Health Cooperative Information Technology Network, Board of Regents 

of the University of Wisconsin System..............................................................................................88

V. Recommendations from Grantees and Conclusion ......................................................................110

Appendix A: Telephone Interview Protocols ...................................................................................114

Appendix B: CAH-HIT Site Visit Interview Protocol ..........................................................................119

Appendix C: Value-Based Return on Investment (ROI) Protocol and Tools .......................................126

Value-Based Return on Investment Analysis...................................................................................142

Appendix D: Flex CAH HIT Grantee Profiles.....................................................................................145

This publication lists non-federal resources in order to provide additional information to consumers. The 

views and content in these resources have not been formally approved by the U.S. Department of 

Health and Human Services (HHS). Listing these resources is not an endorsement by HHS or its 

components. 
ii
FLEX CAH HIT Evaluation- Final Report 1

I. Executive Summary

Nearly 60 million Americans live in rural areas1 and face challenges in accessing high quality health care. 

This represents the vast majority of the approximately 62 million Americans that live in communities 

with shortages of primary care providers.2

The application of HIT has the potential to transform the 

landscape of how health information is collected, 

managed, stored, used, and shared, and has special 

potential to affect the Nation’s rural communities. HIT 

has the potential to help remote communities 

coordinate care, improve disease surveillance, target 

health education and compile regional data, all activities 

that can improve health care and health nationwide. At the same time, rural communities face 

particular challenges to implementing HIT, given the geographic dispersion of its population and the 

many resulting implications for the delivery of health care.

 As policymakers, health care practitioners, and the public 

increasingly demand more widespread use of electronic medical records (EMRs) and health information 

technology (HIT) to increase efficiency, enhance patient safety, and improve coordination of care, 

studies that evaluate its implementation and effects are essential. This is of particular importance in 

rural settings where these challenges are amplified. To this end, the Health Resources and Services 

Administration (HRSA), Office of Rural Health Policy (ORHP) utilized $25 million in one-time funding to 

support 16 rural grantees to develop and implement HIT pilot networks in an 18-month timeframe. This 

report describes an evaluation of these grantees that were charged with designing, creating, and 

implementing functional pilot networks to improve coordination of care in their communities, and 

provide lessons learned for future providers and networks in adopting HIT. The outcomes of this grant 

program are ideally timed to inform and impact the unprecedented HIT funding and programmatic 

supports funded as part of the American Recovery and Reinvestment Act of 2009 (ARRA, or the Stimulus 

Bill) which provides for a total of $19.2 billion for HIT 

programs and incentives. 

OHRP funded grantees through its Medicare Rural Hospital Flexibility (Flex) Critical Access Hospital (CAH) 

Health Information Technology (HIT) Network Implementation 

Program, which promotes the implementation of HIT in CAHs 

and their associated network of providers in States that are 

current Medicare Flex grantees. The grant program funded 

grantees to establish HIT systems, but allowed them to use 

these funds in a flexible way. As a result, some grantee hospital programs already had systems in place and 

were able to build upon and enhance them using grant dollars; absent the funding, some programs would 

never have been able to establish an HIT program. Each grantee was at a different level of maturity when 

the program began, and the grant program allowed each to establish a new system or build upon an 

existing one. 

 

1 U.S. Department of Health Human Services, Health Resources and Services Administration. (2009). Office of Rural Health 

Policy 2008 Annual Report. Retrieved from http://ruralhealth.hrsa.gov/pub/2008annualreport/orhp2008annualreport.pdf. 

2 Ruddy, G., Fryer, G., Phillips, R., Green, L., Dodoo, M., &McCann, J. (2005). The family physician workforce: the special case of 

rural populations. American Family Physician 72(1), 147. Retrieved from http://www.grahamcenter.org/online/graham/home/publications/onepagers/2005/op31-rural-populations.html

“HIT is not an IT project; it’s a clinical 

project that’s supported by IT.”

—Flex CAH HIT grantee

Challenges of Rural Communities and HIT 

- Geographic dispersion of population

- Limited infrastructure, workforce, capital 

and other resources

- Service and provider mix

- Socioeconomic status of patient population
FLEX CAH HIT Evaluation- Final Report 2

Grantees worked on a range of HIT projects, including systems for practice management, disease 

registry, care management, clinical messaging, personal health records, electronic health records, and 

health information exchanges. All shared the common goal 

of using HIT as a tool to improve the safety, quality, 

efficiency and effectiveness of health care delivery by 

ensuring that clinical information of patients served by the 

network is accessible to all providers across the continuum 

of care, from ambulatory care to acute delivery sites. It is 

too early in the process to measure the impact of the grant 

on patient outcomes; most grantees are in the 

implementation phase of the process. However, the ultimate desire and belief is that the use of HIT 

systems will ultimately lead to better health outcomes through improved quality of care—quality that 

improves as electronic management systems promote coordination of information and care treatment 

plans, offer clinicians real-time responses about their patients and national treatment standards, and 

reduce medical errors associated with paper records. 

The Flex CAH HIT program concentrated funding within smaller service areas to increase the likelihood 

of creating sustainable pilot projects. Grant applicants were required to identify up to three CAHs and 

their associated network of providers that together provide a 

full continuum of care for rural residents in their service area. 

To build upon existing patterns of care, it was critical that the 

Flex applicant propose a network that followed common 

patient referral patterns. The network could include local 

partners for the CAH, including but not limited to tertiary / 

referral hospitals, private practice physicians, Medicarecertified Rural Health Clinics (RHCs), Community Health Centers (CHCs) / Federally Qualified Health 

Centers (FQHCs), nursing facilities, home health agencies, public health departments, and emergency 

medical service providers. 

A. Evaluation Design 

A multi-method evaluation approach was developed to achieve the following evaluation goals: 

• Determine the extent to which grantees are able to successfully implement the activities proposed 

in their application;

• Describe the supports and barriers grantees faced in reaching their stated goals and objectives; and

• Describe the project characteristics, and the supportive elements necessary for projects to fulfill the 

goals of the Flex CAH HIT Program, in order to inform future grant funding cycles.

This evaluation utilized the data already collected by the grantees for administrative, monitoring, and 

evaluation purposes. It also collected additional data to support the goals of the evaluation, such as 

return on investment (ROI) and systems analysis. Data collected from each grantee through key 

informant telephone interviews conducted every 3 months supplemented these secondary data. The 

grant administrator participated in each of these quarterly calls which took place from September 2008 

through July 2009. 

In addition, 2-day site visits were conducted with four grantees (University of North Dakota Center for 

Rural Health, Oklahoma State University Center for Health Sciences, South Carolina Office of Rural 

Health, and Board of Regents of the University of Wisconsin System) to explore in-depth their 

administration of the Flex CAH HIT grant, analyze the HIT system implemented, understand the success 

“Small hospitals support a culture of 

working together where people wear 

different hats [and] there’s a much more 

community-oriented environment and 

agreeable attitudes.”

—Flex CAH HIT grantee

“By working together instead of trying to 

be separate all the time, we can show 

rural America that collaboration works and 

is sustainable.”

—Flex CAH HIT grantee
FLEX CAH HIT Evaluation- Final Report 3

and challenges of implementation, and study initial outcomes. Data collection consisted of group 

interviews with staff involved in the implementation of the HIT system. 

B. Analysis and Results

Analysis of these data indicates that many factors 

contributed to supporting the various HIT networks, 

including leadership from health care administrators and 

clinicians, stakeholder buy-in, project design, IT expertise, 

State and national interest in the projects, and supports specific to the CAH environment. At the same 

time, groups encountered common barriers to implementation, including technical issues (i.e., problems 

with interfaces and supports from vendors), unanticipated workforce issues, health system and CAHlevel barriers, and challenges related to financing and administering programs. In addition, some 

grantees experienced barriers because of communication and leadership issues within the networks as 

well as with vendors for scope of work issues.

Grantees learned many common lessons from the process of creating and implementing their HIT 

networks. Chief among these is the importance of good project management that focuses on the 

process of change management and achieving the project’s primary goals. In addition, financial planning 

and fiscal management are essential; administrators must recognize the magnitude of costs associated 

with such a project and carefully manage its costs. Time management is another key element of project 

management; HIT projects can be much more time-consuming than planners anticipate, and budgeting 

time and resources adequately is critical. Grantees pointed to the importance of early planning, which 

can help a project advance and ensure that existing systems and workflows can be translated into digital 

records. Staff buy-in and training are essential to engaging staff, ensuring their ongoing participation and 

support while giving them the tools and information they need to use the HIT effectively. Finally, 

grantees noted that communication and coordination among partners is essential to managing such a 

large and complex project. 

Grantees reported that that implementation of HIT networks had an effect on operations and data 

collection and utilization, including the increased availability of and access to patient data; increased 

provider collaboration; improved physician satisfaction with the ability to access data remotely; and 

improved patient outcomes stemming from better access to charts and improved care coordination.

The evaluation examined supports for and barriers to project sustainability. While some grantees 

reported that they had budgeted for sustainability, including recruiting new partners to the network for 

cost-sharing, others have encountered barriers to funding. Even so, many reported having sought 

additional funding from Federal, State, and private sources.

Grantees were asked to describe whether and how HRSA had been helpful to them throughout the 

process. Most reported that HRSA had been very helpful, particularly with monthly group calls, technical 

assistance, support to work with vendors, and national and regional meetings. 

C. Recommendations

Several recommendations emerged from the results of this evaluation, particularly in terms of grant 

administration and technical support to grantees. Throughout the grant period, grantees experienced 

many challenges in simply administering the grant; future programs could be designed to limit or 

eliminate these challenges. In addition, grantees noted that although they often needed technical 

support or assistance, they were uncertain of how to find or access resources available to them. 

Collectively, these lessons learned and resulting recommendations are relevant for future grantees in 

implementing HIT – the facilities and staff involved in these projects have experience that can be useful 

“You can’t really understate the [need for] 

leadership in working with neighbors in 

the region.”

—Flex CAH HIT grantee
FLEX CAH HIT Evaluation- Final Report 4

to future HIT projects within their singular facility and within their networks. This experience is critical in 

the post-ARRA world of many rapid HIT and HIT-related projects being funded. Additionally, these 

recommendations should serve as points of consideration and revision for future iterations of Flex CAH 

HIT or similar funding.

Grantees emphasized that they had experienced several challenges in administering the grant, including, 

the limited timeframe within which to complete the project, governance issues, unclear evaluation 

expectations, and issues sustaining the activities of the grant. Grantees also raised a series of technical 

support issues. Although many grantees are aware that technical assistance (TA) resources exist, 

knowledge of these resources and how to access them varied widely; access often depends on self 

determination to uncover or utilize one-size-fits-all resources. Some grantees felt that rural grantees 

may have a difficult time articulating their needs or do not know all the relevant questions to ask. In 

addition, many have limited access to resources such as quality broadband services. Collectively, these 

and other grantee needs may best be served by an HIT Primer – a toolkit designed specifically for CAHs 

to aid them in adopting an HIT-based application or system. This HIT Primer should focus on rural health 

settings and in particular CAHs, and build upon other tools such as those encompassed in HRSA’s Health 

IT Adoption Toolbox and the Rural Assistance Center (RAC) guides. Such a Primer would include specific 

engagement and selection modules to address many of the TA needs described below in more detail. 

In short, future HIT implementation programs should consider the following programmatic and resource 

recommendations to benefit CAHs in adopting HIT overall, and to specifically benefit the Flex CAH HIT 

program in future iterations.

Consider a longer grant funding period for future HIT planning and implementation grants. Many 

grantees stated that they needed more time to complete the project. Many found the combination of 

planning and implementation activities expected to be covered in an 18-month period of performance 

to have been unrealistic. 

Acknowledge potential governance issues and facilitate resolution. Grantees reported having widely 

varied governance structures, within which real and perceived challenges existed. While many reported 

experience working with smaller CAHs in an existing collaborative, some found the addition of larger 

hospitals or health systems to their networks to be problematic. Future grantees must be more fully 

cognizant of these potential governance issues. HRSA must be ready to support meaningful mitigation of 

such issues throughout the grant process.

Emphasize the importance and value of ongoing evaluation and clearly state expectations. Grantee 

evaluation intent, methods and execution varied widely across sites. Many sites were conducting or 

planned to conduct staff (user) and/or patient satisfaction surveys. For evaluation to be meaningfully 

conducted by future Flex CAH HIT grantees, HRSA should clearly communicate expectations and intent of 

internal evaluation, and provide technical assistance (evaluation plan examples, selecting appropriate 

measures, data collection methods, etc.) where possible and appropriate. In turn, grantees must 

understand the importance of evaluation and commit to make it a central component of their project 

plans and execution.

Support broader sustainability planning. HIT implementation involves many unknowns that can be 

costly from a financial (and other resource) perspective. Some types of sustainability may be more of an 

acute issue for rural hospitals – where budgets are small and staff and infrastructure resources are also 

seriously constrained. HRSA can help grantees consider different types of sustainability issues –

financial and other resource constraints, operational or workflow challenges, and ways to mitigate these 

challenges. HRSA should also encourage and support grantees considering the total cost of ownership 

(financial and other costs) for HIT systems, rather than focusing on initial purchase prices. 
FLEX CAH HIT Evaluation- Final Report 5

Expand vendor selection, procurement and implementation assistance. Grantees experienced a wide 

range of satisfaction with their selected vendors, and noted specific times and situations during the vendor 

engagement where additional help from HRSA would be extremely helpful. For vendor selection, grantees 

commented they were unsure of what and how to ask vendors meaningful questions tailored to the site’s 

circumstances. Many grantees suggested that HRSA is ideally suited to actively engage with grantees to 

keep expectations between sites and vendors mutually reasonable, and to be an advocate for the 

collective grantees in navigating vendor interactions. They also suggested that HRSA could more readily 

provide or point grantees to resources to develop a site’s needs assessment, RFQ/RFP development, 

vendor selection process, vendor evaluation /comparison tools and contract development tools.

Require key staff succession planning and adequate staff education/training. Grantees identified a 

number of workforce issues that affected project success – namely staff retention and turnover. While 

grantees acknowledged that staff shortages and recruitment challenges in a rural environment are 

inevitable, they also pointed to turnover training and knowledge retention as aspects of larger 

workforce issues that could be readily improved. Grantees need to consider what happens if there is 

turnover in a key position. This “succession planning” can help ensure that not one person wholly holds 

critical information, and that key staff can be effectively replaced. HRSA should provide grantees with 

succession planning tools for key staff, and explicitly encourage more staff education with specific tools 

and estimations for such training, tailored to rural environments.

Establish a mechanism for cross-grantee collaboration, best practices and networking. Throughout the 

evaluation, grantees were appreciative of the networking and collaboration opportunities facilitated by 

HRSA, but grantees consistently wanted additional ways to learn from each other, work together and 

share information. Also, grantees requested the ability to work with future grantees, in the event that 

additional rounds of Flex CAH HIT are funded. Whatever the collaboration mechanism is, grantees 

would benefit from building a stronger consortium across rural care facilities.

Additional supports for successful implementation. Grantees were aware that implementation consists 

of more than just buying a product and installing it, but often noted a lack of confidence in handling 

some of the larger issues inherent in large system implementations. HRSA can support “change 

management” procedures in future grants – to not only aid in a successful system implementation, but 

to uncover dysfunctional procedures and fix them. 

The Flex Program itself helps to sustain the rural healthcare infrastructure, relying on the Critical Access 

Hospital as the hub of an organized system of care. To this end, the Flex Program fosters the growth of 

collaborative rural delivery systems across the continuum of care. Robust, useful and usable HIT systems 

that coordinate patient and treatment information while respecting privacy, and that enable inter-setting 

health care information exchange, are critical to building an infrastructure that supports patients and leads 

to the best possible outcomes for them. The remainder of this report chronicles this model effort to 

establish and sustain HIT systems in 16 rural communities. The report describes the performance of 

grantees as they designed, created, and implemented functioning CAH HIT pilot networks, and describes 

the experiences of partner organizations in this effort. In all, theirs is a remarkable story of success, one in 

which some organizations reported having been entirely paper-based—indeed, with some reporting that 

their clinicians could not even turn on or operate a computer—to moving toward an entirely electronic 

system of medical and health records accessible by providers throughout a rural region. Absent the 

commitment, funding, and support of HRSA, such successes would not have occurred. This successful pilot 

program will point the way for subsequent projects to promote HIT not only in rural networks, but in other 

systems that might be inspired by and learn from the stories told here.
FLEX CAH HIT Evaluation- Final Report 6

II. Introduction

A. Background

The Office of the National Coordinator for Health Information Technology defines health information 

technology (HIT) as “the application of information processing involving both computer hardware and 

software that deals with the storage, retrieval, sharing, and use of health care information, data, and 

knowledge for communication and decision making.” 3

Department of Health and Human Services (DHHS) 

Secretary Kathleen Sebelius states that “expanding the 

use of electronic health records is fundamental to 

reforming our health care system...electronic health 

records can help reduce medical errors, make health 

care more efficient and improve the quality of medical 

care for all Americans.” DHHS has identified furthering 

the use of HIT as a key priority.4 This focus aligns with 

the President’s goal of universal adoption of electronic 

health records for all Americans within five years. 5

The application of HIT has the potential to transform 

the landscape for how health information is collected, 

managed, stored, and shared, and has particular 

potential to affect the Nation’s rural communities. 

According to the National Advisory Committee on 

Rural Health and Human Services (NACRHHS), HIT can

“help disparate rural providers from across the 

spectrum of care to better coordinate care for their patients...help rural communities improve public 

health through disease surveillance and targeted health education... compile regional data that can be 

used to improve the health of rural Americans [and] improve the quality of health care not just in rural 

 

 

3 The Office of the National Coordinator for Health Information Technology. (n.d.). In glossary of selected terms related to ehealth. Retrieved from 

http://healthit.hhs.gov/portal/server.pt?open=512&objID=1256&parentname=CommunityPage&parentid=9&mode=2&in_hi_u

serid=10741&cached=true

4 Office of the Vice President, The White House. (2009, November 17). Vice President announces availability of nearly $1.2 

billion in grants to help hospitals and doctors use electronic health records [Press release]. Retrieved from 

http://www.whitehouse.gov/the_press_office/Vice-President-Biden-Announces-Availability-of-Nearly-12-Billion-in-Grants-toHelp-Hospitals-and-Doctors-Use-Electronic-Health-Records/

5

Goedert, J. (2009, January 8). Obama calls for all Americans to have EHRs within five years. Health Data Management. 

Retrieved from: http://www.ihealthbeat.org/Articles/2009/1/9/Obama-Calls-for-All-Americans-To-Have-EHRs-Within-FiveYears.aspx

Anatomy of a Critical Access Hospital (CAH)

- Medicare Rural Hospital Flexibility Program 

(Flex Program) was created by the Balanced 

Budget Act of 1997

- Flex Program provides grants to implement 

a state’s CAH program and to encourage 

the development of rural health networks

- Cost-based reimbursement from Medicare 

(cost plus 1%)

- CAHs are licensed acute care hospitals at 

least 35 miles from another hospital, or 15 

miles from another hospital in mountainous 

terrain or areas with only secondary roads 

- Maximum of 25 acute care inpatient beds

- 24-hour emergency services
FLEX CAH HIT Evaluation- Final Report 7

communities, but also across America.” 6 According to the 2008 American Hospital Association survey, 

1,998 (40%) of the 5,010 community hospitals are rural hospitals. 7

 Well over half of these rural 

hospitals (1,305) have less than 25 beds and are designated as Critical Access Hospitals (CAH)that bill for 

services to Medicare beneficiaries on a cost basis. 8

However, CAHs face challenges in adopting HIT, which lead to lower HIT adoption rates in rural 

communities.9

 For example, only 20 percent of CAHs have some form of an electronic health record, while 

only 25 percent use electronic prescription order entry. 10 Rural health care systemsface particular 

challenges; their infrastructure, service and provider mix, health status, geographic dispersion of population

and socioeconomic characteristics, are often different and unique to rural communities—rural health 

systems “are not just a small version of suburban and urban health systems.” 11 The NACRHHS notes that 

while “rural communities face many challenges in adopting HIT, including limited access to capital and 

infrastructure, lack of workforce expertise and difficulty in obtaining community buy-in” they “also have 

strengths that may facilitate HIT adoption, including the smaller size and less complex nature of rural health 

care systems.”12

The U.S. Department of Health and Human Services (HHS), Health Resources and Services 

Administration’s (HRSA) Office of Rural Health Policy (ORHP) is charged with informing and advising 

DHHS on matters affecting rural hospitals and health care, coordinating activities within the department 

that relate to rural health care (with particular attention to the Centers for Medicare and Medicaid 

Services and its programs), and maintaining a national information clearinghouse. As part of its mission, 

and to further HIT adoption in rural communities, OHRP funded the Medicare Rural Hospital Flexibility 

(Flex) Critical Access Hospital (CAH) Health Information Technology (HIT) Network Implementation 

Program. The Flex CAH HIT Program promotes the implementation of HIT in CAHs and their associated 

network of providers in States that are current Medicare Flex grantees.

 

13

 

6 The National Advisory Committee on Rural Health and Human Services. (2006). The 2006 report to the Secretary: Rural health 

and human services issues. Retrieved from ftp://ftp.hrsa.gov/ruralhealth/NAC06forweb.pdf

7 American Hospital Association. (2009). Fast Facts on U.S. Hospitals [Fact Sheet]. Retrieved from 

http://www.aha.org/aha/resource-center/Statistics-and-Studies/Fast_Facts_Nov_11_2009.pdf

8 A Critical Access Hospital is a hospital that is certified to receive cost-based reimbursement from Medicare. The 

reimbursement that CAHs receive is intended to improve their financial performance and thereby reduce hospital closures. 

Each hospital must review its own situation to determine if CAH status would be advantageous. CAHs are certified under a 

different set of Medicare Conditions of Participation (CoP) that are more flexible than the acute care hospital CoPs. See the 

following for more information: http://www.raconline.org/info_guides/hospitals/cahfaq.php#whatis. 

9 The Rural Health Research Centers at the University of Minnesota, North Carolina-Chapel Hill, and Southern Maine constitute 

the Flex Monitoring Team. The Team is the recipient of a 5-year cooperative agreement award from the Federal Office of Rural 

Health Policy to continue to monitor and evaluate the Medicare Rural Hospital Flexibility Program (the Flex Program). The 

monitoring project is assessing the impact of the Flex Program on Critical Access Hospitals and their communities and the role 

of States in achieving overall program objectives. See the following for more information: http://www.flexmonitoring.org/

10 Flex Monitoring Team. (2006). The current Status of health information technology use in CAHs (Briefing Paper No. 11). 

Retrieved from http://www.flexmonitoring.org/documents/BriefingPaper11_HIT.pdf

11 The National Advisory Committee on Rural Health and Human Services. (2006). The 2006 report to the Secretary: Rural health 

and human services issues. Retrieved from ftp://ftp.hrsa.gov/ruralhealth/NAC06forweb.pdf

12 The National Advisory Committee on Rural Health and Human Services. (2008). The 2008 report to the Secretary: Rural health 

and human services issues. Retrieved from ftp://ftp.hrsa.gov/ruralhealth/committee/NACreport2008.pdf

13 The Medicare Rural Hospital Flexibility Program is a Federal initiative that provides funding to State governments to 

strengthen rural health by: licensing small hospitals as Critical Access Hospitals (CAH); offering cost-based Medicare 

reimbursement for acute inpatient and outpatient services; encouraging the development of rural-centric health networks; and 

offering grants to States to help implement a CAH program in the context of broader initiatives to strengthen rural health care 

infrastructure. (Source: http://ruralhealth.hrsa.gov/funding/flex.htm)
FLEX CAH HIT Evaluation- Final Report 8

B. Program Purpose

The purpose of the FLEX CAH HIT Network grant was to provide funds to support the development of 

Flex CAHHIT Network pilot programs, sixteen in total, in States receiving the grant (one per State). For 

the purposes of the grant, examples of HIT include practice management systems, disease registry 

systems, care management systems, clinical messaging systems, personal health record systems, 

electronic health record systems, and health information exchanges. The common thread among all 

funded pilot projects is the use of HIT as a tool to improve the safety, quality, efficiency, and 

effectiveness of health care delivery. The Flex CAH HIT program aimed to ensure that clinical 

information of patients served by the CAH HIT network is accessible to providers across the continuum 

of care, from ambulatory care to acute service delivery sites. 

The Flex CAH HIT program concentrated funding within smaller service areas to increase the likelihood 

of creating pilot projects that are sustainable. Grant applicants were required to identify up to three 

CAHs and their associated network of providers, which could include a common larger referral hospital, 

that together provide a full continuum of care for rural residents in their service area. To build upon 

existing patterns of care, it was critical that the Flex applicant propose a network that followed common 

patient referral patterns. The network could include local partners for the CAHs in the network, 

including but not limited to private practice physicians, Medicare-certified Rural Health Clinics (RHCs), 

Federally Qualified Health Centers (FQHCs), nursing facilities, home health agencies, public health 

departments, and emergency medical service providers. The CAH HIT Network proposed by the 

applicant was required to cover a service area that reflects the natural flow of patients served by the 

CAH and its associated providers. Applicants also were required to have past experience with the use of 

quality improvement programs (e.g., the Center for Medicare and Medicaid Services’ Hospital Compare) 

and were asked to select least five performance outcome measures with national benchmarks, including 

the diabetes control and cardiovascular health performance indicators as required by HRSA.

The Flex CAH HIT grantees encompass a variety of HIT implementation projects. Grant projects differ in 

a number of key characteristics, including network structure (e.g., the number of CAHs involved, 

whether a tertiary facility is participating, and what types of ancillary facilities are included); type of HIT 

being implemented (e.g., electronic medical record; health information exchange, telemedicine, or a 

combination thereof); history of collaboration between the grant project partners; and whether the 

proposed project is new or had been planned prior to the Flex CAH HIT announcement. These projects 

might serve as a model to the Nation, particularly rural communities facing similar challenges in 

adopting HIT.

The remainder of this report describes the evaluation purpose, design and methodology. It features the 

analysis and results of the evaluation, particularly in terms of supports and barriers grantees 

encountered during their implementation process, as well as lessons learned in that process. The report 

includes detailed case studies from the four sites selected for extensive evaluation, including:

• Flex CAH HIT Network Implementation Grant, University of North Dakota Center for Rural Health

• Northeastern Oklahoma CAH EHR Network, Oklahoma State University Center for Health Sciences

• South Carolina Office of Rural Health, Lakelands Rural Health Network HIE

• Rural Wisconsin Health Cooperative Information Technology Network, Board of Regents of the 

University Wisconsin System

Results of a value-based return on investment (ROI) evaluation are discussed in detail. The report 

concludes with specific recommendations to HRSA, particularly in terms of administering future grants 

involving HIT expansion activities. 
FLEX CAH HIT Evaluation- Final Report 9

III. Methods

A. Purpose

This evaluation assessed the 16 Flex CAH HIT grantees’ ability to design, create, and implement 

functioning CAH HIT pilot networks that help improve coordination of care for the rural population served 

by the participating CAH(s). The main objectives of the evaluation were to:

• Determine the extent to which grantees are able to successfully implement the activities proposed 

in their application;

• Describe the supports and barriers grantees faced in reaching their stated goals and objectives; and

• Describe the project characteristics, and the supportive elements necessary for projects to fulfill the 

goals of the Flex CAH HIT Program, in order to inform future grant funding cycles.

B. Evaluation Design

To thoroughly and objectively examine the implementation of the Flex CAH HIT program, the evaluation 

employed a multi-method design that incorporated several implementation evaluation strategies. 

Quantitative and qualitative data collection, analysis, and triangulation were necessary to obtain a 

complete picture of program implementation successes, challenges, and areas for improvement. The 

implementation of the Flex CAH HIT program was examined across the 18-month grant period using 

information collection from:

• Review of grant documentation background information, including grantee monthly reports, final 

reports, and grant applications

• Telephone interviews (quarterly and close-out)

• Case study of select grantees, including calculation of value-based return on investment

C. Evaluation Questions

The evaluation questions reflect project implementation from the grantees’ perspective and address the 

critical aspects of planning and oversight, implementation, maintenance and sustainability planning, and 

return on investment. The evaluation questions are drawn from the stated purpose and intent of the 

Flex CAH HIT program and frame the evaluation in terms of the goals and strategies that are most 

important to the success and implementation of the program. Table 1 presents a summary of 

evaluation domains, questions, and data collection methods used to address each question.
FLEX CAH HIT Evaluation- Final Report 10

Table 1. Summary of Evaluation Domains and Data Collection Methods

Domain Evaluation Questions

Document Review 

(e.g., grant 

applications, monthly 

reports, other reports)

Quarterly 

Telephone 

Interview

Group 

Interview with 

Select 

Grantees

Value-Based 

Return on 

Investment

Network 

characteristics

• Who are the members of each CAH HIT network? 

• What is the governance structure of the network? 

• How long has the network been operating? 

• What other quality-related functions does the network perform in 

addition to this HIT project (e.g., joint quality improvement activities, 

collection and analysis of quality measure data)?

X X X

Project maturity

• What types of HIT are already in place at each CAH and at each of 

their network members? 

• What additional HIT investments are needed to meet the goals of the 

implementation project? 

X X X

Planning and 

development

• What types of planning approaches were used by the grantee and the 

network to determine the needs and focus of the HIT implementation 

project (e.g., readiness assessment, workflow analysis, business 

plan)?

X X X

Evaluation 

measures

• To what extent did the technology that was implemented and planned 

for allow patient data to follow throughout the continuum of care?

• How did project implementation improve patient care and help 

providers throughout the network work together more efficiently to 

manage the health care needs of residents in the service area of the 

critical access hospitals that will form the core of the network?

• How will outcomes be measured on specific disease indicators (as 

chosen by grantees)? 

X X X

Assessment of 

project 

implementation

• How is success defined by the CAH network? 

• How successful was the implementation? 

• How successful is the operation of the new HIT functionality? 

• What was the ability of the grantee and the network partners to 

implement information technology in a seamless manner? 

X X X X
FLEX CAH HIT Evaluation- Final Report 11

Domain Evaluation Questions

Document Review 

(e.g., grant 

applications, monthly 

reports, other reports)

Quarterly 

Telephone 

Interview

Group 

Interview with 

Select 

Grantees

Value-Based 

Return on 

Investment

Challenges 

experienced

• What were the scope and impact of barriers encountered?

• What type of supports helped in the implementation of the project? 

X X X

Lessons learned • What lessons have been learned about planning and implementing 

HIT in CAH networks? 

X X X

Sustainability and 

maintenance • How will HIT be sustained and upgraded? X X X

Return on 

Investment

• What was the nature of the return on investment that resulted 

(anticipated to result) from the use of this health information 

technology?

X X
FLEX CAH HIT Evaluation- Final Report 12

D. Methods

1. Document review

The following documentation were reviewed for the purposes of this evaluation:

• Applications for each grant award

• Grantee monthly reports

• Written reports 

These documents were reviewed to obtain an accurate depiction of the project being implemented, 

network structure, network partners, vendors utilized, project maturity, and implementation successes and 

challenges.

2. Telephone interviews, quarterly and at close-out

Information obtained from the applications and reports were supplemented by telephone interviews with 

the grantees on a quarterly basis and at the end of the grant period. Grantees were asked to explain in 

greater detail information shared in monthly reports, in addition to the barriers experienced during project 

implementation, specific supports and elements that have contributed to the implementation of the project 

plan, lessons learned about HIT planning and implementation, and any differences observed between CAHs 

with regard to project implementation. The interview protocols are located in Appendix A. 

3. Case study of select grantees

Site Visits

Group interviews were held with Grant Administrators and other representatives from select grantees to 

supplement the information obtained through the quarterly calls with all grantees and the secondary data 

analysis of grantee monthly reports and other background information. Grantees who participated in the site 

visits were selected based on a combination of several criteria in order to achieve a representative view of 

program implementation: 

• Type of project implemented. Projects could have focused implementation on an electronic health record, 

health information exchange, telemedicine, or a combination of these projects.

• Network structure. This factor takes into account grantee network characteristics such as the number of 

community access hospitals (CAHs) involved in the implementation, the number and type of ancillary 

partners, and the presence of a tertiary hospital.

• Relationship between network partners. This factor indicates the extent to which network partners had an 

existing relationship or collaborated on other projects in the past.

• Project maturity. Some projects were developed upon the announcement of grant funding, while others 

were already under development.

Using these criteria, the following grantees were selected to participate in the site visits:

• University of North Dakota Center for Rural Health

• Oklahoma State University Center for Health Sciences

• South Carolina Office of Rural Health

• Board of Regents of the University of Wisconsin System

Site visits were conducted over a 5-month period near the end of the grant period from March to July 2009. Indepth group interviews were conducted on site with a range of staff, including the Grant Administrator, Principal 

Investigator, Director of Information Technology, Medical Director, Director of Nursing, and other clinical 

services and administrative staff. The interview protocol was designed to collect information about aspects of 

the grant implementation that could not be obtained from secondary data sources. The protocol was 
FLEX CAH HIT Evaluation- Final Report 13

administered in a uniform manner using a standard interview guide and protocol to support analysis across 

grantees. A copy of the interview protocol is located in Appendix B. 

Value-based Return on Investment (ROI)

Adapting and implementing new information systems in live health care settings, and in particular in rural health 

settings, and demonstrating a clear value (e.g., through rates of use), has proven difficult. Because HIT is not 

introduced into a static environment, evaluation must consider the environment in which implementation 

occurs.14

Step 1: Functional Definition Tool

 In terms of evaluating return on investments (ROI) on HIT implementation, financial and disease 

outcome-based ROI can take years to accrue, although more value-based ROI measures can be good predictors 

of these factors. The ROI used for this implementation evaluation focused on how and where users (patients, 

clinicians, administrators, etc.) of an implemented HIT system derive and obtain value. For the grantees 

selected for case study, a suite of tools were adapted and designed to incorporate qualitative and quantitative 

value-based ROI measures to capture important information within the evaluation timeframe. The following 

section describes the steps and measures taken to study ROI of the HIT implementation, and discusses the 

results of that study in the four case-study states (North Dakota, Oklahoma, South Carolina and Wisconsin). The 

full suite of ROI tools is located in Appendix C.

As a first step, the grantees that participated in the site visits used the Step 1 Functional Definition tool to parse 

out individual functions and participants involved in the project’s scope, and to further define the specific 

relationships among data suppliers, data users, and the purpose of the information being shared. The tool 

provided lists of possible HIT functions as well as possible HIT participants; however individual CAHs had the 

opportunity to add or combine up to three user-defined functions and three user-defined participants. CAHs 

completed this tool using their information and then confirmed with the Altarum evaluation team the specific 

functions and participants affected by the Flex CAH HIT grant.

Step 2: Understanding Pre-Grant System(s) and Activities 

In Step 2 of the ROI protocol, grantees used the Understanding Pre-Grant System(s) and Activities questionnaire 

to assess several measurement elements of the pre-grant system(s) and activities involved in the Flex CAH HIT 

project. This tool serves as a baseline or pre-test measure to understand where each CAH is (their “pre-grant 

processes”) as a means to compare the impact of the grant activities (their “to be processes”). In other words, 

this tool serves to evaluate some of the limitations of paper-based information systems.

Step 3: Understanding Implemented System(s) and Activities

In Step 3 of the ROI protocol, grantees used the Understanding Implemented Grant System(s) and Activities 

questionnaire to assess several measurement elements of the implemented system(s) and activities involved in 

the Flex CAH HIT grant. This implementation information collection step then served as a comparison point 

against the baseline measurements taken in Step 2. 

E. Data Analysis

Qualitative data, including interview responses and abstracted information, initially were stored and cleaned in 

Microsoft Word and subsequently imported to NVivo 7 software for content analysis. The data were analyzed 

for key themes and differences in response across grantees. Specific qualitative examples that illustrated 

grantee successes and challenges were stored in NVivo; some are included in this final evaluation report.

Comparison of grantees’ data by type of project implemented (e.g., EHR, HIE, etc.) did not yield significant 

differences in responses. Aggregate results of the analysis are presented in Section III of this report.

 

14 An Evaluation of Electronic Health Records in Indian Health Services
FLEX CAH HIT Evaluation- Final Report 14

F. Limitations

1. Generalizability of findings due to variation in HIT systems. 

The flexible nature of initiatives such as the Flex CAH HIT program poses an evaluation challenge. Each 

grantee designed a different HIT system and implemented that strategy differently in different 

environments, thereby making standardization of the evaluation methods and the Flex CAH HIT program a 

challenge. The time and resources available for the evaluation limited the number of case studies that could 

be conducted. Efforts were made during case study selection to ensure that general characteristics of the 

grantees were represented in the evaluation; however, the grantees that were selected for case study may 

not be representative of all grantees. 

2. Non-experimental design.

An experimental evaluation design was not feasible within the timeframe of the study, in which a site that 

did not receive Flex CAH HIT funding was compared against a Flex grantee on selected indicators. The study 

does not separate out the effects of the Flex CAH HIT program from the myriad other environmental and 

contextual factors that may have influenced the implementation of the Flex CAH HIT grant and initial valuebased return on investment. Thus, this evaluation also does not infer causality. 

3. ROI measures of clinical outcomes and quality improvement.

Beyond the tools developed for Flex CAH HIT value-based ROI evaluation, additional ROI measures of 

specific clinical outcomes and quality improvement are extremely important to HIT implementation. 

However, because clinical and quality improvement measures require a significant observation period to 

appropriately assess implementation and ROI effects, these measures could not be used within the limited 

timeframe of this evaluation project. Given the abbreviated timeline of this grant, the baseline measures 

Flex CAH HIT grantees established in Steps 1 & 2 may be utilized in future ROI evaluations of clinical and 

quality improvement measures. Future evaluations may want to require CAHs to answer additional 

questions regarding specific disease outcomes, workflow, and staffing changes to be affected by the grant, 

including any relevant measurements (e.g. FTE changes, task time, functional or other direct costs).

4. Self-reported data.

The evaluation did not include an independent assessment and observation of the Flex CAH HIT grant 

implementation. All of the data collected for the evaluation were the self-reported perceptions and 

experiences of grantee representatives. Social desirability may have affected participant responses in which 

respondents may have replied in a manner that would be viewed favorably by others. 
FLEX CAH HIT Evaluation- Final Report 15

IV. Analysis and Results

The Flex CAH HIT grantees conducted a variety of HIT implementation projects. Grant projects differed 

in a number of key characteristics, including network structure; type of HIT being implemented; history 

of collaboration between the grant project partners; and project maturity. A summary of the grantees’ 

projects is presented in Table 2 and a detailed profile of each grantee’s project is located in Appendix D.

This chapter presents an analysis of the supports and barriers facing the grantees as they implemented 

their projects, their plans for sustainability and evaluation, the impact their project has had on CAH 

operations and outcomes, as well as a description of the lessons they have learned regarding HIT 

implementation. Data sources for this report are the monthly reports submitted by grantees, as well as 

the September and December 2008 grantee quarterly conference calls and the June 2009 close-out calls 

conducted with all grantees.15

Through the monthly reports and quarterly and close-out calls, grantees were asked to report on:

 

• The supportive elements or factors that were critical in completing the project tasks/activities 

• Any barriers that were encountered in accomplishing the tasks/activities and steps taken to resolve 

them

• Technical assistance or additional resources needed to achieve the goals of the grant project 

• Lessons learned about HIT planning and implementation

• Project impacts on operations and patient care

• Plans and resources needed for sustainability

• Plans for evaluation

• HRSA’s role in the implementation

Inductive qualitative analysis was conducted on aggregated data to identify themes for each topic. 

Findings are presented first on the supports to project implementation, then the barriers to project 

implementation, followed by lessons learned by grantees about HIT implementation. Project impacts, 

plans for sustainability, evaluation, and HRSA’s role in the implementation are also discussed.

 

15 The close-out call topics were covered with the four grantees that were visited in person during the site visit.
FLEX CAH HIT Evaluation- Final Report 16

Table 2. Summary of Flex CAH HIT Implementation Grantees Projects

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Alabama 

Department of 

Public Health

Electronic Health 

Record

Formalize a provider network to include a CAH, rural health center, tertiary 

hospital, and a seven-cardiologist physician group and bring the network 

together for shared information in order to identify more at-risk cardivascular 

disease patients and to provide more coordinated care for those already 

identified.

Although independent, the partners have experience working together. The 

cardiologist group has a strong presence in the community. The participating 

tertiary center is the primary transfer partner, and it has a longstanding 

positive relationship with the cardiologists, who do most interventions there. 

While this was a new project, the CAH had recently invested in cardiac care 

equipment.

One CAH, one rural health center, 

one tertiary center and a physician 

group

Healthland (EHR) 

and Sci-Health 

(performance

tracking)

Hawaii State 

Department of 

Health

Electronic Health 

Record

The goals of the project were to: establish a model CAH and clinic EHR 

capability based on the Indian Health Service Resource and Patient 

Management System (IHS RPMS), a derivative of the VA VistA system; 

implement the RPMS infrastructure with appropriate servers and end-user 

equipment in two CAHs (West Kaua`i Medical Center and Samuel Mahelona 

Memorial Hospital) and three associated clinics; implement security protocols 

and security software in RPMS to enable the HHSC Kaua`i Region CAHs and 

clinics with authorized access to patient records; develop application 

interfaces with patient management and billing systems to ensure adoption 

and improve the efficiency of the EHRs in Hawai`i; and evaluate the 

implementation and usability of the Kaua`i RPMS EHR to provide lessons 

learned for future RPMS implementations in Hawai`i CAHs and clinics.

This pilot FLEX program application was submitted by the State of Hawai`i 

Department of Health (DOH), as a partnership with the Kaua`i Region of the 

Hawai`i Health Systems Corporation (HHSC), and the Telecommunications 

and Information Policy Group (TIPG) of the University of Hawai`i (UH). The 

Partners have been rigorously working over the past two years to develop a 

strategy to implement EHRs in CAHs and clinics to improve patient safety and 

quality, and to improve efficiency and effectiveness for reasons welldocumented in national studies, reports, and executive orders. The DOH has 

been working as a partner with the HHSC and the University of Hawai`i to 

ensure a cost-effective and sustainable EHR capability.

Two CAHs and three clinics An EHR based on 

the Indian Health 

Service Resource 

and Patient 

Management 

System (IHS 

RPMS), a 

derivative of the 

VA VistA system

Illinois 

Department of 

Public Health

Electronic Health 

Record and 

Health Information 

Exchange

Project funds were used to implement electronic health records in the rural 

health clinics of two critical access hospitals; implement a picture archiving 

and communications system at one of the participating hospitals; implement 

health information exchange software; and identify the processes and 

activities that best supported technology implementation in the critical access 

hospital environment.

This project is the result of two years of planning. The CAHs have worked on 

previous cooperative projects, including evaluating HIT and processes to 

improve information sharing, and a joint pharmacy coverage project. The 

CAHs are members of the Illinois Critical Access Hospital Network. The other 

network stakeholders have also worked together in the past. 

Two CAHs and their two clinics NextGen (EHR 

and HIE)

Indiana State 

Department of 

Health

Electronic Health 

Record

St. Vincent Health implemented electronic health record (EHR) technology into 

the outpatient primary care setting (Rural Health Clinics and one Federally 

Qualified Health Clinic), and the inpatient Critical Access Hospital setting, and 

allows them to connect electronically for records and results transfer to the 

tertiary care setting. The demonstration project results were instrumental in 

illustrating how to connect and deliver patient care results into the Indiana 

Health Information Exchange (IHIE).

The project is the result of 12-18 months of planning, which resulted in St. 

Vincent’s organization-wide Information Services and Technology Strategic 

Plan for 2006-2009. Most of the entities involved are owned and operated by 

St. Vincent Health, part of Ascension Health, the parent organization. All CAH 

sites report to the same Chief Operating Staff and Chief Medical Officer.

Three CAHs, connecting to three 

clinics (rural health centers) and an 

FQHC

Eclypsis (inpatient 

EHR) and 

AllScripts 

(outpatient EHR)
FLEX CAH HIT Evaluation- Final Report 17

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Louisiana 

Department of 

Health and 

Hospitals

Electronic Health 

Record and 

Health Information 

Exchange

The Pointe Coupee Parish health system has evolved without a strategic 

system design. The object of the project was to provide a system(s), which 

will allow access to patient data in a secure web enabled environment. The 

ultimate goal was the sharing of appropriate health information among all 

partner organizations. This will be accomplished by connecting all of the 

information systems and processes implemented in a secure, hosted 

environment. This will provide essential data, in an efficient manner, to the 

network’s health care providers. 

The network members have a long history of working collaboratively to 

improve the health care delivery system of Pointe Coupee Parish, and have 

proven to be innovators in rural network development. They have been 

engaged in collaborative sharing of information for several years. 

The Pointe Coupee HIT Network 

includes Pointe Coupee General 

Hospital, a 25-bed CAH; the CAH’s 

transfer tertiary hospital, Our Lady 

of the Lake Regional Medical 

Center (the Lake), a 740-bed 

community hospital located in 

Baton Rouge (approx 45 miles); 

and four local rural health clinics 

managed by the Lake--an FQHC 

with two sites in the parish (Innis 

Community Health Center); one 

local community clinic; two private 

practice primary care clinics; and 

one home health agency, Point 

Coupee Home Bound.

AHS, CNN, CPSI, 

Cerner (CareFx is 

being used 

statewide)

Michigan 

Department of 

Community Health

Electronic Health 

Record and 

Health Information 

Exchange

The primary purpose of Michigan’s FLEX CAH-HIT Network project was to 

create an effective health data exchange between two Critical Access 

Hospitals and their tertiary referral center, in order to improve the safety, 

quality, efficiency, and effectiveness of health care delivery through a full 

continuum of care. The anticipated results will be measurable 

improvements on the health status of local communities including 

improvements in: (a) the safety and quality of care delivery and patient’s 

care; (b) the effective delivery of health services and continuity of patient 

care; and (c) operating efficiencies in terms of time saved, increases in 

revenue, increased tracking and or reporting of patient’s quality and health 

outcomes.

Participating sites are part of the Thumb Rural Health Network (TRHN). TRHN 

is a recently incorporated 15-member organization located in the rural counties 

of Huron, Sanilac, and Tuscola, typically referred to as MI’s “Thumb.” TRHN’s 

membership includes seven CAHs and one sole-community provider; both 

county health departments; six tertiary hospitals surround the region; and one 

Multipurpose Collaborating Council. In 2006, the organization identified the 

need to develop a Health Network Exchange (HNE).

Three CAHs and a tertiary referral 

hospital

Techtime, CPSI, 

and Cerner (HER) 

and Covisent (HIE)
FLEX CAH HIT Evaluation- Final Report 18

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Minnesota 

Department of 

Health

Electronic Health 

Record and 

Health Information 

Exchange

The grantee successfully implemented an electronic health record system 

among its three facilities, Madison Lutheran Home, Johnson Memorial Health 

Services, and Appleton Area Health Services, in order to create an integrated 

community health information system to allow patients served to travel 

seamlessly through the continuum of care and permit reporting of quality 

measurement data. 

The partners collectively make up the Lac qui Parle Health Network, created in 

1998. In 2007, the LqPHN conducted an HIT strategic planning process that 

provided a comprehensive blueprint for the selection and implementation of a 

system-wide electronic health record (EHR) system. The three organizations 

identified a need for patients in the three communities to move seamlessly 

through the continuum of care from emergency services to acute hospitalbased care to assisted living and/or nursing home and referral to Rice 

Memorial Hospital in Willmar or other referring hospitals, such as Avera 

McKennan in Sioux Falls, South Dakota, or Level I Trauma Hospitals in the 

Minneapolis-St. Paul metropolitan area. It also identified a need to continue 

improving patient quality of care and safety by integrating outcomes reporting 

into its EMR environment. 

Three CAHs and their ancillaries Dairyland, HMS, 

CPSI

University of North 

Dakota

Electronic Health 

Record and 

Health Information 

Exchange

The goals of the North Dakota project were: 1) to assist CAHs with the 

implementation of electronic medical records and the sharing of patient data 

with at least one ancillary facility within their respective communities; 2) 

facilitate the exchange of patient information (diabetes) between the CAH 

facility/ancillary and the regional tertiary center; and, 3) strengthen regional 

network to address HIT needs and issues. 

The partners are part of a larger existing network, the North Region Health 

Alliance. There is a long and established relationship of collaboration among 

the members. At the time of the application, North Dakota had 31 critical 

access hospitals (CAHs), none of which was involved in a health information 

technology (HIT) network with its tertiary. Statewide data indicated that CAHs 

were supportive of the need and benefits of HIT; however, they struggled with 

available resources (e.g. financial and IT staff).

Three CAHs (part of an existing 

network) and their ancillary 

facilities, including a Rural Health 

Center, a Community Health 

Center, two long-term care 

facilities, as well as a tertiary 

referral hospital. 

Healthland (2 

CAHS used for 

their HER) and 

American 

HealthNet (1 CAH 

chose this for their 

EHR). Cinical 

Workstation was 

used as the portal 

to the tertiary 

facility.
FLEX CAH HIT Evaluation- Final Report 19

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Nebraska 

Department of 

Health and 

Human Services

Electronic Health 

Record

The main focus of the Nebraska Project was to share and exchange clinical 

health information along a continuum of care. This continuum involved the 

Thayer County Health Services (a Critical Access Hospital and physician’s 

clinic), five additional rural health clinics, one assisted living facility, one 

nursing home, three EMS ambulance units, an independent pharmacy, a 

hospital-owned retail pharmacy, a home health agency, and the network 

hospital (St. Elizabeth Regional Medical Center in Lincoln). The desired 

outcome of the project was to improve the quality, safety, efficiency, and 

effectiveness of the Thayer County Health Care system.

This project was designed to meet the health needs of Thayer County and the 

surrounding communities in Nebraska and Kansas. The population of Thayer 

County is older and the mortality rates for many of the major health problems 

are considerably above the state average. There are also access to care 

issues. Developing an electronic health information exchange system was 

considered a critical tool that would facilitate an improvement in health 

outcomes for patients with diabetes, heart failure, and stroke. It would also 

lead to improvements in adult and childhood immunization rates, a sharp 

reduction in medication errors, and more timely and effective treatment of the 

patients who were transferred to the network hospital. 

A CAH, five rural health clinics, a 

home health agency, a nursing 

home and an assisted living 

facility, several EMS units, a 

hospital owned retail pharmacy 

and an independent retail 

pharmacy, and a network tertiary 

hospital.

Medinotes and 

HMS

Oklahoma State 

University, Center 

for Health 

Sciences

Electronic Health 

Record and 

Health Information 

Exchange

The proposed project was designed to create a Critical Access Hospital (CAH) 

Electronic Health Record (EHR) Network in Northeastern Oklahoma (the 

“Network”). The Network members’ vision for the EHR network was to ensure 

that patient clinical information would be easily accessible to providers within a 

healthcare organization and to other providers as patients migrate from 

ambulatory care to acute service delivery sites within the region. The 

improved access to patient information would be integrated into a coordinated 

system of care that would result in more effective and efficient health care 

delivery that would ultimately lead to improved safety and quality of care for 

patients. 

The partners were not part of a pre-existing network, but they have created

one (the "Northeastern Oklahoma CAH EHR Network") for the purposes of this 

project. The project was new at the time of the application.

Four CAHs and a tertiary center Cerner, Meditech, 

and HMS (EHR) 

and Covisent (HIE)

South Carolina 

Office of Rural 

Health

Health Information 

Exchange

The purpose of this project was to acquire the necessary funding and technical 

resources to implement a regional health information exchange and quality 

improvement initiative in the Lakelands area of western South Carolina. The 

goals for this grant were: 1) To serve as the SC rural pilot site for an already 

developed health information exchange technology platform, that includes a 

personal health record; and 2) to establish a regional quality improvement 

program and reporting function within the health information exchange. 

The Lakelands Rural Health Network (LRHN) is a nonprofit, multi-county 

vertical network that was developed in 2004 with the guidance and financial 

assistance of the South Carolina Office of Rural Health, the state FLEX 

grantee, to achieve efficiencies, expand access, coordinate and improve the 

quality of essential health care services, and strengthen the rural health care 

system as a whole. The network started a strategic planning process in 2004, 

during which time HIT was identified as a priority. The network completed a 

formal needs assessment process that included community and key informant 

interviews to make sure that the LRHN HIT projects are driven by the needs of 

area clinicians.

2 CAHs, an FQHC with 9 family 

practices, 2 rural health centers, 3 

family practices, and a tertiary 

facility

Care Evolution
FLEX CAH HIT Evaluation- Final Report 20

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Tennessee 

Department of 

Health

Health Information 

Exchange

This project endeavored to facilitate enhanced collaboration among health 

care providers within the Middle Tennessee Rural Health Information Network

(MTRHIN) to include three Critical Access Hospitals (CAHs), one Federally 

Qualified Health Center (FQHC) and their referral hospital by supporting the 

electronic exchange of health information. The purpose of this project was to 

support the use of health information technology (HIT) as a tool to improve the 

quality of patient care and minimize health access issues, which in turn 

improves patient outcomes, decreases medical costs and hospital admissions. 

The aims included the adoption and effective use of HIT; the creation of 

sustainable business models for deploying HIT in Medicare Rural Hospital 

Flexibility (Flex) CAH networks; enhancing the ability of safety net providers to 

leverage initiatives and resources as well as improving quality; and promoting 

performance improvement within the Flex program.

The selected CAHs had already developed a workplan and two of the three 

CAHs are within a larger hospital system and were in the midst of formalizing 

a network at the time of grant selection (they have now received 501(c)3 

status). 

Three CAHs, an FQHC, and a 

tertiary referral hospital, collectively 

the Community Health Network

NextGen

Texas Office of 

Rural Community 

Affairs

Electronic Health 

Record, Health 

Information 

Exchange, and 

telemedicine

The goal and primary need for this project has been to improve health care 

delivery and quality of life through Health Information Technology (HIT) 

implementation within two rural communities as a demonstration model for all 

rural Texas communities. The objectives are to improve the safety, quality, 

efficiency and effectiveness of healthcare delivery through implementation of 

health information technology.

There wasn’t a formal network between the CAHs prior to this project. 

However, all the partners have signed MOUs among themselves. This project 

was their second collaborative effort. 

Two CAHs, community partners, 

and a tertiary hospital

OPUS

Virginia 

Department of 

Health

Health Information 

Exchange and 

Telemedicine

Using the American Stroke Association’s “Stroke Continuum of Care” as an 

organizing framework, the VA Critical Access Health HIT (VA CAH-HIT) 

Network served as a test bed for the implementation of HIT interventions 

across the full stroke continuum of care within CAH service areas of the 

Virginia Acute Stroke Telehealth (VAST) statewide network infrastructure. The 

CAH-HIT grant funding was utilized to set up the first regional network for 

VAST. The specific focus of the program was to implement, test and evaluate 

a variety of information and telecommunication technologies to determine how 

technology could be leveraged to address functional requirements--particularly 

at the critical access hospital (CAH) level in highly rural areas of the state. The 

outcome of the effort was intended to not only set up a network in this region, 

but to develop a model to be leveraged statewide. 

In efforts to begin addressing the problem of fragmentation in stroke care, the 

Virginia Department of Health (VDH), Office of Minority Health and Public 

Health Policy (OMHPHP) partnered with the Virginia Telehealth Network 

(VTN) and the Virginia Stroke Systems Collaborative (VSS – which has since 

transitioned to become the Virginia Stroke Systems Task Force) to design a 

model telestroke network for Virginia. The initiative was coined the Virginia 

Acute Stroke Telehealth (VAST) network.

One CAH, one community hospital, 

a tertiary center, and ancillary 

providers

CPSI and 

FastChart (EMR), 

InTouch Health 

(robotics), DR 

Systems and 

CareStream 

(PACS vendor) 

and Medweb 

(teleradiology)
FLEX CAH HIT Evaluation- Final Report 21

Grantee Name

Main Purpose 

(EHR, HIE, and/or 

telemedicine)

Project Description Project History/Background Network Structure Vendor(s)

Washington State 

Department of 

Health 

Health Information 

Exchange

This project set out to connect three CAHs and their respective ancillary 

facilities, all with disparate information systems, to each other and to the 

tertiary referral hospital in Seattle, Harborview Hospital, which is the only Level 

1 Trauma Center in the state, through a health information exchange. This HIT 

project aimed to develop a universal data exchange/interfacing capability on 

multiple levels, and has the potential to serve as the universal EHR exchange 

platform that can be integrated into the statewide telehealth pipeline, thus 

making the platform scalable to CAHs and other providers statewide.

The three CAHs are members of the HIT Implementation Network of Western 

Washington Rural Healthcare Collaborative (WWRHCC), and have a long 

history of collaboration, particularly in the areas of HIT and telemedicine. Since 

2003, all WWRHCC hospitals have been members of the University of 

Washington’s Telehealth Network, and have been working together to 

develop HIT systems to support the delivery of specific specialty services 

through telemedicine. In 2005, WWRHCC performed an internal business 

planning process developed to guide the Collaborative through the next 3 

years in its evolution. Since then, the Collaborative has been actively pursuing 

a consistent set of goals as a cohesive group. 

Three CAHs and their associated 

rural health centers, and a tertiary 

Level 1 trauma center

Orion

Board of Regents 

of the University 

of Wisconsin 

System

Electronic Health 

Record and 

Health Information 

Exchange

This project involves implementing a shared community EHR environment

(with integrated hospital, physician clinic, and skilled nursing applications), 

which is intended to unify information from disparate continuum of care 

settings into a community electronic health record and provide contraindication 

checking and decision support tools that will reduce medication errors, 

facilitate the practice of evidence-based medicine, and improve care quality. 

The RWHC ITN Project was designed to address a variety of critical access 

hospital (CAH) needs and problems. As has been discussed in several 

reports and studies, CAHs face a number of HIT adoption challenges, 

including lack of financing, lack of sufficient HIT professionals, and limited 

expertise to facilitate project management and workflow redesign. These 

challenges are especially pronounced for small-volume CAHs, since they 

generally have tighter margins and smaller proportions of staff to devote to HIT 

and EHR related activities. In many cases the result of these challenges is 

that CAHs are simply not structurally positioned to move forward with the 

technology implementations that can benefit their patients and that will be 

required to meet the demands of the changing healthcare landscape (including 

ARRA, value-based purchasing, and health reform).

4 CAHS and their affiliated

physician clinics and skilled 

nursing facilities

HMS
FLEX CAH HIT Evaluation- Final Report

 22

A. Supports to Implementation

Grantees discussed several core supports to implementation, including components directly related to 

stakeholder buy-in, project design, and supports specific to the CAH environment. 

A.1. Stakeholder Buy-in

Management leadership (10 grantees). Good leadership is necessary to bring both a regional and a 

national perspective to a project, to work with neighboring institutions, and to create an integrated 

system while avoiding territorial issues. As one grant administrator stated “You can’t really understate 

the [need for] leadership in working with neighbors in the region.” Executive leadership and 

involvement can positively contribute to moving a project plan forward. CEOs and CFOs intimately 

involved in the Flex CAH HIT project were instrumental in conveying a sense of urgency and priority to 

staff at all levels, and grant administrators saw this as critical to a smooth and timely implementation. 

Management buy-in was further strengthened by executive backgrounds in areas such as designing 

functional operational systems as well as close working relationships with IT experts. 

Community partner support (7 grantees). The program must have a good relationship with providers; 

one grantee noted that providers “had ideas from years of experience on how to deal with some of the 

issues.” Community partner support can also lead to an expansion beyond hospitals and pharmacies to 

include care providers such as nurse educators, dieticians, clinical work stations, and, in one grantee’s 

case, a regional diabetes center. One grant administrator found that placing advertisements in the local 

newspaper about the project revitalized the small community: “Hospitals are big time employers in each 

community and [our HIT implementation] gets the staff and community pumped. Everyone gets all kinds 

of great publicity by being associated with the effort.”

Clinical leadership and support (6 grantees). Five grant administrators discussed the importance of 

physician buy-in to project success: “The physician champion is obviously critical.” Physician and other 

clinical staff champions can facilitate communication between clinicians and IT staff. One grantee 

administrator felt that the two participating hospitals with a physician and a physician assistant (PA) 

champion stood a better chance of success than those without. In another, clinical input drove vendor 

selection during the project planning stages.

IT Expertise (6 grantees). The knowledge and dedication of IT staff is an invaluable project support, 

especially when the IT staff recognize that there is much to be done with the project beyond what they 

are currently doing, which is important to the long-term success of the project. Having staff who 

understand the breadth and scope of what the project is trying to accomplish is important to its success.

State and national interest in HIT (5 grantees). General awareness of the national HIT environment, as 

well as State awareness of CAH capabilities to integrate HIT, improved grantees’ prospects for planning 

and implementing a successful project. One grant administrator stated that CAHs are “aware that the 

Flex CAH HIT grant is their best chance to get a tested and workable EHR” in the present environment. A 

supportive relationship with the State, including the ability to use its resources and guidance, is 

especially important for project sustainability. The relationship between the grantee and the State can 

be mutually beneficial, as in the case of one grantee whose project provided positive publicity to the 

State in rural newspapers. Another grantee cited the support of the Lieutenant Governor’s office and 

the creation of a Statewide eHealth Council as a support to innovation. State-level officials also have a

stake in the success of innovative HIT implementation efforts because HIT is tied to other issues on the 

State agenda, such as veterans’ care, community mental health, and disaster preparedness.
FLEX CAH HIT Evaluation- Final Report

 23

A.2. Project Components

Vendor support (7 grantees). Seven grantees who had good vendor experiences described their vendors 

as supportive and quick to help with troubleshooting when hospital concerns were voiced. Vendor 

support included a willingness to collaborate to ensure success and sending very qualified people who 

were friendly and helpful to train staff. One grantee wrote that their vendor was “not afraid to hold your 

hand if you’re not good with computers.” Another vendor supported one grantee’s project by being 

knowledgeable about rural culture and communities. Grant administrators praised their vendors for 

attributes ranging from astute strategic planning, to interface capabilities, to willingness to provide 

software support and training. One grant administrator reported that the vendor’s strategic plan was

the foundation for hospital staff buy-in because it designated templates and workforce processes that 

led staff to “stretch their skills to cross over from paper to electronic systems.” 

Partner readiness (6 grantees). A number of grant administrators reported that selecting hospitals that 

already had the infrastructure and/ or network capabilities to implement the project was advantageous. 

One grantee involved all of the area hospitals rather than just the three initially part of the grant. This 

meant that when a few of the initial participating hospitals dropped out of the project, other hospitals 

were prepared to take their place. Another grantee found that its hospitals had already gained 

experience installing ambulatory EMRs and physician monitoring, meaning that the grant didn’t have to 

“re-invent the wheel.” One CAH had already gone through a strategic planning process that allowed the 

grant to take advantage of an existing skill set. Established relationships between partners and experts 

also helped facilitate the project implementation. 

Software supports (6 grantees). Several grantees described how characteristics of the software or 

hardware provided by the vendors, including the ability to connect remotely, a bi-directional database, 

and open architecture solutions, supported project goals. Ease of use was also important: “If you’re in 

various departments in the hospital, you [use] the same program, so the look and feel, the tabs, the pulldown menus, the alarms, are all the same for each department. You don’t have to [re]-train employees 

if they switch departments.”

A.3. CAH-Level Supports

CAH structure and organization (6 grantees). Smaller CAH size can make implementation more 

manageable: “Small hospitals support a culture of working together where people wear different hats 

[and] there’s a much more community-oriented environment and agreeable attitude.” Having fewer 

staff to “get on board,” noted one grant administrator, contributed substantially to project success. 

Another stated that working with smaller facilities and organizations made it easier to generate interest, 

involvement, and buy-in to a new way of doing business. Smaller, less structured health care systems 

provide more flexibility for the exchange of information and quick decision-making. Being in one 

network environment was another support that facilitated networking. The willingness of one CAH to 

dedicate a portion of its budget to staff training had contributed immensely to the success of one 

grantee’s overall implementation and spoke to the value that the institution placed on project 

sustainability.

Collaboration among CAHs (6 grantees). Collaboration among participating CAHs has served as a 

significant project support. Grant administrators found that networking among CAHs and having an 

established CAH network facilitated decision-making, information exchange, and the inclusion of 

different constituencies. One grant administrator hoped that “this will have long-term effects beyond 

this grant in improving stroke systems of care.” Work group collaboration can create the necessary 

cohesion for a network of hospitals while also leaving them the flexibility to work independently on 
FLEX CAH HIT Evaluation- Final Report

 24

issues most important to their facilities. Regular meetings and other forms of communication among 

CAHs helped to emphasize the joint effort required by the implementation.

CAH staff enthusiasm (6 grantees). CAHs that have dedicated and passionate staff willing to commit the 

time necessary to troubleshoot HIT issues have met with greater success than those CAHs that are 

reluctant to confront these challenges head on. For example, one grant administrator described “daily 

huddles” at a CAH where staff meet to discuss areas of frustration and solutions evolving from their EHR 

project. One CAH supported user groups developed from different department staff to facilitate staff 

input and buy-in. Buy-in and involvement from the administrative level to the clinical staff is important: 

“It really has to be a facility-wide buy-in process.” Staff can make an effort to reach out to other CAHs 

and to smaller communities. In particular, projects with champions stand a better chance of success. 

B. Barriers to Project Implementation

Grantees identified a number of barriers and challenges they faced in the implementation of their 

projects. While technical issues comprise a large portion of the barriers facing grantees, other issues 

have also had substantial impact on the grantees’ project implementation. These issues included both 

anticipated and unforeseen issues of workforce, health system and CAH-level barriers, and challenges 

related to finances and administration. One grantee stated that “the whole project is by definition tough 

in the sense that you are introducing high-level technology to rural communities.” These barriers to 

implementation can be broadly grouped in the following manner:

B.1. Information Technology and Related Issues

Vendor/Network Disconnect (9 grantees). The process of implementing the projects was inhibited due 

to a disconnect between vendors and network members resulting in slow vendor negotiations, vendor 

selection, and contracting process delays. The contracting process can be difficult for some hospitals 

because of vendors “promising the moon” but giving hospitals “a huge gap in the contract.” Working 

with vendors who are used to large integrated systems in the context of small and rural systems is 

another difficulty. Many software and hardware applications developed for large medical centers can 

fail to address most or specific, critical needs of a rural setting.16

Furthermore, going into vendor negotiations and expectations for implementation, many CAHs lack 

overarching infrastructure, IT exposure and staff background, and limited resources for training and 

system maintenance – gaps that the CAH may be expecting the vendor to address or at least assist with, 

that are left unaddressed in contracts or worse, are specifically avoided by vendors. Multiple grantees 

were frustrated with trying to get vendors to make their systems interoperable with each other or with 

legacy systems. One grant administrator stated that “It’s one thing to deal with their sales reps; it’s 

another to deal with their programmers and make sure that the product you agreed to have in your 

institutions actually does what they say it can do.” Compounding these interoperability or expectation 

gaps is the contracting process where CAHs will be instructed that contracts are “standard” and certain 

terms may not be negotiated. One grantee thought that the process might be different for large States 

Vendors do not always understand 

CAHs and small facilities; vendors found it difficult to conceptualize how their products work in a small 

setting. Others found the initial selection process cumbersome: “It’s like putting together a 1500-piece 

puzzle.” 

 

16 Demiris, G., Courtney, K., & Meyer, W. (2007). Current status and perceived needs of information technology in Critical 

Access Hospitals: a survey study. Informatics in Primary Care 15(1), 45-51.
FLEX CAH HIT Evaluation- Final Report

 25

with a large IT investment because those States can leverage their size to get the full attention of the 

vendors.

Interfacing different systems (7 grantees). One of the main technology barriers involved the difficulty of 

integrating different systems, which was a “barrier to quick and efficient implementation.” Interfacing 

expenses, difficulty finding compatible software, and different vendor systems at different hospitals fell 

into this category. “Major programming issues” requiring time-consuming and costly “work arounds” for 

Patient Indexes and other essential pieces of the HIT system frustrated implementation. In at least one 

case, even hospitals that shared leadership chose different vendor systems. 

Infrastructure (5 grantees). Both expected and unexpected infrastructure needs were barriers. In some 

cases smaller provider practices could not afford adapters to get onto the data network. The grant in 

some cases became the default for providing resources as simple as cabling in older facilities. One 

grantee found that the hospitals did not have the required interface they had claimed to have. Another 

grantee found it hard to find technical and IT personnel willing to support rural hospitals.

B.2. Partner and health system issues

Partner readiness (5 grantees). The ease of implementation may “boil back to readiness.” In some 

cases, the relationships between CAH HIT partners ended up not being as strong as anticipated. Some 

providers did not have a sense of urgency about their involvement and participation. One grantee found 

that the tertiary center in the project lacked a desire to be flexible with a “very closed and controlling 

system unwilling to engage in discussions about how to make the project work.” Another grantee 

experienced trouble with one specific partner who did not want to change. Finally, one grantee found 

that the biggest challenge was undertaking a project of this magnitude in a rural area where many 

hospitals do not have fully implemented EHRs.

Communications and leadership (4 grantees). Communication and leadership issues were also 

challenges. In one grantee’s CAHs, the IT manager and the CEO did not involve their staff, making staff 

more resistant to challenges. History between particular members in the network can cause 

communication problems when the partners are not supporting transparency, and when political strain 

prohibits some of the necessary networking and connecting. One grantee experienced problems with 

buy-in from the ancillary partners because grant administrators did not have the authority to push the 

smaller non-integrated partners, and the larger, authoritative partner paid little attention to the smaller 

partners. Another grantee experienced problems when the health network board members became 

“more concerned with doing their own thing” after a personnel departure left a leadership vacuum. 

Projects with “a lot of moving parts” like these require that partners communicate about the concept 

and ultimate goals.

Contract negotiations (4 grantees). Grantees experienced difficulties setting up final agreements and 

getting clinics and tertiary hospitals to commit to the project; one experienced a six-week delay in 

contract negotiation with some of the grant’s major partners. Another grantee found it difficult getting 

CAHs to agree to the specific software. Contracts require legal review, a step that may be difficult for 

rural partners who may “interact with their legal team only once a quarter.”

Governance (4 grantees). The complexity of large hospital systems also posed a problem for some 

grantees. One State has regional health boards as well as corporate hospital boards, creating confusion 

over who has the authority to make decisions and creating the potential for “Balkanized EHRs.” Another 

grantee found that the lack of understanding of all the legal barriers and bureaucracies involved in the 

project was the biggest barrier; governance issues with patient compliance, patient consent, and privacy 

officers held up the project timeframe. A final governance issue was the receptivity and involvement of 

the consumer and whether consumers have input into the shape of the system.
FLEX CAH HIT Evaluation- Final Report

 26

B.3. Workforce Issues

Culture (10 grantees). More than half of the grantees experienced some cultural issues: “You’re never 

prepared for the human reaction to the actual implementation.” One grantee found that physicians 

were not as receptive to the technology as had been hoped, and several found that one community or 

hospital was much easier to work with than another. Grantees referenced a lack of general IT exposure 

among many staff, which is a widely recognized challenge in HIT adoption that is more common in rural 

settings. Another grantee experienced pushback from physicians who wanted custom-designed 

templates and had little power to put pressure on the physicians because the hospital would have 

difficulty hiring more. One grantee called doctors “creatures of habit” and another said that physicians 

are “very strong willed.” Elderly, rural, or simply technology-adverse end users presented other cultural 

challenges. While culture is often a hurdle for HIT implementations, it has been noted that provider and 

community buy-in is more difficult to obtain in rural communities, and this particular challenge was 

highlighted in the National Advisory Committee on Rural Health and Human Services 2006 report to 

HHS.17

Shortages and turnover (5 grantees). Workforce issues included staff turnover in rural areas and an 

inability to hire the right person for the job due to a small pool of applicants. Grantees have to take into 

consideration what happens if there is turnover in a key position; one recommended “session 

management” where “somebody knows everything that I know. Not one person holds all the 

information, so if someone quits, the next person can replace that knowledge.” Rural and smaller 

hospitals had particular difficulty attracting and retaining IT knowledge. 

B.4. Fiscal Issues

Lack of funding (8 grantees). Lack of funding was a concern for many of the grantees, ranging from 

unexpected costs to the realization that some sites lack the resources needed to successfully implement 

the project. The current financial climate and strained State budgets worry many of the grantees. 

Financial barriers include hefty maintenance fees for the HIE software, the cost of setting up a Helpdesk, 

needed funds to hire knowledgeable staff, unanticipated costs such as interfaces between records and 

billing systems, and pending bankruptcy of community partners. One grantee had to be “extra careful 

watching small budget items like conference calls and travel.”

Sustainability (7 grantees). Grantees were concerned about future funding. One grantee articulated the 

fiscal challenge as “how do you slow down the freight train?” They have several hundred people 

working on the project currently, and after the grant ends, they will still have 35 clinicians designing the 

in-patient EHR, and a similar number designing the outpatient EHR. They “just can’t stop that process.” 

One grantee is hoping that the hospital corporation will recognize the value of the HIT and ensure that 

the implementation is maintained, and others are searching for new State or Federal grant money to 

continue or expand their systems. 

B.5. Administrative Issues

Timeframe (8 grantees). Many grantees had difficulty with the timeframe: “It became very clear to us 

doing these projects, that by the time the project was up and running, the time to do the assessments 

would not be there.” One grantee thought the timeline was too compressed during the initial 

assessment of the project, leading to inadequate due diligence. The inflexible, structured grant schedule 

was challenging and created extra work for administrators. Staff illness, the need to map and plan 

 

17 The National Advisory Committee on Rural Health and Human Services. (2006). The 2006 report to the secretary: Rural health 

and human services issues. Retrieved from ftp://ftp.hrsa.gov/ruralhealth/NAC06forweb.pdf
FLEX CAH HIT Evaluation- Final Report

 27

workflows, and competing issues and projects all added to timeframe stresses. Implementing the grant 

is “extremely complicated” and if “you’re shoving it through a very aggressive timeline; you’re setting 

yourself up for problems.” 

C. Lessons Learned

Grantees were asked to reflect on the lessons they have learned about HIT implementation. Their 

thoughts are summarized here, organized by theme, ranging from most to least frequently mentioned 

lessons.

C.1. Project Management (11 grantees)

Several grantees said that key lessons learned included flexibility, adaptability, and patience. Project 

managers must be conscious of change management issues and keep in mind their primary goals to ensure 

success: “A lot of times when you get a big grant, you just go out and think you can conquer the whole 

world.” Grant partners must have confidence in the project management. Partners need to think that their 

perspectives are important and that their needs will be addressed and heard. As such, programs must

accommodate the needs of different partners, such as by arranging training sessions so that they are not 

disruptive to the clinical practices. Realism about the readiness of partners and the ability to ameliorate 

personality conflicts between partners are important aspects of project management. One grantee found 

that “there is a lot of sensitivity to how project management discusses and approaches things” and 

managing relationships, conflicts, and flagging motivation is important to keep a project on track.

C.2. Early Planning (10 grantees)

Grantees emphasized the need to plan early and plan well: “You can never plan enough. The meetings 

you have may seem trivial or a waste of time, but as we look back, we understand it was well worth our 

time.” Partners have different levels of knowledge and comfort with technology; a long planning process 

helps the project advance on a level with which all partners are comfortable. Methodical planning is 

required to ensure that existing systems and workflows can be translated to digital records in the most 

efficient and effective way possible; planners need to be “brutally honest” about how well existing 

methods function because “taking an inefficient process and having it run faster is not effective.” It is 

difficult to backtrack at the point of implementation if the team realizes that everyone has not been 

included or the process mapping was not ideal. Early and thorough planning is needed to ensure that 

the technology solutions selected perform as expected and, importantly, are compatible with existing 

legacy systems. Planners must “engage early and engage often.” Project managers and administrators 

must have a plan for the unpredictable.

C.3. Ensuring Staff Buy-in (10 grantees)

Grantees recognize that it is extremely important to have full buy-in and ongoing participating from all 

staff: “If the people who are going to be using the system don’t believe in it, it’s going to fail.” End users 

must be involved in the decision-making process to help create a level of comfort with and buy-in to the 

system. HIT is “not an IT project; it’s a clinical project that’s supported by IT.” Another grantee agreed 

that HIT is a clinical project and a quality of care project that requires buy-in from both the 

administrative and the Chief Medical Officer levels. Top-level buy-in is critical to ward off resistance 

from lower-level staff. Two grantees emphasized the need for a physician champion or “evangelist” at 

each organization to secure buy-in. One grantee felt that it would have been helpful to have a liaison or 

champion to go between the IT staff and the physicians to overcome terminology barriers. Program 

planners should not assume that some staff will resist the new technology: one grantee was surprised to 

find that the older nurses loved the new system.
FLEX CAH HIT Evaluation- Final Report

 28

C.4. Financial Planning and Fiscal Management (9 grantees)

Grantees emphasized the need to recognize the magnitude of expenses involved in a project of this 

nature and carefully manage the budget. One grantee learned that its grant writers should have 

allocated funding more carefully for a larger tertiary facility, and another learned that IT costs can drain 

even budget surpluses relatively quickly. One grantee was stymied by a delay getting legislative 

authority to spend funds and had to entirely rework the budget due to the delay. Unexpected expenses 

can cause stress, as one grantee that had to divert funding to infrastructure needs and to the 

development of interfaces between vendors found. HIT implementation involves many unknowns that 

can be costly from a financial perspective. One grantee found that fiscal management was a drain on 

resources, requiring several staff to track financial resources. Finally, planners must consider the 

financial resources of partners because rural hospitals may lack the money or resources to sustain HIT.

C.5. Time Management (5 grantees)

Five grantees noted that HIT projects can take more time than expected. The 18-month timeframe for 

this grant was difficult for some grantees. One felt that this process usually takes place over three or 

four years of negotiation and securing clinical involvement. The timeframe is especially challenging for 

small hospitals without extensive resources and staffing. Implementation in CAHs that have limited 

financial and personnel resources needs to be at a pace that the hospital can absorb. Additionally, the 

implementation can lead to a temporary reduction in operational efficiency as partners become used to 

the new workflow and processes. State processes can also create unexpected delays and grant 

administrators need to be more realistic about the timeframe, allot more time to the startup period of 

the grant, and “tread lightly before engaging fully.”

C.6. Communication and Coordination (5 grantees)

Ongoing communication is a necessity for a project with many different partners. First, obtaining input 

regarding participants’ wants and needs is important to ensuring that expectations are met. One 

grantee found that this process includes explaining to the community what a health information 

exchange (HIE) is. Planners need to recognize that articulating a clear message is critical to getting the 

point across. Second, several factors can make it difficult to get multiple CAHs working together and 

moving in the same direction. Communication is a critical success factor because there are a number of 

steps, such as coordinating downtime for system maintenance, which must be executed concurrently. 

Finally, communication between the technical side of the project and the clinicians can be difficult; one 

grantee administrator felt that this project taught her that she should not assume that clinicians know 

everything about HIE and that she will be sure to serve as a conduit between the technical side and the 

clinical side in the future.

C.7. Training (4 grantees)

Staff require adequate training; one grantee found that they had not budgeted enough for training. 

Another found that its vendors had underestimated the training needed and learned that it is important 

not only to purchase the applications but also to dedicate resources to ongoing support until the staff is 

comfortable using the resources. One facility whose staff lacked computer experience set up a 

classroom of computers and allocated time for staff to “fiddle around” with computers. This is a creative 

solution for supporting staff who may have a fear of new technology. Finally, ensuring that some 

internal staff can service and maintain the hardware and software can cut down on costs and ensure 

that the system is in the hands of people who are dedicated to patient safety: “It’s better to put money 

into staff education than rely on an outside vendor.”
FLEX CAH HIT Evaluation- Final Report

 29

D. Impact on Operations and Outcomes

D.1. Availability of Data (9 grantees)

The most common impact on operations was the increased availability of and access to patient data. 

The ability of staff to see patient information quickly from any computer in the hospital is helpful, and 

replacing faxed information with real-time access to data has had an impact on operations. Some 

grantees have not yet fully implemented their systems but expect to see an increase in the availability of 

data. One grantee expects video conferencing, telemedicine, and electronic document transfers to 

greatly affect the availability of data. Another grantee anticipates having more ease of use, less time 

spent pulling patient records and making paper copies and a system that reduces duplicative tests 

between local and tertiary hospitals which will improve the continuum of care between the rural sites 

and the tertiary facilities. Finally, one grantee has observed that the quality of data is improving and 

becoming more reliable as the data fields become standardized between providers.

D.2. Provider collaboration (8 grantees)

Three grantees feel that provider collaboration has been improved by the project; one said that 

clinicians are especially satisfied with the ability to access information from external locations. Another 

grantee emphasized its “team development model” and noted that the way that the relationships 

developed have allowed for quicker decisions about transfers. 

Five grantees feel that provider collaboration will be improved once their project has been fully 

implemented. One grantee said that between 20 and 30 percent of its clinics cannot exchange 

Continuity of Care Record (CCR), Continuity of Care Document (CCD) or Clinical Document Architecture 

(CCA) data formats currently, and another is trying to install a monitoring system. The clinicians will have 

access to information they did not have previously when a patient was referred from another clinician. 

Finally, one grantee has found that the implementation process had started for the first time a 

Advice to Other CAHs

• “Don’t think that you have to solve everything yourself. The resources are available outside – having a 

knowledgeable consultant to advise [our project] with vendor selection was invaluable and saved money and 

grief in the long run.” 

• “Identify [your] champions early on. Especially in the CAH environments where there is a shortage of 

physicians, identifying nurses that will be the clinical applications coordinators and giving them the time they 

need to spend on the project is very important.”

• “Make sure the entire team and hospital staff gives input at the very beginning of the process on what they 

feel like needs to be accomplished, so they feel like it’s their system also, rather than something that’s just 

been forced down their throat.” 

• In terms of vendor selection, make sure that the RFQ/RFP has all the necessary information. “Don’t get 

drawn into some vendor salesperson giving you ‘everything’s great’ – make sure you do your vendor 

evaluation and selection in a very detailed way.” 

• “The most important thing, especially when dealing with a short timeframe, is selecting the right community 

and ensuring that these communities are ready to deal with change and make the projects successful. Do an 

in-depth culture assessment to ensure that the full community, not just a few individuals, are ready for 

change.”
FLEX CAH HIT Evaluation- Final Report

 30

discussion about how different entities chart and how they use EMR. This grantee feels that this will lead 

to more information-sharing and help providers work more efficiently.

D.3. Patient Outcomes (7 Grantees)

Seven grantees described the positive patient outcomes generated by the implementation. Positive 

impacts include constant access to patient charts for physicians, which lead to better care coordination 

and better outcomes. Grantees also noted marked improvements in point of care areas such as 

treatment plans, medical reconciliation, and medical errors. For some grantees, the implementation 

has led to improved staff development and relationships between the CAH and network hospitals, 

allowing quicker decisions about patient transfers.

D.4. Data tracking (5 grantees)

Three grantees felt that there was a positive impact on data tracking; the ability to share data between 

remote clinics is important. One grantee found that radiology results are returned much more quickly 

and that use of couriers has been reduced. Two grantees felt that it is too early in their projects to 

measure the impact on data tracking. 

D.5. Data entry and management (3 grantees)

Two grantees believe that the electronic health records and other technological improvements will 

improve hospital reporting, data entry, and the flow of the patient record. The data entry will be 

streamlined and duplicative entries will be reduced. A third grantee feels that once providers and staff 

are used to the system, there will be more rapport between the nurses entering patient data and the 

patients. 

D.6. Quality of Care (2 grantees)

One grantee feels that there will be a positive impact on quality of care because there will be less 

chance of miscommunication between medical staff. With the paper-based system, physicians never 

directly touched the patient record and instead gave verbal instructions to the staff which increased the 

likelihood of medical error. Another grantee’s rural hospital has used the technological improvements as 

a marketing tool, advertising local state-of-the-art stroke treatment rather than having patients rely on a 

three-hour drive or air transport to a larger hospital for treatment. This project also used its system to 

make provider training available at any point in time, which will enhance the delivery of care in the rural 

area. In rural areas, “you basically had to go in physically to take a course, and spend a lot of time 

traveling a great distance. It makes perfect sense to use the technology to make stroke training more 

accessible and efficient for EMS providers in remote areas.” 

D.7. Efficiency (2 grantees)

Two grantees discussed the impact of the implementation on efficiency. Both discussed the upfront 

costs of such a change in processes that “a lot of people may not realize” such as a period of reduced 

productivity due to disruptive changes. However, one of the grantees notes that of its 18 providers all 

but 3 are back up to their baseline levels of productivity and the remaining three are nearly recovered. 

The project does not yet have good evidence that buy-ins are increasing above the baseline, but they 

are seeing the operational efficiency increase at the practice-level. For instance, all relevant information 

is documented in the chart, without the reliance on transcription and medical records that there was 

before the EMR/HIE implementation stage.
FLEX CAH HIT Evaluation- Final Report

 31

E. Evaluation

Grantees were required to conduct an evaluation of their individual Flex CAH HIT project, and also 

participate in the national evaluation of the Flex CAH HIT program.

Five grantees have conducted formal assessments of the HIT implementation: 

• Staff satisfaction. Four grantees measured staff satisfaction. Two grantees took baseline satisfaction 

surveys. One posted that data to the AHRQ website to share with other projects. A third grantee 

also used a staff satisfaction survey and plans to survey each partner six months after the 

implementation date. One hospital found that satisfaction levels have increased dramatically in the 

year since implementation. Finally, the fourth grantee looked at the preliminary reactions of the 

physicians to the technology and is considering continuing the assessments. This assessment found 

that there were differences in satisfaction depending on the type of site: rural physicians were 

generally more satisfied because the smaller environment made it easier for physicians to access the 

system. At the large State hospital, doctors found it more challenging to use the system in a bigger 

and more hectic environment. 

• Patient satisfaction. Two grantees are either using or will use a formal measure for patient 

satisfaction. One is currently collecting patient satisfaction data for comparison, and the other has 

measured patient satisfaction continuously. This grantee found that patient satisfaction, for a brief 

period of time, went down. However, it is now above the baseline level because patients are 

noticing the advantages of having technology in the practice, and they like that physicians have 

more access to their information. The doctors can even see information on patients who have had

something done at another hospital within the county.

• Other assessments. One grantee had a vendor assessment and negotiated a week of training with 

the vendors as a result of that assessment. 

Three grantees conducted informal assessments of their system. One grantee administrator is aware of 

some informal feedback from providers; most providers do see the value of the system, and the more 

resistant physicians are becoming more convinced that EMR are a positive. Another grantee agreed that 

most staff are coming around to the system, but that it is still new, especially for the older staff at more 

rural hospitals: “These people were used to doing things a certain way for the past 10 to15 years.” The 

informal feedback for the third grantee has also been generally positive. Four grantees do not have any 

kind of assessment. 
FLEX CAH HIT Evaluation- Final Report

 32

F. Sustainability

Grantees discussed their plans for sustaining their implemented systems, including the barriers they 

face, the funding they have sought to complement or continue projects implemented with Flex CAH HIT 

funding, and other ways in which they have sought to sustain their projects.

F.1. Barriers to Sustainability

Funding (7 grantees). Grantees most commonly cited funding barriers. The lack of money and the 

expenses associated with maintenance and interfacing were common concerns. One grantee noted that 

the infrastructure for HIT is present and that all problems can be solved with the right funds: “It’s just 

going to come down to if they can afford it.”

Other (5 grantees). Other barriers included a prohibitive local environment and the need for discipline 

and “doggedness” on the part of project managers and partners. The problem of the maintenance of 

software upgrades over time was mentioned by two grantees. Finally, a few grantees had operational or 

workflow barriers, including problems with vendors’ abilities to follow through with service agreements 

and software upgrades.

F.2. Funding Sought

Six grantees have sought grant money to help with sustainability. Sources include Federal, State, and

private funding, including the Medicare Flexibility Grant, grants from the Agency for Healthcare 

How Grantees Define Success

Utilization (7 grantees). Utilization was the most commonly mentioned definition of success: a system that “does 

get used and handles people’s needs.” Success is “getting the system up and information flowing into the portal” 

but this is only the first step in a long process of expanding to all hospitals in the system. One grantee stated that 

the primary success will be when a clinician at the tertiary hospital sees a transfer patient and all of that patient’s 

records from the referring hospital are available. After the patient has been treated and returned to the local 

community, the attending physician will be able to access that patient’s updated records seamlessly.

Clinical outcomes (4 grantees). Clinical outcomes, including HRSA-specific measures, are a longer-term 

measurement of success, which may takes months or even a year or two to demonstrate impact. Due to the small 

size of rural hospitals, there are not yet enough data for one grantee to draw clinically significant measures. The 

measures to track impact on patient care include medication errors, medication reconciliation, diabetes, stroke, 

congestive heart failure, and improved communication with EMT throughout the continuum of care.

User satisfaction (4 grantees). Physician and user satisfaction with the technology is a necessary measure of 

success: “Our measure of success is that we have a product implemented that is a valuable tool for our providers 

and that it qualifies us for our incentive payments.” One grantee would like to do a survey of this measure, and 

another has already developed user satisfaction surveys.

Patient satisfaction and care (3 grantees). Patient satisfaction and quality of care were important measures of 

success for three grantees: “Our success is first and foremost – that we can improve our patient care, and that 

the system will allow us to give better patient care.” 

Interoperability (2 grantees). Standardization of various pieces of the implementation and getting critical 

interfaces between systems build are measures of success for two grantees.

Meaningful use (2 grantees). Two grantees specifically listed compliance with the provisions for meaningful use 

of an EHR in an inpatient setting as a goal.
FLEX CAH HIT Evaluation- Final Report

 33

Research and Quality, State funding to address annual maintenance fees for the installed software, and 

private funding to purchase additional infrastructure.

Four respondents either had not sought additional funding or were not aware if additional funding had 

been sought. One has had budgeting discussions but no grant seeking, and another was considering 

pursuing a grant for the development of a community report card that would compare data locally to 

national-level HIE data.

F.3. Planning and Support

Budgeting for sustainability (9 grantees). Most of the grantees discussed how they had incorporated 

sustainability into project or hospital budgets: “We knew we couldn’t go forward with this unless we 

could sustain it.” Hospitals in particular were said to “have budgeted for everything” and were expected 

to support their own EMRs. One grantee said that each of its partners understands and will be able to 

sustain their individual applications. The ability to sustain the grant was one of the hospital selection 

requirements for one grantee. Another’s community hospital board included one year of sustainability 

in their budget. Finally, one grantee included sustainability in their cost support and plans to reinvest 

their depreciation returns: “If you’re going to be sustainable, you’ve got to be disciplined.”

Expansion to new partners (5 grantees). Five grantees have sustainability plans that include bringing in 

new network members to spread expenses, hosting fees, maintenance fees, and upgrade fees across 

hospitals. Using a model and applying it statewide helps create a project that can be sustained within 

the State system. One grantee had plans to bring in other hospitals using Federal earmarks or stimulus 

funds. One grantee noted, “By working together instead of trying to be separate all the time we can 

show rural America that collaboration works and is sustainable. If each facility could find their niche and 

we could build off each other we could help sustain each other. Because, in these rural areas, if the 

hospitals and schools go away, the communities themselves dry up!”

G. HRSA’s Role in the Implementation

Twelve grantees discussed ways in which HRSA was helpful during their implementation:

Monthly group calls (8 grantees). Overall, the monthly group calls were the most commonly mentioned 

help provided by HRSA. Grantees appreciated the opportunity to share with other States and learn 

about their projects. One grantee would like to continue to network with other similar and dissimilar 

projects from around the country in the future, to get “in step” with how projects are progressing. 

Providing technical assistance (5 grantees). Five grantees felt that the technical assistance provided by 

HRSA, including access to an evaluator and other resources, was helpful. HRSA provided language to use 

on an end-user agreement for the HIE and provided other information and resources as requested by 

the grantees.

Help with a vendor (3 grantees). Three grantees referenced an issue with one of the vendors that HRSA 

helped to resolve. The HRSA Project Officer “arbitrated” with some of the participants and HRSA was 

“willing to step in when some issues with a couple of the vendors occurred.”

In-person meetings (2 grantees). Two grantees felt that the national and regional meetings were 

helpful, providing them with “face time” and giving HRSA staff a chance to meet with the State staff 

involved in the project.

Other (4 grantees). Other ways that grantees described HRSA as being helpful include:

• “HRSA has been active, not intimidated by the scope of the program, and deserve a lot of credit, 

including for bringing objective evaluation folks on...”
FLEX CAH HIT Evaluation- Final Report

 34

• HRSA gave updates and communicated clearly. HRSA has provided “invaluable information and 

resources.”

• Having a personal project officer (PO) was helpful – “whereas with some of the other grants, they 

barely even knew who we were.” The POs were very responsive to the grantees.

Only two grantees described ways in which HRSA was not helpful. One grantee found it difficult to 

process the amount of information that HRSA provided: “I personally couldn’t juggle learning that 

website and all the tools that were out there in addition to all the additional things that were going on.” 

This grantee felt that HRSA needed to be more liberal on its timelines. The other grantee felt that CVD 

and diabetic measures were irrelevant to what its grant was trying to accomplish and thought that 

hospitals needed to use quality indicators that specific to them.

H. Suggestions from Grantees for Future Grant Programs

Grantees provided suggestions to HRSA for future grant programs of a similar nature:

More technical assistance (6 grantees). Many grantees had requests for specific TA that would have

been useful in the project, including:

• Mechanism for allowing grantees to share resources and documentation, such as a knowledge 

center for HIT

• Ability to network with prior HIT grantees in the event that another round of Flex CAH HIT is funded

• Additional TA for rural areas—rural grantees have a difficult time articulating what they need, do 

not understand what questions to ask, and lack access to resources such as quality broadband 

services

• Help building a consortium among rural care facilities

• Development of a national privacy model as privacy and security are an issue that “we’re going to 

plow into in a big way”

• Model for and TA on sustainability and funding justification

More time (5 grantees). Many grantees felt constricted by the 18-month timeline and felt that more 

time allotted would have allowed the implementation to proceed “at a more orderly pace.” The tight 

timeline resulted in “a lot of corners that needed to be cut” because grantees did not have time to 

properly deal with unexpected personnel and change management issues. 

Vendor help (3 grantees). Three grantees specifically suggested that HRSA develop issues and policies for 

dealing with vendor selection and contract development: “There is a lot of vendor activity promising the moon” 

but hospitals are not used to understanding vendor contracts, meaningful use rules, and other vendor-related 

issues. One grantee felt that HRSA has an opportunity to “push the vendors and keep them in check.” 

Funding (3 grantees). Two grantees specifically requested increases in the travel budget so that grantee 

staff can travel to key national IT meetings and other valuable learning opportunities such as grantee 

meetings to share results and best practices, and tease out results that others around the country were 

having with their projects. The other funding related suggestion was simply more funding to cover the 

unexpected costs of interfacing.

The following sections describe in detail the case studies conducted in four select sites (University of 

North Dakota Center for Rural Health, Oklahoma State University Center for Health Sciences, South 

Carolina Office of Rural Health and Board of Regents of the University of Wisconsin System), as well as 

the results of a return on investment evaluation conducted at these sites. 
FLEX CAH HIT Evaluation- Final Report

 35

Case Study 1: Flex CAH HIT Network 

Implementation Grant, University of North Dakota

Center for Rural Health

A. Introduction

The goal of the North Dakota Flex CAH HIT Network Implementation Project was to facilitate the 

exchange of health information by implementing a patient-centered electronic medical record (EMR) 

along the continuum of care --facilitating patient safety, efficiency, and effectiveness of health care 

services. The objectives of the project were to:

• Assist CAHs with the implementation of EMRs and the sharing of patient data with at least one 

ancillary facility within their respective communities.

• Facilitate the exchange of patient information between the CAH, ancillary facility, and the regional 

tertiary center.

• Strengthen the regional network to address HIT needs and issues. 

Three North Dakota CAHs, (Northwood Deaconess Health Center in Northwood; Pembina County 

Memorial in Cavalier; and First Care Health Center in Park River — all part of the North Regional Health 

Alliance), one tertiary referral hospital (Altru Health System in Grand Forks), and several ancillary 

providers associated with the selected CAHs (Valley Community Health Centers, Wedgewood Manor 

[long-term care facility], CliniCare, and First Care Rural Health Clinic) as the network in this pilot project. 

The participating CAHs did not all choose the same provider for their EMR, and therefore individually 

focused on the implementation of those different EMRs and in the implementation of Clinical 

Workstation – a one-way portal between the rural facilities and the tertiary facility, Altru Health System.

Figure 1 below depicts the three EMRs implemented within the three rural facility sites and the 

information-sharing relationships with those CAHs and their clinic and long-term care facility partners. 

Also depicted are the paths of the Network’s information exchange implementation of Clinical 

Workstation.
FLEX CAH HIT Evaluation- Final Report

 36

Figure 1. North Dakota Implementation Map

B. Background

The Flex CAH HIT grant opportunity was aptly timed for the North Dakota Network – the planning and 

implementation project would build upon previous and ongoing health IT projects statewide in North 

Dakota, and at the specific facilities themselves. Upon learning about the Flex CAH HIT grant 

opportunity, the State Flex program director and the State Office of Rural Health (SORH) program 

director, both of the North Dakota Center for Rural Health at the University of North Dakota, identified 

two networks within the State that they felt were working well together and would be best qualified to 

participate in the Flex CAH HIT grant. The Center for Rural Health pre-selected Northwood Deaconess 

Health Center, a CAH, to participate in the Flex grant, as it was believed the community health center 

associated with Northwood, Valley Community Health Centers, would help make North Dakota’s 

proposal to HRSA more competitive nationally. Additionally, Altru Health System, the tertiary facility, 

was willing to work with the rural providers in the North Regional Health Alliance. A competitive process 

was then used to select the two remaining hospitals; all eligible hospitals in the Network applied. Two 

hospitals were ultimately selected: First Care Health Center in Park River and Pembina County Memorial 
FLEX CAH HIT Evaluation- Final Report

 37

Hospital in Cavalier. Based on their applications, both were deemed to be at an appropriate level of 

readiness and commitment for the Flex project. 

C. Existing Infrastructure

Prior to the Flex CAH HIT-funded EMR implementation, each of three CAHs used an electronic system for 

financial purposes. Pembina had a PACS medical imaging system, which had not yet been integrated into 

the full EMR as of May 2009. First Care had software in its exam rooms that could be used to pull up the 

PACS system, as well. Both Pembina and First Care also had an electronic lab system. Northwood had a 

completely paper-based clinical system prior to the Flex grant activities. 

In preparation for installing an EMR system, First Care Health Center had recently wired its new facility 

to support electronic health information processes. This IT infrastructure update began in 2005 

concurrently with an addition and renovation of the hospital building and construction of a clinic, 

funded with a USDA loan, capital campaign, and local sales tax. 

The providers at all the participating hospitals and clinics also had portal access to patient information 

through Altru's Clinical Workstation application prior to the Flex grant, but First Care and Pembina were 

not using it. The Community Health Center at Northwood was previously an Altru clinic, so Northwood 

staff were familiar with Clinical Workstation. 

In addition to existing infrastructure in individual facilities, Blue Cross Blue Shield funded the North 

Regional Health Alliance with a networking grant of $85,000 to collaborate on HIT. The Network is using 

these funds to build a larger regional HIE. 

D. Planning Approaches

Pembina and First Care underwent markedly different planning processes. Pembina’s process was 

informal and relied on the chosen vendors for design of the facility’s EMR workflow. Pembina staff 

reported, however, that given the progress of the EMR implementation to date, it would have been 

advisable, during the planning stages of the implementation, to examine the workflow in detail; an 

updated workflow (adjusted for EMR use) should properly meet the needs of the facility’s existing 

policies and procedures. 

In contrast, First Care respondents felt that it would be difficult at best to be fully prepared for the 

changes in workflow resultant of an EMR implementation. First Care’s CIO designed a readiness 

assessment for the facility and ultimately chose the EMR vendor. This decision was driven by the fact 

that the previous electronic system used for the financial portion of hospital operations did not have the 

ability to support a clinic EMR. 

Involvement of clinical stakeholders in planning

Both Pembina and First Care involved clinical stakeholders, including physicians, specialists, and nurses, 

in planning, to various degrees. In terms of champions for HIT, the nursing staff (reported specifically at 

Pembina) were very supportive during the EMR planning phase. Various clinical stakeholders at both 

Pembina and First Care pushed for the incorporation of lab, radiology, medical records, and nursing 

paper-based records to the electronic system. At Pembina, all physicians and one PA were involved in 

planning, as well as several physicians at a competing, but interconnected, clinic. These providers were 

included in staff meetings in which plans for the EMR implementations were discussed. At First Care, a 

PA, family physician, and surgeon participated in the project planning phases. The CEO of First Care 

directly discussed the Flex grant opportunity with their providers, and requested input and support for 
FLEX CAH HIT Evaluation- Final Report

 38

the implementation. First Care respondents added that all of the facility staff were involved to some 

degree with the application demonstrations before the vendor was selected. 

While EMR planners sought out their perspectives and participation, the enthusiasm and level of 

participation in planning varied widely among clinical stakeholders. For instance, one provider at 

Pembina wrote a list of wants and needs for an EMR; several made time to watch the vendor 

presentations and were quite involved in terms of examining and comparing vendor workflows; while at 

least one provider wanted no involvement in the implementation. Similarly, at First Care, one provider 

was resistant to the idea of implementing an EMR. The sentiment expressed by this provider was 

essentially, ‘if we are going to have to do it, we might as 

well do it now when the grant money is available.’

Vendor selection

Originally, the North Regional Health Alliance anticipated 

that installing the same vendor system at each facility 

participating in the Flex CAH HIT grant would be easier and 

more sustainable, given the potential to use a sharedstaffing model. However, the facilities were unable to reach a consensus about a single vendor, in part 

because facilities had decided to move forward with selecting an EMR at different times and with 

consideration for the variety of legacy systems in place at those facilities. First Care and Pembina chose 

the same vendor; Northwood chose a different vendor. CAH respondents reported that this difference 

in vendor selection between hospitals was “really not an issue” given that data interactions were to 

occur almost solely with Altru through the HIE, rather than directly between hospital EMR systems. 

Pembina CAH examined three vendors as candidates for the EMR implementation. The CEO and CFO, in 

addition to clinical stakeholders, contributed to the selection process. Ultimately, the final selection was 

chosen because this vendor already hosted the facility’s financial system, and CAH planners felt that it 

was important to integrate the existing system with the clinical EMR under a single vendor. 

First Care staff developed a number of elements which planners agreed were vital to an EMR system. 

These included price, Certification Commission for Health Information Technology (CCHIT) certification, 

the ability to interface with systems already in place, and sustainability. One vendor was eliminated 

simply because the EMR came in well over the CAH’s budget of $700,000 to $800,000. Another vendor 

considered was also decided against because while it was an “impressive” system, according to 

respondents, it would have likely been hard to interface with other systems. Clinical stakeholders 

(including the PA, most nurses, and staff from both lab and radiology departments, as well as the 

business office), chose their vendor because it met all the basic criteria for cost and interfacing, and had 

minimal hidden costs—an indicator of long-term sustainability.

Northwood respondents explained that because their facility is associated with a long-term care facility, 

they chose to consider a vendor that would offer lab, clinic, long-term care, and therapy components. 

Aside from these criteria, Northwood also was interested in choosing a vendor system based upon 

relative value (cost) and hardware and database language requirements. One vendor offered a “deep 

discount” relative to another vendor that Northwood planners were considering. Additionally, four 

other area hospitals not immediately involved in the activities of the Flex CAH HIT grant notified 

Northwood that they would be willing implement this vendor’s system if the vendor was capable of 

consolidating all the facilities’ information into one datacenter. Combined, these factors were enough to 

prompt Northwood to implement the EMR with this vendor. 

CAHs selecting different vendors for their 

EMR was “really not an issue” given that 

data interactions were to occur almost 

solely with the tertiary system rather than 

between CAH EMR systems.
FLEX CAH HIT Evaluation- Final Report

 39

E. Staff Training

Northwood. At Northwood, staff training on the EMR was rolled out in three phases: in the first phase, 

six or seven users were sent off-site to the vendor’s headquarters in Omaha, Nebraska. These were 

“super-users,” identified to train other end-users when they returned from Omaha. The Northwood CIO 

reported that the training went very well, while it was “a lot to learn in a short time.” Following the 

training, super-users were given practice time on a test database at Northwood. The second phase of 

training consisted of clinical super-users working with other end-users at Northwood. The third phase 

focused on training staff at the long-term care facility. 

Every clinician at Northwood was trained on the EMR system. Super-users worked with employees oneon-one or in small group settings, without a training manual. Respondents reported that there were 

multiple ways to navigate the EMR system, and therefore training was customizable depending on the 

end-users being trained, and their clinical responsibilities. Rather than relying on a training manual, 

training was therefore developed based upon departmental needs using printout screens (screen shots) 

of the system. Both of Northwood’s physicians were assigned to nurses who had already undergone 

training. 

First Care. At First Care, three financial staff members received training initially. The CFO reported the 

financial training went well, and noted the importance of having staff trained on both the financial and 

clinical modules of the EMR simultaneously. The selected vendor assigned an onsite clinical specialist to 

train end-users on the EMR, but respondents also felt they had been able to learn much of the system 

components on their own, through trial and error. Two weeks after the lab and radiology components 

went live (June 2009), the vendor provided a follow-up training to clarify any remaining issues and staff 

concerns. First Care created a computer lab onsite to facilitate easy access to computers and a 

designated area for vendor training. The lab was later disassembled with hardware being placed 

throughout the facility.

Pembina. At Pembina, the financial part of the EMR was already in place by the time the Flex CAH HIT 

grant was disbursed. For the other eight clinical components, the selected vendor trained at least two 

super-users from each department onsite. Like First Care, Pembina created a computer lab onsite to 

facilitate easy access to computers and a designated area for vendor training. The lab was later 

disassembled with hardware being placed throughout the facility.

Altru - Clinical Workstation. Altru trained clinical staff from the CAHs to use Clinical Workstation, 

providing phone numbers and technical support for staff who encountered problems gaining access to 

patient records. Respondents reported that CAH staff found the training very helpful and accessing Altru 

staff for assistance to be simple. Presently, clinical staff, including nurses and dietitians, use Clinical 

Workstation.

F. Governance, Policies, and Procedures

Confidentiality and Privacy Policies

All hospitals stressed the importance of data confidentiality for staff. Pembina and Northwood set up 

guest accounts to accommodate temporary nurses or nursing assistants who do not need access to full 

patient records. IT is also vigilant about requiring end users to routinely update their passwords. First 

Care also views patient privacy as a challenge, as there has been at least one patient privacy complaint 

(which was found to be unsubstantiated). At Northwood, IT staff have examined their “acceptable use” 

policies.
FLEX CAH HIT Evaluation- Final Report

 40

Security

The CIO at Northwood stated that data are stored on a terminal server and are therefore extremely 

secure. The facility is looking to make contingency plans for the data center located in Grand Forks. A 

backup center in Omaha is being planned. . This set up is unique to Northwood.

Interoperability

Northwood plans to interface with the Community Health Center (CHC) when it implements its own 

EMR system in the future. Altru Health System has a 3-year plan to implement an EMR which will allow 

for a regional HIE system should CAHs within their service area (N=20) wish to participate. The State of 

North Dakota is also engaged in a statewide planning process to develop an HIE using lessons learned 

from this project.

G. Impact on Hospital Operations

Staff Satisfaction with the EMR 

Respondents from each hospital agreed that there had been some resistance from physicians and it 

would likely take time to get the physicians fully using the EMR systems. To address physicians’ 

concerns, the clinical coordinator at First Care showed physicians an EMR screen shot and asked them to 

tell her what was wrong with it. She reportedly found this interaction effective in incorporating 

physician buy-in and stimulating interest in the capabilities of the EMR system. The main problem, 

according to one CAH respondent was that, “the physicians want [the EMR] to be simple to use and to 

be right the first time.” First Care respondents also said that some nurses who threatened to leave the 

facility when the EMR was installed are now satisfied with it. 

Usage

Pembina has about 180 staff and about 120 use various modules of the EMR system on a daily basis. In 

comparison, Northwood has about 140 users out of about 180 staff and reported that the facility has 

about 30 users at a time on the system during peak hours. Nurses, physical therapists and laboratory 

clinicians use the EMR the most heavily. Nursing assistants use the system somewhat less, primarily just 

for charting, while as reported above, physicians are beginning to use it as they become more 

comfortable with it. At First Care, about 85 or 90 of the 110 to 120 staff use the EMR. During peak hours, 

there are about 30 to 35 users at a time. The physicians at 

First Care use the EMR only on the inpatient side to retrieve 

information, but are not yet inputting information. 

Respondents from First Care reported that laboratory 

clinicians are the heaviest users of the system. 

Tracking Data Through the Continuum of Care

First Care developed a form to track how nurses and 

dietitians were using Clinical Workstation. Reportedly, dietitians and nurses at First Care and Pembina 

use the Altru Clinical Workstation to access medical record information from Altru for inpatient, 

outpatient, and swing bed patients. First Care’s dietitian gave several examples of using Clinical 

Workstation to obtain information about diabetes patients who had received care at Altru. For example, 

she was able to check the record of a patient who had seen an endocrinologist at Altru and was 

uncertain about her insulin schedule. She was also able to check that another patient had laboratory 

work (i.e., HbA1c and lipid testing) completed at Altru and did not duplicate it, and that another patient 

had had a “swallow test” done at Altru. The dietician at Pembina found the EMR efficient for use by a 

The EMR enables a dietician consultant, 

who works only one day onsite at the 

hospital, to access patient records 

remotely, communicate with nutrition staff 

located outside the hospital, and access 

emergency room and clinic records from 

her desk.
FLEX CAH HIT Evaluation- Final Report

 41

consultant dietician who worked only one day each week at the hospital. For instance, the electronic 

system enabled the dietician to access patient records remotely, and to communicate with nutrition 

staff located outside the hospital. She was also able to access emergency room and clinic records from 

her desk—easing the review of charts and the design of 

care plans.

Communication Between Providers and 

Clinical Staff

As the CIO at Northwood stated, “poor documentation 

before [the EMR] means poor documentation after EMR.” 

He emphasized that the EMR serves the same purpose, 

organizationally, as a paper-based system, but exposes weak linkages and identifies workflow issues 

within an organization that a paper-based system simply would not. The EMR implementation has 

initiated discussion between nursing and financial/business staff that never occurred prior to the 

implementation. Respondents reported that clinical and business staff alike now view the computer as a 

tool and have become more engaged—“they are not afraid anymore.” The EMR also clarifies who is 

responsible for various situations, and with the use of user identifications, there is greater accountability 

among all hospital staff. Finally, according to First Care’s CEO, the Flex grant and subsequent EMR 

implementation has improved First Care’s working relationships with Altru. The CEO stated his belief 

that the EMR “really started people thinking about what they can do together instead of, ‘what can I do 

so that I survive and you don't.’”

Feedback from Patients

According to First Care’s Medical Records Director, patients were neutral in regards to the paper-based 

chart. Immediately following the implementation at Pembina, patients expressed a negative sentiment 

toward the EMR, because nursing staff were slower admitting patients as they learned how to navigate 

the system. However, at present, patients appreciate the system, particularly the e-prescribing 

functionality and patient care instructions. Patients’ concerns about the security and confidentiality of 

the electronic record system have also lessened, particularly as a result of newsletters and public 

education initiatives that have helped to increase awareness. 

H. Evaluation

The State Office of Rural Health contracted with John Snow Inc. (JSI) to conduct the evaluation, which 

focused on workflow, process, and lessons learned at each CAH. According to JSI’s March 2009 report, 

“Lessons Learned from Electronic Health Record Implementation at Three North Dakota Critical Access 

Hospitals,” the evaluation included onsite interviews with administrative, information technology, and 

clinical staff; reviews of facilities and infrastructure; and consideration of pertinent background 

documentation. The evaluation was designed to identify perceived successes and challenges of the 

EHR/EMR implementation, as well as any outstanding HIT issues facing each organization. 

I. Supportive Factors 

• Investment in training. CAH respondents emphasized the importance of training to a smooth and 

efficient EMR implementation. Northwood reported that “the best thing [we] did was to send 20 

people to Omaha for training” by the vendor. While training was very expensive, it was highly 

effective and well worth the cost. One respondent added that having computers available for staff 

to practice using the EMR before going live worked well. 

One CIO noted that the EMR serves the 

same purpose, organizationally, as a 

paper-based system, but exposes weak 

linkages and identifies workload issues 

within an organization that a paper-based 

system simply would not.
FLEX CAH HIT Evaluation- Final Report

 42

• Adequate IT staffing. Many rural hospitals do not have adequate IT staff. However, Pembina, First 

Care, and Northwood employed sufficient IT staff. 

• Communications between hospital departments. Throughout the implementation, ongoing 

communication and interaction between hospital departments served as a support, in addition to 

maintaining open lines of communication with the vendor. Weekly meetings from the beginning of 

the planning phase and periodic presentations to clinical staff about the status of the Flex CAH HIT 

project activities were very helpful.

• State environment. Respondents noted that the implementation benefited because North Dakota is 

a small state in terms of population, with a smoothly functioning State Office of Rural Health and a 

Flex Program with well-established relationships with its rural hospitals.

• National environment. Participating hospitals and clinical staff understood that given the Federal 

stimulus dollars and national push for EMR systems, HIT “is here to stay” and it was consequently 

important to support the EMR implementation and “figure out how to make it work.” 

J. Barriers and Solutions

• Staff turnover and training. Training new employees to use the EMR system was challenging. For 

example, at Pembina, training was problematic at the long-term care facility, because turnover 

tends to be high among nursing assistants and dietary staff. To address this drain on training 

resources, the facility planned to train key departmental personnel who were unlikely to leave and 

who would then pass on their knowledge as new employees were hired. However, one respondent 

noted that in some ways, training new staff who are not wedded to the workflow of the historically 

used paper-based system can be easier. Respondents also noted that it had been their experience 

that some nursing assistants might have benefited from a basic computer usage course to better 

familiarize themselves with the EMR and more easily navigate the system.

• Rushed timeframe. Adhering to the Flex CAH HIT grant timeframe of 18 months took a “tremendous 

effort and staff time.” Respondents believed a two-year timeframe would have been more realistic 

given the number of activities they needed to complete and the amount of planning involved prior 

to the EMR implementation. Unexpected barriers also played a role in the rushed timeframe. For 

example, the project lost 90 days because a tornado hit one community. 

• Limited vendor support. Respondents reported that occasionally, the vendors were unresponsive or 

did not address issues immediately and this further slowed the implementation and bogged down 

the 18-month timeframe. For example, at First Care the pharmacy component had to be rebuilt, 

and over 1500 medication reentered because the vendor had given clinicians incorrect instructions 

with regards to entering brand and trade names. Respondents also expressed frustration that the 

vendor passed off questions or requests for explanations about some component of the EMR as 

unnecessary or frivolous, stating, “you don’t need to know that now.” Despite these complaints, the 

vendor was overall very patient with facility end-users, and quick to respond to major problems. 

• Lack of HIT expertise. Respondents explained that “small rural hospitals don’t know what a vendor 

should do for them” and “don’t know how to translate a list of what they do to workflow.” They 

reported that it would have been helpful to know what questions to ask of the vendors; for 

example, “How do you order a diet for a patient in the EMR?” Oftentimes, the facilities simply did 

not know from whom to request help during the grant period. 

K. Lessons Learned

• Do not underestimate the time, work, and costs required to implement an EMR. The grantee 

noted that “no one seemed to anticipate all of the work involved in implementing varying modules 
FLEX CAH HIT Evaluation- Final Report

 43

of software—it seemed like one more thing after another instead of what people envisioned—

having an EMR installed as one product.” As such, it was necessary for all staff involved to be 

flexible, as much changes throughout the course of the project (e.g., vendor and staff availability, 

weather, and emergencies.) In addition, the grantee emphasized the need for adequate time to 

plan, as well as to allow the staff time to adapt to the new environment. The grantee also noted that 

unseen costs were “a surprise” (i.e. ongoing maintenance and additional modules with updates).

• Invest in planning and readiness activities prior to implementation. The idea of conducting a 

workflow analysis and redesign proved to be a challenge from the beginning of the project, with the 

principal investigators (PIs) seeking guidance from their quality improvement organization, 

consultants, and other CAHs. The hospitals were advised that their vendors would walk them 

through the process, and numerous efforts to look beyond vendor assistance were rejected. As the 

project progressed, the need for workflow analysis continued to surface; the PIs decided to 

reallocate a portion of the project budget to support consultants to work with each of the three 

CAHs to evaluate how the implementation related to increased workflow efficiencies and improved 

data flow. While this occurred after the implementation, the benefits were nonetheless substantial 

and provided each of the health facilities with a report outlining current processes and 

recommendations for improvement. 

• Anticipate the need for connections between the various participants and facilities. The grantee 

noted that participating CAHs did not anticipate the needed connection between their IT and clinical 

staff—that both needed to understand and appreciate the EMR’s connection to patient care and the 

logistics of IT infrastructure. At the facility level, the grantee noted that the Flex and State Office of 

Rural Health Care programs are well suited to assist with these types of initiatives, having 

established relationships with the facilities and the trust needed to assist with network activity 

between large and small facilities. The grantee further noted that the process was invaluable in 

terms of building relationships and trust in the region; for example, now IT staff from approximately 

ten facilities, both large and small, meet quarterly to share information and experiences with each 

other.

L. Sustainability

All facilities involved in this project are committed to sustaining their EMRs in the future. All three CAHs 

have budgeted for maintenance fees and system replacements and upgrades. Pembina respondents 

added that the facility employs two IT staff, one of whom is also the CIO at another area hospital. 

Northwood built sustainability into its cost report, and is planning for the next software system in five to 

six years. The CIO estimated that Northwood’s support costs are flat because the facility already

employed several IT staff. The CIO further estimated that about half a full-time equivalent (FTE) is still 

needed to work with the network hospitals who are implementing the clinical EMR. 

Respondents also reported an interest from hospitals in neighboring States in visiting and learning from 

their implementation and ongoing project activities. Further, the State Office of Rural Health program 

director, who chairs the State HIT Committee, recently had an opportunity to talk with State legislators 

concerning State HIT legislation. Each of these factors bodes well for the long-term sustainability of the 

network. 

Impediments to Sustainability 

The hospitals will need to be able to connect with Altru to have information exchange. 

Complementary Funding and Plans for Expansion

The Blue Cross Blue Shield of North Dakota grant paid for the server that is used by the other facilities.
FLEX CAH HIT Evaluation- Final Report

 44

M. Value-Based ROI18

1. Clinician commitment

North Dakota’s CAH HIT Network respondents were asked to rate user clinician commitment before and 

after implementation of Flex CAH HIT grant activities along the Likert scale shown below. 

Baseline (orange) measure versus implemented system (green) measure:

User training and support

Collectively, North Dakota respondents used both vendor-supported training materials and staff, and 

supplemented this with internal hospital staff for additional support. All sites referenced a need for 

training and support beyond what was supplied by the vendors. Respondents noted a particular 

challenge among system users, especially clinicians, who lacked computer skills and confidence: “This 

group had not had to work with entering and retrieving data electronically before.” At the outset of the 

project, Northwood Deaconess surveyed user staff to assess their computer skills and confidence. Those 

survey results led to additional supports, called “Computer 101 Training,” being put in to place for users. 

All facilities also noted concerns with initial, vendor-supplied training support. Staff reported finding the 

training too rushed, having too little one-on-one interaction, and needing more training reinforcement 

once actual use began. Sites noted that in-house IT staff were utilized to supplement vendor trainings 

and provide basic system support to troubleshoot issues before contacting the vendor. Pembina staff 

noted that of particular help were the scenarios set up with test patients in a test system to familiarize 

staff with actual workflows and system use.

User feedback

For Pembina and First Care facilities, staff feedback was obtained through utilizing existing mechanisms 

for gathering staff input. These sites pointed to standing staff meetings (involving different types of 

staff), emails and memos as their main feedback mechanisms. Northwood Deaconess set up regular 

EMR implementation-related meetings. 

For all facilities, working daily and directly with system users was noted as critical. As smaller facilities, 

respondents shared that it is easiest for staff to have issues reported and responded to as they come up 

– usually immediately. As one Northwood Deaconess staff member put it, “we have discovered that we 

need to over-communicate. If we don’t, people quickly jump to conclusions or will stop until 

communication is resumed.” Internal IT and other staff evaluate issues to see whether they can be 

handled in house, or if contacting the vendor is warranted. Users are encouraged to work with vendor 

tech support as well as internal support. 

Implications

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

greater information availability post-implementation – moving responses from a pre-grant level of 

 

18 Refer to Appendix C for a detailed description of each variable used to calculate the value-based ROI.
FLEX CAH HIT Evaluation- Final Report

 45

Somewhat Committed to Very Committed. From the outset, respondents indicated methods to 

specifically engage clinicians in planning and implementation efforts. Pembina respondents shared that 

user clinicians were verbally supportive of the grant activities and willing to informally discuss issues 

(e.g. ‘wants and needs’) but were reluctant to attend meetings. At other facilities, clinicians were 

reported as being more engaged during planning – participating in vendor demonstrations. 

North Dakota respondents shared a collective sense of inevitability for many of their clinicians and 

other staff regarding the HIT implementation – as one staff member put it, “this is the future so we may 

as well be on the forefront.” Also, staff shared that as implementation has continued, clinicians have 

stopped looking for ways to work around the system, and are even seeking ways to utilize it more.

2. Information availability

Within the context of information affected by Flex CAH HIT grant activities, North Dakota respondents 

were asked to rate how available information is, when and where it is needed, along the Likert scale 

shown below. 

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

significantly greater information availability post-implementation – moving responses from a pre-grant 

level of Neutral to Very Available. Respondents shared that the paper-based system was all that they 

knew going into this implementation, so the paper-based system at the time was seen as acceptable in 

terms of facilitating the availability of information. However, comparing it to the EMR systems, staff 

were able to recognize how much time it had taken to track and locate medical records and to validate 

the information contained in them. While staff are still learning where and what are the best ways to 

access information with the new systems, respondents noted information needed daily and in a timely 

manner is very accessible and available.
FLEX CAH HIT Evaluation- Final Report

 46

3. System usability

For the HIT system(s) implemented as part of the Flex CAH HIT project, North Dakota respondents were 

asked to rate system usability prior to and post system implementation along the Likert scale shown 

below. 

Baseline (orange) measure versus implemented system (green) measure:

User perceptions of ease of use

North Dakota respondents collectively reported some apprehension on the part of system users during 

planning and before full implementation, noting that staff were unsure as to how all the pieces of the 

systems would fit together and work together. 

At the time of the North Dakota site visit, implementation had progressed significantly, but was not 

complete. Therefore, respondents shared that user’s perceptions of ease of use were still evolving as 

they learned how to properly use the systems. Pembina and First Care respondents noted that clinicians 

reported having to spend more time to document patient encounters; these users felt the new systems 

were less useful and not as easy to use as the paper system. However, these respondents also noted 

that clinician responses to these documentation challenges were improving, and that clinician feedback 

on ease of use was mixed between positive and slightly negative. 

Pembina laboratory staff found the system very easy to use, reporting that the system followed their 

workflow and has helped them to share the responsibility of ordering accuracy with clinicians. All 

facilities noted some challenges with merging existing administrative workflows and processes with the 

new systems – specifically noting technical interface challenges – but shared a commitment to continue 

to work through these challenges with the assumption that these interactions would improve. 

System use

Respondents expressed some surprise in terms of system use: –users were using the systems more than 

had been anticipated, and staff had started to look for ways to use the system beyond the initial 

training. Further, staff shared a forward-looking view that as users become more comfortable with the 

systems, any gaps in knowledge and use that exist will quickly be closed. Respondents recognized that 

users are not yet using the new systems to their full potential, and it will likely take a good deal more 

time to fully utilize them.

Implications 

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

system usability improvement from Neutral for pre-grant systems to Somewhat Usable for the 

implemented system. One Pembina staff member noted that now using a system of electronic medical 

records represents a not unanticipated, but still a “different reality [for] our users.” Further, Pembina 

respondents shared a collective desire to constantly look for ways to incorporate documentation not 

collected directly by the system into the EMR. Staff reported examining how, where and why data are 

collected and searching for alternative solutions so that it can be incorporated into the EMR. This 
FLEX CAH HIT Evaluation- Final Report

 47

process includes adding or modifying electronic forms, scanning and importing information into the 

system.

Other respondents shared some surprise with the complexity of their implemented systems, and 

referenced this complexity as hindering full utilization.

4. System efficiency

For the HIT system(s) implemented as part of the Flex CAH HIT project, respondents were asked to rate 

system efficiency prior to and post implementation along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

greater system efficiency post-implementation – moving responses from a pre-grant level of Neutral to 

Somewhat Efficient. As in other discussions above, respondents noted some difficulty in responding to 

this question fairly, because the interview occurred while implementation was still underway. 

Laboratory respondents in particular noted that turnaround time for their work has not yet improved 

and that they are still struggling with incorporating external data and interfacing with external systems 

(reference lab data). 

Respondents noted many inefficiencies with the paper-based system that were expected to improve as 

part of the EMR implementation – notably a reduction in time spent tracking and collecting data; 

manual copying, faxing and record updating; and compiling data for reporting purposes. Facilities shared 

that many changes had been made, but that they are not finished incorporating all of the tools available 

in the systems, and that they anticipate significant gains in efficiency (in the areas noted above) over 

time and as users become more comfortable with the system. 

5. Clinical outcomes

Respondents were asked to rate their perceptions as to the pre- and post-implemented systems in 

terms of supporting positive clinical outcomes along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

that the implemented system increased perceived support for positive clinical outcomes – moving 

responses from a pre-grant level of Neutral to Somewhat Supportive. Again, as discussed in previous 

sections, respondents found evaluating the implemented system difficult as it had only been in place for 

a few months. Collectively, North Dakota respondents shared expectations for significant gains in their 
FLEX CAH HIT Evaluation- Final Report

 48

ability to evaluate health outcomes in the future. Pembina staff referenced already having the ability to 

compare and trend information alongside clear and consistent documentation. Further, Pembina 

reported that they will be using the system to evaluate clinical outcomes in their next Quality 

Improvement Cycle, in which they will be focusing on the EMR system and its impact. First Care 

clinicians reported seeing improvements in medication reconciliation and patient care instructions – two 

clinical outcome factors that they feel the EMR system has significantly impacted.

6. Non-financial values

In the case of First Care, their existing financial information system was not going to be supported after 

2008, and the timing of the Flex grant was ideal to push for finding a system that could not only support 

the financial management needs of the facility, but to implement and interact with an EMR as well. 

Pembina staff referenced the increased readability and clarity of chart information and orders as 

providing significant value for their staff. At Northwood Deaconess, one respondent shared that they 

have received great value from the EMR acting as a one-stop source for information and facilitating 

stronger collaboration among clinical staff. 

Patient satisfaction

Respondents were asked to report patient satisfaction of the pre-grant and post-implementation 

systems along the Likert scale shown below.

Both the baseline measure and implemented system measure were noted at Very Satisfied: 

Implications 

Measured from pre-grant implementation to post-implementation, North Dakota respondents reported 

that the EMR system has increased patient satisfaction – moving responses from a pre-grant level of 

Neutral to Somewhat Satisfied. Pembina staff shared that they focused primarily on the financial values 

of the implemented system, noting that eventually they would also likely see improvements in accuracy, 

readability and timeliness of information as part of the EMR. Staff then reported surprise in realizing

improvements in these non-financial factors so soon after implementation. Conversely, First Care staff 

reported not yet realizing expected gains in time, money and quality, but noted that they are still 

committed to the EMR implementation and that it will be their new standard of care. For all North 

Dakota facilities, formal patient satisfaction surveying had not yet been implemented, but is being 

considered at some facilities.
FLEX CAH HIT Evaluation- Final Report

 49

Case Study 2: Northeastern Oklahoma CAH EHR 

Network, Oklahoma State University Center for 

Health Sciences

A. Introduction

The Oklahoma Flex CAH HIT grant project created a CAH Electronic Health Record (EHR) Network in 

Northeastern Oklahoma (the “Network”). The Network participants engaged in the Flex project were 

Oklahoma State University (OSU) Medical Center (urban tertiary hospital to which other partners 

primarily refer patients); Drumright Regional Hospital (CAH) and Drumright Medical Clinic; Holdenville 

General Hospital (CAH), Holdenville Rural Health Clinic, Physician Health Clinic, and Allen Health Clinic; 

Fairfax Memorial Hospital (CAH); Bristow Memorial Hospital; and the private practice physicians who 

staff each hospital and are responsible for the majority of the hospital’s referrals (Figure 2). This site visit 

consisted only of discussions with representatives from Drumright and Holdenville hospitals and OSU 

staff, so the majority of the case study reflects responses from these facilities and participants.

The Network members’ vision for the EHR Network was to ensure that patient clinical information 

would be easily accessible to providers within a healthcare organization and to other providers as 

patients migrated from ambulatory care to acute service delivery sites within the region. Improving

access to patient information by integrating it into a coordinated system of care was expected to lead to 

more effective and efficient health care delivery and ultimately to improved safety and quality of care 

for patients. 

Each CAH received $291,500 and provided $8,500 matching funds to purchase and implement an EHR 

system at their hospital and in the physician offices of their staff physicians. OSU Medical Center, the 

tertiary referral hospital for the Network CAHs, received $350,000 and provided $650,000 in matching 

funds to purchase and implement an EHR system at its hospital. 
FLEX CAH HIT Evaluation- Final Report

 50

Figure 2. Oklahoma Flex CAH HIT Network Implementation Map

Drumright Regional Hospital and Holdenville General Hospital both began the Flex project with no 

electronic clinical system, but did have electronic systems for billing. As a result of the Flex CAH HIT 

project, Drumright has now implemented their EMR system, including all acute care modules except for 

the emergency department and ambulatory systems. Holdenville has currently implemented all modules 

including clinical, billing and registration, except for the emergency department, prescription drugs and 

ambulatory modules. Holdenville plans to continue documenting emergency department visits on paper 

and will scan the note into the EMR, and is in the process of trying to interface the current prescription 

drug system with the EMR. The associated clinics were expected to go live with the clinical modules 

before July 2009. 

Fairfax Memorial Hospital and Bristow Memorial Hospital are both owned by the same management 

company – Community Partners. Both hospitals joined the Flex project part way through the period of 

performance (when Cleveland Area Hospital left the project). Fairfax and Bristow came into the project 

with some EMR modules in place, and decided to expand that functionality under the same vendor 

rather than start over, and both are in process of implementing the new EMR modules. 

All hospital sites (CAH and ancillary) are in the process of determining what data will be shared within 

the HIE. OSU Medical Center and OSU physician clinics currently do not have an EMR system that will 

directly connect to the HIE. One portion of the purchased HIE system is a portal that will query 

information from all the disparate systems, utilizing HL7 messaging, and allow physicians as well as 

clinical staff to view, update and send requests for information from other locations. OSU is in the 

process of negotiating the integration of other third-party systems such as labs, e-prescribing, and 

payers into the HIE. This portal will allow physicians and staff real-time access to patient data.
FLEX CAH HIT Evaluation- Final Report

 51

B. Background

The Director of Grants and Resource Development at the Center for Rural Health directed the 

application for the Flex CAH HIT grant as the mission of the Center (part of Oklahoma State University 

Center for Health Sciences) includes bringing technology to rural populations. 

C. Planning Approaches

From the very beginning of the Flex grant period, OSU mapped out workflows from all the departments 

of participating CAHs. OSU utilized an extensive questionnaire in addition to a vendor scorecard to 

determine what needs the CAHs might have for a vendor solution. OSU disseminated the 

questionnaires, conducted interviews, and provided workflow assessment templates to the CAHs. Staff 

interviews or end-user assessments were identified as particularly important in gathering what endusers were envisioning for the EMR. Interviews were conducted by department (i.e., physicians, lab, 

radiology, pharmacy, and nursing), based upon personnel job descriptions. OSU also interviewed 

departmental heads. 

Involvement of Clinical Stakeholders in Planning

OSU reported that part of the work plan included organizing Hospital EHR Workgroups, which included 

CEOs and clinicians, to assess hospital-level needs and secure project buy-in. The Medical Records 

Manager, CFO, Business Manager, Lab Director, Nurse Director, and CAH Administrator were involved in 

planning at Drumright CAH. A number of clinical staff from Drumright CAH also traveled to Northern 

California to examine an EHR system in a live hospital environment. 

OSU reported that Holdenville CAH providers were involved in planning; however, these were 

contractual physicians not directly employed by the hospital. Other Holdenville staff were connected 

with the project implementation process through updates during staff meetings. A meaningful 

proportion of clinical staff from both sites participated in the vendor scorecard process. When 

Holdenville CAH was initially considering the vendor products, clinical staff were very involved. For 

example, the Director of Nursing went on-site to a hospital where the vendor’s ancillary, clinical, and 

business components were all represented. In addition, a number of staff participated in the visit to 

Bedford Regional Medical Center in Indianapolis to see the vendor’s system. Staff who attended 

included the CFO, Director of Nursing/Pharmacy, Radiology Manager, Lab Manager, Controller/IS/ 

Payroll/AP, Business Office Manager/Clinic Billing, Nurse/Quality Improvement, HIM Director, and the 

OSU Project Director and Project Manager. Similar personnel attended the site visit to Coon Memorial 

Hospital to see the EHR system. The CFO and OSU Project Manager wanted to include physician 

representation on the vendor site visit, but physicians were needed at the hospital and could not take 

time away from their clinical practice. 

Physician Participation in Project Planning

While physicians were invited to be a part of all aspects of the project, some respondents reported that 

they generally did not participate. Holdenville CAH’s CFO and OSU remarked that the physician response 

to the HIT implementation was generally, “whatever you want” and “if [HIT implementation] needs to 

be done, let’s do it.” The CFO noted that such a response was typical in any CAH environment, especially 

one without a strong physician HIT advocate. CAH respondents reported that although clinical staff 

were, for the most part, very open and fairly easy to work with as far as training prior to 

implementation, and that nursing staff were “on board” with the implementation, physicians were 

slightly more difficult throughout the process. However, the CFO reported that there was a “good 
FLEX CAH HIT Evaluation- Final Report

 52

effort” made to ensure that the physician perspective was captured throughout the entire planning 

period. 

Vendor Selection

The OSU Project Director reported that several vendors were considered for the Flex CAH HIT 

implementation. Ultimately, Holdenville and Drumright chose different vendors. According to OSU, 

each participating CAH was allowed control in selecting the EHR system that met the needs of its 

particular hospital. OSU stated that its task was to ensure that the vendor’s vision and strategy matched 

that of the respective CAHs; to evaluate all the software options offered by each vendor; to acquire and 

check vendor references and take current users on site visits to watch the product in operation; and to 

review the packages of several vendors for comparison and negotiation purposes. 

CAH-Specific Planning Approaches

Holdenville General Hospital. The Holdenville project kicked off in October 2007. The OSU Project 

Manager and the Project Director began by gathering information about technologies available; creating 

the RFP; and narrowing the prospective vendors down to three. Holdenville desired a system to 

encompass everything from clinical to financial/billing modules. This narrowed the vendor selection to 

two finalists. From there, OSU scheduled site visits and saw the vendor systems in use at hospitals in 

both Indiana and Texas.OSU and Holdenville staff then selected one vendor that would best meet 

Holdenville’s core requirements. Shortly thereafter, Holdenville began its implementation. The system 

went live on December 1, 2008.

Holdenville CAH’s CFO reported that the decision to implement with the selected vendor was driven by 

the desire to have an integrated EMR for its clinical and financial informatics system. However, from 

OSU’s standpoint, the business-side modules were not a priority, given that the Flex grant paid for only 

the clinical aspects and activities of the implementation. Consequently, the vendor scorecard was 

designed to weigh more heavily on the clinical components of vendor systems. The “practice 

management,” or financial component of the system was not funded by the grant, since the hardware 

involved was specific to the practice management system. 

The factors identified as most critical in the selection of the selected vendor included cost, availability of 

staff training, company stability, IT support, company long-term goal, and application functionality 

based upon physicians, nursing, and other levels of staff. These factors were all detailed and weighted in 

the OSU-designed vendor scorecard. An additional criterion in choosing the vendor was a system that 

could integrate the CAH’s associated clinics, including one rural health clinic and two provider-health 

clinics.

Holdenville’s workflow analyses (of various hospital departments) were planning tools used to assess 

current processes. The workflows were helpful for the vendor to review in order to plan the 

implementation. From the standpoint of OSU management, the workflow analyses were helpful in 

identifying various departmental needs within the CAH and to ascertain what needed to be 

accomplished to get Holdenville ready to implement the 

EMR. 

Drumright Regional Hospital. CAH respondents reported 

that Drumright began the HIT implementation project with 

“literally no money.” Therefore, one of the biggest drivers in 

vendor selection was the cost of the EMR. Although staff 

were interested in other vendors initially, these were quickly 

eliminated due to price. 

“Anybody in health care sees the writing 

on the wall that you’re going to have to do 

this with an EMR. We didn’t know how we 

were going to make it work because 

money is so tight.”

– Drumright CAH Administrator
FLEX CAH HIT Evaluation- Final Report

 53

Initially, Drumright narrowed its vendor selections to the most affordable vendors. One vendor was 

eliminated because it was not designed for interfaces to ancillary systems. Originally, Drumright chose 

another vendor as the “winner,” however, since this vendor was strictly an ambulatory system there 

was only one remaining option within their price range. OSU and several Drumright staff conducted a 

site visit to a 40-bed hospital in Northern California that had implemented this vendor’s system. OSU 

and Drumright staff liked this vendor’s system because it was an application service provider (ASP)

model, and therefore hiring in-house IT staff was unnecessary. The Drumright Administrator reported 

that choosing this vendor was a “no brainer,” because the monthly maintenance fees were minimal.

The CAH Administrator’s original goal had been to avoid monthly vendor/system upkeep fees. However, 

OSU was able to negotiate down the vendor’s monthly fee by considering only add-ons or irreplaceable 

components of the EMR. After negotiating with the vendor, the CAH Administrator realized that while 

the existing payments looked cheaper on paper, there were many hidden costs. For example, Drumright 

was paying $600 per month, at the time, for exit care and discharge planning, $800 per month for the TSystem, as well as $2,000 per month for 1.0 FTE used for a data entry staff person. Since the selected 

vendor was all web-based, the only maintenance would be “from the wall out.” Drumright negotiated a 

support contract with the vendor which was built into the monthly fee, as well as a contract with a 

Tulsa-based IT support group.

Fairfax Memorial Hospital and Bristow Memorial Hospital. As mentioned previously, Fairfax and 

Bristow hospitals joined the Oklahoma Flex project following Cleveland Area Hospital’s departure from 

the project. OSU reported “significant challenges” with Cleveland CAH’s management company, 

Community Partners, throughout the vendor selection process. Consequently, OSU dissociated with 

Cleveland CAH midway through the grant period. However, Bristow and Fairfax, also managed by 

Community Partners, took Cleveland’s place, and intend to be a part of the HIE network housed at OSU. 

Vendor Perceptions

Holdenville General Hospital. The CFO reported that Holdenville made a wise choice of vendor, and that 

a similar vendor selection process would have been used over again because, overall, the selection 

process went smoothly. Respondents reported that while the vendor has “had their fair share of ups and 

downs, [the vendor] has tried every inch of the way.” However, negotiating with the two final vendors 

was difficult, especially because the vendor not selected was “ready to give their [system] away.” 

Despite the Flex grant’s emphasis on the clinical portion of HIT, CAH respondents reported that the 

clinical modules of the systems under consideration were not the deciding factor of the final selection. 

Rather, the choice came down to “some very minute details of the business side.” In addition, the intent 

of purchasing the EMR was to unify the clinic and the CAH systems, to have one point of entry for all. 

Respondents reported that indeed, the vendor accomplished this by integrating Holdenville’s modules 

with the clinics’ systems. 

Drumright Regional Hospital. For the most part, Drumright staff agreed that everything has been going 

well with the vendor as the implementation is wrapping up. Overall, Drumright was identified as being

“very happy” with its vendor selection, however staff were nervous with their selection at the start of 

the Flex grant. The Nurse Manager reported, “Something’s changed with [the vendor]. I think it’s the 

possibility of the CAH market. They’ve always been good to us, but there’s been a real change.” 

The CAH respondents emphasized the importance of having the vendor on-site 24/7 during the 

implementation period. Additionally, the OSU Project Manager reported that Drumright’s Administrator 

“did a good job on the ROI,” in his determination of how Drumright could recoup some of their upfront 

costs. He further reported that the Administrator’s vision has, for the most part, been realized to this 

point. 
FLEX CAH HIT Evaluation- Final Report

 54

D. The EHR System

Established Infrastructure at Participating CAHs

Prior to the Flex CAH HIT grant, both Holdenville and Drumright CAHs existed almost entirely on paperbased systems. Drumright had teleradiology and pharmacy systems, but surgery documentation was 

completed in an Excel spreadsheet. In March 2009, after the EHR implementation had been completed

at Drumright CAH, respondents reported that the old, paper-based system had been “awful.” 

Phases of HIT Implementations at Holdenville and Drumright CAHs

Drumright. Drumright implemented nursing documentation on January 21, 2009, and modules for 

physician orders, transfers, discharges, and patient education on February 16, 2009. At the time of the 

site visit, Drumright was working toward the implementation and interfacing of the medication and 

pharmacy portions of the EMR, which were expected to go live on April 6, 2009. The radiology portion of 

the EMR had gone live as of March 2009, and the vendor was continuing work on the interface with 

radiology transcription. CAH respondents reported that no definite implementation date was planned 

for either the surgery or radiology modules; rather, implementation was occurring gradually as staff 

became ready. 

On a related note, CAH respondents reported that the emergency department (ED) portion of the EMR 

had not yet been implemented because staff were not happy with the way the nursing portion of it was 

designed, believing the navigation to be “messy.” According to the Nurse Director, successive screen 

pages within the nursing emergency department module did not flow logically into one cumulative 

report, and that “it would be a nightmare to go to ten different pages” to input information. As a 

consequence, Drumright requested a hold on the go-live date until the system was considered more 

user-friendly for clinical staff. The vendor has not yet built nursing care plans, which it is now supposed 

to be working on. From a management and nursing point of view, the nursing care plans must be 

developed and implemented very quickly.

Holdenville. Holdenville reported being on target for the revised July 31, 2009, project end date. The 

December 2008 go-live included the patient accounting/business module, as well as lab, pharmacy, 

order entry, and clinical modules. The physician documentation module, however, has been delayed. 

Additionally, the current version does not have a way of “noting nursing orders,” so the vendor has 

announced plans to implement an order entry module to cover both lab and radiology orders. This is the 

only separate module that has not yet been implemented within the CAH itself. However, the EMR 

system has not yet been implemented in the clinics. 

Holdenville management is still considering implementing an ED module; however, the ED system might 

always be stand-alone due to a contractual agreement for ED coverage and turnover of physicians (lack 

of continuity of ED medical staff). They currently use the T sheets (T sheets are used to document ED 

visits, based upon CMS guidelines), which are scanned into the medical record.

Health Information Exchange with the Oklahoma State University. The OSU Project Manager reported 

that he has been in contact with the HIE vendor about moving forward with the interfaces at Drumright 

and Holdenville. Drumright has begun talks with the vendor, and went live with the HIE in March 2009.
FLEX CAH HIT Evaluation- Final Report

 55

E. Staff Training

Holdenville. At Holdenville General Hospital, the vendor was responsible for scheduling trainings with 

hospital staff. CAH respondents reported that the vendor trained staff concurrently for five days after 

the clinical modules were installed. Approximately ten staff were trained on the floor, and four staff 

were trained in surgery. Six CAH-employed and two contractual physicians were also trained. In total, 

ten nursing staff were trained and presently use the vendor’s EMR. 

Holdenville’s staff and contractual physicians received training in early December 2008. They were 

trained on how to operate the EMR and “where to find things” in that system. Nursing staff were trained 

in the pharmacy and order entry modules in late August and early September 2008, in preparation for 

the scheduled go-live on September 15, 2008. The vendor trained three head nurses first, and then 

those nurses trained the remaining staff. The vendor offered several sessions for each module for nurses 

only and physicians only. 

Training on the surgery module was set up in three stages. For example, a test module was conducted, 

where nursing staff “played with test patients” and the head nurse checked all their documentation. 

Even after the surgery module went live, the head nurse took responsibility for checking all the 

documentation. In general, the nurses were able to access the modules before the go-live, if they had 

free time on the floor. However, there was not a lot of downtime for staff to practice because the 

patient census went up dramatically during the initial go-live phase. The vendor’s clinical expert was onsite when the system went live. 

Drumright. In December 2008, the vendor began “super-user” training for the charge nurses at 

Drumright. This training comprised a two-hour overview of the EMR. However, CAH respondents 

reported that this amount of information packed into a one-time training was overwhelming. 

Consequently, the vendor offered two additional trainings prior to the implementation of the nursing 

documentation modules, and a third training prior to the order entry go-live. In addition, the vendor 

planned to conduct at least one more training session for the Medication and ED modules, dependent 

on staff requests.

The vendor also offered trainings for the physicians and PAs on how to build reports. As of March 2009, 

two surgeons had yet to be trained, because they resisted training until 100 percent of the EMR 

modules were fully implemented. CAH respondents noted that one surgeon refused training, but only 

visits Drumright once weekly. 

Staff Satisfaction with Vendor Trainings

Holdenville. It was reported that all Holdenville staff have been trained and are using the EMR. Staff did 

not initially receive training on the clinical documentation application, and had to request a second 

training to fully understand how to build reports and “the whole digital patient record.” The CFO 

reported that while implementation went smoothly through the initial stages, Holdenville’s vendor’s 

project manager has “dropped the ball and the following stages have been more lackadaisical.” 

The plan for staff training by the vendor going forward includes free training, and “teaching how to use 

the system above and beyond the standard.” This training will occur every year in October. The group 

reported that the vendor also offers plenty of opportunities for free webinars, and the vendor’s site is 

full of helpful hints and suggestions. Holdenville “just has to figure out how to work it in for staff.”

It was reported that there was a great staff response to the trainings. Specifically, overall nursing staff 

were very pleased the system. With regard to the physicians, their technology skills were “not great,” so 

their satisfaction was not as pronounced. One physician did not know how to type, and did not know 
FLEX CAH HIT Evaluation- Final Report

 56

what a pull-down menu was. In this case, additional one-on-one assistance has been helpful, and 

physicians have been receptive overall to help. After testing out some group training scenarios, the

nurse trainers learned that one-on-one training was most effective for teaching purposes.

Drumright. The vendor was “fabulous at being on the floor when they went live.” The vendor placed 

two or three employees at Drumright during the day shift, and at least one during the night shift. For a 

small hospital, “it is hard to do all this training.” Training has been very expensive as far as staffing, and 

has taken a lot of overtime. One notable challenge was in training pain resource nurses (PRNs) – as they 

do not work many shifts, getting them scheduled for trainings was particularly difficult.

Anonymous staff evaluations were distributed after the super-user and phase training. The purpose of 

the evaluation was to discern how employees felt about the impending go-live and to gauge their 

comfort level. It was unanimous that the super-user sit-down training was too overwhelming. 

Consequently, two more training sessions were offered. After the ensuing trainings, the nurses’ anxiety 

levels “really settled down.” Any issues encountered were submitted to the vendor, which was 

reportedly helpful, because the vendor was not aware that such issues existed. For example, the vendor 

had set up certain alerts as reminders for the next nurse on shift to complete a task, but the times were 

not set up to properly align with the shifts. This led to nurses re-entering reminders, and the vendor 

staff “pulling [their] hair out” because there were double reminders for everything. The problem was 

rectified by the vendor redefining alert times. There were also some language differences, similar to 

issues encountered at Holdenville, where staff wanted to keep the language in line with what CAH staff 

were accustomed to seeing.

Charge nurses were strongly encouraged to take advantage of the vendor’s on-site support staff while 

the implementation is still in the go-live phase. The importance of being proactive and asking questions 

has also been stressed, and nurses were instructed that they would be accountable for information 

when the vendor staff have completed the implementation. 

System Testing

The vendor was in Holdenville for “two weeks straight” to build the interfaces and set up standardized 

templates with existing workflow. The templates were “built from the ground up” and customized for 

Holdenville. The lab and radiology departments built their procedures internally, and the vendor tested 

those procedures during their two-week standardization period. The vendor used mnemonics similar to 

those already in use to minimize staff confusion during training. 

Some gaps that have been identified pertain to the duplicative lab process. For instance, staff must 

enter information into the EHR and then into the LabCorps computer. The lack of communication was 

related to interface with the reference lab. However, the CFO reported that the capabilities for 

interfacing the lab system were included in the EHR vendor package, and was surprised to hear from 

other staff of the duplicative nature of the work. 

Drumright conducted testing with the vendor at each “phase.” Physician ordering was tested, with the 

assistance of the vendor; for example, the vendor verified that an order was sent to the correct places in 

the chart. During the go-live, the vendor provided 24-hour, on-site support. One respondent reported 

that the vendor was very confident because they had, in the past, spent months testing every different 

scenario. 

Drumright staff expressed some concerns about physicians placing medication orders/missed orders. 

The EMR displays a small icon that reads ‘you’ve got an order.’ However, the way the system is currently 

designed, the order is directly transmitted to the patient’s chart and the nurse may dispense the 

medication without prior physician review and approval. When Drumright staff approached the vendor 
FLEX CAH HIT Evaluation- Final Report

 57

about the medication order problem, the vendor replied, ‘this is just the way our [system] is built,’ 

because the vendor is accustomed to working with larger hospitals that can handle quick medication 

turnover. However, the vendor accommodated Drumright’s problem by developing a template to 

enable review and approval of the medication before transmittal to the patient chart.

In small CAHs like Drumright, there is less “departmentalizing” than in large hospitals. Therefore, the 

vendor had to rebuild the Drumright’s EMR to accommodate the multi-faceted responsibilities of its CAH 

staff. For instance, CAH nurses essentially take on other roles, which may include jobs such as 

respiratory therapy; in larger hospitals, individuals are hired to do very distinct jobs. 

Technical Support

At Holdenville, the nurses act as the support staff. The head nurse reported keeping a log book with staff 

questions and concerns, and also noted that if a call needs to be placed to the vendor immediately, she 

is responsible. The head nurse places several calls a week to the vendor, to resolve “little glitches” and 

she reported that the response is generally good. The vendor’s Website and online support has also 

been helpful in troubleshooting problems. 

At Drumright, the vendor and nursing director meet and generate a list of problems with the system. 

Drumright staff are able to express problems with the system through an “issue notebook” which is then

communicated to the vendor. Additionally, each week, Drumright staff receive an update from the 

vendor, and new information regarding the system is shared at staff meetings. 

F. Governance, Policies and Procedures

Holdenville and Drumright CAHs approached modifications to their facility policies and procedures in 

different ways. Drumright’s Nurse Manager reported reviewing over 500 policy forms/templates offered 

by the vendor, and choosing the forms which most closely corresponded to Drumright’s existing 

processes. She also ensured that each selected form included the CAH’s rules and regulations. These 

new forms initially had a “big impact” on workflow, until CAH staff became accustomed to the modified 

processes. While the Nurse Manager had designed a “cheat sheet” as a stand-in for Drumright CAH’s 

policies and procedures, she reported that there was no mechanism to hold staff accountable for 

appropriately identifying and properly filling out the modified forms. Nevertheless, she emphasized, 

“Our cheat sheet is our lifeline” to guiding clinical staff through the new forms. Conversely, Holdenville 

had not begun to update their policies and procedures as of March 2009. The CFO reported that this 

“process update is one of our administrative goals,” but anticipated that making changes to the CAH’s 

policies and procedures would be “the worst part of all, because it’s one thing to learn the system, but 

another when all the processes change.” 

Data Confidentiality 

At Holdenville, concerns cited with regard to data confidentiality included the vigilance of staff to sign 

out of the EMR when they were finished, and the loss of data after program timeout. The program 

timeout (e.g., the active report “times out” after five to ten minutes of inactivity) was reportedly 

cumbersome because staff are not permitted to save patient information until they are finished with the 

entire report. Respondents suggested an auto-save function so staff would not have to save as often, 

and to prevent data loss. Drumright respondents reported that all staff signed a HIPAA form before the 

go-live, and that staff only have access to the EMR while physically at the hospital. A confidentiality issue 

of note at Drumright was the necessity of positioning the computer portal cart so patient information 

was not visible to anyone passing down the hall next to the cart. Drumright staff were also trying to be 

careful to close the session as soon as they were finished entering patient information.
FLEX CAH HIT Evaluation- Final Report

 58

Interoperability

The CAHs described a number of problems related to interoperability and interfacing. At Drumright, the 

existing patient registration system has experienced issues interfacing with the EHR. Drumright staff 

reported that “[Vendor] is usually very on top of [interoperability] issues.” However, a major problem is 

the T-1 connection for day-to-day services. Drumright has low bandwidth and a lot of latency, causing 

the system to run rather slow. That has been resolved by the vendor, who installed a dedicated T-1 to 

the vendor itself. However, there have been redundancy problems with the vendor-dedicated T-1, 

because no matter how many communication lines enter the hospital, if one goes down, they both go 

down. This occurred once, and staff reverted to using paper documentation. At Holdenville, the vendor 

is in the process of building customized templates for reporting in order to fully integrate the EMR. 

Terminology has not been an issue, since the vendor built the system based on Holdenville’s existing 

systems. Holdenville reported that they have “had nothing but problems” with the pharmacy system 

which was purchased 1.5 years ago, prior to going into the EMR conversion. The pharmacy system has 

been very problematic in the past and will not “talk” to the pharmacy module.

G. Impact on Hospital Operations

Tracking Data through the Continuum of Care

While Holdenville “originally had a pretty efficient paper-based system,” CAH staff reported that nursing 

documentation is going very well and that the EMR has been improving the quality of nursing 

documentation. Staff are now able, for instance, to generate a cumulative report on different labs, 

whereas as prior to the implementation, a single sheet of paper was required for each individual lab. 

Transfers between facilities have also been easier and faster post-implementation. 

Availability of Data

Holdenville respondents reported that functionality of the EMR was “disjointed.” However, they also 

reported that the vendor has stated that this “disjointedness” would be rectified in the next software 

update. Holdenville staff also reported disappointment that “they have to go to different places to get 

the big picture,” and, while re-entering the data has become “three to four times more cumbersome at 

this point,” once the data are entered “everything flows smoothly.” Holdenville respondents believed “it 

will always be [duplicative] unless technology changes, especially if there is a high volume of patients.” 

In contrast, Drumright respondents agreed that the implementation has had a “phenomenal” impact on 

the availability of data. The Nurse Director was able, for instance, to pull up patient information 

remotely to check whether everything is being done correctly with a patient. She reported that it has 

made her job a lot easier to be able to develop “points of intervention” during the process, where a 

manager can look to see who is overdue for their nursing interventions, instead of just asking, “Are you 

guys doing okay?” Staff are also more easily able to adjust to the changes in volume. 

Feedback from Staff

Most staff are positive about the implementations and satisfied with the system. However, at present, 

the system has increased the cost of care and required staff to do extra work and training in addition to

their existing responsibilities. At Drumright, clinical staff completed a survey related to the vendor’s 

training, but have not yet taken a survey related to the overall EMR project or their satisfaction with the 

implementation. Overall, respondents reported satisfaction with the EMR implementation. Drumright 

respondents in particular were satisfied with the vendor’s technical support and responsiveness to staff 

concerns. One physician reported that from an operational standpoint, e-prescribing has been very 

helpful, as well as the ability to see the full patient record and medical history remotely. In this respect, 
FLEX CAH HIT Evaluation- Final Report

 59

the EMR has made providers’ jobs easier and more streamlined than the paper-based system. 

Respondents stated that overall, “physicians are complaining less” and while “everything was tense for a 

while...it’s moving forward again. The physicians just needed some time to figure [the EMR] out.” 

Physicians are still figuring out how to deal with medication reconciliation, which will be facilitated by 

additional training. The CFO reported that since the implementation team was “scared to throw 

physicians [into the EMR], and didn’t want to overwhelm them,” they have initially limited physicians’ 

use of the system to order entry.

The impact of the EMR differed by provider type. For example, it changed the process of physician 

rounds, because it took physicians longer, and physicians had a more difficult time finding necessary 

information. Now, physicians are changing the way they do their rounds. Initially, the nurses saw some 

physicians look at the computer screen and copy all the information down so they could then get that 

information in the patient rooms and return and input the information into the EMR. The laptops are 

supporting their morning rounds better and physicians are becoming more comfortable with their use. 

Additionally, at the outset of the implementation, physicians were unwilling to trust what they saw in 

the electronic system. If they did not see something filled out in a field, they thought it might be an 

omission. They just were not sure if they were supposed to be looking somewhere else. This mistrust of 

the system is gradually eroding with increased use, however. For nurses and other ancillary staff, the 

implementation has been seen as more positive because they are able to complete documentation 

faster. 

Further, respondents reported that the availability of information between physicians and other 

providers has increased dramatically, and information is much more retrievable. This has also 

contributed to better communication between individual providers and across departments. Clinical 

staff were also pleased to note that they are now able to “call the doctor and ask them to look at 

information while they are off-site, and the physician is able to do that from their desk wherever they 

are. There is no longer that morning scramble to get the charts on the doctor’s desk.”

Impact on Patient Care

Drumright staff believe the EMR will be “a huge factor for patient quality of care” and emphasized that 

patient care should be the number one priority. Drumright’s Nurse Director explained that her biggest 

fear was that patient care would be ignored. “It just takes one bad outcome, or one bad person. It’s 

been a challenge,” she said. 

Feedback from Patients

No patient complaints had been reported from either Drumright or Holdenville as of March 2009. There 

have been several articles in the local paper about the phases of the EMR project, and some positive 

comments related to this publicity, but nothing directly from CAH patients. The EMR publicity and the 

good reputation the facility has maintained since recently opening its new building is of critical 

importance for the financial stability of the CAH, especially since Tulsa is only 30 minutes away. The 

short distance to Tulsa is not a deterrent for patients who may choose to drive to a bigger referral 

hospital if Drumright’s service suffers due to the EMR implementation.

Additionally, there will be a formal assessment by JL Morgan and Associates at Holdenville, although the 

CAH has always administered patient satisfaction surveys at discharge. However, there are no plans to 

change the way this survey will be implemented or questions added about the EMR. 
FLEX CAH HIT Evaluation- Final Report

 60

H. Evaluation

The goal of the grantee-level evaluation was to develop a solid Quality Improvement (QI) foundation, 

which would enable participating facilities and end-users to move forward with implementing more 

advanced HIT systems in the future. According to the Flex CAH HIT application submitted by OSU, 

baseline data were to be extracted from records prior to September 2008, as well as CMS Hospital 

Compare. All the Network hospitals report quality data to CMS Hospital Compare, and demonstrated a 

commitment to improving quality and reporting data, according to the grant application. 

CAH respondents noted that as of December 2008, Duke University Medical Center and the HIE vendor 

had been extracting real-time data to send to Duke for analysis. Duke was using an SITR grant to develop 

a disease management system, and has partnered with OSU for a pilot project. 

As for specific evaluation of the Flex project effects, evaluators hired by OSU participated in the site visit 

interviews, but as of the time of the site visit (March 2009) reported having done “nothing formally” yet 

towards the evaluation. Although plans for the evaluation were reported as including a “pre-post 

snapshot”, evaluators commented that this evaluation structure would then make it difficult to “tease 

out the impact of HIT without a nice econometric model.” When pressed for specifics for the pre-post 

evaluation (metrics, roll-out plans), the evaluators did not identify definite structure or plans for the 

evaluation, and reported no information gathering had taken place to inform the evaluation to date. 

I. Challenges and Solutions

• Pre-existing facility infrastructure. The CAHs struggled 

with low bandwidth and an outdated or inadequate T-1 

system, which negatively impacted the implementation. 

• Overextension of staff resources. Pinched staff 

resources made it difficult to set aside time solely for 

staff training sessions and limited the times at which 

staff could take time on their own to practice using the 

EMR systems.

• Vendors’ unwillingness to design interfaces and modify 

templates. Holdenville reported encountering 

resistance from the vendor when facility staff requested 

modeling templates for the electronic modules on existing facility workflows. The vendor required 

Holdenville to pay for this type of standardization, which was not cost-effective for the CAH, nor was 

it a contractual agreement CAH staff were aware of at the time of vendor selection. Staff reported 

being unsure of the vendor’s flexibility to make changes to workflow templates and the vendor’s 

commitment to make the changes required by law.

• Modifying CAH policies and procedures. Addressing how Drumright’s existing policies and 

procedures would fit into a newly implemented electronic system was “a lot of work,” according to 

the CAH Administrator. Drumright underestimated the amount of time necessary to convert existing 

policies and procedures into a format that would work with the EMR. The Administrator reported, 

“It was naïve to think that the vendor would have it all set for us.” Holdenville also expressed 

concerns about its capabilities to design templates to capture documentation and build reports 

without undergoing an intensive conversion of its existing policies and procedures.

• Limited grant timeframe under which to select vendors. Staff from both CAHs as well as grant 

management at OSU cited the limited timeframe as a major stressor on the HIT implementation. 

The original timeline imposed under the 

Flex CAH HIT grant “made it real stressful 

and difficult, and that’s one of the reasons 

why it got easier as it went along; 

because the timeline has been pushed 

back. If you can go into it knowing you 

have twelve to eighteen months [to 

complete the implementation], it would 

make things a lot easier, especially for a 

CAH with limited resources.” 

–– Drumright CAH Administrator
FLEX CAH HIT Evaluation- Final Report

 61

The Drumright Administrator reported that Drumright staff felt rushed during the vendor selection 

process.

• Staff turnover. Staff turnover at rural CAHs is difficult because several larger hospitals that offer 

higher wages and greater flexibility are located within an hour’s drive. Respondents at Holdenville 

reported that the CAH was struggling to compete for qualified staff with these larger hospitals. 

Consequently, the threat of key clinicians and hospital personnel leaving during or following the 

EMR implementation was a major concern. The CFO reported that ongoing staff training to address 

turnover issues would be factored into the budget, but was something that needed to be thought 

about in more detail.

• Detachment of management-level staff. Holdenville’s Board of Directors reportedly “had not gotten 

a good feel for the quality pieces of the EMR,” and were more concerned with the potential return 

on investment a business/financial-focused electronic system could bring to bear. Holdenville’s CFO 

reported wanting the Board to be more concerned with the clinical and quality aspects of the 

implementation, particularly because these were the components funded by the Flex CAH HIT grant. 

The CFO further admitted that partially because of the Board’s emphasis on financial systems, the 

implementation wasn’t “quite as organized because [the CFO] was pulled in the business direction.”

J. Supportive Factors 

• Collaboration between CAH, OSU, and vendor staff. Collaboration among CAH personnel, OSU 

project management, and the HIT vendor to ensure a tight reading of the project schedule, acted as 

a support to completing the implementation activities in accordance with the stringent HRSA 

timeline.

• Facility-level project management role. Drumright’s nurse manager served as the de facto project 

manager and HIT champion. The CAH Administrator praised her time and devotion to the project as 

a critical element to its success. The administrator also recognized his own role in understanding and 

accommodating the demands of such a role, which the nurse manager undertook in addition to her 

significant clinical responsibilities.

• Facility management-level buy-in. Related to the necessity of accommodating management for a 

clinical project “champion,” management-level buy-in and advocacy were considered integral to the 

success of the implementation activities. The CFO at Holdenville, as well as the CAH Administrator at 

Drumright, both served to move the HIT projects forward through their leadership of clinical staff 

and influence with the hospital boards and other management-level staff. 

• Small hospital environment. Respondents reported a positive benefit of planning and implementing 

HIT systems within a small hospital environment. Few and close-knit staff streamlined 

communication and consensus about the content of vendor modules as well as the direction of 

training and implementation activities. CAH administrators and clinical staff reportedly had mutual 

respect for and confidence in one another, which enabled decision-making and acceptance among 

the EMR end-users.

• Established relationships with or knowledge of similar vendor implementations. OSU project 

management as well as CAH respondents acknowledged the importance of having a relationship 

with Owasso, a large tertiary hospital which had previously implemented the vendor’s EHR on a 

larger scale. Advice and reports of vendor experiences from staff at Owasso were reportedly very 

helpful when considering how to best approach vendor oversight, planning, and implementation at 

both Drumright and Holdenville.

• Attentive vendor training and technical support. Drumright’s vendor has been supportive 

throughout the implementation and this support, along with the nurse manager’s role as 

Drumright’s champion/nurse director, contributed to the success of the project.
FLEX CAH HIT Evaluation- Final Report

 62

K. Lessons Learned

• Designate a clinical or facility-level champion. Drumright’s nurse manager has, since June 2008, 

essentially acted as the project manager on top of her nurse management role, and has been “on 

heavy duty” with the implementation since October 2008. Respondents encouraged other small 

facilities to designate such a staff person who had committed to “be on board with the project and 

have a good positive attitude” throughout the process. However, CAH respondents warned that it 

might be “easy for someone in [the nurse manager’s multifaceted] role to end up with a bad 

attitude,” presumably because the task was overwhelming for the time allocated to one FTE. It 

would be important to identify, they reported, the structural aspects of the situation and the 

leadership qualities that would best serve to support an EMR implementation before designating 

such a champion at the facility-level. The CAH administrator noted that someone at the facility level 

was especially important to be attentive not only to the “show stoppers,” but the minutia of the 

daily routine. He suggested that several CAHs implementing a HIE system underwrite such a 

dedicated staff person familiar with hospital processes, policies, and procedures, to take on a 

project management role and serve as a nucleus of communication within the institution.

• Employ a knowledgeable HIT consultant. One of the drawbacks encountered during both the 

project planning and implementation phases was the lack of facility staff expertise in or experience 

with HIT. Holdenville CFO reported that because of this knowledge and experience gap, it should be 

a top priority of small hospitals, particularly those without dedicated IT staff, to hire a consultant 

who knows the “right questions” to ask of the vendor. One CAH respondent reported, “You don’t 

think about the compatibility of the different [vendor] systems, [or] how many screens you have to 

go through” to see all the patient information. Staff should have asked more probing questions 

about the capabilities of the vendor systems and application processes, but simply did not know. A 

specialized HIT consultant would make a good addition to any facility-level implementation team. 

• Establish relationships with other facilities familiar with the selected vendor’s reputation. The OSU 

project manager reported that one of the big problems with HIT implementations in CAHs is that the 

vendor systems are not designed to merge with the workflows of small facilities. He reported, “Even 

if you do ask the questions, the vendors might not know [the answers] because they’re not used to 

[implementing the system] in a particular way. It makes it difficult given the complexities.” The 

project manager offered that one of the best responses to vendor unfamiliarity with small hospital 

processes is to build a relationship with other, larger or tertiary hospitals, where the vendor has 

done business in the past. If CAHs build such a relationship and introduce some dialogue as they are 

implementing about past workflow issues or difficulties encountered during planning or 

implementation, this communication may work to their advantage. 

• Recognize staff training needs. Drumright CAH’s Nurse Director reported the importance of 

“overstaffing” during the go-live period. She brought on an extra nurse and CNA for each shift to 

ensure that clinical staff duties were fully covered, even with the new work required by the EMR. It 

is important to keep in mind that not all staff are computer savvy, and learning curves among staff 

vary drastically. Drumright has been taking these differences into account, and “training the heck 

out of the younger administrative assistants” to support the nursing staff, and assisting those who 

were least used to computers. 

L. Sustainability

A portion of Drumright and Holdenville’s annual fees for maintaining their EHR systems will be eligible 

for cost-based reimbursement from Medicare. The CEOs anticipate using funding from their Small 

Hospital Improvement Program (SHIP) grants each year to cover annual maintenance costs. Holdenville 
FLEX CAH HIT Evaluation- Final Report

 63

reportedly had not designated funding for sustainability; however, the Board has plans to consider 

sustainability funding through the July 2009 budget process. The Board also approved the purchase of a 

larger, mid-size server to “grow” the EHR system for at least seven more years. The upgraded server has 

the capacity to hold both clinical and business documentation. Holdenville included an FTE for an IT 

specialist in the budget, as well as funds for hardware, software, and infrastructure. Holdenville is 

supported by sales tax and city property taxes, and has established a relationship with OSU (beginning 

with the Flex project), and OSU will continue to look for grants. Holdenville is a Public Trust 60, so gets 

its T-1 line free of charge, and will be able to get up to two more T-1 lines free, as well.

Impediments to Sustainability

CAH finances were the biggest impediment to sustainability. According to Holdenville respondents, the 

biggest single issue in regard to sustainability will be “keeping up” with modifications to the

infrastructure. In contrast, at Drumright, the vendor’s Web-based ASP model allows the CAH to maintain 

the EMR with minimal costs. Holdenville respondents also raised the need for funds for a backup system 

or “disaster recovery system,” which was not offered by their vendor. There is, however, a duplication of 

the system as a temporary backup, but if the server failed for some reason, Holdenville will not have 

access to the clinics’ electronic system. In this case, the clinics would be obliged to use “downtime” 

procedures until the system was again accessible. 

Complementary Funding or Support for the HIT Implementation

Aside from the Small Hospital Improvement Program sought each year to cover annual EHR system 

funding, neither of the CAHs has sought outside funding for sustainability. Holdenville’s CEO stated she 

would “love to get a hold of something” and “will continue to search for anything that’s out there grantwise” but reported she had not yet heard of any newly applicable grants. Neither does Drumright have 

ongoing funding to sustain the Flex CAH HIT activities, but, as discussed above, is confident that monthly 

feeds for the EMR will remain low and costs recoupable by increased staff efficiency and greater billing 

accuracy. 

M. Value-Based ROI19

1. Clinician commitment

Northeastern Oklahoma CAH EHR Network (the Network) respondents were asked to rate user clinician 

commitment before and after implementation of Flex CAH HIT grant activities along the Likert scale 

shown below. 

Both the baseline measure and implemented system measure were noted at Very Committed: 

However, respondents did note that clinician commitment at Fairfax and Bristow hospitals was varied, 

due to a lack of satisfaction with the implemented system.

 

19 Refer to Appendix C for a detailed description of each variable used to calculate the value-based ROI.
FLEX CAH HIT Evaluation- Final Report

 64

User training and support

Oklahoma Network respondents reported a wide range in terms of the quality and quantity of user 

training and support conducted as part of the different hospitals’ implementations. Drumright 

respondents collectively noted being very happy with the training and support they have received from 

their vendor, Cerner. Clinicians received several hours of training for each system module, such as 

documentation, orders, scheduling, and so on. Cerner also trained laboratory staff. In general, this 

support was seen as complete and very responsive. However, respondents were quick to note 

perceptions that Cerner was trying to make a particularly good impression with this implementation. 

Two comments received were that this project was Cerner’s first CAH implementation and that it 

wanted Drumright to become a “poster child” for future, similar business development, and that 

Oklahoma State University is looking to implement an HIT system and Cerner wanted to go “above and 

beyond” to make a good impression. Grant administrators did not think hospitals would be treated so 

well in the future.

For Holdenville, training conducted by its vendor, HMS, occurred for one week, and then additional 

training was offered at an additional cost. Staff noted that overall, this training was not very efficient. 

However, staff did find that using a “train-the-trainer” approach worked much better – initial trainings 

were conducted by the vendor on their suggested schedule, then ongoing trainings for current and new 

users are set up by the respective hospital department, based on training needs. Fairfax and Bristow 

respondents reported similar training and support to Holdenville, noting that internal training staff are 

very committed to making sure all users are appropriately trained and capable of using the implemented 

systems.

User feedback

Again, Oklahoma Network participants reported varying methods for seeking and considering user 

feedback, depending on the individual facility. Holdenville reported weekly meetings throughout the 

implementation to discuss issues or concerns and have them addressed. As part of its implementation, 

Holdenville is striving to keep ongoing communications with all user staff; for example, clinical staff keep 

and share a log book of issues or recommendations. One respondent noted that “being a small hospital, 

communication is daily and we continue to find new and/or better ways to work through a process.”

Drumright designed written feedback reports/surveys to assess user satisfaction and comfort with the 

implemented system. Fairfax and Bristow respondents noted that open communication is encouraged, 

and as questions or concerns are raised, they are dealt with as quickly as possible. If needed, additional 

training is then made available.

Implications

Measured from pre-grant implementation to post-implementation, Oklahoma respondents reported the 

same level of user clinician commitment for both points in time – Very Committed. Each facility 

reported significant staff commitment to the activities of the implementation. Drumright respondents 

noted having received positive feedback from staff – that they are excited about the improvements and 

efficiencies of the new system. As a result, staff have been persistent about moving forward with 

implementing additional components of the system. At Holdenville, the group nature of deciding to 

pursue the project and following through together as a team was reported as key to staff excitement 

during the project and in implementation. Ancillary and clinical staff, financial/revenue cycle staff, and 

administrative staff were engaged in product demos and site visits to decide on a final vendor. Staff 

reported excitement about getting a new system in place to help with everyday job tasks as well as to 

help with patient safety. 
FLEX CAH HIT Evaluation- Final Report

 65

Fairfax and Bristow hospitals reported strong commitment by internal training staff to making sure that 

all users are trained and capable of using the systems, and implementation has gone well and has been 

successful. 

The Oklahoma Network facilities reported different methods of staff training, somewhat dependent on 

the level of training included by their vendor. Staff satisfaction was reported to be somewhat tied to 

training – in that facility resources devoted to training, support, and feedback were received well, when 

available.

2. Information availability

Within the context of information affected by Flex CAH HIT grant activities, Network respondents were 

asked to rate how available information is, when and where it is needed, along the Likert scale shown 

below. 

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, Oklahoma Network respondents 

reported greater information availability post-implementation – moving responses from a pre-grant 

level of Neutral to Very Available. Respondents from Holdenville and Drumright hospitals reported 

frustration from clinical staff with searching for records in the paper-based system. Many times, 

information was tracked manually in each department, so information was technically available, 

although it required a lengthy process to retrieve it. At the time of the site visit, Fairfax and Bristow 

hospitals were at a point in their project implementation where they were still primarily using a paperbased system.

3. System usability

For the HIT system(s) implemented as part of the Flex CAH HIT project, Network respondents were 

asked to rate system usability prior to and post system implementation along the Likert scale shown 

below. 

Baseline (orange) measure versus implemented system (green) measure:

User perceptions of ease of use

Network respondents noted that their expectations of the implementation focused heavily on the new 

system’s ability to help streamline processes within the hospitals and ancillary departments and clinics. 

Holdenville in particular referenced their existing electronic lab information system and challenges 

associated with disparate, pre-grant systems (some paper, some electronic). With the implementation, 

Holdenville respondents reported that streamlining these systems was an expectation, and that the 
FLEX CAH HIT Evaluation- Final Report

 66

benefits of such streamlining helped the perception of the implementation – that the end results would 

help to ease the challenges of implementation. 

Respondents did raise some limitations to understanding how workflows would change with 

implementation of the new system, noting that staff knowledge is limited to their workflows with paper 

documentation. 

System use

At the time of the site visit, all sites reported implemented systems were being used as anticipated. 

Fairfax, Bristow and Drumright reported that they had not yet run across any unexpected challenges or 

benefits associated with system use – that their focus was the change in use from paper processes to 

electronic. While Holdenville respondents shared that system use was occurring as expected, 

respondents indicated challenges with interfaces between different systems as an unexpected 

challenge. At the time of the site visit, Holdenville was working with the pharmacy interface –

respondents shared that at the time of procurement, they tried to save money on the pharmacy system, 

but now they are finding many interoperability issues with the EMR system. 

Implications 

Measured from pre-grant implementation to post-implementation, Oklahoma respondents reported 

system usability improvement from Neutral for pre-grant systems to Somewhat Usable for the 

implemented system. Drumright respondents were particularly pleased with the usability of the 

implemented system. One staff member shared, “There’s no way in this world you could ever talk [us] 

into going back to paper. Everything came out better than anticipated.” 

Other hospitals have experienced varying usability of their implemented systems. As referenced above, 

Holdenville has experienced issues with interfaces between existing systems and the newly 

implemented EMR system. Holdenville is also still working through scanning in historical medical 

records (particularly relevant for their emergency department), and referenced this example as a 

hindrance to fully advancing system utilization. Respondents reported that these gaps in utilization 

were being addressed, but staff mentioned that they can only do so much to address these issues – that 

they are somewhat at the mercy of the technology vendor. As another example, staff noted 

mechanisms to integrate their fee-based pay structure with the HMS emergency department system is 

still in process, and is therefore making work more difficult in the short term.

4. System efficiency

For the HIT system(s) implemented as part of the Flex CAH HIT project, respondents were asked to rate 

system efficiency prior to and post implementation along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, Oklahoma respondents reported 

greater system efficiency post-implementation – moving responses from a pre-grant level of Neutral to 
FLEX CAH HIT Evaluation- Final Report

 67

Somewhat Efficient. All sites referenced the efficiency of the paper-based system as being acceptable 

at a basic level for singular episodes of care. The real gains noted for the implemented system were in 

the larger value of providing aggregate care or transferring information. With the paper-based system, 

staff reported a lack of efficiency in getting all the treatment information into the paper chart in a timely 

manner to satisfy the physician and/or nurse caring for the patient. Staff also noted issues with charts 

being issued to physicians for chart completion (or other administrative function) and then the patient 

presenting to the emergency department for care, where treating staff then had to treat without the 

chart information or spend time tracking down documentation. Holdenville also noted that for its 

existing lab system, all information had to be entered manually (from paper charts as opposed to 

information being electronically transferred or populated), thus leaving much room for error.
FLEX CAH HIT Evaluation- Final Report

 68

5. Clinical outcomes

Respondents were asked to rate their perceptions as to the pre- and post-implemented systems in 

terms of supporting positive clinical outcomes along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, Oklahoma respondents reported 

that the implemented system increased perceived support for positive clinical outcomes – moving 

responses from a pre-grant level of Not Very Supportive to Somewhat Supportive. Holdenville staff 

shared a perception that collectively, staff knew they were doing the best they could with the system 

(paper-based) they had. One respondent said staff shared an acceptance of “pretenses that we have to 

wait for things and it may or may not delay the care for the patient.” Collectively, staff referred to a 

longstanding hope to have more timely information to and from the lab down the hall, as well as from 

the radiologist reading reports miles away. Staff further noted that manual records, charting and 

ordering did increase the likelihood for incorrect or misinterpreted information being used to care for 

the patient; they noted that while computer-based systems have room for error in data entry, 

automated checks and balances can help to detect errors immediately. 

6. Non-financial values

Oklahoma respondents noted several non-financial values of the new system, including gains in 

information accessibility, completeness, accuracy and efficiency; use of less paper; improved ease of 

documentation; and improvements in patient safety. The Oklahoma grant administrators said that “all 

the facilities knew [the value of accessibility of patient records] going in, but didn’t conceptually realize 

how good that was going to be.” Physicians were noted as being the most leery or skeptical user group 

of the technology implementation, but “they have been pleasantly surprised.” This staff satisfaction and 

more efficient workflows for staff were noted as specific non-financial values. Since implementation, 

paper workflows were reported as restricted to downtime procedures only.

Respondents noted that as result of the implementation, they anticipate decreases in malpractice 

premiums and in adverse drug events, though neither has happened yet. Also reported were 

expectations that the system will facilitate clinicians’ ability to code correctly and submit proper charges 

to the billing systems to further increase efficiency. Respondents noted that charts are simply more 

accessible and less redundant—illegible handwriting is no longer an issue, and duplicate data are no 

longer collected. 
FLEX CAH HIT Evaluation- Final Report

 69

Patient satisfaction

Respondents were asked to report patient satisfaction of the pre-grant and post-implementation 

systems along the Likert scale shown below.

Both the baseline measure and implemented system measure were noted at Very Satisfied: 

Implications 

Measured from pre-grant implementation to post-implementation, Oklahoma respondents reported the 

same level of patient satisfaction for both points in time – Very Satisfied. The hospitals noted that 

patient surveys have been and will continue to be part of their internal assessments, and that the 

facilities have added questions to their normal patient satisfaction surveys (encounter follow up surveys) 

to appraise activities related to the Flex CAH HIT grant implementation. At Holdenville, staff reported 

that patients have not noticed a difference in their experience at the facility, other than during the 

registration process. As of yet, there have not been any complaints.
FLEX CAH HIT Evaluation- Final Report

 70

Case Study 3: Lakelands Rural Health Network HIE, 

South Carolina Office of Rural Health 

A. Introduction

The purpose of the Lakelands Rural Health Network (LRHN) Flex CAH HIT grant project was to acquire 

the necessary funding and technical resources to implement a regional health information exchange and 

quality improvement initiative in the Lakelands area of western South Carolina. The LRHN is a nonprofit, 

multi-county vertical network that was developed in 2004 with the guidance and financial assistance of 

the South Carolina Office of Rural Health (the State FLEX grantee) to achieve efficiencies, expand access, 

coordinate and improve the quality of essential health care services, and strengthen the rural health 

care system as a whole. The goals for this grant were to:

• serve as the SC rural pilot site for an already developed HIE technology platform that includes a 

personal health record; and

• establish a regional quality improvement (QI) program and reporting function within the HIE.

Figure 3. Lakelands Rural Health Network Implementation Map
FLEX CAH HIT Evaluation- Final Report

 71

The LRHN Flex project consisted of both EMR and HIE implementations, as depicted in Figure 3 above. 

As of May 2009, Self Regional Healthcare (tertiary hospital), Montgomery Center for Family Medicine 

(ancillary clinic), and Carolina Health Centers FQHC were connected through the HIE. Abbeville Area 

Medical Center (CAH) and Lauren’s County Medical Center were in process of connecting with the HIE. 

Also, as part of Flex project activities, Abbeville and Edgefield County Hospital (CAH) implemented an 

EMR.

B. Background

The Lakelands region of South Carolina is very rural and lowincome, with declining economic health. The population is 

aging, diverse, poor, underemployed, and suffers from 

health disparities. The region has the highest rate of infant 

mortality in South Carolina, as well as high rates of diabetes. 

LRHN is a six-county partnership of key players and safety 

net providers in the Lakelands region designed to achieve 

efficiencies, expand access, coordinate and improve the 

quality of health care services, and strengthen the rural health care system as a whole (Figure 4). It was 

developed in 2004 with the guidance and financial assistance of the South Carolina Office of Rural 

Health. Partner organizations include Abbeville Area Medical Center, Carolina Health Centers Inc., 

Edgefield County Hospital, DHEC Public Health Region 1, Laurens County Health Care System, 

Montgomery Center for Family Medicine, Self Regional Healthcare, The Self Family Foundation, and the 

South Carolina Office of Rural Health. The activities of the Flex CAH HIT grant impact nine primary 

partner organizations in addition to 25 practice sites that comprise LRHN. 

Figure 4. Lakelands Rural Health Network Organizational Structure

The mission of LRHN is to “develop a collaborative, economically viable health network to improve the 

quality of care in the Lakelands service area.” Several objectives targeted by the collaborative include:

• Establishing a regional QI council to lead regional QI initiatives. 

• Developing a regional HIT plan that includes expansion of an existing electronic health record (EHR) 

to rural primary care providers.

LRHN critical assumption

“Technology is not an end but a means to 

an end. Technology is one of many 

valuable tools that will achieve our goal of 

improving the health of residents 

throughout the Lakelands region of South 

Carolina.”
FLEX CAH HIT Evaluation- Final Report

 72

• Installing a HIE to help rural primary care providers fulfill the “Meaningful Use” requirement by 

reporting on quality indicators.

Prior to the release of the Flex CAH HIT grant in 2007, LRHN oversaw a number of activities that helped 

set the stage for and complement the Flex CAH HIT-funded activities to follow. For instance, the Clinical

Leadership Council developed a physician-led Electronic Medical Record (EMR) Task Force to evaluate 

ambulatory EHR systems and recommend products that would best meet the needs of LRHN providers.

C. Planning Approaches

Role of the Physician-Led EMR Task Force

In 2006, LRHN conducted an inventory of readiness of 300 physicians located in the Lakelands, South 

Carolina region, the results of which indicated that the Network had only an 18 percent EMR adoption 

rate, but that 40 percent of inventoried practices wished to adopt an EHR within one year. 

Consequently, a task force comprised of physicians (the ongoing EMR Task Force) was formed and began 

meeting under the assumption that Self Regional Healthcare, the Lakeland region’s tertiary hospital, 

would serve as the host of an EMR system for the Network. The initial plan was for Self Regional to 

“open the system up” to all LRHN physician practices and clinics, but this arrangement encountered 

considerable opposition from local physicians who wished to select an EMR that would best meet their 

individual needs. In response, LRHN developed and distributed a survey to Network physicians to 

document their “wants and needs,” and the EMR Task Force examined and narrowed the multitude of 

HIT vendors under consideration. The Task Force used a standard assessment tool as well as the analysis 

of aggregate physician survey responses to make a recommendation, deciding upon the vendor that 

would best meet the needs of LRHN providers to lead the EMR implementation.

Planning the Health Information Exchange 

The co-Principal Investigator (PI) of the Flex CAH HIT grant and de facto “physician champion” of LHRN’s 

Flex CAH HIT implementation project first became interested in pursuing the Flex CAH HIT grant 

opportunity because it included a number of elements that fit closely with the LHRN’s established 

mission and was a logical progression from the EMR implementation project. His personal goal was to 

establish QI with a focus on education and health promotion, and clinical intervention at the 

participating sites using HIE and comparative effectiveness research supports. However, many of LRHN’s 

facilities at that point had not yet implemented EMRs, with some working off a completely paper-based 

system.

The Co-PIs and other key personnel from the LRHN attended two strategic planning retreats at the 

Georgia Health Policy Institute and reviewed the Markle Foundation’s Common Framework for health 

information sharing to learn more about the technology and policy implications of implementing an HIE. 

The co-PIs engaged in a formal process as part of the Clinical Leadership Council (CLC), a physician-led 

multi-county forum, to discuss the clinical elements necessary for an HIE. The CLC reported its major 

findings to the LRHN Board of Directors, who were amenable to the implementation plan. It was 

reportedly a six-to-seven month process, directed by the CLC, to determine the components of a final 

HIE product for the Lakelands Network. 

South Carolina Health Information Exchange 

Unlike the EMR vendor selection, planning tools were not utilized to select an HIE vendor because a 

vendor had already been selected by the State as the vendor for the South Carolina Health Information 

Exchange (SCHIEx) platform. The State designated the selected vendor to implement its HIE in 

partnership with the South Carolina Office of Research & Statistics (ORS). SCHIEx functions much like a 
FLEX CAH HIT Evaluation- Final Report

 73

Master Patient Index (MPI), which works with ORS to leverage existing data. SCHIEx data will include 

claims data only, from Medicaid, the South Carolina immunization registry, and LabCorp reference lab. 

The Flex CAH HIT grant award was an important step because it allowed the LRHN to purchase and test 

the adapters required to implement a two-way data exchange of claims as well as clinical data with ORS. 

Essentially, the Flex grant enabled clinical data to be layered on top of existing claims data held by and 

continuously submitted to ORS under a legislative mandate to collect hospital claims information.

Involvement of Clinical Stakeholders in Planning

The HIE QI Coordinator and Flex CAH HIT project co-PI is also the Director of the Montgomery Center for 

Family Medicine, a teaching hospital, and has experience implementing the Chronic Care Model and 

measures from the National Health Quality Report (NHQR). He reported that as co-PI, his role was to 

“push the QI agenda” and “push what physicians want out of the HIE,” and to expand the HIE to 

incorporate other LRHN clinics. His co-PI on the Flex project is the Director of the South Carolina Office 

of Rural Health. Both co-PIs worked to ensure provider buy-in for the HIE implementation. 

CAH respondents reported that at its core, LRHN is a physician-driven QI initiative. LRHN personnel 

visited every physician in the region to gauge their level of buy-in to the EMR and later, to Flex-funded 

HIE implementations. Physicians reported that the Flex award had moved them “to a point of 

engagement that we otherwise wouldn’t have been able to get to so quickly.” While LRHN providers 

were not involved in the vendor selection process because the decision for the HIE was reached at the 

State level, many were involved in the EMR vendor selection process, as the CLC sought their counsel 

and ultimately selected a vendor after thorough consideration of physician requests for various 

components of an electronic system.

Vendor Selection Process

The HIE vendor was written into the Flex CAH HIT grant 

application following approval by the LRHN Board, based 

upon the existing use of and established relationship with the 

vendor at the State-level. CAH respondents reported that 

there were no challenges with the HIE vendor selection 

process because “it was a no-brainer” and they “didn’t want 

to recreate the wheel.” 

The Flex CAH HIT grant funded the 

connection of the HIE system with EMR 

systems already selected and installed at 

LRHN partner facilities. The HIE 

implementation project served as a 

support for and add-on to these EMR 

systems, which pre-date the Flex award.
FLEX CAH HIT Evaluation- Final Report

 74

Connecting to HIE and the State Record Locator Service

In keeping with the State’s choice of HIE vendor, LRHN desired to link to the core technology of the State 

for purposes of cost-savings and long-term sustainability. LRHN chose the HIE vendor because the 

Network had already contracted with South 

Carolina’s statewide Record Locator Service (RLS). 

The RLS adapter technology linking individual 

health care providers’ EMR systems to the HIE had 

already been fully developed at a test site in 

Charleston. Since the initial system design was 

complete, the cost-savings for connecting to the 

HIE vendor were significant (near $1 million). 

Further, LRHN received approval from the South 

Carolina Data Oversight Council in March 2009 to 

become the second site in the State to connect to 

the RLS’ existing data holdings. The LRHN plans to 

expand the Flex-funded HIE to enable real-time 

clinical information sharing from disparate 

electronic data sources. 

A subset of provider-level clinical data is attached 

to and shared by the RLS, providing a “thin” level 

of data immediately to LRHN and other State 

providers with access to the Internet. This 

capability will allow for the measurement and 

comparison of high-risk populations in the 

Lakelands region, particularly individuals with 

diabetes and cardiovascular disease (CVD). The 

Flex-funded HIE, therefore, will be instrumental in 

enabling physicians to generate quality reports to 

gauge how their patients compare nationally 

(using data from the National Health Quality 

Report). This capability of the State HIE model to 

report and compare provider-level clinical data 

with the RLS offers an immediate benefit to coordination of care. As a result, LRHN considered an 

important component of the HIE to be the system’s ability to improve the Network’s quality indicators, 

while delivering care in the most cost-efficient manner possible. 

D. The EHR/HIE System

Phase of HIT Investments at Participating Facilities

A variety of EMR systems were installed within LRHN facilities at the time of the LRHN application for 

the Flex CAH HIT grant. Each of the three Network CAHs also had received funding to choose and 

implement a vendor to meet their needs prior to the release of the Flex grant. While the co-PI reported 

that the HIE vendor “was comfortable” connecting its HIE applications with one of the EMR systems, he 

was unsure about its ability to connect to the other EMR system. 

By June 2009, the HIE was interacting with Self Regional Hospital and Self Physician Practice. A July 2009 

“go-live” was planned for the CAHs/clinics. The co-Principal Investigator reported delays in the initial HIE 

SCHIEx. 

The South Carolina Health Information Exchange 

(SCHIEx), gives providers the ability to view clinical 

data to include medications, diagnoses, procedures, 

and common problems, thereby positively affecting 

continuity and quality of care provided, as well as 

assisting with controlling cost. SCHIEx will also 

include lab results, immunizations, discharge 

summaries, e-prescribing, and operative and clinical 

notes. These clinical data are driven from 10-years 

of paid SC Medicaid claims, as well as information 

shared from participating providers' EMR systems. 

Through SCHIEx, clinicians get the healthcare 

information they need, when and where they need it, 

to support the best possible outcomes for their 

patients. SCHIEx's unique structure allows data to 

be organized around a person -- not a transaction, 

provider, or organization. By centralizing relevant 

patient data, SCHIEx helps practitioners make more 

informed clinical decisions at the point of care. 

SCHIEx enables clinicians to reduce duplication and 

potential errors by providing them with information 

on services previously rendered. Through SCHIEx, 

providers can reduce inappropriate and unnecessary 

orders, paperwork and treatment delays. Providers 

gain insight into the care their patients have received 

at other locations - all at no extra cost.

Excerpt from http://www.schiex.org/index.php
FLEX CAH HIT Evaluation- Final Report

 75

connection with Carolina Health Center because its EMR vendor was not providing optimal service and 

the HIE vendor was struggling to connect to a marginally interoperable system. Additionally, delays were 

anticipated connecting Edgefield County Hospital because it had not yet implemented all modules (such 

as order entry) necessary for the facility’s EMR to connect with the HIE. Edgefield had also purchased 

but not yet installed a practice management system, as well as modules for nursing and physical 

therapy, as of May 2009. 

As of May 2009, the Network had three physician practices ready to go live; however, it is ideal for all 

ten to go live simultaneously. Eventually, any physician will be able to log on to the HIE and have access 

to primary care data. However, that has not yet happened, since the network administrators are waiting 

for the ten practices to be ready; until then, with only a few hospitals participating, the value is just not 

there for the clinicians. The LRHN would like the HIE rolled out as a network for all the communities 

rather than piecemeal; if one physician logs into the HIE and the data are not there, then the physician 

only “gives it a few shots” before he stops using it. At the last LRHN Board meeting, it was decided that 

they will go live with the HIE rollout by July 2009. 

Features and Functional Components of the HIE System

Implementation involved connecting a live HL7 interface of demographic data visit information, lab data, 

medication lists, and reports to an HIE adapter. The HIE adapter sends fourteen unique identifiers to the 

ORS in Columbia, which then matches and responds back through the adapter to the other LRHN sites 

where the identified patient has been located. Then, other LHRN adapters send the patient’s visit 

information into a longitudinal record. Self Regional Health Care built the HL7 architecture to support six 

hospitals and 22 sites, each of which has an adapter. Once a patient encounter is requested and the visit 

sent, the data are dropped from the central system. Patient information is secure because the only 

information sent is the information that the Web server requests and the database recognizes one Web 

server. Under this federated model, there is no central repository – each hospital, clinic, or physician 

practice site owns its own data. While the member sites are hosted by Self Regional Health Care at 

present, this functionality is movable. The facilities share responsibility for monitoring and auditing the 

information exchange.

System Testing

The HIE patient matching process is based on seventeen variables, and IT personnel reported a very high 

degree of confidence in records matching. To verify the system’s reliability, a number of “dummy” 

records were mixed in with the legitimate patient information. The HIE successfully filtered out the false 

information. The vendor has also been testing algorithms.

Technical Support

LRHN employs IT technicians to staff a round-the-clock helpdesk and hotline to answer HIE trainingrelated questions. Several layers of Helpdesk expertise are available—the users’ first place to look for 

help is the site administrator, followed by the system administrator. If either of these parties cannot 

resolve an issue, it will be passed to the HIE vendor developers. 

Site administrators are responsible for collecting and documenting issues and identifying recurring 

problems. A number of such (recurring) questions were covered during the site administrator training, 

enabling them to troubleshoot the most basic issues without deferring to the systems administrator or 

vendor developers for assistance. Early during the initial trainings targeting the site administrators, 

LRHN and project personnel collected a list of potential issues to be addressed, to pass on to the HIE 

vendor. The vendor also offers a “provide feedback” button on their Web site. CAH respondents thought 
FLEX CAH HIT Evaluation- Final Report

 76

that technical support had, to date, been sufficient for their needs, and though at times they 

encountered vendor delays, any issues were ultimately resolved.

E. Governance, Policies, and Procedures

LHRN Network Policies

As a result of the Flex CAH HIT award, LRHN developed specific data host policies and new legal 

agreements. This section accounts for changes to LHRN policies and procedures anticipated by CAH 

respondents, as the HIE implementation was planned to go-live in late summer 2009.

In order to enter a “live mode” within the planned timeframe, LRHN developed a policy framework. 

Respondents explained that initially, the LRHN took responsibility for “policing” the Network, but 

ultimately decided to return the responsibility for enforcing policies and procedures to individual 

Network facilities. It was decided that each partner should have and be accountable for its own security 

and HIPAA privacy policies. However, the legal structure is such that the greater LRHN monitors the HIE 

and can approach facilities if “something goes awry.” For example, in the instance of a security breech at 

a Network facility, the facility’s site administrator would be notified by the LRHN, whereupon that 

facility would hold sole responsibility for dealing with the problem according to its own established 

policies and procedures.

Security

Network Architecture 

Self Regional Health Care built the architecture for the HIE system to support six hospitals and twentytwo LHRN sites, each of which has an HIE adapter to communicate and share information with other 

sites. The idea behind this architecture was for patient data to stay within facility source systems. For

instance, once a patient encounter is requested and the visit information is sent, the data is “dropped.” 

Patient information is therefore secure 

because the only information sent is 

information that the Web server requests. 

There is no central repository: each site owns 

its own data. The benefit of this federated 

model is that in the case that any one facility 

has a failure, patient information is secure 

because it is warehoused separately and not 

“owned” by the Network. 

User Agreements

Prior to the HIE implementation project, 

everyone who used the EMR systems had to 

sign a user agreement. However, because the 

HIE falls under Network ownership, individual 

end-users no longer need sign an agreement; 

instead, site administrators (usually the IT 

manager) at each facility sign one agreement, 

as does the system administrator housed at 

Informed Consent and the Option to Opt Out of the HIE 

Part of the LHRN “governance redirection” was driven by 

the concept that informing patients about the HIE should 

be a provider-led conversation. Patients are given a 

brochure describing the HIE and the option to opt out, in 

hopes that this information will initiate a provider-patient 

interaction during a primary care visit, rather than at the 

level of office staff.

The LHRN held off the go-live date of the HIE in part to 

allow enough time for patients to have a conversation with 

their providers about how the HIE will impact privacy and 

confidentiality. One CAH respondent noted, “This is the 

quietest rollout of HIE on the planet, because we want 

people to come to their physician to talk about why this is a 

good thing or a bad thing for them. It keeps them from 

thinking this is a big government conspiracy” [to capture 

patient data].
FLEX CAH HIT Evaluation- Final Report

 77

Privacy and Confidentiality Components of the Lakelands Rural Health Care HIE System

• Health care providers will ask for patients’ written permission to view their health information.

• Access to health information is restricted to only authorized users.

• Health providers are required to follow all the same privacy and security laws that they do now.

• HIE participation is completely voluntary; patients can opt out at any time.

• There is not a single “combined” data base. Each provider’s information is kept separate and a “view” of the 

patient’s information is created only when needed for care. 

• Except for medical emergencies or special situations, health care providers must have an established 

relationship with a patient to view his health information. 

• Health care providers must include information about the HIE in their notice of privacy practices.

• Lakelands HIE System is subject to robust audit guidelines.

Self Regional Healthcare tertiary hospital. The system administrator is then responsible for assigning 

user identification codes and passwords. Each end-user, is, however, obligated to sign a security form 

obligating them not to share their password. 

End-users not directly caring for a particular patient have only limited access to patient data, and are not 

able to identify or locate demographic information on the patient. The level of access to patient data is 

typically in four tiers: the site administrator, the clinician, the audit record, and finally, the demographic 

record, available to front office personnel. If an end-user does not have an established electronic 

relationship with the patient, the patient’s identity is hidden. The HIE system offers an advanced 

confidentiality feature which documents who has viewed the components of a patient record, as well as 

when and what components were viewed. This audit log essentially serves as a deterrent to people who 

should not or need not be looking at the patient chart. It is open to anyone who has been given access 

to the patient record. 

Data Privacy and Confidentiality 

CAH respondents explained that initial concerns about protecting patient privacy were unduly amplified. 

Key LRHN and facility personnel realized that the facilities would still be exchanging the same 

information as had been exchanged before the HIE implementation, just along a different (electronic) 

continuum. Therefore, key personnel decided that the only needed policy change related to privacy was 

a notification to patients about the HIE go-live, and a vehicle to allow patients to opt-out of participating 

in the electronic data exchange system. The Public Information Collaborative, one of the functional 

subgroups of the LRHN, developed material for use by Network facilities, including recommended 

verbiage for a notice of privacy practice, and public education material for data privacy and 

confidentiality, including a brochure of FAQs, an informational poster for each site, and a Website 

notice. The LRHN also crafted a Global Confidentiality Agreement in concert with all HIE-participating 

facilities, to prompt these facilities to abide by the specified policies according to HIPAA law.
FLEX CAH HIT Evaluation- Final Report

 78

F. Staff Training 

As of May 2009, a designated site administrator at each participating facility was trained to be “familiar” 

with the features and functionality of the HIE system. Site administrators included the co-PIs, as well as

IT Managers. Super-users, including medical managers, were to be trained by the co-PIs. One co-PI was 

responsible for training the residents, as well as other providers at the Montgomery Center. Other site 

administrators were tasked with simultaneously 

training providers and nurse super-users at their 

sites. The super-users were then tasked with 

spreading the information to all clinical staff 

using a “train-the-trainer” approach in order to 

accommodate the geographically disparate 

nature of the sites. This approach will also allow 

super-users to apply lessons learned from prior 

trainings as they conduct trainings at successive 

sites. 

Site administrators reported that prior to the 

super-user and clinician trainings, each LRHN 

facility built awareness of the HIE through 

informational updates and general staff 

meetings. There were also plans to conduct 

subgroup staff meetings to address the concerns 

of specific clinical personnel. For instance, 

training for medical mangers was intended to target both the clinical and patient education and HIPAA 

components of the HIE, whereas providers would receive training only on the clinical applications. 

Training of front office staff would be focused on how to present notices of privacy practice to patients, 

since these personnel would likely receive the majority of patient questions regarding the electronic 

storage and transmission of their health information. One respondent noted the ability of front office 

staff to respond to patient questions could “make or break” patient buy-in to the HIE federated model.

Types of Staff Trained

“Those staff who have need will be given access” to the HIE, reported a site administrator. Physicians, 

nurses, billing staff, and clerks will be given access to the system. The current LHRN policy for access to 

the HIE is to allow access only to those personnel who are inputting information. However, key LRHN 

personnel reported they would be amenable to the use of a de-identified patient server to make 

information available in the aggregate for research purposes. CAH respondents noted expectations of 

the percentage and type of staff that use the HIE routinely will depend on the robustness of the system. 

Training Schedule

The Network determined that training should be done close to the time at which the HIE becomes 

functional so immediately following training, clinicians would have access to the system. As one site 

administrator explained, “We all have been sitting around waiting until we got enough [electronically 

accessible historical patient] information. If we showed the system to providers tomorrow, and they 

couldn’t use it right away, they’d forget about it.” While Self Regional Healthcare, the Montgomery 

Center, and Carolina Health Centers had the capabilities and infrastructure to go-live with the HIE, the 

actual date of implementation (and with it, training), was reportedly on hold until Abbeville, Edgefield, 

and Laurens CAHs/clinics reached a stage of connectivity. 

Training to Respond to Patient Privacy Concerns

One of the biggest challenges to overcome during the 

HIE go-live is ensuring that front office staff 

understand how to explain the federated model of 

data management to patients. Patients received a 

brochure describing the EMR and HIE and given the 

opportunity to opt out of these systems. While key 

project personnel hope that patients will discuss the 

relative merits of the EMR and HIE systems directly 

with their providers, they are aware of the importance 

of addressing patient concerns at the first point of 

contact with office staff. Therefore, training front office 

staff how best to articulate the privacy and 

confidentiality components of the electronic 

transmission and storage of data in a federated model 

will serve as a cornerstone of staff training.
FLEX CAH HIT Evaluation- Final Report

 79

Site administrators reported that they preferred to minimize refresher-like trainings, which contributed 

to the decision to hold the staff trainings as close as possible to the date of implementation. However, 

after the HIE go-live day, site administrators and super-users anticipate offering ongoing training. 

Training that encompasses the EMR and HIE system at each facility must become a part of new 

employee or resident physician orientation. 

Satisfaction with Training

The LRHN conducted a pre-HIE staff satisfaction survey at Abbeville Area Medical Center and Laurens 

County Health Care System, and plan to roll out a post-live HIE evaluation, as well. However, since the 

CAHs/clinics received generally favorable responses to the pre-HIE (post-EMR) survey, a significant 

improvement in staff attitudes subsequent to the implementation was not expected. Despite the delay 

on staff trainings, the co-PI reported that hospital staff have been very energized and excited about the 

HIE implementation process, particularly young, tech-savvy doctors and resident physicians.

G. Impact on Hospital Operations

Tracking Data Through the Continuum of Care

Planners believe that the implementation of the 

LRHN HIE will not interfere with physician daily 

practice due to the behind-the-scenes extraction 

of data. The EMR implementations at each 

participating facility have positively impacted the 

availability of data, and CAH respondents 

expected the HIE go-live to further increase their 

ability to track data through the continuum of 

care. One respondent stated, “Obviously [the 

HIE will increase] availability, especially for the 

patients who don’t have any of their records with them, or who [are] incompetent or unresponsive [to 

information requests].” 

Quality Improvement

Many of the more progressive LRHN providers had been interested in doing their own quality reporting, 

but either did not know how, or did not have the hardware or software necessary. Those clinicians who 

tracked QI within their own EMR systems prior to the HIE implementation had no means of validating or 

comparing their patient data to other Network providers or nationally, or of using QI to make ongoing 

improvements in health care delivery within the Network. In particular, some types of quality measures, 

especially those not driven by lab data or prescriptions (i.e., aspirin use or smoking status), may not have 

been recorded in the patient health record at all. This type of data will begin being captured by either 

clinicians or support staff with the HIE go-live. QI will also improve data tracking through the continuum 

of care, as physicians and nurses work more closely to document patient information. The capacity to 

track QI data will also facilitate linkages between providers though competition, and will serve as a 

communication mechanism among clinicians. 

While access to information is helpful at certain times with 

certain physicians and certain rules, it is not a fix for 

everything in health care. According to the co-PI, “Rural is 

the difference... because rural has space. Even distance, 

now [with the HIE], does not mean lack of health care. The 

HIE gives [providers] the ability to connect to clinics, and 

get a discharge summary, and be part of quality 

improvement even when they are one physician out in the 

middle of nowhere.”
FLEX CAH HIT Evaluation- Final Report

 80

H. Evaluation

Diabetes Measures and Case Management 

Diabetes is a major problem in South Carolina, and was an ideal chronic condition for tracking by the 

HIE, because the measures covered a wide range of metrics, including clinical data e.g., foot exam), lab 

data (e.g., LDL cholesterol), vital signs (e.g., blood pressure), outside data (e.g., eye exam), and other 

measures such as smoking cessation advice and status. The standards for the evaluation measures 

originated from HRSA, but it was decided to use the more rigorous, NCQA point standards as a guide. 

CAH respondents anticipate that within a year, diabetes outcomes will show a marked improvement 

from baseline data collected from 2007 to 2009 across sites. They also are confident that the HIE system 

will contribute to decreased incidence of myocardial infarction and stroke.

The use of a rigorous, national-level diabetes grading 

system has generated interest at the provider level. The 

co-PI projected that at least forty LRHN providers, in 

addition to residents, will send their individual and 

practice-level data to NCQA in order to become certified 

in diabetes care (diabetes certification is reimbursable 

by Blue Cross Blue Shield). The co-PI further described 

his ability to do case management using de-identified 

patient diabetes information. The HIE will allow for the 

tracking and comparison of disease measures between providers and physician practices for the 

identification and documentation of individual or systemic problems. He anticipates that following the 

HIE go-live, physicians or nurses will be able to manage patient cases in this way themselves.

I. Supportive Factors 

• Strong Administrative/Network Leadership. The LRHN Coordinator, the CEO of Carolina Health 

Centers and Board Chair of LRHN, the Associate Director of the South Carolina Office of Rural Health, 

and two physician champions, the Medical Director at Carolina Health Centers and the Director of 

the Montgomery Center for Family Medicine, each worked to advance the Flex CAH HIT 

implementation. Together, they served as strong advocates to enlist provider and hospital staff buyin, to collaborate with the EMR and HIE vendors, and to generally ensure the success and 

sustainability of the project.

• Engaged Physicians. LRHN administrators reported that having two physician champions intimately 

involved in the construct of the implementation was immensely useful. As one co-PI explained, 

“Physicians help bring the impact of the EHR down to earth – where we are, what we can do, and 

where we can take this in the future.” Essentially, physicians serve to ground an HIT project in 

practicalities, and to guide implementation activities with a pragmatic, clinical-centered approach. 

Additionally, CAH respondents reported the two physician champions to be very hands-on with the 

vendor in the development of various HIE modules, workarounds, and forms. 

• Consistent Vendor Terminology. Negotiating IT is extremely difficult for staff who are not familiar 

with IT jargon. Therefore, respondents reported that clear, consistent, and well-defined vendor 

terminology was very helpful.

• Support from the State Office of Rural Health. The South Carolina State Office of Rural Health 

(SORH) was accommodating and directly supportive, according to key personnel involved in the Flex 

grant activities. For instance, the SORH collaborated with LRHN staff to work through budgetary 

How Do You Define Success?

CAH respondents defined success as meeting 

NCQA standards for diabetes management 

across the Network. Monthly provider and 

physician practice-level data will be used as 

an important tool to get clinicians to meet 

yearly diabetes management goals.
FLEX CAH HIT Evaluation- Final Report

 81

issues, and was helpful in communicating directly with HRSA to get timely answers to facility and 

staff questions about the HIE implementation and vendor support. 

J. Challenges and Solutions

• Planning and Vendor Selection. CAH respondents reported that one of the biggest challenges has 

been ensuring that the network technology partners understand the “real world” and the limitations 

within hospital systems. Many challenges with the HIE implementation were related to the 

“philosophical framework that is the gap between the IT world and the health care world.” From a 

process standpoint, the Network has had to take “a few steps back” in the timeline of the 

implementation when it was thought the technology would be able to accommodate a particular 

approach, but in fact was not yet ready to do that. “You don’t just turn a switch and have this stuff 

work perfectly.” 

• Interoperability. The LRHN found one existing EMR vendor system troublesome because of 

difficulties encountered interfacing with the HIE system. A facility IT manager reported their EMR 

had a 24-hour delay when uploading information to the HIE HL7 server. The Network has attempted 

to deal with this delay by designing a “work-around,” or querying program that automatically 

uploads information to the HIE server twice daily. CAH respondents also reported a number of 

challenges with the corporate structure of this particular vendor. The vendor is sun setting its EMR 

application in favor of another EMR; however the HIE implementation was delayed in large part due 

to persistent interoperability problems.

• Coordinating the HIE Implementation Timeframe. Key personnel reported that “HIE timelines 

[were] kind of crazy,” whereas implementing EMR systems was a much more standard, cookiecutter process. HIE implementation involved many more players simultaneously. “If one player is 

dragging its feet, then the whole process may be held up that much further. That’s why 

implementing an HIE is much more complicated and harder to predict.”

• Governance. One of the major governance challenges with which LRHN grappled, termed “breaking 

the glass,” was how to approach situations where clinicians require immediate access to patient 

information not within their jurisdiction. The ED is one department where providers often need to 

have this type of information access. Another scenario might include a consult, where a treating 

physician discusses a patient condition with another physician not typically involved in the case. The 

Lakelands Executive Director reported that the Network is taking “a very conservative route” to 

patient privacy and data accessibility, and has approached other HIEs/Regional Health Information 

Organizations (RHIOs) like the LRHN to consider alternative solutions. 

• Quality Improvement. While the HIE system is capable of collecting QI data, planning for how to 

effectively evaluate the impact of the HIE system on patient outcomes has had “all kinds of 

challenges.” For example, the co-PIs spent significant personal time querying provider-level patient 

data and uploading it to one master spreadsheet. As one co-PI reported, he “spent all his time 

turning data” from the approximately fifty LRHN providers. While the result –shareable QI data 

across sites – was considered well worth the effort – it was possible only after much complicated 

and time-consuming backend work.

• Staff Turnover. Staff turnover at the individual facilities has been a real challenge, and has reportedly 

affected the Network’s philosophical approach to governance and privacy. Because LRHN’s HIE is a 

federated model where each facility owns its data, the facilities themselves are ultimately responsible 

for maintaining standards. The LRHN, however, is still responsible for producing training materials that 

can be implemented and adapted at the site level, as a way to maintain relative consistency among 

members. The co-PI added that he is not sure that staff turnover will impact sustainability significantly. 

LRHN respondents also expressed concern about staff turnover in relation to some key hospital 
FLEX CAH HIT Evaluation- Final Report

 82

personnel who essentially championed the project. For instance, the co-PIs position was “not 

sustainable” if he were to leave the Network for some reason. 

• Limited IT Staff At Rural Facilities. IT staff are hard to come by at small, geographically isolated 

hospitals. 

K. Lessons Learned

• Build on Established Vendor Relationships. Building an HIE is difficult, and having pre-existing 

relationships with vendors is important. While conducting vendor assessments, LRHN respondents 

noted that the most important component is looking at which vendors are already established at 

facilities within the Network. 

• Technology is Only One Part of the Equation. Respondents noted that while technology is an 

important component of quality health care, it is only a means to an end. While implementing the 

HIE, participants also remained focused on more “low-tech” components of health care, such as 

patient education materials, which help to change patient behavior.

• Plan HIE Implementation. Implementing an HIE requires the collaboration and coordination of many 

providers and entities; the respondents feel that an HIE implementation is therefore more 

complicated and harder to predict. Providers in the Network had been communicating with one 

another since 2003 regarding the EMR implementation. Respondents felt that this prior 

communication, established relationships, and previous work done on the EMR implementation 

bolstered their efforts to implement an HIE.

L. Sustainability

The LRHN, as the first rural test site for the HIE system, expressed hopes that other parts of South 

Carolina will connect to and eventually form a Statewide HIE network. LRHN respondents reported that 

when thinking about sustainability early on during the project planning phases, they envisioned an 

associated fee that would generate a revenue stream. However, because SCHIEx is offered to providers 

free of charge, this plan was abandoned. The LRHN Board also decided on the importance of offering a 

free or reduced-fee HIE to providers because the organization’s mission was to establish an electronic 

system that would improve quality of care, not one that would be tenuous because of providers’ 

inability or refusal to pay an associated fee. Other more viable approaches to sustainability mentioned 

by LRHN respondents, possibly dependent on Federal stimulus incentives, included:

1. Connecting the LRHN with a Regional Extension Center for technical assistance and support in 

the form of grant-writing and outreach.

2. Selling de-identified patient data for research.

3. Selling access to a Personal Health Record (PHR) to the public. For example, CAH respondents 

cited an initiative in California where individuals are reimbursed by their insurance companies 

for access to their PHRs. One co-PI believes that after a robust EMR/HIE system is established at 

the CAHs (Abbeville Area Medical Center and Edgefield County Hospital), patients might be 

willing to pay for a Web-accessible PHR. 

Impediments to Sustainability

The cost of maintaining the HIE system was quoted at $200,000 to $300,000 annually. A large portion of 

this maintenance fee is related to the cost of maintaining the adapters at each facility. Given the present 

state of the economy, it is uncertain whether patients and physicians will be willing to pay for quality 

reporting. This is also dependent on the State financial environment and the Federal Government’s 
FLEX CAH HIT Evaluation- Final Report

 83

willingness to pay for quality. CAH respondents also noted that training is a barrier to sustainability, 

because the resources and staff time required for adequate and routine training need to be built into 

the maintenance costs for the HIE. 

The tertiary hospital and the HIE vendor have funded a great deal of the EMR and HIE implementation 

projects with in-kind donations. However, Edgefield has “a long way to go” before it has a fully 

functional EMR, and needs to find more grant money before it will be able to connect to the HIE. 

Complementary Funding or Support for the HIT Implementation

The LRHN routinely monitors for grants that would sustain the HIE and assist facilities with annual 

maintenance and staff training costs. LRHN respondents reported that a QI project funded by USC 

Preventative Health and Family Medicine will bring in additional funding. LRHN also has earmark 

requests in queue through a local Congressman.

Access Health, a Duke University endowment project attempting to build a Statewide model for 

establishing business partnerships, is looking to approve between two and five applications in South 

Carolina with the goal of building sustainable networks. The LRHN HIE project is a viable candidate for an 

award. The Access Health project will conduct a comprehensive community needs assessment, and 

provide within the first year, technical assistance around building sustainability for the Network. In years

two and three, funding for infrastructure and implementation will become available. Respondents 

reported that the technical assistance offered by Access Health would be especially useful to sustain the 

project. 

eHealth South Carolina is an initiative convened by the Office of Research & Statistics that is comprised 

of a number of coordinating entities, including an HIE workgroup considering how to link sub-network 

organizations. Since the Lakelands Network is the most rural HIE initiative in the State, the Network 

plans to be a key player in eHealth South Carolina. The LRHN is spreading the word about the Network’s 

HIE implementation at State and national conferences in hopes of learning about new funding 

opportunities, as well.

M. Value-Based ROI20

1. Clinician commitment

LHRN respondents were asked to rate user clinician commitment before and after implementation of 

Flex CAH HIT grant activities along the Likert scale shown below. 

Baseline Likert scale (orange) measure versus implemented system Likert scale (green) measure:

User training and support

South Carolina reported that system user groups – primarily clinicians, laboratory workers and 

administrative staff– were receiving initial and ongoing trainings. These trainings included formal 

(structured) interactions with system features. Additional user groups would be considered for training 

 

20 Refer to Appendix C for a detailed description of each variable used to calculate the value-based ROI.
FLEX CAH HIT Evaluation- Final Report

 84

if and when they were to interact directly with the system. At the time of the evaluation site visit, 

physicians had not yet received HIE training. Administrators planned to have lead physicians train other 

physicians as “physicians wouldn’t take other staff seriously.”

User feedback

LRHN respondents reported myriad methods and forums for soliciting user feedback, including a “Wants 

and Needs” inventory sheet, meetings, weekly scheduled conference calls, breakout groups, governance 

summits and emails, followed up with meeting minutes, agenda setting and action items. Within each 

facility, the IT Director and CEO receive feedback from clinicians and administrators, and who in turn 

receive information from IT Steering Committee meetings (where issues are discussed formally) in the 

form of meeting minutes. Additionally, historical system data are available to system users with proper 

authorization, allowing for better coordination of information and feedback across sites.

Implications

Measured from pre-grant implementation to post-implementation, South Carolina respondents 

reported that clinician commitment to Flex CAH HIT grant activities moved from Somewhat Committed

to Very Committed. Comments received during interviews further explained that clinician knowledge of 

system offerings affected their commitment, as did the stage of system implementation at the time of 

the survey. 

Further, at the time of the site visit, LRHN’s plan for system user training and support was a work in 

progress. Plans for both structured and more informal peer-to-peer training were under development, 

with some user groups – primarily clinicians (non-physicians), laboratory workers and administrative 

staff–receiving initial trainings. Further support for system users includes many avenues for soliciting 

feedback on the overall implementation and specific system functions (e.g., a “Wants and Needs” 

inventory sheet). Additionally, the IT Steering Committee shares information about the process and 

outcomes of addressing specific system issues with clinicians and administrators.

2. Information availability

Within the context of data affected by Flex CAH HIT grant activities, LRHN respondents were asked to 

rate how available information is, when and where it is needed, using the Likert scale shown below. 

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, South Carolina respondents 

reported greater information availability post-implementation – moving responses from a pre-grant 

level of Somewhat Available to Very Available. CAH respondents reported that before implementation, 

information was somewhat available, but fragmented, and that data not in-house were difficult to 

obtain, as access was only possible through a variety of manual means such as telephone or fax. Upon 

system implementation, data were not only very available for clinical data, but for de-identified 

(population) data as well.
FLEX CAH HIT Evaluation- Final Report

 85

3. System usability

For the HIT system(s) implemented as part of the Flex CAH HIT project, South Carolina respondents were 

asked to rate system usability pre- and post-system implementation using the Likert scale shown below. 

Baseline (orange) measure versus implemented system (green) measure:

User perceptions of ease of use

One grant administrator, talking about the paper-based medical records system said, “It’s like asking 

how you did wash before a washing machine? You just expected patient encounters to be a disaster.” 

Mark One respondent noted that the paper-based medical records system was problematic for new 

resident physicians who are mostly familiar with EMRs, and who find the transition to a rural community 

with only paper records very difficult. “Residents are horrified – it makes recruiting in medically 

underserved rural areas much harder.” 

As for the overall HIT implementation, LRHN reported that physicians perceived that the EMR and HIE 

systems would be useful tools for integration, giving them easier access to data that would otherwise 

not be able.

System use

South Carolina’s sites were not all fully implemented (particularly in terms of populated data in their 

HIE), impeding full use of Lakelands Connect. Additionally, governance and policy procedures were also 

identified as inhibitors to staff fully utilizing the HIE and EMR systems. 

Implications 

Measured from pre-grant to post-implementation, South Carolina respondents reported significant 

system usability improvement from Not Very Usable for pre-grant systems to Very Usable for the 

implemented system. Comments detailed that the pre-grant, paper-based system was not convenient 

and duplicative. Outside of an individual hospital site, information was fragmented, although though 

staff within a hospital found their own records and other information useable for their needs.

South Carolina noted that paper-based systems added to the significant challenges with recruiting 

resident physicians in rural, medically underserved areas. Thus, the Flex CAH HIT implementation 

project is seen as an opportunity to affect this information access disparity, and it was reported that 

overall, LRHN-affiliated physicians perceived that the EMR and HIE systems would be useful tools for 

integration and easier access to data that would otherwise not be available.

4. System efficiency

For the HIT system(s) implemented as part of the Flex CAH HIT project, LRHN respondents were asked to 

rate system efficiency before and after implementation using the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:
FLEX CAH HIT Evaluation- Final Report

 86

Implications 

Measured from pre-grant implementation to post-implementation, South Carolina respondents 

reported significantly greater system efficiency post-implementation – moving responses from a pregrant level of Extremely Inefficient to Somewhat Efficient. Comments on the pre-grant systems 

mentioned stop-and-go processes for information access that impeded efficient decision making and 

action. Information/data were fragmented and compartmentalized with cumbersome, if any, access. At 

this point in implementation, respondents see the system as being somewhat efficient but significantly 

improved over the pre-grant systems, and expect that as it is further deployed, additional efficiencies 

will accrue. The Web-based nature of the systems and consolidated view of the EHR, arranged in tabs, 

enabled users to understand information more easily and make decisions more efficiently.

5. Clinical outcomes

LRHN respondents were asked to rate their perceptions as to the pre- and post-system implementation 

in terms of supporting positive clinical outcomes using the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, South Carolina respondents 

reported that the implemented system significantly increased perceived support for positive clinical 

outcomes – moving responses from a pre-grant level of Not Very Supportive to Very Supportive. 

Respondents shared additional comments that the implementation has had a huge impact on outcomes 

such as motivating providers to more effectively manage chronic conditions (diabetes in particular) via 

thorough documentation and clinical information capture. The grant administrator reported that the 

whole process has helped show how clinicians were collecting data – and stated that the HIE/EMR has 

helped him go through the documentation and care management processes more meticulously to 

ensure accuracy, completeness, and efficiency. 

In a specific example of the implementation’s effect on clinical outcomes, South Carolina reported that 

before the grant, clinicians did not look closely at longitudinal diabetes numbers. However, since 

implementation, providers have noted that the HIE and EMR have made “a huge difference.” The grant 

administrator reported that he was not doing a good job of sharing diabetes-management information 

with other providers, but with the activities of the grant, suddenly, “we got down to individual offices 

having their own information and seeing what the other offices are doing and comparing themselves to 

everybody else, and going a step further to physicians. This motivated people much more in trying to 

improve their outcomes and make their patients better than everybody else.” 
FLEX CAH HIT Evaluation- Final Report

 87

6. Non-financial values

With their EHR and HIE systems fully implemented and functioning, respondents reported that South 

Carolina is on the cutting edge of technology and is meeting national targets for information exchange 

and electronic health data with timely governance and leadership.

Patient satisfaction

LRHN respondents were asked to report patient satisfaction of the pre-grant and post-implementation 

systems along the Likert scale shown below.

Both the baseline measure and implemented system measure were noted at neutral: 

Implications 

Measured from pre-grant implementation to post-implementation, South Carolina respondents 

reported the same level of patient satisfaction for both points in time – Neutral. 

One LRHN respondent shared their perception of patient satisfaction as “patients don’t have a clue 

about what is happening on the other side,” referencing a paper-based or an electronic system. If 

patients have to wait for an hour due to a lack of information availability, then they can be really 

dissatisfied. The grant administrator shared that “the [patient] assumption in the paper-based world, 

was that docs had all the information, but they didn’t” further, that “[patients] don’t appreciate the 

benefits so much, but they also didn’t see the drawbacks.” Respondents suggested that the bottom line 

is that patients are neutral to information systems, whether they are paper-based or electronic when 

there are no problems, and similarly highly dissatisfied when there are problems with either type of 

system. LRHN reported that they will continue to monitor and assess patient satisfaction through their 

Press Ganey patient satisfaction surveys upon discharge.
FLEX CAH HIT Evaluation- Final Report

 88

Case Study 4: Rural Wisconsin Health Cooperative 

Information Technology Network, Board of Regents 

of the University of Wisconsin System 

A. Introduction

The goal of the Wisconsin Flex CAH HIT grant project was to implement a collaborative electronic health 

record (EHR) environment (initially consisting of a hospital information system and a physician practice 

EMR system) that is shared by multiple CAHs from a common datacenter and supported by a pooled 

staff. In this model, a single CCHIT certified vendor provides the EHR software, and a collaborative nonprofit entity supports a shared staffing structure in addition to a shared data storage center. 

The Rural Wisconsin Health Cooperative Information Technology Network (ITN) is the nonprofit 

consortium organization that operates the collaborative EHR environment. The ITN’s mission is “to 

provide community hospitals and their affiliates with HIT applications and support services that promote 

high quality, cost effective healthcare.” 

Figure 5 depicts the EMR information paths and relationships implemented as part of Flex CAH HIT grant 

activities. The secondary datacenter in Sauk City supports the Madison datacenter where the EMR 

application is hosted. Information is accessed at each facility from the datacenter through the use of 

high speed telecommunications funded by the FCC Rural Healthcare Pilot Program. While sites are not 

yet able to access medical record information outside of their own facility, program development is in 

process to allow clinicians at one ITN member facility to access patient information from other ITN 

member facilities.

Figure 5. Rural Wisconsin Health Cooperative Network Implementation Map
FLEX CAH HIT Evaluation- Final Report

 89

B. Background 

As a result of an extensive planning process begun in 2005, the ITN recognized the need to channel 

resources into small hospital systems otherwise lacking the capacity to participate in EHR 

implementation and HIE. ITN invited a number of small, rural facilities to participate in the Flex CAH HIT 

grant after deciding that the formation of a collaborative of multiple small facilities using the same 

vendor platform was the best approach to EHR implementation. ITN involved four rural CAHs and three 

clinics in the activities funded by the Flex grant. The CAHs included Tomah Memorial Hospital, Memorial 

Hospital of Lafayette County (Darlington), Boscobel Area Health Care, and St. Joseph’s Community 

Health Services (Hillsboro). The three participating clinics are associated with St. Joseph’s. 

C. Planning Approaches

Prior to the release of the Flex CAH HIT grant in 2007, driven by its vision of a collaborative HIE 

environment among multiple small facilities, a Wisconsin collaborative of rural hospitals (four of which 

would eventually become founding ITN members) initiated a goal-setting and vendor selection process 

in 2005. According to the ITN CIO, cost and quality were the overarching goals of vendor selection. The 

criteria deemed necessary to consider a vendor included the vendor’s ability to support a multi-facility 

EHR with an integrated physician EMR and to develop data exchange capabilities between participating 

facilities. 

After establishing these fundamental criteria, the ITN organizers solicited input from interested facilities 

to develop a vendor RFP. Departments such 

as lab, pharmacy, and nursing were given the 

opportunity to contribute to the RFP by 

outlining departmental goals in order to 

ensure that the chosen vendor system best 

met their needs. After compiling facility- and 

departmental-level input, the ITN 

incorporated the results into their vendor selection tool and RFP.

In addition to the release of the vendor RFP, the ITN organizers took several other steps to establish a 

functioning network, the intent of which was to anticipate and address future challenges:

• Development of a legal approach. Facilities interested in participating in an EHR collaborative 

agreed upon a 501(e) framework and developed a number of documents related to tax-exempt 

status. This work eventually led to the establishment of the ITN.

• Establishment of staffing levels. The ITN organizers estimated FTEs at each facility for the 

implementation by examining staffing levels at hospitals of similar size and revenues. The ITN 

ultimately recommended a combined 7 to 8 FTEs across the four participating facilities for a 

successful implementation. 

• Establishment of workgroups. The ITN launched and facilitated a variety of monthly workgroups 

that provided a “collaborative stimulus” to guide the activities of the implementation. The 

workgroups included a monthly “Board of CEOs” to guide the overall strategic approach and engage 

hospital leadership; a “Project Workgroup” that identifies needs (for instance, how to approach 

order verification when facilities lack coverage by a certified pharmacist); a “Medical Records 

Workgroup” that works to define policies (for instance, a policy to describe the transition from a 

“hybrid” record to a complete electronic record); a “Quality Improvement Workgroup,” a “Patient 

Care Workgroup,” and a “Patient Accounting Workgroup.”

“The biggest piece of this is collaboration.” The sharedstaffing model enabled by the choice of a single vendor by 

multiple facilities saved costs and increased the feasibility 

of the HIT implementation project in a small hospital 

environment.
FLEX CAH HIT Evaluation- Final Report

 90

• Development of a vendor selection tool. The ITN organizers developed a vendor selection 

scorecard, the “Hospital Information System Evaluation Methodology.” The ITN organizers 

established a recommended weight for the clinical, financial, ancillary, and physician EMR 

applications of an EHR system, as well as an “other” category to include factors such as cost. While 

the ITN recommended that facilities take into account the importance of clinical applications, the 

final scorecard weighted equally clinical and financial indicators, because, according to the ITN CIO 

“everyone could ultimately ascribe whatever they wanted, and the CFOs in the room felt that 

[clinical and financial] should be equally ranked.” The ITN organizers then distributed the Evaluation 

Methodology to each facility to assist them in choosing a vendor. 

Involvement of Clinical Stakeholders in Planning

The ITN CIO’s role was to facilitate a process where each facility identified its preferred vendor, and then

to foster collaboration among facilities if there was enough “energy” generated around any individual 

vendor. The ITN organizers invited the participation of all area facilities in the vendor selection process, 

and established a workgroup called the “Shared HIS Task Force.” This Task Force began with nine 

facilities, each with several staff deeply engaged in the criteria development and vendor selection 

process. Staff from each facility provided input to develop a set of evaluation questions to be asked of a 

number of vendors invited to showcase their systems at a “vendor fair.” Additionally, facilities were 

encouraged to submit individual staff questions and address facility priorities that were not adequately 

addressed in the collaborative process of the Shared HIS Task Force or other planning workgroups. 

In 2007, after the HRSA Flex CAH HIT grant was announced, four of the original nine facilities involved in 

the vendor selection process agreed to participate in the shared EHR environment. The Flex CAH HIT 

grant project, according to the ITN CIO, seemed like an ideal match, given the detailed planning and 

vendor selection process the ITN organizers had undertaken prior to its release. In October 2007, the 

ITN incorporated and received its 501(e) designation. 
FLEX CAH HIT Evaluation- Final Report

 91

Anatomy of a Vendor Fair

The ITN developed a variety of tools to assess potential vendors, including a “vendor fair” to formally present 

vendor products to interested facility staff and stakeholders. The ITN employed “Media Site,” to record and capture 

the presentations to allow facility staff unable to attend the fair to get a sense of vendor capabilities from their 

home facilities. Based on the criteria developed by the Shared HIS Task Force during initial planning for the 

implementation (ability to support a multi-facility EHR and develop data exchange capabilities between facilities), 

five vendors were invited to present at the vendor fair. Drawing from facility priorities and common themes 

compiled from the vendor scorecard, the Shared HIS Task Force developed a script organized by three 

categories: clinical-ancillary, financial-administrative, and physician-patient EMR. For each category, the Task 

Force developed a 90-minute script in order to demonstrate how each of the five vendor systems might handle a 

patient encounter from start to finish. The nine facilities involved in the vendor fair brought a variety of end-users to 

rank the vendor presentations using a scale that addressed each of the three categories in detail. Additionally, 

each vendor also set up a booth to allow end-users to “test drive” the system after the formal presentation and 

ranking. 

Organizing the vendor fair involved a detailed process of invitation and follow-up. The organizers articulated the 

need to use the vendor fair as a vehicle for developing staff buy-in to the implementation by encouraging staff 

attendance. Staff representation from several of the involved facilities was “really impressive,” and according to 

the ITN CIO, some facilities had as many as 30 staff people attend. However, other facilities sent only a few 

representatives to the vendor fair. These differences in attendance projected the respective level of involvement in 

and commitment to the implementation as the project progressed.

Vendor Selection

As the initial five vendor options were narrowed by the determinations of end-user evaluations from the 

vendor fair, the two finalist vendors returned for a full day question-and-answer session. End-users also 

had the opportunity to participate in site visits at other locations to see the vendor system up and 

running. Four facilities ultimately agreed to collaborate--to use a shared platform, shared set of 

efficiencies and staff, and shared training method across sites--if one finalist was chosen as the vendor. 

Three facilities, in contrast, agreed to these conditions with the other finalist as the vendor of choice. 

Because one vendor was preferred by an additional facility, it was the final choice for the 

implementation because having a greater number of participating facilities maximized project benefits 

while reducing costs. 

Vendor Perceptions

“Given how the project has been implemented, [the selected vendor] does a lot of things well, 

technology-wise. With some other vendors, it may have been a little more onerous,” reported the ITN IT 

Manager. ITN respondents agreed that while there were pros and cons to the vendor implementation, 

participating facilities would likely have chosen the same vendor if given the opportunity again, 

particularly in the context of comparable price tags for other vendor systems. 

One of the participating facilities, which had worked with the vendor’s applications prior to the multifacility implementation, reported that the vendor support was good. However, other ITN respondents 

surmised that because the vendor had never before conducted a multi-facility implementation, it failed 

to devote needed resources to strengthening the collaborative structure. The vendor viewed the project 

as four independent implementations, according to the ITN CIO and consequently, he was obliged to “do 

some real fighting at the ‘upper levels’ [of the vendor management] to get some of the resources for the 

specific needs of the [collaborative] project.” Many of the issues with the vendor were resolved, 

according to the ITN CIO, when he requested the vendor implementation team return to Wisconsin to 

address several facility complaints. He reported the situation was “triaged pretty well.” 
FLEX CAH HIT Evaluation- Final Report

 92

Additionally, end-users expressed some frustration about the nature of “departmentalization” of the 

implementation. Hospital staff reported that individual facilities were burdened with determining the 

way in which departments should communicate information within the EHR, a challenging prospect 

given inexperienced staff. At the outset of the project, it was difficult to impose a higher structure over 

vendor workflows, according to ITN respondents. As the implementation progressed, the ITN reportedly 

developed the expertise and familiarity with each facility and its processes to better address workflow 

compartmentalization issues. 

D. The EHR System

Applications of the implemented system include:

• A real-time Web-based EMR Portal application which offers “sortable” lab results, transcriptions, 

medication information, vital signs, patient care plans, historical documentation, and is primarily 

used for viewing patient information. Clinicians have the ability to see which end-users have viewed 

patient data, which orders have been placed and what resulted, as well as who placed the order and 

when it was placed. Documentation is entered via transcription in the Medical Records or Radiology 

departments, as well as the Patient Care Documentation system. 

• A report function which includes cumulative lab results, a reporting function for medications, and an 

electronic signature function. 

• A Patient Care documentation application which enables nurses to input vital signs and other data, 

as well as design and implement care plans to best meet patient needs. 

• An application that allows facilities to add patient information at intake, including diagnoses and 

drug allergies, which will cross over with 

discharge information. 

Tomah Memorial Hospital was the most 

advanced in terms of EMR at the start of the 

Flex grant process, as the facility had installed 

an electronic record for its Lab, Radiology, 

Pharmacy, order entry and inpatient nursing 

departments prior to participating in the 

project (self-identified as HIMSS21

Boscobel Area Health Care began the 

implementation with all electronic ancillaries and 

EMR 

Adoption Scale level of 3 – 4). The 

implementation required clinicians at Tomah to 

access new admissions from the EMR portal 

(called Clinical View) rather than a paper chart. 

In addition, it moved Tomah from an older 

version of Nurse Documentation to a newer, 

Java-based version. Additionally, Tomah is 

scheduled to implement Computerized Order 

Entry (CPOE) to supplement its wrist 

verification barcode system. 

 

21 HIMSS Analytics. (2009). U.S. EMR Adoption Model. Retrieved from http://www.himssanalytics.org/stagesGraph.html

HIMSS Analytics EMR Adoption Model is used to evaluate 

the progress and impact of EMR systems in acute care 

delivery environments, where:

• Stage Zero: All three ancillaries not installed

• Stage One: Ancillaries – Lab, Radiology, Pharmacy –

All installed

• Stage Two: Clinical Data Repository, Controlled 

Medical Vocabulary, Clinical Dec, may have 

Document Imaging

• Stage Three: Clinical documentation (flow sheets), 

CDSS (error checking), PACS available outside 

Radiology

• Stage Four: CPOE, CDSS (clinical protocols)

• Stage Five: Closed loop medication administration

• Stage Six: Physician documentation (structured 

templates), full CDSS (variance & compliance, full RPACS

• Stage Seven: Medical record fully electronic; HCO 

able to contribute CCD as byproduct of EMR; Data 

warehousing in use
FLEX CAH HIT Evaluation- Final Report

 93

order entry in place (HIMSS EMR rating 1 – 2). Boscobel has since added the functionality of Document 

Imaging (scanning) to its EMR in Medical Records and at Registration. Boscobel has also implemented the 

EMR portal, which allows clinicians to electronically access lab results, transcriptions, medication information, 

and scanned images. The facility is scheduled to implement a nurse documentation and electronic signature 

function. Currently, Boscobel lacks a Picture Archiving and Communications System (PACS). According to the 

ITN CIO, the cost of a PACS rivals the cost of an entire EMR, and because of the current economic 

environment, the facility will likely not be able to purchase a PACS in the near future. 

Memorial Hospital of Lafayette County, the smallest participating facility, began the implementation 

without electronic Lab, Radiology or Pharmacy applications (HIMSS 0). During the project period they 

implemented financial and administrative applications, each ancillary, order entry, scheduling, patient 

care documentation, eMAR, Surgery Management, and the EMR portal. Memorial Hospital is also slated 

to implement an Exit Care discharge instruction application as part of the implementation. Memorial 

currently lacks a PACS system.

St. Joseph’s Community Health Services did not have electronic lab or pharmacy systems prior to the 

Flex grant (HIMSS 0 – 1); these were installed along with financial and administrative applications, RIS, 

OR Management, scheduling, patient care documentation, the EMR portal, and the Exit Care discharge 

instruction application. Additionally, the facility’s clinics implemented a physician practice management 

system. St. Joseph’s had a pre-existing relationship with the University of Wisconsin, through which it is 

provided PACS functionality. Prior to the implementation, St. Joseph’s bypassed Radiology by sending 

films directly to the University for reading. ITN and St. Joseph’s are currently working on an interface to 

bring Radiology results from the UW system back into the EMR portal.

System Testing

During the “file build” process, the vendor’s applications were tested by end-users and refined to best 

meet the needs at each facility before the go-live. The timeframe of the test period for each application 

was generally several weeks, but variable and dependent on the application and facility. Staff involved in 

system testing included 100 percent of planned end-users. The vendor relied on the training and test 

period to identify gaps in system functionality. 

Technical Support

Technical support concerns were heaviest during the transition period from system testing to go-live. 

Not all the facilities were equipped with internal IT support; therefore, these facilities either used a 

contractor or passed issues to the shared ITN Helpdesk as needed. The Helpdesk maintains a list of 

hundreds of request tickets, including troubleshooting end-user questions about computer, printer, 

scanner, and security settings, as well as software glitches or configuration issues. While feedback from 

an end-user satisfaction survey evaluating IT support was complimentary, ITN IT support staff 

emphasized the need for additional resources in order to increase responsiveness and engage end-users 

in additional collaborative efforts. For example, “a big workflow problem” occurred as a result of a 

clinical staff person sharing a system password with a contract physician who could not access the 

system because IT support was not available “around the clock” to deal with the issue as it played out. 

E. Staff Training

End-users were trained to use the EMR applications following a “train the trainer” model. First, vendor 

representatives trained ITN staff in November 2008, who then took full responsibility for training facility 

staff. ITN staff identified and recruited a number of super-users from nursing, as well as one ancillary 

staff person from each shift, who underwent specialized training before they, in turn, held trainings with 
FLEX CAH HIT Evaluation- Final Report

 94

all other staff. Those end-users identified as “super-users” also met weekly to suggest changes and 

provide feedback as Patient Care Assessments were being developed prior to the implementation, or 

“go-live” period. Super users involved in the Patient Care Workgroup built files after the initial trainings, 

and continued to help staff throughout the go-live period.

In all, 100 percent of clinical staff at the four facilities were trained, including all nursing, dietary, 

occupational therapy, physical therapy, and OR staff. Depending on the facility, end users from some 

departments (i.e., ED, OB, OR) were not trained because there was a delay in implementing various 

applications. 

RNs underwent four-hours of specialized training and ancillary end-users two hours of specialized 

training, in addition to an eight-hour training required of all clinical staff. Physicians received one-on-one 

training of variable length. Training time was flexible and dependent on the availability of staff and 

patient admit volume at each facility. Re-trainings and “touch-ups” were planned on an as-needed basis, 

and were necessitated in some but not all areas of the implementation. For instance, continued training 

on the clinical view application was reported to be necessary, as well as ongoing training for new 

employees.

Satisfaction with Training

While pre- and post- tests of staff attitudes of HIT were not administered to assess staff attitudes and 

knowledge of the applications, St. Joseph’s Community Health Services assessed staff IT savvy, and 

provided computer-based trainings for staff deemed less adept at computer use. Tomah Memorial 

Hospital also rolled out a quarterly exercise to encourage staff to practice their IT skills, which ITN staff 

planned to share with other facilities. Staff can exercise their skills using online resources, such as an 

online training library. While each clinical staff person participated in ITN-facilitated trainings, the 

expectation was that these staff spend additional time prior to go-live to practice their skills in the 

training library.

Despite these efforts to acclimate end-users to functional use of the applications, ITN respondents 

reported end-user concerns that the initial trainings were too broad, simply covering too much 

information to absorb in one session. In several cases, staff felt overwhelmed or frustrated because 

even after training, they judged themselves unequipped to handle the dramatic changes to workflow set 

in motion by the installation of an electronic system. Nevertheless, ITN respondents believed the 

trainings were effective overall, and increasingly so as the training cycle progressed. 

F. Governance, Policies and Procedures

At the organizational level, the ITN applied for and received 501(c)3 status, developed numerous legal 

documents and policies pertaining to Form 990 filings with the IRS, and established an effective 

governance structure. The ITN also engaged in ongoing strategic planning and balanced scorecard 

processes that utilize data from various tools to ensure that the project “stays on track.” As 

circumstances change, the ITN recognizes the need to change its documentation of policies and 

procedures. The ITN has provided model policies for the four participating facilities that may be used in 

the absence of an applicable facility policy. ITN staff have completed drafts of a number of policies 

crafted for the CAH HIT network, however; several practical changes have been made but not yet 

articulated on paper as the implementation has progressed. 

Security

The ITN implemented various technologies to protect the integrity and availability of the ITN Network, 

and developed detailed HIPAA-related policies and procedures that were found to be effective by an 
FLEX CAH HIT Evaluation- Final Report

 95

independent auditing firm in their Type 1 Statement on Auditing Standards (SAS) #70 audit.22

There are two data centers, one in Madison, and a smaller, backup data center at the RWHC ITN building 

in Sauk City, Wisconsin. The Madison data center is on two power grids to eliminate the risk of going 

down. The pair of data centers enables the provision of a redundant environment that is highly available 

and positively affects quality. ITN staff monitor “uptime,” “downtime,” connection speeds, and overall 

facility usage, and receive notifications in case of downtime with the goal to eliminate such periods. This 

type of redundant capability would be impossible with a single-facility implementation. 

Policies 

and procedures at the forefront of consideration included disaster recovery and contingency planning. 

For instance, the ITN is developing a risk mitigation strategy, which measures potential risks to HIT 

equipment and connections based on probability of occurrence. Security and auditing concerns will also 

be centerpieces of newly developed CAH HIT network policies and procedures. 

Data Confidentiality 

While the ITN is “strong” on physical security (for example, the user identification and biometrics 

required to enter the server storage units offer reliable protection), it is working to better define its 

remote access policy by identifying encryption and other security assessment strategies. In a rural health 

care environment, security is a challenge because end-users play a multitude of roles within an 

organization, and consequently, access must be more malleable. 

The four participating facilities expect that the ITN will be very careful with data confidentiality, 

especially as the project moves closer to implementing the health information exchange (HIE) 

component. The hospitals decided rather than using a single Master Patient Index to store patient data, 

to maintain control of their data as four individual entities. The ITN’s focus on auditing is a key 

component of confidentiality that goes hand-in-hand with appropriate user accessibility to the EHR. The 

ITN is working to facilitate a collaborative arrangement in which each facility employs a process to 

address confidentiality and potential security incidents. Presently, each hospital has intrusion-detection 

devices funded by the Rural Health Care (FCC) Pilot Program, to monitor unauthorized network 

penetrations.

Interoperability

Because the facilities don’t have HIE functionality, related policies are not yet applicable and will not be 

until the project is closer to the HIE go-live. However, the ITN will “definitely have to consider 

interoperability” and will be reassessing all of these policies as the implementation progresses. 

G. Impact on Hospital Operations

Tracking Data Through the Continuum of Care

The Flex grant activities have had a positive impact on data tracking and end-user accountability. Before 

the system go-live, respondents reported that paper charts were routinely misplaced, or handwriting 

was illegible, whereas after the go-live, data available electronically was accessible in a centralized 

repository and information was clearly legible. Also following the go-live, patient care became linked 

with individual end-users who completed the care. Consequently, the level of documentation is 

expected to improve as end-users adapt to use of the electronic system and as problems inherent in the 

paper-based system are eliminated. 

 

22 SAS #70 is an auditing standard and indicates that a service organization has been through a thorough audit of their control 

objectives and activities, often including controls over information technology and related processes.
FLEX CAH HIT Evaluation- Final Report

 96

Clinical respondents reported satisfaction with the flow of data through the electronic system, and 

noted that ongoing improvements are being made since the go-live. Staff reported a number of benefits, 

including the ability to:

• View and compare results,

• View an entire patient encounter and historic record, and 

• Review a patient record or patient complaint with a physician over the phone (where both parties 

have access to the information).

ITN IT staff reported that it has been very rewarding to see clinicians using the system and having all 

patient information accessible at their fingertips. ITN staff believe physicians will continue to have a 

positive experience with the electronic system, especially as it eases demands when other providers are 

on-call or unavailable for in-person consultation. 

Efficiency

In terms of streamlining and automating reporting, the electronic system is expected to free up staff 

hours, thereby benefiting daily time management. However, concerns about changes in workflow were 

expressed, including the time consuming nature of writing notes longhand and later transcribing them 

into the electronic system. This was a process that was necessary for some staff, because at least one of 

the participating facilities had yet to purchase an adequate number of computer portals for all clinicians 

to carry into patient rooms while conducting rounds. One nurse reported that staff satisfaction with the 

efficiency of the electronic workflow was evolving as end-users develop workarounds to better navigate 

their new electronic responsibilities. She reported, “There are days when I could literally tear the hair 

out of my head, but that is part of the process. We have lots of work to do – people are on the right path 

– people are buying in. But, there is still some angst, especially when it gets fast and there are lots of 

patients.” Nevertheless, end-users noted that facility struggles with workflow were perhaps a result of 

hospital design rather than the EHR system.

At the start of the project, physicians in particular “dug in deep” with the existing, paper-based system. 

They were reticent to adopt a new workflow when they believed the existing workflow was functional. 

Now, according to ITN and clinical respondents, there is complete buy-in to the electronic system, even 

from physicians who protested the loudest at the outset of the implementation. While there have been 

challenges, physicians and nurses have generally worked together to troubleshoot issues and learn the 

“ins and outs” of the applications. One clinical respondent reported that a year ago, she was unsure 

whether the project would continue because of staff hesitancy, but was presently “very happy” because 

with some “handholding” from ITN and nursing staff, physicians have “jumped on.” She explained, 

“Physicians have to see a purpose and [the electronic system] cannot affect their time management. If it 

is going to take more time and effort on their part, they are going to be more resistant.” 
FLEX CAH HIT Evaluation- Final Report

 97

H. Impact on Patient Care

Patient Safety 

The general perception of end-users is that the implementation will improve quality of care as a result of 

more complete documentation and information availability, and fewer medication errors. 

From a quality perspective, Tomah respondents reported they “are satisfied and much better off.” 

Clinical respondents reported that patient safety had already improved since the various facility go-lives. 

For example, standardization of the nomenclature has improved a lot among staff; as one end-user put 

it, “we are calling things by the same name.” Clinical respondents noted that while they had not 

necessarily realized it prior to the implementation, medication administration was unsafe without barscanning. Prior to the implementation, staff “probably didn’t recognize that this feature was so 

significant.” The medication barcoding system has been most useful in improving medication safety 

numbers. The results have been “remarkable,” according to clinical respondents, with a dramatic 

improvement already in effect for the safety of patients. The medication barcoding system “is absolutely

the right thing to do for your patients. It increases safety dramatically,” reported an end-user.

Feedback from Patients

According to end-users, it has been difficult to ascertain the impact of the implementation on patients 

because facilities have not received a necessary quantity of patient satisfaction surveys to draw 

significant conclusions. To alleviate concerns about the new electronic system, during the go-live 

process nurses explained what was happening to patients and apologized in advance for spending time 

looking at a computer at the bedside. Patients reportedly seemed satisfied with the explanations and 

conveyed the “general sense,” according to end-users, of feeling safer. Noticeable benefits to patients 

described by end-users included:

• Physicians are able to show patients data in real-time. 

• Patients are able to take their medication reports with them at the time of discharge.

• Physicians may produce radiology images on a compact disc for patients to take home with them.

• Electronic transmission of radiology allows physicians to review films before patients’ arrival at a 

facility, expediting the patient visit and eliminating the chance of films being lost during transport 

between referring facilities.

I. Evaluation

The goal of the project evaluation is to develop a solid QI foundation, enabling facilities and end-users to 

move forward with implementing more advanced HIT systems in the future, while avoiding an 

extraordinary amount of burden. The evaluation is examining four overarching measures. It is an 

ongoing process that ITN staff hope will serve as the framework for a formal QI program. The four 

measures are:

• Medication Error Rates. Medication error rates are being used as an evaluation metric because 

these data were already being collected, though whether consistently among sites is unknown. 

Medication error rates were also chosen because at least three applications were implemented 

related to patient safety: pharmacy, eMAR (which will potentially reduce communication errors 

between hospital departments), and medication bar-coding. Medication error rates in a paperbased system are reportedly “incredibly complicated to monitor,” in part because the onus to 

detect, report, and complete all the required documentation falls on point-of-care or frontline staff. 

In selecting this metric, the ITN hoped to understand how facilities capture their medication error 
FLEX CAH HIT Evaluation- Final Report

 98

rates, with the goal of bringing a more “systems approach” to monitoring error rates and detecting 

more errors following the implementation. Indeed, at St. Joseph’s Community Health Services, the 

medication error rate has increased from 0.65 to 0.79 percent post-implementation. This metric was 

pointed to as an example of heightened error rates presumably not because of more errors, but of 

better capture of the information in total. 

• Discharge Instructions. Discharge data will be monitored in hopes of improving standardization of 

patient care. The discharge application has gone live at Memorial Hospital at Lafayette County and 

at St. Joseph’s Community Health Services. The automated software will help guide staff in order to 

raise core measure scores in this area. 

• Patient Satisfaction. ITN staff decided to address the following three questions with regard to 

patient satisfaction:

1. The extent to which staff checked the patient’s ID bracelet before treating him/her;

2. The extent to which the patient felt the use of computer systems improved his/her care 

experience; and

3. The extent to which the patient felt the use of computer systems improved the safety of his/her 

care.

The ITN expected a dip in patient satisfaction as staff were trained, but also expect an improvement 

over time. The ITN is still waiting to receive pre-implementation patient satisfaction reports from 

the facilities. 

• End User Satisfaction. End user satisfaction will be measured in the form of a Web-based survey, 

the focus of which is satisfaction with the RWHC ITN program, which was also used in the Balanced 

Scorecard. 

J. Supportive Factors

• Facility Investment in Training. A noted support to a successful implementation was the time and 

energy invested in training prior to the go-live period. Clinical respondents from Tomah Memorial 

Hospital, whose Board and Administration were reportedly supportive of training, reported that “golive day went very smoothly, because the training was good, problems were anticipated, and the 

fires were put out before they began.” End-user training encompassed two eight-hour days, plus 

another four-hour and two-hour training session – a significant investment in time that “really made 

a big difference.” “You can’t say enough for the dollars spent in training,” emphasized one end-user.

• IT Support. Clinical respondents praised the constant support provided by the ITN Clinical 

Application Specialists. Facilities lacked an adequate number of IT staff or expertise to provide 

readily-accessible support onsite, and the Clinical Application Specialist and Helpdesk functionality 

were reported to be invaluable resources. As one nurse noted, “having ITN trainers on the floor to 

assist with end-user questions and provide support was essential.” Another clinical respondent 

reported that having an IT person to serve as the “resident expert” for how to maintain the system 

was important to reduce fragmentation among departments and facilities, and necessary for 

success.

• Staff Buy-in. The degree of group buy-in contributed to the success of the implementation. A key 

success factor noted by respondents was end-user engagement in the vendor selection process. 

According to the ITN CIO, developing staff buy-in must begin in the early stages of planning to carry 

through the implementation. Staff need to “feel they had some say or stake in the decision upfront” 

to remain committed to the goals of the project even when encountering challenges later in the 

process. Valuing clinician engagement, in particular, early in the process, led to a smoother “go-live” 

later on in the implementation. Consensus among all types of clinical staff is important, particularly 
FLEX CAH HIT Evaluation- Final Report

 99

because changing one component or application has an effect on other departments. In addition, 

garnering support from a wide range of staff was critical when requesting technical assistance from 

the vendor. “When we need something from [the vendor], we all get together, so it’s four facilities 

asking for a change” instead of one. “That little bit of extra clout” was very helpful for leveraging 

purposes. 

• Learning from Best Practices. Tomah Memorial Hospital, the first facility to implement the system, 

helped the remaining three facilities set up patient care and intervention plans. Access to templates 

from past go-lives was very beneficial to the success of future implementation efforts, according to 

clinical respondents. Staff from the other three facilities visited Tomah Memorial Hospital to 

practice using Tomah’s forms. End-users found the ability to ask questions and discuss concerns with 

their colleagues at other participating facilities to be reassuring, and this process improved staff buyin across the spectrum.

• Sharing Lessons Learned. The ITN CIO reported that the monthly HRSA all-grantee conference calls 

were useful for identifying common concerns across projects and enabled the project directors to 

reach out to each other to learn from shared experiences.
FLEX CAH HIT Evaluation- Final Report

 100

K. Challenges and Solutions

Training

Participating CAHs experienced various challenges pertaining to end-user training, such as sudden high 

patient census during planned training time prior to or immediately following go-live, and trouble 

scheduling re-trainings with end-users who needed a “refresher.” Other barriers encountered in training

clinical staff included:

• Super users, those staff handpicked and trained by ITN trainers, encountered some difficulty training 

newly hired nurses. One super user said that there was a need to integrate traditional nursing skills 

with multi-tasking skills to efficiently enter information into the EHR. This respondent reported 

struggling to teach some of the less experienced nurses how to properly use the applications. 

• Clinical respondents noted that a particular challenge was keeping contract physicians, who used 

the EHR only periodically, up to date. Additionally, nurses, who often worked shifts in several 

departments, were more apt to forget components of the system that differed between 

departments. Therefore, routinely engaging end-users with the EHR was important to preventing 

information loss.

• Access to training materials was a significant challenge, given limited staff time to use the training 

library as a resource to prepare for the go-live.

Physician Resistance to Perceived Inefficient Workflow

Clinical and ITN respondents reported that physician perceptions of the EHR led to resistance in 

adopting the electronic system. Physicians tended to be more hesitant to use the EHR because some 

elements were not as efficient as the paper-based system. For instance, template-driven documentation 

required in the electronic system, while more thorough and exportable, also takes longer. It would be 

much faster for a physician to jot notes down than to log into an application, especially order entry or 

CPOE. These slowdowns in large part affected physician workflow, which reduced their buy-in. ITN 

respondents reported that over time, physician acceptance is expected to gradually increase to a level of 

consistent buy-in. 

Staff Turnover

Staff turnover is an especially profound issue in small hospital environments, and this held true for the 

ITN project. In fact, respondents reported staff turnover to be “a nightmare” and cited several examples 

of hospital super users who were very involved in implementation planning and strategy leaving the 

CAH. The ITN CIO reported that he was “very worried” about staff leaving the ITN. “It’s critical for the 

ITN to keep its people, because the industry is such that expertise is built over years and cannot be 

learned overnight.” The CIO further noted the critical role of managers to balance staff sustainability 

with the achievement of project goals and objectives.
FLEX CAH HIT Evaluation- Final Report

 101

L. Lessons Learned

Foster a Shared Staffing Model

Project management at the ITN repeatedly emphasized the benefits of using a shared staffing model to 

approach the implementation. Collaboration among facilities was essential in learning from others’ 

experiences, prioritizing resources across the project, garnering across-the-board support for vendor 

selection and file-building, and getting input from the workgroups with representatives from all four 

participating facilities.

Mitigate the Risk of Staff Turnover

The ITN tries to hire the “kinds of people who are energized by challenges and want to make a 

difference.” Respondents reported that RWHC ITN staff are very mission-driven, and that the ability of 

ITN project management to inspire facility-level staff has likely hindered end-user turnover as a result of 

the EHR implementation. Keeping careful documentation and detailed work plans was cited as a method 

to protect against turnover-related inefficiencies. 

Invest in Planning Phases and Staff Preparation for Change

Respondents agreed that it was of critical 

importance to invest heavily in planning for 

the implementation, and to prepare facility 

staff from the outset that there would be 

several months of stress immediately prior to 

and following the go-live period. A mentality 

of preparedness for challenges was thought 

to better enable end-users to effectively respond to unexpected issues. Painstaking file building and 

template design during the planning stages reap significant benefits, as well.

Invest in Training and Go-Live Support

A number of lessons learned were offered in regard to training and go-live support. ITN trainers 

recommended leaving ample time—at least two or three weeks—between scheduled trainings and golive day, in order to give end-users a chance to absorb and practice their newly learned skills. ITN 

trainers also recommended putting a hold on end-user trainings until all applications were installed and 

available to reduce the propensity of end-users to forget elements of the trainings during the lag time. 

Other lessons learned about training included:

• The benefits of site visits to facilities that 

had already implemented EMR to 

supplement “spoon fed” trainings 

consisting only of PowerPoint screen shots 

from the applications. Allowing as many 

end-users as possible the opportunity to 

get practical experience was advised. 

Clinical respondents also encouraged the 

design of “cheat sheets,” or tools with screen shots explaining how to navigate the applications 

step-by-step.

• Clinical staff suggested that read-along manuals, which would allow them to focus their attention on 

the training content rather than note-taking, would be helpful during the trainings. Also suggested 

“If you are the primary person in your organization, you 

feel very isolated and alone. In [the shared staffing model], 

because it was a collaborative effort between four facilities 

and the ITN, this was not the case. It is nice to bounce 

things off of each other and understand that ‘this is not 

unique to you, this is part of the process.’”

“Don’t just have the department director explain the 

implementation plan to staff, but have forums and group 

discussions, and be honest about the difficulty of the 

transition [from paper to an electronic system]. Change is 

hard! You have to plan for change. Flexibility – the 

willingness to change frequently, and on the spot--is key.”
FLEX CAH HIT Evaluation- Final Report

 102

as potentially useful was the provision of an EHR workflow outline of “what is done and how to do 

it” with a new patient.

• Fully staff facilities on the day of, and for several weeks following, the go-live. Provide around-theclock IT support in case the need arises. 

• Ensure that support is available to end-users aside from full-time staff. For instance, a trainer cannot 

be a full-time nurse who has responsibilities on the floor if a trainee has questions. Hospital 

administration must provide the requisite FTEs for a trainer who is fully accessible and dedicated to 

that task. 

Continue “Tweaking” and Improving the System

Since changes made to one application could quickly create problems with successive functions, the 

grantee benefited by having mechanisms in place to serve as checks that all the systems were working 

properly. It was considered important to make modifications when needed and to develop an audit 

process to ensure that “everything you think is there is really there.” To maintain a level of awareness 

about system glitches or concerns occurring at the facility-level, end-users keep a notebook to 

document concerns, and make these available to the ITN and vendor. “Communication is absolutely 

key,” as is listening and responding to staff concerns in a timely manner.

Strong Project Management and Staffing

Project leadership and standardization directed by the ITN was a success factor that streamlined the 

implementation. The ITN served several important functions, such as monitoring care plans to see that 

patient charts were accurate and complete, and providing ongoing informatics assistance and support to 

participating facilities. The ITN CIO recommends the inclusion of a CIO or a clinical-minded IT person, at 

the “administration table.” He explained that oftentimes, as the director of the IT department, the 

hospital CFO has a strong influence in developing the strategic plan, which consequently may be very 

financially driven. The expertise of the ITN in both health and IT enabled it to fill in this clinical gap; 

however, oftentimes it is difficult to find someone to fill the CIO role in a small hospital environment.

M. Sustainability

Collaboration and Shared Staffing Impact on Sustainability

During the vendor selection process, the ITN created a cost-modeling exercise to project the price to 

implement a single EHR vendor at the participating facilities, compared to implementing several unique 

vendors at one or more facilities. The model identified every cost of the implementation and essentially 

compared the relative cost for facilities to buy their own servers, vendor support, and additional staff 

time to effectively support an electronic 

environment and a number of other 

telecommunications issues. Then, it 

compared these individual facility costs to 

those generated in a collaborative 

environment, and found a savings of twenty 

percent, due to the collaborative nature of 

the implementation, shared infrastructure, 

and negotiated discounts among the facilities. The cost savings were dependent on the number of 

participating facilities, so as more facilities were added, savings increased. As a result of this process, the 

ITN has “hard evidence” that implementing an EHR in a collaborative environment results in significant 

cost-savings, of which long-term sustainability is a key component. 

The ITN offers support services otherwise inaccessible to 

the CAHs and other small facilities participating in the Flex 

grant activities. The ability to take advantage of clinical and 

financial strategic planning, database expertise, and IT 

and Wide Area Network specialists provided by the ITN 

will play an important role in maintaining the viability of HIT 

implementation over time.
FLEX CAH HIT Evaluation- Final Report

 103

From a practical perspective, the CAH HIT project is sustainable because all the facilities signed a binding 

contract. Additionally, each facility organization committed to five years of participation. In other 

examples of this around the country, ITN research determined that facilities tend not to drop out of 

collaborative HIT implementation projects after five years. The ITN CIO stated a belief that the 

sustainability for this implementation is indefinite, and depends only on the ability of participating 

facilities to pay their monthly ITN service fees. 

Impediments to Sustainability

The cost of supporting HIT goes beyond the initial capital investment required to customize vendor 

applications and install hardware. Particularly at small facilities, the cost of supporting an electronic 

system is generally much more burdensome. According to the ITN CIO, the return on investment (ROI) of 

an electronic system is almost always predicated on volume, where “bigger volume gets the bigger ROI.” 

In illustrating this point, he described the workflow at a small facility, where a single transcriptionist may 

keep his or her job, even after an HIT implementation, while at a large facility with a dedicated 

transcriptionist department, a number of staff may be eliminated and consequently, more significant 

cost savings incurred. “In a smaller hospital setting, there isn’t the volume, ROI, or level of IT expertise 

[whereas in a larger facility], there is essentially a support person for every application. A small hospital 

simply cannot afford that size of an IT department. This is an issue the ITN has attempted to mitigate by 

employing the collaborative model.” The ITN CIO emphasized that without the expertise and support 

services offered by the ITN, the facilities participating in the activities of the Flex grant would simply not 

have access to critical CIO strategic planning skills, as well as database expertise, Wide Area Network 

(WAN) specialists, and clinical, financial, and other IT expertise. “These are skills that a small hospital 

couldn’t support internally, but by collaborating, there is more of that potential.” 

Complimentary Funding or Support for the HIT Implementation

Rural Health Care Pilot Program 

This grant program, initiated by the Federal Communications Commission (FCC), is intended to facilitate 

the creation of a nationwide broadband network dedicated to health care by providing access to 

broadband in rural and urban areas. Because the cost of receiving broadband is significantly higher for 

rural areas (a 20 MG broadband connection may cost $3,000 monthly for the participating facilities), the 

RWHC ITN was awarded $1.5 million under the Rural Health Care Pilot Program. The FCC grant also 

funds a variety of equipment related to WAN management that has been incorporated along with the 

hardware paid for by the Flex CAH HIT grant. Following the award, the ITN worked with the FCC to revise 

the plan to meet the specific needs of the CAH HIT project, based on the more intensive needs of the 

four facilities that decided to participate.

Other Funding Requests

The ITN has received a Congressional Appropriation, has applied for a USDA DLT loan/grant, and has put 

in a request for State funding allocated from Federal stimulus dollars, to be spent to expand the CAH HIT 

collaborative environment from four to six, or even eight, facilities. The ITN also has submitted requests 

for interface support, additional support for WAN costs, and a shared PACS environment beyond the 

scope of the CAH HIT network participants. 

Plans for Expansion

Four additional facilities are currently considering joining the ITN collaboration, so there may be 

expansion through additional facility participation.
FLEX CAH HIT Evaluation- Final Report

 104

Participating hospitals are planning to expand HIT functionality; one noted that “it is always a building 

process. No one ever finishes.” Tomah and the other hospitals are still working toward CPOE 

functionality. The Project Workgroup has an ongoing agenda item of collaborative technology 

implementation (“what can we do together, what should we be looking at...”). The ITN CIO emphasizes 

that only a handful of facilities that have achieved an HIMSS level 7, and the CAH HIT facilities are only at 

a level 3 or 4, leaving room to expand HIT functionality/capability in the future. 

Now that the facilities have a solid foundation, the ITN CIO is available to meet with any facility to the 

extent that it is interested in developing a longer-term strategic process. The ITN CIO states, “A lot of 

people are now waiting to see what the definitions of the stimulus Medicare incentives are, because you 

want to take advantage of that opportunity, so you need to understand what qualifies as a certified EHR 

cost, which is very important for CAHs.” When the definitions of the incentives are released, the ITN CIO 

will recommend that he meet with the CEOs to begin sketching some strategic plans for the next few 

years. “As long as they understand their continuum – the strategy that they’re on – and they’re adding 

to that in a consistent way, then they are going to be fine. The problem becomes when people don’t 

have that long-term strategy, and they’re doing things piece-meal, and they don’t really know where 

they’re going to.”

N. Value-Based ROI23

1. Clinician commitment

RWHC respondents were asked to rate user clinician commitment before and after implementation of 

Flex CAH HIT grant activities along the Likert scale shown below. 

Both the baseline measure and implemented system measure were noted at Very Committed:

User training and support

RWHC respondents reported many methods for user training and support, as well as methods for 

assessing user satisfaction with trainings. Post-training surveys for the e-MAR implementation indicated 

that the training provided was seen as effective, useful and helpful, but that there was interest in more 

time being devoted to expand the training and support programs. Respondents also noted that those 

end users who spent time practicing were the most successful in utilizing the system appropriately. 

Several hospital representatives noted generally that training and support are done well when there is a 

new implementation, but that ongoing support (i.e., for upgrades, training new hires) is troublesome. 

Additionally, a particular issue raised repeatedly by hospital representatives was that the quantity (i.e., 

time spent) and quality (i.e., detailed documentation) of vendor-supported training were below 

expectations. 

In terms of support and training beyond what the vendor, HMS, provided and that of individual facility’s 

efforts, several RWHC respondents noted that the RWHC Information Technology Network (ITN) made a 

 

23 Refer to Appendix C for a detailed description of each variable used to calculate the value-based ROI.
FLEX CAH HIT Evaluation- Final Report

 105

significant difference in the success of the Flex CAH HIT implementation. ITN staff were able to support 

CAH staff during planning and implementation and serve (as one respondent put it) as translators from 

the vendor into everyday life. ITN’s role allows for sharing knowledge between facilities through 

workgroups and in collecting information related to a problem, then translating it to the vendor so it can 

be resolved effectively.

User feedback

RWHC relies on a variety of committees and processes to gather and consider general and ongoing user 

feedback. For example, for day-to-day end user feedback at Tomah Hospital, notebooks are left on 

every nursing cart, so users can write down suggestions. Designated staff address these suggestions 

promptly, and the nurses feel heard. Other hospitals rely on weekly educational sessions that are 

available for providers, as well as surveys and learning portals to train users and identify gaps in training 

and support. As referenced above, post-training surveys for the e-MAR implementation were 

implemented and responses indicated that users found the training to be effective, useful and helpful. 

User feedback has been part of the Wisconsin Flex CAH HIT implementation from the outset – a group of 

clinicians was involved in the initial vendor selection processes and discussions. To feed into those 

discussions, information was gathered from super users and department directors about processes that 

needed improvement. 

Implications

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported the 

same level of user-clinician commitment for both points in time – Very Committed. 

RWHC hospitals reported significant commitment throughout the Flex CAH HIT planning and 

implementation efforts. All hospitals noted that commitment to the project began first with the 

administrative staff, with other staff (clinicians, management) following their lead. Some sites reported 

a critical staff member—a champion-- who greatly helped planning and implementation efforts along.

This commitment was identified as vital to implementation success, as the hospitals reported struggling 

to overcome significant challenges. For example, one hospital reported that when the implementation 

began, some users could not even turn on a computer. Another challenge reported was a lack of time 

by people who were identified as super users, which then caused significant problems – it was shared 

that “people had full time jobs, and then were being asked to learn how to use computers, e-mail, 

SharePoint, and all the HMS applications, so it was a real challenge.” The high level of staff commitment 

to the project and responsiveness to addressing problems were factors in addressing and meeting these 

workflow changes and challenges.

2. Information availability

Within the context of information affected by Flex CAH HIT grant activities, RWHC respondents were 

asked to rate how available information is, when and where it is needed, along the Likert scale shown 

below. 

Baseline (orange) measure versus implemented system (green) measure:
FLEX CAH HIT Evaluation- Final Report

 106

Implications 

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported 

greater information availability post-implementation – moving responses from a pre-grant level of 

Somewhat Available to Very Available. 

RWHC respondents reported that prior to implementation, information was incomplete (particularly in 

charting) and dispersed, making information difficult to manage. Also, care plans were difficult to follow 

completely when operating in the paper-based system, as these protocols needed to be remembered to 

be followed entirely. In the new, electronic system, respondents noted quicker access to lab orders and 

results, greatly decreasing lag time. Also, care plans were more completely followed as the 

implemented system prompts for missed vaccinations, referrals, and other documentation. Finally, 

respondents noted that electronic access to the chart has greatly diminished “fighting” over access to 

the paper chart.

3. System usability

For the HIT system(s) implemented as part of the Flex CAH HIT project, RWHC respondents were asked 

to rate system usability prior to and post system implementation along the Likert scale shown below. 

Baseline (orange) measure versus implemented system (green) measure:

User perceptions of ease of use

RWHC respondents noted several user perceptions of the implemented system in terms of ease of use. 

CAH staff noted the following improvements upon system implementation: viewable information, 

without needing to flip through the paper chart or needing to track down information; easier access to 

previous encounter information; compliance on information date/time/authorship stamp is largely 

alleviated; and legibility of documentation.

Some challenges were also noted in using the implemented system. Respondents noted the paperbased system was easy to use for patient interviewing, but that it also often ended in incomplete 

interviewing, so the implemented system provided for more complete documentation. However, this 

more thorough documentation has called attention to the actual time it takes to perform appropriate 

documentation. 

System use

As discussed above, the implemented system has been reported as very useful in facilitating more 

complete adherence to patient care plans, and utilization of care plans throughout the patient stay. 

Hospital staff reported preferring scanned information as opposed to pulling records, particularly in 

finding the right information within a record.

Respondents also reported that although increased information can be helpful, it can also pose 

challenges in some situations. Auto-generated reminders were generally seen as helpful, but also 

sometimes bypassed, and Memorial Hospital delayed the auto faxing of lab results because providers 
FLEX CAH HIT Evaluation- Final Report

 107

were getting more information than they wanted. Also, with the implemented system, staff reported 

that forms are not always completed online because staff forget the form; some forms are filled out that 

do not have to be. Sometimes workload affects which forms are completed or not.

Implications 

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported 

significant system usability improvement from Neutral for pre-grant systems to Very Usable for the 

implemented system. As detailed in the sections above, RWHC respondents generally found the 

implemented system easy to use and that system elements (for the most part) were used effectively as 

intended. The major area where impacts were noted for both ease of use and use as intended was in 

documentation – that the implemented system aided in gathering and finding appropriate, complete 

documentation, but that this thorough documentation took longer than it had with the paper-based 

system. One respondent noted “appropriate documentation isn’t always easy to do, whether on paper 

or electronically.”

4. System efficiency

For the HIT system(s) implemented as part of the Flex CAH HIT project, respondents were asked to rate 

system efficiency prior to and post implementation along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported 

greater system efficiency post-implementation – moving responses from a pre-grant level of Not Very 

Efficient to Somewhat Efficient. Comments on the implemented system referenced an ongoing 

learning-curve for users of the grant. At the time of the site visit, the Wisconsin CAHs were still in the 

process of helping users effectively use the electronic system, affecting the overall efficiency users 

experienced with the system. Respondents at Memorial Hospital did note that the change process they 

underwent in planning and implementation forced staff to examine different department’s needs and 

communication paths, resulting in more effective and efficient work processes.
FLEX CAH HIT Evaluation- Final Report

 108

5. Clinical outcomes

Respondents were asked to rate their perceptions as to the pre- and post-implemented systems in 

terms of supporting positive clinical outcomes along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:

Implications 

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported 

that the implemented system significantly increased perceived support for positive clinical outcomes –

moving responses from a pre-grant level of Not Very Supportive to Very Supportive. Respondents 

shared additional comments that the implementation has had significant impact on specific outcomes 

such as: capturing and sharing information related to patient transport; more complete charting; 

catching missed immunizations and vaccinations; conducting dietary evaluations and smoking cessation 

more frequently; and reducing early, late, or missed medication administration. Also, respondents 

mentioned that the patient discharge instruction process will continue to improve, as it is standardized 

and automated based on diagnosis. Overall, staff reported a general improvement toward working 

together towards common patient care goals: “The new system made us realize that we depend on one 

another for information, and we have to speak the same language.”

6. Non-financial values

With their EMR and e-MAR systems fully implemented and functioning, Wisconsin respondents reported 

many benefits outlined in detail in the sections above in terms of information organization, availability, 

and completeness. In addition, respondents noted a greater sense of individual responsibility for 

completing information documentation, and staff seeing the value of timely charting. Staff also liked 

that multiple people can view a record at one time, without having to track down or share a single paper 

record. 

Beyond these benefits, respondents also noted that the activities of the implementation help the 

affected hospitals with their reputation, as “technology savvy is perceived as an organizational 

accolade,” and, that by going paperless, there is the benefit of using less paper and being “greener.”

Patient satisfaction

Respondents were asked to report patient satisfaction of the pre-grant and post-implementation 

systems along the Likert scale shown below.

Baseline (orange) measure versus implemented system (green) measure:
FLEX CAH HIT Evaluation- Final Report

 109

Implications 

Measured from pre-grant implementation to post-implementation, Wisconsin respondents reported 

greater system efficiency post-implementation – moving responses from a pre-grant level of Neutral to 

Very Satisfied. Respondents reported ITN facilities have collaborated on some common patient 

satisfaction surveys. These surveys include questions regarding: (1) extent to which the patient felt that 

computer systems improved the safety of their care; (2) extent to which the patient felt computer 

systems improved their care experience; and (3) extent to which staff checked patient identification 

bracelets before treatment. A specific area of improvement noted by one respondent was in patient 

medical history information – that with the paper system, patients always had to repeat their allergies, 

medications, and other information from their medical history, and would sometimes complain about 

having to do so. 

Hospital staff reported some negative impact on patient satisfaction during implementation transition 

phases, when staff are still not completely comfortable with the system. However, staff also reported 

anticipating that this dip in satisfaction would be temporary, and over time patients would experience 

better care and coordination of care as result of the EMR and e-MAR systems.
FLEX CAH HIT Evaluation- Final Report

 110

V. Recommendations from Grantees and 

Conclusion

The results of the evaluation pointed to several recommendations from grantees, particularly in terms 

of grant administration and technical support to grantees. Throughout the grant period, grantees 

experienced many challenges in simply 

administering the grant; future programs could 

be designed to limit or eliminate these 

challenges. In addition, grantees noted that 

although they often needed technical support or 

assistance, they were uncertain of how to find 

or access resources available to them. This 

theme of TA gaps and more proactive support 

to ensure HIT implementation success and 

meaningful adoption is the most significant 

lesson learned highlighted by grantees, and is 

particularly relevant given the significant 

investment and emphasis on HIT in the postARRA world of health care. Flex CAH HIT 

grantees are poised to be catalysts in their 

states for all manner of HIT projects. For 

individual facility or communities investing in 

HIT, these grantees offer key perspective and 

experience to inform similar projects. For large 

regional or statewide HIT projects funded by 

ARRA – the State HIE Cooperative Agreements, 

Regional HIT Extension Centers, Medicaid EHR 

Incentive Programs, and HIT workforce 

development initiatives – the Flex CAH HIT grantees can serve as resources to engage priority providers 

and offer guidance and perspective in the execution of these projects in a rural setting. Following this 

idea, an HIT Primer has been developed to benefit future Flex grantees but also all rural providers in 

approaching HIT implementation projects. This Primer is a toolkit designed specifically for CAHs to aid 

them in adopting an HIT-based application or system and builds upon other tools such as those 

encompassed in HRSA’s Rural Health IT Adoption Toolbox24 and the Rural Assistance Center25

 

24 More information about the Rural Health IT Adoption Toolbox developed by the Office of Rural Health Policy can be found at 

(RAC) 

http://healthit.ahrq.gov/portal/RuralHITtoolbox

25 Information guides offered by the Rural Assistance Center can be found at http://www.raconline.org/

HIT Adoption Primer

• Engagement: Evaluate organizational readiness 

- Assemble multidisciplinary implementation team

- Review and document current systems and 

processes

- Define clear goals and objectives to prepare 

CAH/CAH network for change process

- Detail an overall strategy to increase technology 

acceptance and diffusion

• Selection: Thorough organizational understanding 

and effective vendor relationships

- Review HIT needs and current capabilities to 

inform RFI/RFP

- Building the business case (i.e. estimated benefits 

and life cycle costs)

- Guide for soliciting product/vendors to suit CAH 

needs

o Support services for negotiation of appropriate 

contract terms
FLEX CAH HIT Evaluation- Final Report

 111

guides. This Primer includes modules to address many of the specific TA needs described below in more 

detail, and is intended to benefit the front-end of HIT adoption challenges: engagement and selection. 

A. Recommendations from the Grantees

Grant Administration

During the evaluation, grantees emphasized several grant administration challenges they had 

experienced, including, the limited timeframe within which to complete the project, governance issues, 

unclear evaluation expectations, and issues sustaining the activities of the grant. These challenges lead 

to the following recommendations:

Consider a longer grant funding period for future HIT planning and implementation grants. Many 

grantees stated that they needed more time to complete the project well. Many found the combination 

of planning and implementation activities expected to be covered in an 18-month period of 

performance to have been unrealistic. Grantees emphasized the need to plan early and plan well, noting 

that network partners have different levels of knowledge and comfort with technology; a thorough 

planning process helps an HIT project advance on a level with which all partners are comfortable. 

Grantees noted that planners must be “brutally honest” about existing functions and processes. One 

grantee noted that “taking an inefficient process and having it run faster is not effective.” Good HIT 

implementations should include change management, and time for these activities should be included in 

planning and in implementation timeframes. Project managers and planners must “engage early and 

engage often,” and must have a plan for the unpredictable – particularly in rural environments where 

HIT implementations are often blazing new trails with limited staff and expertise, and thus require 

additional time to do things properly to ensure project success.

Acknowledge potential governance issues and facilitate resolution. Grantees reported having widely 

varied governance structures, within which real and perceived challenges existed. Some reported a 

culture of working together as being common among smaller CAHs and/or with existing collaboratives, 

reporting that it was easier to generate interest, involvement, buy-in, and decision making. One noted, 

“Small hospitals support a culture of working together where people wear different hats [and] there’s a 

much more community-oriented environment and agreeable attitude in smaller hospitals.” However, 

for many sites, the inclusion of larger hospitals or health systems in the Flex CAH HIT network added 

additional complexity to ongoing governance challenges. Structural issues were exacerbated when 

specific governance policies were considered – such as patient compliance, patient consent, and privacy 

and security matters. Future grantees must be more fully cognizant of these potential governance 

issues. HRSA must be ready to support meaningful mitigation of such issues throughout the grant 

process.

Emphasize the importance and value of ongoing evaluation and clearly state expectations. Grantee 

evaluation intent, methods and execution also varied widely across sites. Many sites were conducting 

or planned to conduct staff (user) and/or patient satisfaction surveys. The benefits of HIT are 

theoretically and academically clear, but assessing new information systems in live health care settings, 

and in particular in rural health settings, have proven difficult. Disease outcome(s) and financial 

evaluations can take years to show outcomes, but process and value based evaluations can indicate 

successful projects (using intermediate evaluation), and projects likely to show disease-based and 

financial improvements in a longer evaluation timeframe. Trying to assess disease and/or financial 

outcomes prematurely can lead to successful projects being labeled as ineffective or unsuccessful, 

simply because not enough time has passed for these measures to be assessed. For evaluation to be 

meaningfully conducted by future Flex CAH HIT grantees, HRSA should clearly communicate
FLEX CAH HIT Evaluation- Final Report

 112

expectations and intent of internal evaluation, and provide technical assistance (evaluation plan 

examples, selecting appropriate measures, data collection methods, etc.) where possible and 

appropriate. In turn, grantees must understand the importance of evaluation and commit to make 

evaluation a central component of their project plans and execution.

Support broader sustainability planning. HIT implementation involves a lot of unknowns that can be 

costly from a financial (and other resource) perspective, and many grantees acknowledged this issue. 

Some types of sustainability may be more of an acute issue for rural hospitals – where budgets are small 

and staff and infrastructure resources are also seriously constrained. Many grantees reported at least 

some detailed plans to address sustainability – either by incorporating sustaining costs into existing 

budgets, and/or broadening the project to include new partners to spread expenses such as hosting 

fees, maintenance fees, and upgrade fees. HRSA has provided guidance to grantees in considering

different types of sustainability issues – financial and other resource constraints, operational or 

workflow challenges, and ways to mitigate these challenges. HRSA has encouraged and supported 

grantees in considering the total cost of ownership (financial and other costs) for HIT systems, rather 

than focusing on initial purchase prices. 

Technical support

Also raised by grantees throughout the evaluation were a series of technical support issues. While many 

grantees recognize many technical assistance (TA) resources do exist, knowledge of these resources and 

how to access them is extremely varied and often depends on self determination to uncover or utilize 

one-size-fits-all resources. Also raised was a sense that rural grantees have a difficult time articulating 

their needs and don’t know all the relevant questions to ask, and have limited access to resources such 

as quality broadband services. These challenges, detailed in Section X, lead to the following 

recommendations:

Expand vendor selection, procurement and implementation assistance. Grantees experienced a wide 

range of satisfaction with their selected vendors, and noted specific times and situations during the 

vendor engagement where additional help from HRSA would be extremely helpful. For vendor 

selection, grantees commented they were unsure of what and how to ask vendors meaningful questions 

tailored to the site’s circumstances. As rural hospitals, grantees noted (in many cases) serious 

inexperience in understanding and negotiating contracts and what recourse sites had during 

implementation to resolve issues. HRSA has been suggested by many grantees as ideally suited to 

actively engage with grantees to keep expectations between sites and vendors mutually reasonable, and 

to be an advocate for the collective grantees in navigating vendor interactions. Also, HRSA could more 

readily provide or point grantees to resources to develop a site’s needs assessment, RFQ/RFP 

development, vendor selection process, vendor evaluation /comparison tools and contract development 

tools.

Require key staff succession planning and adequate staff education/training. Grantees identified a 

number of workforce issues that effected project success – namely staff retention and turnover. While 

grantees acknowledged that staff shortages and recruitment challenges in a rural environment are 

inevitable, they also pointed to turnover training and knowledge retention as aspects of larger 

workforce issues that could be readily improved. Grantees need to consider what happens if there is 

turnover in a key position. This “succession planning” can help ensure that no one person solely holds 

critical information, and that key staff can be effectively replaced. Additionally, many grantees shared a 

desire to have planned/budgeted for more staff training, and certainly beyond the training provided by 

the vendor. Particularly in rural environments, vendor and marketplace estimations of time to train 

staff can be grossly underestimated. Many grantees shared a need to allow staff time to learn basic 

computer skills and combat a general fear of new technology. Grantees would like HRSA to provide
FLEX CAH HIT Evaluation- Final Report

 113

succession planning tools for key staff, and explicitly encourage more staff education with specific tools 

and estimations for such training, tailored to rural environments.

Establish a mechanism for cross-grantee collaboration, best practices and networking. Throughout the 

evaluation, grantees were appreciative of the networking and collaboration opportunities facilitated by 

HRSA, but grantees consistently wanted additional ways to learn from each other, work together and 

share information. Also, grantees requested the ability to work with prior grantees, in the event that 

additional rounds of Flex CAH HIT are funded. However this collaboration mechanism is created or 

expanded, grantees indicated that they would benefit from building a stronger consortium across rural 

care facilities.

Additional supports for successful implementation. Grantees were aware that implementation consists 

of more than just buying a product and installing it, but often noted a lack of confidence in handling 

some of the larger issues inherent in large system implementations. Ensuring a facility or group of 

facilities is ready to implement and understands the technical environment in which that 

implementation will take place is a far more complicated and critical series of tasks. Understanding 

existing/legacy technology systems and infrastructure (or lack thereof), and related functions results in a 

complex “asset map” that defines the implementation environment that any large implementation (such 

as the Flex grants) will have to work within. As mentioned previously, successful implementation also 

should involve “change management” – where existing workflows and processes are reconsidered for 

efficiency and efficacy as a new system is implemented. HRSA can support “change management” 

procedures in future grants – to not only aid in a successful system implementation, but to uncover 

dysfunctional procedures and fix them. 

B. Conclusion

The Flex Program itself helps to sustain the rural health care infrastructure, relying on the Critical Access 

Hospital as the hub of an organized system of care. To this end, the Flex Program fosters the growth of 

collaborative rural delivery systems across the continuum of care. Robust, useful and usable HIT systems 

that coordinate patient and treatment information while respecting privacy, and that enable intersetting health care information exchange, are critical to building an infrastructure that supports patients 

and leads to the best possible outcomes for them. This report describes this model effort to establish 

and sustain HIT systems in 16 rural communities. The report describes the performance of grantees as 

they designed, created, and implemented functioning CAH HIT pilot networks, and describes the 

experiences of partner organizations in this effort. In all, theirs is a remarkable story of success, one in 

which some organizations reported having been entirely paper-based—indeed, with some reporting 

that their clinicians could not even turn on or operate a computer—to moving toward an entirely 

electronic system of medical and health records accessible by providers throughout a rural region. 

Without the commitment, funding, and support of HRSA, such successes would not have occurred. This 

successful pilot program will point the way for subsequent projects to promote HIT not only in rural 

networks, but in other systems that might be inspired by and learn from the stories told here. 
Appendix A: Telephone Interview Protocols
115

FLEX CAH HIT GRANTEE QUARTERLY PHONE CALL

GRANT PROJECT DIRECTOR PROTOCOL

Grantee Name:

Reporting Quarter:

Date Conducted:

Interviewee (name and contact info):

In addition to discussing any issues that have been described and for which there is required follow-up in 

the monthly reports for the quarter, the following questions will be asked of all grantees:

1) If you are experiencing barriers to accomplishing and completing activities [refer to monthly 

reports], what are the predominant reasons for them? Please describe your plan for resolving 

them. If you do not see a clear resolution to the barriers at this time, please describe what it 

might take to resolve them. 

2) What is your biggest challenge to date and what challenges do you see on the horizon?

3) What do you feel are the staff, resource, logistical, environmental (organizational) or 

serendipitous elements which have contributed to date to the implementation of your project 

plan? How have each of these contributed? How do these differ by CAH?

4) What lessons have you learned about HIT planning and implementation? Do you think these 

lessons are transferable to other sites?

5) What are the differences between CAHs with regard to planning, implementation, acceptance, 

and (as appropriate) utilization?
116

CAH-HIT Close-Out Interview Protocol

Grantee Name:

Date Conducted:

Interviewee (name and contact info):

Activities to Date

1. What activities in the workplan for this project have been completed? What activities are 

currently underway? 

Technology 

2. What, if any, technology is still needed to meet the goals of this project? 

Utilization

3. What percentage of staff (by type) have been trained to use the HIT system? 

4. What percentage of staff is using the HIT system, by staff type? Do you have a plan to achieve 

100% use? 

Outcomes and Impact

5. How do you define success? What measures are currently in place or will be put in place to 

track success? (probe for HRSA-specific measures and other measures chosen by the site).

6. What impact, if any, has the implementation had on your patients? 

7. To what extent has the HIT implementation impacted the availability of data?

8. To what extent has implementation of the project allowed patient data to be tracked through 

the continuum of care? 

9. Has the HIT implementation helped providers to work more efficiently with other providers to 

manage the health care needs of patients? If so, how? 
117

10. Have any formal assessments been done of satisfaction with the project implementation among 

medical staff, nursing staff, other types of staff, and patients? If so, what were the results? If 

not, have you received informal feedback on the process and are there plans to do a formal 

assessment in the future? 

11. What effect did this project have on the operations of your hospital and other community 

providers that participated in it? Probes: What impact did the project have on quality of care? 

The cost of care provided? Did the impact differ by type of provider? What factors influenced the 

choice of providers for the HIT network, and how did the project affect linkages across providers? 

Vendor Selection

12. What has been your experience with the vendor(s) chosen for the EHR/HIE/other HIT products 

implemented for this grant? Would you select the same vendor(s) and product(s) if you were 

doing it again? Why or why not? 

13. What criteria/factors were used in selecting your vendors (e.g. sustainability, cost, availability)?

Sustainability

14. How do you anticipate that the HIT implemented through this grant will be sustained both in 

terms of maintenance and financial, and upgraded? What type of support and maintenance has 

been planned for? 

15. What do you think the impediments to sustainability will be? 

16. Have you sought funding to complement, support, sustain or enhance the Flex activities? From 

whom, when, and for what purposes was the funding sought? Was funding received? 

HRSA

17. What was HRSA’s role with regard to your project? Was HRSA’s involvement in your project 

helpful? If yes, how? If no, why not? What other assistance did you need from HRSA? 

18. What advice would you give to HRSA with regard to any future funding (e.g. grant processes, 

funding limitations, structure or focus of grant program) for rural HIT projects? 
118

19. What future types of evaluation or studies would be most valuable to you (i.e. outcome 

measures) as you implement and expand HIT activities? 

Successes, Challenges, and Lessons Learned

20. What challenges have you encountered in implementing this project? How were they 

addressed? Probes for possible challenges: Staff attitudes? Resources? Time? IT infrastructure? 

Others? 

21. What types of supports assisted in the planning and implementation of this project? How?

22. What has gone particularly well in the implementation of this project? Why do you think that 

is? 

23. What lessons have you learned about HIT planning and implementation? Do you think these 

lessons are transferable to other sites? 

24. What advice would you give to other CAHs about planning and implementing HIT? 
119

Appendix B: CAH-HIT Site Visit Interview Protocol
120

Introductions and Project Overview

Topics: Network Background, Vendor Selection, Organizational and Staff Development, Strategic Plan, 

Sustainability, Value Based ROI Preliminary, Q&A

Introduction

1) What is your role in this organization, and how long have you worked here? (ASKED OF ALL 

RESPONDENTS, BUT THERE IS A SLIGHTLY DIFFERENT VERSION OF THIS QUESTION FOR EACH 

TYPE OF PERSON INTERVIEWED)

2) What is your involvement in the current HIT implementation project? How long have you been 

involved?

3) What prompted you to apply for/participate in the Flex CAH HIT grant project? (PD, CAH CEO)

4) Which providers are participating in the project activities? (PD, CAH CEO)

5) How have the medical staff/local physicians and nursing staff been involved in planning and 

implementing the HIT activities funded by this grant? (MEDICAL AND NURSING DIRECTOR)

6) What types of planning approaches were used to determine the needs and focus of your HIT 

implementation project (e.g. readiness assessment, workflow analysis, strategic/business plan)?

7) What types of relevant HIT existed at each CAH and network member prior to implementation?

Vendor Selection

8) What criteria/factors were used in selecting your vendors (e.g. sustainability, cost, availability)?

9) Who was involved in the process of selecting vendor(s) for the EHR/HIE/other HIT for this grant? 

For those involved, what challenges were encountered in that process? How were they 

addressed? Would you use a similar process if you were doing it again? (IT DIRECTOR)

10) What has been your experience with the vendor(s) chosen for the EHR/HIE/other HIT products 

implemented for this grant? Would you select the same vendor(s) and product(s) if you were 

doing it again? Why or why not? (IT DIRECTOR)

11) Was the same vendor/system selected by all CAHs or other participating entities? How has that 

impacted the project implementation? (IT DIRECTOR)

12) What providers were involved in vendor selection?

Sustainability

13) How do you anticipate that the HIT implemented through this grant will be sustained both in 

terms of maintenance and financial, and upgraded? What type of support and maintenance has 

been planned for? (PD, CAH CEO)
121

14) What do you think the impediments to sustainability will be? (PD, CAH CEO)

15) Have you sought funding to complement, support, sustain or enhance the Flex activities? From 

whom, when, and for what purposes was the funding sought? Was funding received? (PD, CAH 

CEO, IT DIRECTOR)

Funding 

Source

When 

requested

Timeframe 

of funding

Funding 

received?

Purpose of Funding

16) Are there plans to expand HIT functionality/capability at the hospitals that are currently part of 

the network or to additional hospitals? If so, what will the expansion involve? If not, why not? 

(IT DIRECTOR)

17) What is the impact of staff turnover on the sustainability of the system? (PD, CAH CEO)

HRSA

18) What was HRSA’s role with regard to your project? Was HRSA’s involvement in your project 

helpful? If yes, how? If no, why not? What other assistance did you need from HRSA? (PD)

19) What advice would you give to HRSA with regard to any future funding (e.g. grant processes, 

funding limitations, structure or focus of grant program) for rural HIT projects? (PD)

20) What future types of evaluation would be most valuable to you (i.e. outcome measures)? 

Network/Telecommunications Infrastructure

Topics: Datacenter Tours, Server Configuration/Redundancy, Wide Area Network (WAN) Configuration, 

Network Security, Q&A 

Confirm network implementation map

System Functionality

Topics: Demonstration of Various Applications in Use, Including Patient Care Documentation, E-MAR, 

Clinical View, ExitCare, and Others, with Focus on Clinical Applications; Q&A

Use this time to learn about system.
122

Functionality – review Step 1 of ROI and use to confirm understanding

1) As part of your project, what functional components have been implemented in your hospital 

and/or clinic (e.g., laboratory data, pharmacy data, radiology data, surgical procedures, clinic 

and inpatient notes etc.)? (PD, CAH CEO, MEDICAL DIRECTOR, NURSING DIRECTOR)

Technology 

2) What testing of system functionality has occurred? What measures were used? What are the 

results? What problems or gaps have been identified? (PD, CAH CEO)

3) What, if any, technology is still needed to meet the goals of this project? (PD, CAH CEO, IT 

DIRECTOR)

4) To what extent has the HIT implementation impacted the availability of data?

5) To what extent has implementation of the project allowed patient data to be tracked through 

the continuum of care? (PD, CAH CEO, IT DIRECTOR)

6) Has the HIT implementation helped providers to work more efficiently with other providers to 

manage the health care needs of patients? How? (HIE DIRECTOR)
123

Implementation Activities

Topics: Recap of Activities To Date/Workplan Review; File Building Process Discussion; End-User Training 

and Go-Live Discussion, Next Steps; Q&A

Activities to Date

1) What activities in the workplan for this project have been completed? What activities are 

currently underway? (PD, CAH CEO)

Staff Training 

2) What type of staff training has been conducted? What types of staff have attended? Was the 

training required? How many staff have participated in the training, by staff type? (PROJECT 

DIRECTOR, CAH CEO) 

3) When did the training occur? How frequently has it occurred? (PROJECT DIRECTOR, CAH CEO)

4) How has it been determined what trainings should be held? (PROJECT DIRECTOR, CAH CEO)

5) Have pre- and post-tests of staff attitudes and/or knowledge of HIT been conducted? If so, what 

were the results? (PROJECT DIRECTOR, CAH CEO)

6) Was staff satisfied with the training? Has there been any assessment of staff satisfaction with 

the training? If so, please describe. (PROJECT DIRECTOR, CAH CEO)

7) How effective do you feel the trainings have been? (PROJECT DIRECTOR, CAH CEO)

8) What additional training, if any, would be useful? (PROJECT DIRECTOR, CAH CEO)

9) How was the medical staff and nursing staff trained to use the EHR? What percentage of 

medical staff (and nursing staff) participated in training? How effective do you feel the training 

has been? Was the medical staff (and nursing staff) satisfied with the training? What additional 

training, if any, would be useful? (MEDICAL AND NURSING DIRECTOR)

Utilization

10) What percentage of staff (by type) have been trained to use the system? 

11) What percentage of staff are using the system, by staff type? Do you have a plan to achieve 

100% use? (PD, CAH CEO, MEDICAL DIRECTOR)

12) What is the average number of users on a daily basis, by staff type? (PD, CAH CEO)
124

Helpdesk/Application Support and Education

Topics: Helpdesk Structure; PhaseWare and Sharepoint Demonstration; Educational Activities and 

Tools; Q&A

Technical Support 

1) How is technical support provided to the medical staff (e.g., internally by CAH staff or externally 

by the network or another entity)? What types of technical support have been requested? 

Please describe support requests by type of staff, including physicians, nurses, clerical/support 

staff, and accounting. How effective do you feel the technical support has been? What 

additional support, if any, would be useful? (MEDICAL DIRECTOR, NURSING DIRECTOR, and IT 

DIRECTOR, PD, CAH CEO)

Policies and Procedures

Topics: Review of Various Operational Procedures, with Focus on Operational Security, Q&A

1. How have policies and procedures changed since the HIT implementation? What new types of 

policies and procedures have been put in place?

2. How has the project implementation handled interoperability issues?

3. How has the project implementation handled data confidentiality issues? (IT DIRECTOR)

Project Evaluation

Topics: QI Plan: Discharge Instructions, Medication Errors, End-User Survey Results, Patient Satisfaction, 

and QI Submission Facilitation; Levels of Utilization; Next Steps; Q&A

Outcomes and Impact

1) What is the status of the individual site evaluation for the grant? What methods were used? 

What have been the results to date? (PD)

2) How do you define success? What measures are currently in place or will be put in place to 

track success? (probe for HRSA-specific measures and other measures chosen by the site).

3) Is your system capable of collecting data to report on the outcome measures you have chosen? 

(PD, CAH CEO) If not, how and when do you anticipate you will be able to collect this data? (IT 

DIRECTOR)

4) Do you have baseline data on your outcome measures? (PD, CAH CEO)

5) What impact, if any, has the implementation had on your patients? (PD, CAH CEO)

6) Have any formal assessments been done of satisfaction with the project implementation among 

medical staff, nursing staff, other types of staff, and patients? If so, what were the results? If 
125

not, have you received informal feedback on the process and are there plans to do a formal 

assessment in the future? (CAH CEO, MEDICAL AND NURSING DIRECTOR)

7) What effect did this project have on the operations of your hospital and other community 

providers that participated in it? Probes: What impact did the project have on quality of care? 

The cost of care provided? Did the impact differ by type of provider? What factors influenced the 

choice of providers for the HIT network, and how did the project affect linkages across providers? 

(CAH CEO)

Staff Attitudes and Lessons Learned: Panel Discussion 

Topics: Panel Discussion of Challenges, Barriers, and Benefits of EHR Implementation Initiative. Topic 

Detail to be Developed and Distributed Prior to Meeting. 

Participants: Network Staff; 3 Representatives from Each of the Network Member Facilities (Including 

Nurses, Physicians, IT Managers, Executives, Pharmacists, QI Managers, etc. to the extent available.)

1) Is staff satisfied with the system?

Probe for: relevance of available data, timeliness, accessibility, impact on patients, facilitation of 

workflow/daily tasks, and responsiveness to requests for support.

2) Has the implementation of EHRs (or specific components if not all have been implemented) 

improved the ability of clinicians to provide patient care? How? ( e.g., track quality measure 

data for patients with chronic conditions and follow up)? Which system components are most 

useful to your clinical practice? (MEDICAL AND NURSING DIRECTOR, PD, CAH CEO)

3) What impact has implementation had on patients? What feedback have you received from your 

patients? (MEDICAL AND NURSING DIRECTOR)

Successes, Challenges, and Lessons Learned

4) What challenges have you encountered in implementing this project? How were they 

addressed? 

Probes for possible challenges: Staff attitudes? Resources? Time? IT infrastructure? Others? 

(PD, CAH CEO)

5) What has been the most challenging aspect of this HIT project for clinicians? (MEDICAL AND 

NURSING DIRECTOR)What types of supports assisted in the planning and implementation of this 

project? How?

6) What has gone particularly well in the implementation of this project? Why do you think that 

is? (PD, CAH CEO)

7) What lessons have you learned about HIT planning and implementation? Do you think these 

lessons are transferable to other sites? (PD, CAH CEO)

8) What advice would you give to other CAHs about planning and implementing HIT? (CAH CEO, IT 

DIRECTOR)
126

Appendix C: Value-Based Return on Investment 

(ROI) Protocol and Tools
127

There are many methods and approaches to consider when evaluating return on investment (ROI) of 

health information technology (HIT) implementations. Financial and disease outcome-based ROI can 

take years to accrue, but more value-based ROI measures can be good predictors of future financial and 

disease outcome-based ROI. The focus of the Flex CAH HIT ROI evaluation is on how and where users 

(patients, clinicians, administrators, etc.) of an implemented HIT system obtain value. For the Flex CAH 

HIT projects, the following tools are designed to incorporate both qualitative and quantitative valuebased ROI measures to capture important information in an appropriate and reasonable manner within 

the evaluation timeframe. 

Steps 1 & 2 are mostly concurrent and serve as tools for establishing baseline measures for comparison 

with the implemented grant system(s) and activities. Step 1 is a clarification and understanding exercise 

that also serves as the context for completing Steps 2 & 3. See attachments for further detail.

For each project to be evaluated, Flex CAH HIT grantees will first utilize the attached Functional 

Definition tool. This tool is used to parse out the individual functions and participants involved in the 

project’s scope, and to further define the specific relationships among data suppliers, data users and the 

purpose of the information being shared. 

Step 1

The tool provides lists of possible HIT functions as well as possible HIT participants, however individual 

CAHs will have the opportunity to add or combine up to three user defined functions and three user 

defined participants.

CAHs will fill in this tool to the best of their ability prior to a conversation with a member of the Altarum 

team. The result will be confirmation between the CAH and Altarum as to the specific functions and 

participants affected by the Flex CAH HIT grant.

Next, grantees will utilize the Understanding Pre-Grant System(s) and Activities questionnaire to assess 

several measurement elements of the pre-grant system(s) and activities involved in the Flex CAH HIT 

project. This tool serves as a baseline measure to understand where each CAH is (their “pre-grant 

processes”) as a means towards comparing the impact of the grant activities (their “to be processes”).

Step 2

CAHs will fill in this questionnaire to the best of their ability prior to a site visit with a member of the 

Altarum team. The result will be confirmation between the CAH and Altarum as to the current state of 

the specific systems and activities to be affected by the grant. 

Finally, grantees will utilize the Understanding Implemented Grant System(s) and Activities 

questionnaire to assess several measurement elements of the implemented system(s) and activities 

involved in the Flex CAH HIT grant. This implementation information collection step would then be 

compared against the baseline measurements taken in Step 2. 

Step 3

CAHs will fill in this questionnaire to the best of their ability prior to a conversation with a member of 

the Altarum team. The result will be confirmation between the CAH and Altarum as to the implemented 

state of the specific systems and activities affected by the grant. 

Beyond the tools currently developed for Flex CAH HIT evaluation, it should be noted that additional ROI 

measures of specific clinical outcomes and quality improvement are extremely important effects in HIT 

implementation. Clinical and quality improvement measures require a significant observation period to 
128

appropriately assess implementation and ROI effects, and therefore are not reasonable to directly 

consider in the limited timeframe of this evaluation project. 

Given this, the baseline measures Flex CAH HIT grantees will establish in Steps 1 & 2 may be utilized in 

future ROI evaluations of clinical and quality improvement measures. Additionally, it may be desirable 

to have CAHs answer additional questions regarding specific disease outcome, workflow and staffing 

changes to be affected by the grant, including any relevant measurements (e.g. FTE changes, task time, 

functional or other direct costs).
129

Step 1 – Functional Definitions

The first step in this tool is to define the list of HIT functionalities that your Flex CAH HIT project is 

implementing. Below is a list of possible HIT functionalities, however, you may not find the specific 

function you are looking for on the list. Or, you may wish to combine several functions under a single 

label. If so, you may enter up to three of your own function labels in the yellow cells (text in these cells 

reads “Enter User Defined...”) below. The goal is to try and find your project's functions and participants 

on the list as much as possible, so as to be consistent. Please read the full list before you begin making 

selections.

List of HIT Participants Additional Detail

Ambulatory surgery Ambulatory surgery (hospital-based and free-standing)

Ancillary staff Ancillary staff (such as cleaners, porters, kitchen staff, etc.)

Behavioral/Mental Health inpatient 

providers

Behavioral/Mental Health outpatient 

providers

CMS/Medicare

Community outreach clinics Community outreach clinics (public health, school health, 

nursing clinics, express care)

Durable Medical Equipment providers

Employers/Plan Administrators

Health plan enrollment stations

Hospitals Inpatient hospitals

Inpatient pharmacies

Inpatient physicians Medical/surgical inpatient physicians

Inpatient nurses Medical/surgical inpatient nurses

Laboratories Laboratories (including hospital-based, public health and 

free-standing)

Local public health agency Local public health agency (excluding clinics and labs)

Non-radiology procedure centers Non-radiology procedure centers (including hospital-based 

and free-standing)

Other therapists Other therapists (e.g., occupational therapy, physical 

therapy)

Outpatient pharmacies Outpatient pharmacies (including hospital and freestanding)

Outpatient physicians Medical/surgical outpatient physicians

Outpatient nurses Medical/surgical outpatient nurses

Patients or caregivers

Pharmacy Benefit Managers (PBMs) Pharmacy benefit management companies

Pharmaceutical and Medical Device 
130

List of HIT Participants Additional Detail

manufacturers

Primary care physicians and clinics

Private health plans

Quality improvement personnel

Radiology centers Radiology centers (including hospital-based and freestanding)

Regulators FDA, DEA, state regulators

Researchers

Skilled nursing facilities

Specialty Care Physicians and clinics

State Medicaid

State public health agency State public health agency (excluding clinics and labs)

Enter User Defined Participant #1

Enter User Defined Participant #2

Enter User Defined Participant #3

Case management and care 

coordination

Claims management

Clinical document availability Clinical document availability (e.g., populating an EMR)

Clinical document exchange Clinical document exchange (e.g., discharge summaries 

sent between providers)

Health plan enrollment verification

Historical allergy list

Historical procedure list

Historical visit/hospitalization list Historical medical encounter (hospitalizations, visit list)

Medical record Medical record (remotely stored, internet-served EMR)

Medication decision support Medication decision support (safety or other alerts)

Medication formulary check

Medication list Historical medication list 

Medication prescription delivery Medication prescription delivery (e-prescribing)

Patient access to health records

Patient alerts or reminders

Patient sending health information to 

providers

Patient-provider e-mail

Price/cost information
131

List of HIT Participants Additional Detail

Procedure or test ordering

Procedure/referral authorization

Provider-to-provider e-mail

Public health information to clinicians Public health information to clinicians (alerts)

Public health reporting

Referral and/or consultation 

management

Registry for disease care 

management/improvement

Registry for public health surveillance Registry for public health surveillance (e.g., cancer, 

injuries)

Registry of immunization, medication or 

devices

Reporting and analysis of quality or 

safety performance

Results delivery Results delivery (lab, radiology)

Results list Historical results list (lab, radiology)

Enter User Defined Function #1

Enter User Defined Function #2

Enter User Defined Function #3

Functional Definition Tool- Step 1 of 3

Starting with row 6, below, please indicate who will be exchanging data with whom, and for what 

purpose, as part of your Flex CAH HIT implementation. Multiple Data Suppliers and/or data users may 

be selected for each functionality by selecting each participant in a new cell moving down the list. For 

additional participants, Functionality should only be listed once, at the first line of the relationship. 

Please change the Func # column (first column) to be numbered appropriately for each Functionality. 

Next, indicate the current level of interoperatibility for each function. Fill in as many rows as needed. 

Please read the full list before you begin making selections.

In addition to this specific relationship assignment, please describe the expected impact of the HIT 

intervention and how the project will exert this impact (read text on next page).

Func 

#

Data Supplier Data User Functionality Pre-Grant 

Level of 

Interoperability

Expected Level 

of 

Interoperability

1 Inpatient nurses Ancillary staff Procedure or 

test ordering

Level 1 Level 3

Other therapists 

2 Ancillary staff Inpatient physicians Results 

delivery

Level 1 Level 3

Inpatient nurses
132

Func 

#

Data Supplier Data User Functionality Pre-Grant 

Level of 

Interoperability

Expected Level 

of 

Interoperability

Other therapists 

3 Inpatient nurses Inpatient physicians Clinical 

document 

availability

Level 1 Level 3

Other therapists Inpatient nurses

Ancillary staff Other therapists 

Quality 

improvement 

personnel

4 Inpatient 

pharmacies

Inpatient 

pharmacies

Medication list Level 1 Level 3

Inpatient 

physicians

Inpatient physicians

Inpatient nurses Inpatient nurses

Other therapists 

Quality 

improvement 

personnel

5 Ancillary staff Ancillary staff Medical record Level 1 Level 3

Inpatient nurses Inpatient nurses

Inpatient 

physicians

Inpatient physicians

Other therapists Other therapists 

Quality 

improvement 

personnel

Narrative Description of Intervention

1. Procedure or Test Ordering: Rather than writing down orders and sending or faxing to ancillary 

departments (Lab, Radiology), nursing areas input orders through the HMS Order Entry System. The 

ancillary departments immediately (in real time) receive orders electronically. This eliminates the 

process of nurses writing and delivering and then ancillary staff transcribing the paper information into 

ancillary systems. In addition, orders go through various contraindication, duplicate order and therapy 

checking decision support logarithms. Also, orders are checked against diagnosis to ensure medical 

necessity for coverage, and advanced beneficiary notices (ABNs) are automatically generated in case of 

potential non-coverage. For verbal or phone physicians orders, the HMS system requires readback 

verification documentation, which is routed to the physicians electronic signature inbox in Clinical View. 

Orders (and results) are available to be viewed as a component of the patient's medical record 

2. Results Delivery: As by-product of HMS order entry system, results are delivered and available in a 

variety of ways. They can automatically print out at the ordering departments network printer, and also 
133

are available through HMS's Monitor and Clinical View applications, and are sent to the physicians 

electronic signature inbox in Clinical View, where a warning notification is given when new result/orders 

are available. Out-of-range results are flagged as priority items. Through Clinical View, results can be 

graphed to indicate changes over time. Results are over the life of a patient and no data is archived, 

which allows clinicians easy access to past data 

3. Clinical Document Availability: Using the HMS multidisciplinary Patient Care Documentation 

application, inpatient nurses and other therapists input initial assessments, progress notes, vital signs, 

and a variety of other clinical documents directly into the Patient Care application rather than using 

paper forms. This intervention allows for the use of logic driven processes, including skip and branch 

logic, required fields, reminders, and careplans and orders being generated as byproducts of 

documentation. Care plans include goals and interventions to achieve completion; and careplan 

worklists with time intervals are created automatically. Accurate authenticated user and 

date/timestamp information is automatically logged through all phases of documentation. And the 

information generated is available for multiple caregivers from multiple locations to review, rather than 

in a single paper chart location, as was previously the case 

4. Medication List: Rather than maintaining multiple paper medication lists, pharmacists and nursing 

staff use the HMS Pharmacy and Patient Care applications to collaborate on a single electronic 

medication administration record that is updated in real-time as medications are added, removed, and 

administered. This allows for accurate user and timestamping, and also for the most current medication 

information to be available to all providers. The system captures home medications and provides for 

medication reconciliation as part of the CPOE (not yet implemented) and discharge process 

5. Medical Record: Through the HMS Clinical View application, caregivers can effectively view the 

patient's electronic medical record, including the information related to the above interventions, as well 

as physician dictations, various scanned documents, clinical information from previous visits, and other 

information. Clinical View also allows physicians to e-sign off on dictations, and will be the application 

from which they perform CPOE. PACS images can be integrated through Clinical View (PACS integration 

is currently implemented at only one of the facilities), and there is flexibility as to the presentation of 

the various data elements, which may be customized by provider preference. This intervention allows 

multiple clinicians simultaneous flexible access (including remote access) to much of the information 

they need to make care decisions; whereas previously the information was only available in a single 

paper chart.

Interoperability Levels

Level 0 Data not exchanged prior to grant activities

Level 1 Phone and mail (paper based) communications

Level 2 Machine-transportable data (standard fax)

Level 3 Machine-available data (EMR, email and electronic 

messaging)

Level 4 Machine-interoperable data (interoperable data exchange with standardized message 

formats and content)

List of HIT Participants List of HIT Functions
134

List of HIT Participants List of HIT Functions

Ambulatory surgery Case management and care coordination

Ancillary staff Claims management

Behavioral/Mental Health inpatient providers Clinical document availability

Behavioral/Mental Health outpatient providers Clinical document exchange

CMS/Medicare Health plan enrollment verification

Community outreach clinics Historical allergy list

Durable Medical Equipment providers Historical procedure list

Employers/Plan Administrators Historical visit/hospitalization list

Health plan enrollment stations Medical record

Hospitals Medication decision support

Inpatient pharmacies Medication formulary check

Inpatient physicians Medication list

Inpatient nurses Medication prescription delivery

Laboratories Patient access to health records

Local public health agency Patient alerts or reminders

Non-radiology procedure centers Patient sending health information to providers

Other therapists Patient-provider e-mail

Outpatient pharmacies Price/cost information

Outpatient physicians Procedure or test ordering

Outpatient nurses Procedure/referral authorization

Patients or caregivers Provider-to-provider e-mail

Pharmacy Benefit Managers (PBMs) Public health information to clinicians

Pharmaceutical and Medical Device 

manufacturers

Public health reporting

Primary care physicians and clinics Referral and/or consultation management

Private health plans Registry for disease care management/improvement

Quality improvement personnel Registry for public health surveillance

Radiology centers Registry of immunization, medication or devices

Regulators Reporting and analysis of quality or safety 

performance

Researchers Results delivery

Skilled nursing facilities Results list

Specialty Care Physicians and clinics Enter User Defined Function #1

State Medicaid Enter User Defined Function #2

State public health agency Enter User Defined Function #3
135

List of HIT Participants List of HIT Functions

Enter User Defined Participant #1

Enter User Defined Participant #2

Enter User Defined Participant #3
136

Step 2 – Understanding Pre-Grant System(s) and Activities

1. In the context of activities affected by the Flex CAH HIT project, how did pre-grant information 

systems fit with the workflow of the following groups?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

2. In the context of activities affected by the Flex CAH HIT project, how will the implemented 

information systems change the workflow of the following groups, including those who support 

them?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

For each affected group, are these workflow changes clearly understood? Please explain.

o Narrative follow-up

3. For the following groups, what are the user perceptions of the pre-grant information systems in 

terms of usefulness or ease of use?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

4. For each user group (clinicians, public health officials, laboratories, payers, administrators) what 

training and support will they receive as part of the system implementation?

o Narrative response for each user group

5. Currently, how committed are the user clinicians to the activities of the grant?

 Very committed

 Somewhat committed
137

 Neutral

 Not very committed

 Extremely uncommitted

Please explain how you arrived at your answer above – why do you feel this way?

o Narrative follow-up 

6. As part of the grant activities, how is user feedback sought? How do you let users know their 

feedback is considered/incorporated?

o Narrative response

7. For the pre-grant system, what is the level of patient satisfaction?

 Very satisfied

 Somewhat satisfied

 Neutral

 Not very satisfied

 Extremely unsatisfied

Please explain how you arrived at your answer above – how do you assess patient satisfaction?

o Narrative follow-up 

8. In the pre-grant system, how available was information when and where it was needed? 

 Very available

 Somewhat available

 Neutral

 Not very available

 Extremely unavailable

Please explain how you arrived at your answer above – how do you know?

o Narrative follow-up

9. How usable was the pre-grant system?

 Very useable

 Somewhat useable

 Neutral

 Not very useable

 Extremely unusable

10. Why and how was the pre-grant system used or not used?

o Narrative response

11. What non-financial values did various stakeholder groups (clinicians, public health officials, 

laboratories, payers, administrators) receive from the pre-grant system? What do they 

anticipate receiving from the implemented system? What steps will be taken to support/assure 

use of the new system?
138

o Narrative response for each stakeholder group

12. For each user group (clinicians, public health officials, laboratories, payers, administrators) how 

efficient was the pre-grant system?

 Very efficient 

 Somewhat efficient

 Neutral

 Not very efficient

 Extremely inefficient

Please explain how you arrived at your answer above.

o Narrative follow-up

13. For each user group (clinicians, public health officials, laboratories, payers, administrators) what 

is their perception of the pre-grant system in terms of supporting positive clinical outcomes?

 Very supportive 

 Somewhat supportive

 Neutral

 Not very supportive

 Extremely not supportive

Please describe what clinical outcomes you are basing your answer on.

o Narrative follow-up
139

Step 3 – Understanding Post-Grant System(s) and Activities

1. In the context of Flex CAH HIT project activities, how do the implemented systems fit with the 

workflow of the following groups?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

2. In the context of activities affected by the Flex CAH HIT project, how do the implemented 

information systems change the workflow of the following groups, including those who support 

them?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

For each affected group, are these workflow changes clearly understood? Please explain.

o Narrative follow-up

3. For the following groups, what are the user perceptions of the implemented information 

systems in terms of usefulness or ease of use?

o Narrative response for each

 Clinicians (e.g. physicians, nurses, PA’s)

 Public health officials

 Laboratories

 Payers 

 Administrators (e.g. executives, QI/process staff, support staff)

 Others

4. At this point in the implementation process, how do users (clinicians, public health officials, 

laboratories, payers, administrators) feel about the training and support they are receiving as 

part of the system implementation?

o Narrative response for each user group

5. At this point in the implementation process, how committed are the user clinicians to the 

activities of the grant?
140

 Very committed

 Somewhat committed

 Neutral

 Not very committed

 Extremely uncommitted

Please explain how you arrived at your answer above – why do you feel this way?

o Narrative follow-up 

6. As part of the grant activities, how is user feedback sought? How do you let users know their 

feedback is considered/incorporated? How has this changed from your answer in Step 2?

o Narrative response

7. For the implemented system, what is the level of patient satisfaction?

 Very satisfied

 Somewhat satisfied

 Neutral

 Not very satisfied

 Extremely unsatisfied

Please explain how you arrived at your answer above – how do you assess patient satisfaction?

o Narrative follow-up 

8. With the implemented system, how available is information when and where it is needed? 

 Very available

 Somewhat available

 Neutral

 Not very available

 Extremely unavailable

Please explain how you arrived at your answer above – how do you know?

o Narrative follow-up
141

9. How usable is the implemented system?

 Very useable

 Somewhat useable

 Neutral

 Not very useable

 Extremely unusable

Please explain how the implemented system’s usability compares with what was anticipated, 

the reason for any gaps and mitigation strategies to close these gaps.

o Narrative follow-up

10. Why and how is the implemented system used or not used as anticipated?

o Narrative response

11. What non-financial values have various stakeholder groups (clinicians, public health officials, 

laboratories, payers, administrators) realized as part of the implementation? How does this 

compare to what was anticipated? What steps will be taken to support/assure use of the new 

system? What non-financial values have various stakeholder groups realized as part of the 

implementation?

o Narrative response for each stakeholder group

12. For user groups (clinicians, public health officials, laboratories, payers, administrators) how 

efficient is the implemented system?

 Very efficient 

 Somewhat efficient

 Neutral

 Not very efficient

 Extremely inefficient

Please explain how you arrived at your answer above.

o Narrative follow-up

13. For each user group (clinicians, public health officials, laboratories, payers, administrators) what 

is their perception of the implemented system in terms of supporting positive clinical outcomes?

 Very supportive 

 Somewhat supportive

 Neutral

 Not very supportive

 Extremely not supportive

Please describe what clinical outcomes you are basing your answer on.

o Narrative follow-up
142

Value-Based Return on Investment Analysis

1. Clinician commitment 

Several studies of HIT implementation as well as direct-user feedback in using HIT point to the critical 

need for end-user commitment to the activities of implementation and especially to the specific 

technology being implemented. In the case of the Flex CAH HIT projects, the main end-user group is 

clinicians – including physicians, nursing staff, PAs and other clinical staff. To measure this important 

factor in HIT implementations, the Flex CAH HIT Value-Based ROI protocol asked questions of CAHs to 

assess clinician commitment to the grant activities. 

In addition to a specific question regarding pre- and post-implementation rating of clinician support for 

the activities of the grant, CAHs were asked to respond to two other issues related to clinician 

commitment – user training and support, and user feedback. Well-designed user training can improve 

the likelihood that end users, particularly clinicians, will accept new HIT; in general, effective training is 

well-established as key to promoting end-users embracing new technologies.26 Lessons from eprescribing initiatives and other HIT projects have shown strong support for one-on-one training and 

support including on-site software support, customization by specialty, and office workflow integration 

as part of successful HIT implementations.27 Similarly, user feedback needs to be developed and 

implemented with a focus on continuous improvement based upon end-user feedback. Given the 

complexity of health care, the need for iterative and continual refinement of HIT based upon user 

feedback may be even more critical for success than in IT implementations in other industries.28

2. Availability of information

HIT systems, appropriately implemented and used, enable clinicians to store, retrieve, and access more 

information than what is available to them in a paper-based system. Electronic means of recording 

and/or sharing health information for patient care may provide clinicians a mechanism to better 

coordinate care and information, and as a result, improve continuity across transitions in providers and 

settings. Studies have reported that clinicians working in treatment environments that had completed 

an HIT implementation such as an EHR/EMR and/or HIE were significantly more likely to report high 

levels of both clinical information availability and timeliness.29 For the Flex CAH HIT projects, an 

important element to capture in assessing the value of the implemented system was end-user 

perception of information available to them. CAH respondents were asked about the availability of 

information before HIT implementation (pre-system) and after implementation (post-system). 

 

26 Karsh, B. (2004). Beyond usability: designing effective technology implementation systems to promote patient safety. Quality 

and Safety in Health Care 13(5), 388-94.

27 Halamka, J., Heinold, J., Fournier, G., Stone, D., & Berry, K. (2006). E-Prescribing in Massachusetts: Collaboration leads to 

success. Patient Safety & Quality Healthcare, September/October 2006. Retrieved from http://www.psqh.com/sepoct06/eprescribing.html

28 Doebbeling, B. & Pekny, J. (2008). The role of systems factors in implementing health information technology. Journal of 

General Internal Medicine 23(4), 500-501.

29 Graetz, I., et al. (2008). Care coordination and health information technology: Information availability and timeliness across 

care transitions. Clinical Medicine & Research 6(3-4), 149.
143

3. Importance of system usability

End-user perceptions of usability of HIT systems are another critical issue in their implementation. 

Unless clinicians deem it usable and useful, clinicians will not use an HIT system. Usability has been 

defined as a measure of how efficient, effective, enjoyable, and safe an IT system is to use.30 Many 

studies have documented the importance of usability in terms of its impact upon the adoption of HIT 

systems by clinicians; indeed, there are many documented cases of organizations that have deployed 

HIT systems that were later turned off, boycotted, or were not used to their fullest extent because of 

clinician dissatisfaction with them.31, 32

4. Importance of system efficiency

 To measure perceptions of system usability for Flex CAH HIT 

projects, the value-based ROI protocol asked for user measurements of pre- and post-grant systems in 

terms of usability. 

Implemented HIT systems – EMR/EHRs in particular – can contribute to improvements in care 

organization and delivery efficiency. Most Flex CAH HIT grantees reported that the decision to 

implement HIT was aimed, in part, to improve site efficiency, workflow, and clinician productivity. 

Increased clinical efficiency and provider productivity are frequently cited as benefits of EMR/EHRs and 

HIE – e.g. tracking down previous lab results, eliminating reams of paper records, reducing time required 

for chart reviews--– thereby making workflows more efficient. During implementation or “go live”, sites 

will experience some slowdown in productivity as clinicians and administrative staff become familiar 

with the HIT system(s). 33, 34

5. Supporting positive clinical outcomes 

CAH respondents were asked about how efficient they found the pre-grant 

system and following implementation, about the efficiency of the implemented system.

HIT holds enormous promise for supporting positive clinical outcomes; demonstrating these effects is 

extremely important in demonstrating the value of an HIT investment. Many studies have 

demonstrated that implementation of HIT, particularly EMRs, has led to improvements in morbidity, 

mortality, complication rates, prescribing errors and disease management associated.35, 36

 

30 Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994). Human-computer interaction: Concepts and 

design. 

 However, as 

discussed previously, specific clinical outcomes and quality improvement measures related to a 

particular HIT implementation require a significant observation period to appropriately assess effects; 

for the purposes of this evaluation, it was not possible to study this outcome. Instead, as part of the 

31 Borycki, E. & Kushniruk, A. (2005). Identifying and preventing technology-induced error using simulations:Application of 

usability engineering techniques. Healthcare Quarterly 8(Sp), 99-105.

32 Ash, J., Stavri, P., & Kuperman, G. (2003). A consensus statement on considerations for a successful CPOE implementation. 

Journal of American Medical Informatics Association 10(3), 229-234.

33 Moiduddin, A. & Gaylin, D. (2007). Health information technology adoption among health centers: A digital divide in the 

making? [Background paper]. Retrieved from http://www.nhpf.org/library/background-papers/BP_HealthCenterIT_07-23-

07.pdf

34Gaylin, D., Goldman, S., Ketchel, A., & Moiduddin, A. (2005). Community health center information systems assessment: Issues 

and opportunities. Retrieved from http://aspe.hhs.gov/sp/chc/chc.pdf

35 Amarasigham, R., Plantinga, L., Diener-West, M., et al. (2009). Greater use of information technology impacts mortality, 

complication rates, and costs. Journal of Clinical Outcomes Management 16(3), 108-109.

36Koppel, R., Metlay, J., et al. (2005). Role of computerized physician order entry systems in facilitating medication errors. 

Journal of the American Medical Association 293(10), 1197-1203. 
144

value-based ROI protocol, site respondents were asked for their perceptions of the pre-grant (paperbased) and post-implementation (electronic system) in terms of supporting positive clinical outcomes, 

with the idea that their perceptions would be likely indicators of substantive clinical outcome changes 

that would eventually be measurable over time.

6. Other non-financial values of HIT

In order to solicit input on other areas of value identified by system users, respondents were asked to 

share other non-financial value they had derived from their pre-grant system(s) and then from the 

implemented system(s). 

System implementation and patient satisfaction is one such non-financial value of HIT to which 

respondents were specifically asked to respond. Theoretically, greater access to information about prior 

care, medication prescriptions, laboratory test results, or clinical guidelines and faster information 

availability could allow physicians to support more productive discussions about medical issues and 

leave more time to thoroughly explain diagnoses and treatments or address patient concerns. At the 

same time, there could be unintended consequences in terms of time spent navigating the computer 

system, searching for information, exam room computing, and documenting visit activities that might 

limit time to address patient needs. 37 It is important to note that similar to the discussion on efficiency,

potential adverse effects such as slowdowns in productivity or time lost to navigating new systems tends 

to be prominent in the period shortly following implementation. For these reasons, Flex CAH HIT 

grantees were asked to estimate patient satisfaction prior to and post-implementation for the systems 

affected by grant activities.

 

37Hsu, J., Huang, J., et al. (2006). Health information technology and physician-patient interactions: Impact of computers on 

communication during outpatient primary care visits. Journal of the American Medical Informatics Association 13(2), 236. 
145

Appendix D: Flex CAH HIT Grantee Profiles
146

Alabama

Project Title: Flex CAH HIT Network Implementation Program

Project Summary: Randolph Medical Center (RMC) is a Hill-Burton era Critical Access Hospital in 

Roanoke (Randolph County), Alabama. The over-arching problem the project was intended to solve is 

the high mortality rate due to cardiovascular disease (CVD) in the hospital’s three county service areas. 

The hospital had started on a solution with an investment of over a million dollars in a 64 Slice CT 

Scanner but needed a Health Information Technology (HIT) major upgrade and follow-up system to 

attract the at risk patients to the hospital to use the scanner that this grant opportunity provided.

Project Background: ADPH and the management team at RMC applied for the Flex CAH HIT Program to 

make a dramatic and positive impact on cardiovascular disease (CVD) in the hospital’s service area. CVD 

is one of the major causes of death in rural America and in rural Alabama. ADPH selected RMC because 

they have a sound vision to improve cardiovascular care, they have put down money towards realizing 

that vision, and they have a passion for their vision. RMC has the necessary staff, as well as an 

established referral pattern with a tertiary center for complex invasive cardiac disease. However, they 

lack an HIT system to integrate their efforts from the rural setting in Roanoke, AL to the tertiary medical 

center 45 miles away. 

Network Participants:

• Randolph Medical Center (RMC) is one of three Critical Access Hospitals in Alabama. It is located 

in Roanoke in rural Randolph County in the Eastern part of the state. It is a not-for-profit 

hospital owned by the city of Roanoke. RMC was the lead agency for implementation of the 

project including selecting the vendors, managing the on site installation of the software and 

hardware, providing the personnel to be trained, and numerous other daily tasks.

• East Alabama Medical Center (EAMC) is the 352 bed tertiary hospital that RMC refers most of 

their patients to for higher level care. EAMC contracted for software changes to their database 

to enable the networking component. They were critical to the successful resolution of network 

privacy and HIPAA concerns.

• Institute for Advanced Cardiology (IAC) is a cardiologist physician group with offices in Randolph 

and Lee Counties. They use the 64 Slice CT Scanner at RMC for prevention procedures. They will 

be a node on the network when additional funding is obtained.

• Southern Family Health Care (SFHC) is a Rural Health Center in Roanoke that is a node on the 

network.

Project Management: The Alabama Department of Public Health (ADPH) is the largest state agency in 

Alabama with subordinate units in all 67 counties. As the grantee, ADPH performed several important 

functions including grant administration, contracting, fiscal agents, top level project management, 

evaluation, and liaison with the HRSA Office of Rural Health Policy. All of these were accomplished by 

the Office of Primary Care and Rural Health except for fiscal agent which was done by Health Finance. 

Other ADPH central office units such as General Counsel and Computer Systems Center made occasional 

but important contributions.

Project Goals: 

• Raise the HIT capabilities at RMC. RMC cannot effectively share data with other providers when 

its systems are several generations old and still very much reliant on paper. Therefore, the 
147

primary need is to move RMC further along on a path to acquiring a cost effective, functional 

EHR.

• Improve data sharing among network providers. Define and integrate the critical data to be 

shared among network participants that will allow true insights into CVD. These data will be 

housed in a disease registry and data warehouse available for access by network providers 

across the continuum. 

• Install reporting and trending capabilities. It is imperative that data can be retrieved, analyzed, 

and used as a basis for additional action.

Project Objectives:

• Formalize a provider network to include Randolph Medical Center, East Alabama Medical 

Center, and a seven-cardiologist physician group that has offices in both cities and privileges at 

both hospitals. 

• Bring the network together for shared information in order to identify more at-risk CVD patients 

and to provide more coordinated care for those already identified.

• Develop and implement specific interventions aimed at improving patient compliance with best 

practice medicine. These interventions include but are not limited to: initial and follow-up lab 

tests; health fairs and other screenings to identify at-risk patients; reminders of periodic care 

requirements; educational sessions on diet, exercise, and medications; and care assistance to 

ensure the patient is being appropriately treated within the continuum.

• Frequently report and analyze performance around key metrics such as blood pressure, HbA1c, 

and cholesterol.

Expected Project Impact: Improved effectiveness, efficiency, safety, and outcomes, including improved 

health, as evidenced by an increase in outpatient utilization and decrease in more costly inpatient 

events due to a better focus on compliance. 

Catchment Area and Target Population: The service area encompasses Randolph, Chambers, and Clay 

Counties. According to 2000 Census figures, 17.0 percent of all Randolph County residents were living on 

incomes below the FLP compared to 16.1 percent in Alabama and 12.4 percent nationally. According to 

2005 estimates of per capita personal income, $22,189 was available for spending per person in 

Randolph County, the 11th lowest among all 67 AL counties. This level is $12,282 less per person than the 

$34,471 national per capita personal income. Cardiovascular disease is killing people at an alarming rate 

in all three represented counties.

Evaluation: RMC is currently using a database called Sci-Health to track a host of collectable data to 

measure success. RMC has begun to track the core measures and compare them to past months. RMC 

is tracking census levels in various departments including psychiatry, swing bed and OR. RMC is using 

this data to spot trends and determining reasons for these trends. RMC is also using this software to 

track other things such as numbers of radiology exams performed as well as customer approval and the 

customer’s willingness to return to or recommend RMC to a friend or family member. RMC is also 

tracking mortality numbers and health performance indicators

Metrics: 

• Access to physicians, measured by a count of unassigned at-risk CVD patients that are identified 

and tracked in the disease registry, assigned to a doctor and are verified to have visited the 

physician.
148

• Missed appointments, measured by a percentage of CVD patients in the disease registry that 

failed to show up for scheduled visits.

• High blood pressure management, measured by trending patients with blood pressure <140/90 

mm/Hg.

• Cholesterol Management, measured by trending patients with LDL cholesterol levels <130 

mg/dl.

• Incidence of smoking, measured by a percentage of active smokers within the CVD registry.

• Diabetes compliance (as a risk factor for CVD), measured by average HbA1c.

• Other measures under consideration: triglyceride trends and calcium scoring.

Project Status at End of Grant Period: The original goal was the implementation of an EHR system in a 

rural CAH setting using existing grant dollars. This task was accomplished using the grant money for the 

CAHHITN program as well as using alternate grants such as Flex and SHIP. However, the initial plan was 

to have a network that included RMC, EAMC, IAC, SHRC and a data warehouse. When all of the 

requested funding was not awarded, the data warehouse was dropped in favor of a more elegant (and 

cheaper) solution, direct data links. The link to IAC has not been implemented due to a lack of funding.

Sustainability: RMC has the ability to continue use of the EHR as-is for the foreseeable future. RMC 

ensured during the initial contract procedure that it received a cost for ongoing maintenance and 

support that the hospital could afford, independent of any grants. The only possible anticipated issues 

are a changing of technology in the future that would cause this system to have to be replaced. There 

will also be the cost of replacement for the hardware not covered under the maintenance contract (e.g., 

desktop computers) due to normal wear and tear. In addition, with the American Recovery and 

Reinvestment Act (ARRA) funding and mandate for meaningful use, including networking, the expansion 

of this network to other CAHs and providers is a near certainty.
149

Hawaii

Project Title: An Electronic Health Record Implementation for Critical Access Hospitals (CAH) and Clinics 

in the Kauai Region of the Hawaii Health Systems Corporation (HHSC)

Project Summary: The purpose of the project was to: 

• Establish a model CAH and clinic EHR capability based on the Indian Health Service Resource and 

Patient Management System (IHS RPMS), a derivative of the VA vista system; 

• Implement the RPMS infrastructure with appropriate servers and end-user equipment in two 

CAHs (West Kaua`i Medical Center and Samuel Mahelona Memorial Hospital) and three 

associated clinics; 

• Implement security protocols and security software in RPMS to enable the HHSC Kaua`i Region 

CAHs and clinics with authorized access to patient records; 

• Develop application interfaces with patient management and billing systems to ensure adoption 

and improve the efficiency of the EHRs in Hawai`i; and 

• Evaluate the implementation and usability of the Kaua`i RPMS EHR to provide lessons learned 

for future RPMS implementations in Hawai`i CAHs and clinics.

Project Background: This pilot FLEX program application was submitted by the State of Hawai`i 

Department of Health (DOH), as a partnership with the Kaua`i Region of the Hawai`i Health Systems 

Corporation (HHSC), and the Telecommunications and Information Policy Group (TIPG) of the University 

of Hawai`i (UH). The Partners have been rigorously working over the past two years to develop a 

strategy to implement EHRs in CAHs and clinics to improve patient safety and quality, and to improve 

efficiency and effectiveness for reasons well-documented in national studies, reports, and executive 

orders. The DOH has been working as a partner with the HHSC and the University of Hawai`i to ensure a 

cost-effective and sustainable EHR capability.

Network Participants: The Hawai`i Health Systems Corporation is a public benefit corporation 

established by the State of Hawai`i in 1996. The HHSC is the largest health provider for the “Neighbor” 

Island residents and provides acute, rural, and long-term care services with 1,260 licensed beds on five 

islands. The hospitals of the HHSC have approximately 23,000 inpatient admissions annually. The sites 

included in the implementation were Kauai Veterans Memorial Hospital (KVMH), Samuel Mahelona 

Memorial Hospital (SMMH), and the West Kauai Medical Clinics (WKMC). 

Project Management: The Telecommunications and Information Policy Group (TIPG) of University of 

Hawaii led the project implementation and facilitated the communication between all other 

organizations involved. TIPG provided project oversight and formed groups and committees within the 

other organizations to lead various aspect of the project. TIPG also established a Cooperative 

Agreement with the Indian Health Service to provide consultative support and assistance to the project.

In addition, a Steering Committee, consisting of administrators and department heads from the facilities 

involved in the project, was formed to guide and manage the project from the facilities point of view. 

The primary function of the Steering Committee was to take responsibility for the feasibility, business 

case and the achievement of outcomes of the Kauai RPMS project. The Kauai RPMS Steering Committee 
150

monitored and reviewed the project status, as well as provided oversight of the project deliverable 

rollout. 

There were regularly scheduled meetings for the various teams to facilitate the communication and 

open communication and sharing of information to the facilities using tools such as Microsoft 

Sharepoint.

Goals: To improve health care, patient safety, and the efficiency of health care delivery through the 

cost-effective implementation of EHR and other HIT system in the rural communities in Hawaii by 

substantially implementing the RPMS in the Kauai region of the Hawaii Health Systems Corporation 

(HHSC).

Objectives: 

• Establish a model CAH and clinic EHR capability based on the Indian Health Service Resource and 

Patient Management System (IHS RPMS), a derivative of the VA vista system; 

• Implement the RPMS infrastructure with appropriate servers and end-user equipment in two 

cahs (West Kaua`i Medical Center and Samuel Mahelona Memorial Hospital) and three 

associated clinics; 

• Implement security protocols and security software in RPMS to enable the HHSC Kaua`i Region 

cahs and clinics with authorized access to patient records; 

• Develop application interfaces with patient management and billing systems to ensure adoption 

and improve the efficiency of the ehrs in Hawai`i; and 

• Evaluate the implementation and usability of the Kaua`i RPMS EHR to provide lessons learned 

for future RPMS implementations in Hawai`i cahs and clinics.

Expected Impact: A successful implementation was expected to result in improved quality of health 

care, patient safety, and cost-effectiveness of the Kauai CAHs and clinics. It was also anticipated that 

other CAHs in Hawaii would have a cost-effective model for EHR implementation, and CAHs nationwide 

will be able to learn from their experience.

Catchment Area and Target Population: The catchment area was the Island of Kauai; no specific target 

population was identified.

Evaluation: At the time of the final report, the evaluation was described as “in progress.” The grant 

application described the following evaluation activities:

• Formative Evaluation: life cycle systems development methodology

• Evaluation of Implementation Process: assessment of the implementation of both the core 

HOV implementation and the enhancements provided under this grant via satisfaction 

survey and personal interviews

• Evaluation of Clinical Measures: Health Disparities Collaborative measures as the primary 

method to track and report outpatient clinical measures. Secondary clinical measures will 

focus on the existing Health Plan Employer Data and Information Set and the Ambulatory 

Quality Alliance.
151

Project Status at End of Grant Period: The project has gone through several evolutionary processes 

since it was initially planned due to significant problems encountered during the course of this project 

which caused major delays in the implementation of the EHR system. First, it was initially planned to 

build upon an existing EHR infrastructure (Hui OpenVista), however the project has needed to make 

adjustments to build the entire infrastructure for the HER (using the Indian Health Service’s Resource 

and Patient Management System (RPMS)). These changes had a major impact on the project by shifting 

focuses, redirecting resources, impacting timelines, and adding a significant amount of work. If the 

existing EHR infrastructure was in place as anticipated we would not have suffered such major delays. 

An existing network communications infrastructure was in place which was necessary to support the 

project. Without the network communications infrastructure it would have been very difficult to 

implement the EHR network. 

The second significant issue that was encountered was the delay in open source release of the 

proprietary Graphical User Interface (GUI) for RPMS, VueCentric. Without the GUI the RPMS is not an 

acceptable system to be implemented in the HSC Kauai Region. The GUI was originally planned for its 

open source release in early January of 2009, however to date it has still not been released. Initial 

attempts to license the GUI software have been denied from the company, Medsphere, as they did not 

have a model to license the software outside of the Indian Health Service. With the inability to license 

the GUI and unavailability to get it via an open source method, it became questionable if the RPMS 

would be an EHR solution for the HHSC Kauai Region. Through discussions with the HIS, the project 

administrators are in the process of obtaining a royalty free license for the use of the RPMS GUI in the 

participating facilities.

Sustainability: The project will be sustained after CAHHIT funding has ended. Since the project is using 

open source software the ongoing costs have been minimized to an affordable level. The program will 

be sustained using operating funds of the facilities. The CAHHIT project has provided the significant 

funding required to begin the project and get the base system implemented. There are expansion 

projects planned to begin following the CAHHIT project with funding obtained though other grant 

programs. 
152

Illinois

Project Title: Patient Health Information Network (PHIN)

Project Summary: The purpose of the project was to implement electronic health records in the rural 

health clinics of two critical access hospitals; implement a picture archiving and communications system 

at one of the participating hospitals; implement health information exchange software; and identify the 

processes and activities that best supported technology implementation in the critical access hospital 

environment.

Project Background: The Patient Health Information Network project was designed to develop a 

replicable process for the implementation of electronic health records and health information exchange 

technologies in the critical access hospital environment. Another goal of the project was to overcome 

the limited selection of available vendor options for electronic health records for small, rural hospitals 

by modifying an ambulatory product for use in the inpatient setting of a critical access hospital, thus 

keeping costs to a minimum and making implementation of the electronic health record as streamlined 

as possible for the users.

Network Participants:

• Good Samaritan Hospital, a tertiary hospital with 141 beds.

• Washington County Hospital, a CAH, with 25 acute and swing beds, and 33 long-term care beds.

• Salem Township Hospital, a CAH with 25 acute and swing beds.

Project Management: Network management built upon the relationships that already existed between 

the participating hospitals, the Illinois Critical Access Hospital Network, and the Illinois Department of 

Public Health Center for Rural Health. The relationship among all was governed by a memorandum of 

understanding created before the grant proposal was submitted and that functioned without 

amendment throughout the project. Each entity maintained autonomy and decisions related to project 

implementation were agreed to by group consensus. Project management was provided by Illinois 

Critical Access Hospital Network staff, in an arms-length relationship with its member hospitals.

Project Goals: 

• Seamless continuum of care, improved coordination of care, reduction of medication 

complications, and improved patient safety

• Reduction in healthcare costs for patients, payers and providers as a result of the reduction in 

duplicative services

• Dissemination of results of HIT pilot project to CAHs

Project Objectives:

• Finalize HIT implementation plan (administrative operations, partner selection, establishment of 

baseline data for performance measures)

• Test and modify HIT implementation plan (secure staff, purchase and install hardware and 

software)

• Implement community wide secure portal for data exchange (identify secure data exchange 

provider, disseminate equipment and connectivity requirements)
153

• Document that HIT has improved clinical outcomes (develop five performance measures with 

national benchmarks)

• Implement community wide electronic health record

• Increase knowledge database of PHIN representatives

Expected Project Impact:

• Common EHR repository allows providers to more effectively treat patients, make a more 

accurate diagnosis, and develop a more effective treatment plan.

• Reduction in duplicative orders, automatic generation of recall reports and reminders to 

promote patient compliance with provider recommendations

• Reduction in healthcare costs that results from inefficiency, medical errors, inappropriate care, 

incomplete information, and duplicate testing

• Providers will be able to treat more patients without adding additional hours to their workday 

by having the complete record available during the patient encounter, use a template driven 

format to document their encounter, and electronic communication with pharmacies

• Functionality improved by defining scheduling templates, controlling over-bookings, resolving 

appointment conflicts, scheduling recurring appointments, and improving billing and collection 

procedures. More efficient workflow.

• Patient safety measures such as pharmaceutical recalls can be standardized and evaluated in a 

more quantitative fashion with actionable outcomes. 

Catchment Area and Target Population: The rural, low-income population of Washington, Marion and 

Jefferson Counties.

Evaluation: Performance measures focused on process – successful electronic health record software 

implementation and PACS installation – and on disease management performance measures for the 

multiple chronic diseases required by the project. The chronic disease management indicators 

addressed patients with diabetes and those with cardiovascular disease. Screening indicators were 

collected for breast cancer screening, prostate cancer screening. Influenza immunization prevalence 

indicators were evaluated.

Project Status at End of Grant Period: A process review of the project indicates a significant level of 

achievement by project participants and electronic health records implementation is supporting the 

activities described below.

• Centralized, enterprise-wide appointment scheduling is in use.

• Point-of-care clinical documentation is in use in the ambulatory rural health clinics operated by 

the hospitals.

• Point-of-care documentation is in use in the physical therapy departments of the hospitals. 

Project staff continue to work together to identify template modifications needed to more 

closely match the requirements of a hospital-based physical therapy department rather than the 

ambulatory-based documentation format of the software product.

• Project hospitals expect to participate in the development of a new inpatient product to be 

announced soon.
154

• Health information exchange software has been installed and tested. Electronic health records 

data are being pushed to the central data repository, using CCR/CCD standards, and have been 

accessed by other authorized users.

• The health information exchange software is self-hosted by Illinois Critical Access Hospital 

Network on a temporary basis until all testing is complete and additional users are identified. 

• Capabilities of the health information exchange software have been demonstrated to other 

Illinois Critical Access Hospital Network members in multiple Webinars and in-person 

presentations. Planning is underway to add another critical access hospital’s 11 medical 

practices to the health information exchange. Those practices, all members of a PHO, all use 

NextGen electronic health records. Following that implementation, three more hospitals that 

have expressed interest in joining the health information exchange will be connected. All use 

CPSI for their hospital information system and electronic health records, requiring the 

development of an interface to the health information exchange.

• Development of the necessary HL7 interface is underway to link the State of Illinois 

immunization database to the Illinois Critical Access Hospital Network’s health information 

exchange.

• Preliminary discussions have begun to possibly link our health information exchange to a similar 

exchange in a neighboring state.

• One of the clinical informatics specialists has met all requirements to be certified by the 

electronic health records software vendor at the highest level of proficiency.

The installation of a picture archiving and communications system at one of the project hospitals has 

been completed with the financial support of the project funds. The PACS at both hospitals will be 

interfaced to the health information exchange software.

A chronic disease management review process has been established that uses data collected from the 

electronic health records on the last workday of the month for each of the clinical indicators. The 

hospital’s Quality Improvement team will be responsible for the review and follow-up of the indicator 

values. The initial comparison of data selected from the electronic health records to the baseline data 

abstracted at the beginning of the project period identified inconsistencies, most likely due to the 

baseline having been developed from a one-month sampling of patient records while the first end-ofmonth report was based on total patient electronic health records created over a one year period. 

Future monthly reviews are expected to document improved outcomes.

The development of the inpatient electronic health record software is the most significant unresolved 

issue. However, the grantee expects that issue will be resolved by the participation of the project 

hospitals in a beta project with a software vendor. Several functionality issues remain with the health 

information exchange software but the vendor is working with the grantee and has expressed its 

commitment to address the issues. The grantee continues to have weekly calls with vendor 

programmers and administrative staff and feel confident the issues will be resolved.

Sustainability: The project is sustainable beyond the grant period. Program activities will continue at 

the close of the project and the health information exchange network will be expanded. The project 

hospitals have included within their annual budgets future software annual maintenance fees for the 

software purchased with grant funds. Both hospitals have increased staff to support the electronic 

health record use and its future expansion; infrastructure enhancements also have been budgeted.
155

The health information exchange network also will be expanded and the first new user will be 

connected soon. Costs associated with the software will be shared by all users. Other administrative 

costs will be built into a user fee, most likely on a subscription basis. The information exchange software 

is licensed to the Illinois Critical Access Hospital Network, which most likely will develop a distinct 

limited liability corporation to manage the health information exchange. 

Several grant opportunities will be pursued by the Illinois Critical Access Hospital Network related to 

supporting the adoption and use of health information technologies by its members: a statewide health 

information exchange development process has been implemented by the state’s Department of 

Healthcare and Family Services and the Illinois Critical Access Hospital Network is an invited participant 

in those developments.

Illinois Critical Access Hospital Network is participating in a consortium’s application for funds from the 

recently announced HITECH Health Information Technology Extension Program. If funded, this 

consortium will assist critical access hospitals and rural primary care professionals, among others 

statewide, implement electronic health records and participate in health information exchange 

opportunities. Illinois Critical Access Hospital Network’s experience in this CAHHIT project has prepared 

it for participation in this expanded effort.
156

Indiana

Project Title: Flex CAH-HIT Grant

Project Summary: The overall purpose of the project was to improve the safety, quality, efficiency, and 

effectiveness of health care provided in rural areas, and among rural providers. St. Vincent Health 

developed a project that would roll out electronic health record (EHR) technology into the outpatient 

primary care setting (Rural Health Clinics and one Federally Qualified Health Clinic), inpatient Critical 

Access Hospital setting, and connect electronically for records and results transfer to the tertiary care 

setting. The demonstration project results were instrumental in illustrating how to connect and deliver 

patient care results into the Indiana Health Information Exchange (IHIE) which is being recognized as one 

of the most successful HIE’s in the country.

Project Background: The literature on HIT delivery clearly indicates a significant shortage of technology 

penetration in rural health care clinic (outpatient) settings as well as rural critical access hospitals. 

Without dissemination of technology in rural settings, quality reporting and improvement efforts will be 

greatly delayed in comparison to their urban counterparts. Currently within all of the St. Vincent CAHs 

there are no electronic medical record or patient care management systems in place, only only isolated 

systems. The purpose of the project was to create a ‘hub’ to attach these spokes to.

Network Participants:

• St. Vincent Mercy Hospital: the CAH location in Elwood, IN. works closely with the FQHC 

in Elwood. There are several service agreements in place that include reduced rent, 

laboratory and radiology services at reduced rates, physician support services from the 

hospital, insurance and benefits for the FQHC administratively have all been offered 

between the CAH and FQHC locations. 

• Madison County Community Health Center: the FQHC location, with home office 

located in Anderson, IN. This FQHC also has a location in Elwood, IN. that works closely 

with the St. Vincent CAH location. It should be noted that this location is separately 

owned and operated from the St. Vincent Health operations. Allscripts was installed into 

the FQHC location and is submitting data to IHIE at the State level for the IHIE / results 

delivery component of the project. 

• St. Vincent Randolph Hospital: the CAH location in Winchester, IN. that also operates a 

RHC location next to the hospital. Allscripts was installed into the RHC location and the 

clinic is also submitting data to IHIE at the State level for the IHIE / results delivery 

component of the project. 

• St. Vincent Jennings Hospital: the CAH location in North Vernon, IN. also operates a RHC 

location within the hospital medical office building structure. Allscripts was installed into 

the RHC location and they are submitting data to IHIE at the State level for the IHIE / 

results delivery component of the project. 

• St. Vincent Indianapolis Hospital: this is the tertiary care hospital located in 

Indianapolis, which serves as the transfer partner for the CAH locations.
157

Project Management: In general, the CAH HIT project was managed through the St. Vincent 

Health project management office which manages all HIT related work effort across the 

delivery system. 

Goals:

• Serve as a viable CAH HIT demonstration project for HRSA and for the Indiana State Office of 

Rural Health

• Link information management systems of Emergency departments, Imaging and Radiology 

departments, Picture Archival and Communications (PACS), Pharmaceutical management, and 

Laboratory information management systems with the new clinical management system in each 

CAH

• Establish connectivity to the Indiana Health Information Exchange (IHIE) network to share 

patient vital patient vital quality of care to patients.

Objectives:

• Implement centralized clinical documentation that is easily accessible regardless of location the 

patient was originally admitted

• Facilitate easy access to patient chart for historic and previous admission information

• Provide an effective tool for care providers to record and measure objective and quality data 

related to a patient’s vital signs

• Provide an effective tool for care providers to record and measure objective and quality data 

related to a patient’s intake and output

• Provide an effective tool for care providers to view and graph lab results in a variety of ways for 

optimum patient safety and user ease

• Provide an effective tool for care providers to view transcribed documents

• Facilitate the transition of intake and output times

• Configure core admission components to create a patient list that incorporates all inpatient 

beds and locations with patient demographic data

• Replace specific non-physician clinical documentation currently captured on paper

• Provide availability to print out all clinical documents

• Design documentation headings and sort order within document review chart section

• Results interface for Lab, Radiology and Transcription

• Develop all ancillary results interfaces to send results to SCM

• Develop a clinical summary view to enhance current reporting process use by nursing and 

provide current patient information to appropriate clinicians

• Create and deliver quality training to staff prior to go live activation

• Configure and deploy core clinical documentation flow sheets for Vital Sign and I&O

• Additional high-level goals
158

Expected Impact:

Indiana State Office of Rural Health (SORH) anticipated the following improvements:

• Effectiveness of chronic disease management for primary care patients

• Performance on published measures of hospital quality and pay for performance

• Care coordination across the continuum of services

• Quality and safety of care provided in acute care settings

Saint Vincent Health anticipated the following improvements:

• Improving access to tertiary and quaternary health care services

• Integrating its health services in a manner that focuses on stewardship of its ministry dollars

• Assisting these rural locations and affiliated clinics with a higher level of care

• Overall increase of patient, physician, and associate satisfaction

• Improved communications between providers

• Decrease in patients seeking inappropriate care due to improved communications by the ED 

physicians, the CAH, and the participating clinics

• Decrease in known and unknown medical error incidents

• Decrease in duplication of testing throughout the organization

• Improved communications throughout the local primary care delivery system will lead to higher 

quality and a reduction in patient ‘hand-off’ errors

• Improved medication management systems

• Potential increase in patient volumes hat occur due to positive word of mouth by locally 

satisfied customers

• Reduction in pharmacy costs over time and improved safety

• Recruitment and retention of qualified health care providers

• Potential increase in clinical sites from area colleges

• Reduction of charges that lead to bad debt due to the reduction in duplication of testing

• Reduction of bad debt that is generated by inappropriate patient visits

• Reduction of supply overhead in many cases

• Cost shifting will occur versus dramatic cost reductions.

Catchment Area and Target Population: St. Vincent Health has a service area of approximately 50 

counties throughout central Indiana. The CAH HIT demonstration project was focused on a rural health 

clinic and CAH location in Randolph County market in eastern Indiana, a rural health clinic and CAH 

location in the Jennings County market in south-central Indiana, and a FQHC location in Elwood, Indiana 

that has clinical service linkages to a CAH location in Elwood, located within Madison County north of 

Indianapolis. 
159

Madison County is specifically noteworthy due to its’ economic challenges, unemployment 

rates, chronic disease prevalence, high levels of uninsured & under-insured, extremely high bad 

debt and charity care delivery levels. Upon completion of this project, it is believed that 95% of 

the county primary care and hospital providers were submitting patient data to IHIE, thus 

allowing physicians and hospitals to obtain this valuable information to help manage utilization, 

while attempting to reduce the extremely high cost of care to the residents of Madison County 

in Indiana.

Evaluation: The goals of the overall EHR design were to improve the effectiveness of chronic diseases 

management and preventive care for primary care patients, improve care coordination across the 

continuum of services, and reduce the potential for adverse drug events. To accomplish the goals of this 

project St. Vincent Health developed both a quantitative and qualitative evaluation model. Specific 

performance measures that were addressed included:

• Chronic Disease Management: Required Diabetes and Cardiovascular Disease Measures and 

Pharmacology Therapy for Pediatric Asthma;

• Preventive Care Measures: Adult Flu Vaccination Status, Pneumococcal vaccination;

• Care Coordination: Use of IHIE patient summaries by hospital and clinics;

• CPOE Adoption: Percent of inpatient orders entered by a physician; and,

• Medication Safety: Percent of unreconciled medications upon admission. 

As anticipated, St. Vincent was not able to demonstrate a significant amount of value upon completion 

of the project because not enough time had transpired for SVH to measure the results of the technology 

installation. For this reason, grant administrators sought to develop a balanced set of measures that 

either currently exist within the St. Vincent Health Critical Access Hospitals (CAH) or can be generated 

through post-implementation electronic reporting systems. Because of the relatively small number of 

patients seen in rural healthcare settings, they also focused on more process-based measures that have 

been demonstrated elsewhere to improve patient outcomes (e.g., measures of adoption and 

satisfaction, effectiveness of chronic disease management, coordination of care, performance on 

hospital-based quality of care measures, and improved quality and patient safety). The grantee states 

that “this should still provide credible evidence to CAH peers that the improvements associated with HIT 

are meaningful, yet not impose an undue burden on the collection of data to demonstrate reductions in 

mortality and significant morbidity.”

Project Status at End of Grant Period: According to the grantee’s final report, the project “was 

a successful implementation EHR software and integration of health care delivery in primary, 

critical access, and tertiary care providers.” St. Vincent Health met the project goals and 

deliverables. The data collected in this project demonstrate the success of the implementation 

and provide information on areas that need continued intervention. Continued monitoring of 

these measures and comparison with baseline rates will provide feedback on the success of the 

HIT systems and processes implemented.

Sustainability: The support funding was added to the operations budget for the next fiscal year. The 

grantee has also budgeted for the expense of expanding the network, and will further expand the EHR 

strategy for the entire network.
160
161

Louisiana

Project Title: LA Pointe Coupee HIT Network

Project Summary: The Pointe Coupee Parish health system has evolved without a strategic system 

design. The object of the project was to provide a system(s), which will allow access to patient data in a 

secure web enabled environment. The ultimate goal was the sharing of appropriate health information 

among all partner organizations. This will be accomplished by connecting all of the information systems 

and processes implemented in a secure, hosted environment. This will provide essential data, in an 

efficient manner, to the network’s health care providers. 

Project Background: The Pointe Coupee Parish health system has evolved without a strategic system 

design. In an attempt to bring “systemness” this CAH-HIT Network pilot program was proposed.

Network Participants: The Pointe Coupee HIT Network includes Pointe Coupee General Hospital, a 25-

bed CAH; the CAH’s transfer tertiary hospital, Our Lady of the Lake Regional Medical Center (the Lake), a 

740-bed community hospital located in Baton Rouge (approx 45 miles); and four local rural health 

clinics managed by the Lake--an FQHC with two sites in the parish (Innis Community Health Center); one 

local community clinic; two private practice primary care clinics; and one home health agency, Point 

Coupee Home Bound.

Project Management: The PCGH CAH HIT network can be described as an affiliation rather than a 

structured network. The CEO of the hospital is on the board of the Federally Qualified Health Center. 

Goals: The purpose of the proposed network is to fully implement functional electronic health records 

with practice management system capabilities within each network partner organization and enable 

appropriate health information exchange among all partner organizations.

Objectives:

• Develop capacity for business and patient information sharing and clinical decision support

• Ensure effectiveness of network infrastructure

• Efficient deployment of full scale network architecture

• Have a fully functional electronic information capture, tracking, reporting, and sharing network

Expected Impact: HIT systems that will enable the hospital and physician practices to exchange 

information, track patients, define weak areas in the chain of care will benefit the entire health care 

system and more importantly the patient population. Project administrators expected improved 

coordination of care, increased quality of care, and cost savings to the system. 

Catchment Area and Target Population: The Pointe Coupee HIT Network serves the Pointe Coupee 

Parish service area, which is located in south central Louisiana. Pointe Coupee’s land area is 557.4 

square miles. Agricultural production is a mainstay of the economy. Pointe Coupee is one of the most 

diverse agricultural parishes in the state. Close to 165,000 acres of land are used to farm cotton, 

sugarcane, soybeans, corn, milo, wheat, cattle, hay, vegetables, rice, crawfish and pecans (LSU 

Agricultural Center, 2007).

The 2006 estimated population for Pointe Coupee Parish is 22,648 (U.S. Census, 2007). The largest town 

in Pointe Coupee Parish is New Roads (home to Pointe Coupee General Hospital), with close to 5,000 

residents (U.S. Census, 2000). According to the 2000 U.S. Census estimates, 49% of the population is 

male; 61% is white, and 38% is black. Only 69% of the population 25 years and older has earned a high 
162

school diploma. The unemployment rate is high with only 55% of residents at least age 16 in the labor 

force. Consequently, 23% of all individuals and 19% of all families in the parish are below the poverty 

level (U.S. Census County Quick Facts, 2000). Over 40% of area residents receive some form of public 

assistance and 27% of the community is enrolled in Medicaid (Louisiana Medicaid Annual Report, 

2004/2005).

Evaluation: According to the grantee’s final report, an external evaluator is conducting the evaluation of 

the project. A final project evaluation report was to be submitted separately. The grant application 

described four main evaluation components: policy review and strategy, process review, program 

review, and health outcomes measures.

Project Status at End of Grant Period: At Pointe Coupee General Hospital, no major issues were 

encountered. The personnel quickly became involved and implemented on time and very well. 

However, the failure to engage one of the software vendors eliminated the home health agency data 

from being a part of the portal. In addition, the FQHC had a very difficult time and is still not 

implemented. They did not have nor acquire the resources necessary to do a timely implementation. 

Finally, after many attempts to meet with the the tertiary center, proect administrators were unable to 

obtain any level of participation.

Sustainability: Information not available.
163

Michigan

Project Title: Michigan’s Thumb Area Flex CAH HIT Network Implementation Grant

Project Summary: The primary purpose of Michigan’s FLEX CAH-HIT Network project was to create an 

effective health data exchange between two Critical Access Hospitals and their tertiary referral center, 

so that the electronic health records of a transferred patient could be shared electronically between the

referring physician in the 3 rural communities and specialty physicians at the tertiary hospital. It was 

anticipated that the project would improve the safety, quality, efficiency, and effectiveness of health 

care delivery through a full continuum of care. The anticipated results will be measurable 

improvements on the health status of local communities including improvements in: (a) the safety and 

quality of care delivery and patient’s care; (b) the effective delivery of health services and continuity of

patient care; and (c) operating efficiencies in terms of time saved, increases in revenue, increased 

tracking and/or reporting of patient’s quality and health outcomes.

Project Background: According to the Michigan Department of Community Health, the service area 

experiences high levels of hospitalizations for heart diseases, injuries, cancer, pneumonia, 

cerebrovascular diseases, and osteoarthrosis. The area has a higher-than-average rate of deaths due to 

diabetes. According to the region’s hospital officials, relevant barriers in the service area include: (a) at 

least a 70-minute travel time to the closest tertiary care services; (b) poor communications between 

primary and tertiary care professionals; and (c) poor access to needed specialty care services.

Network Participants: Two Critical Access Hospitals, Deckerville Community Hospital in Sanilac County, 

and Harbor Beach Community Hospital in Huron County were selected as the rural target sites to be 

electronically connected with a tertiary hospital partner. Although each hospital faced unique local 

challenges, they both shared common problems including: (a) Federal designation as a Critical Access 

Hospital; (b) both are located in Michigan’s rural “thumb” region; (c) both send patients requiring 

tertiary care services to the same regional medical center; and (d) both seek to develop common 

regional solutions to their local HIT, access, and quality-of-care issues and challenges. In the spring of 

2009, McKenzie Memorial Hospital in Sandusky was added to the project as a third Critical Access 

Hospital partner site. Mercy Hospital in Port Huron was selected as the tertiary partner for the CAH HIT 

Network. 

Project Management: The Thumb Rural Health Network (TRHN) provided overall long-term governance 

and direction to the project. TRHN is a 16-member organization located in the rural counties of Huron, 

Sanilac and Tuscola in Michigan’s Thumb region. TRHN’s membership includes seven CAHs and one 

sole-community hospital; the region’s two Health Departments; six tertiary hospitals serving the region; 

and one Multipurpose Collaborating Council. By using the TRHN as an oversight body for the project, 

the participants took advantage of an existing, successful regional collaborative body to: (a) provide 

project oversight; (b) communicate with other health care providers in the region; and (c) facilitate 

subsequent expansion of the HIE Network to other CAHs in the Thumb as well as other tertiary facilities. 

The TRHN also helped to identify other related HIE services that could generate a revenue stream to 

help support long-term sustainability of the HIE Network.

However, to date, the lack of an operational HIE Network has not required management and oversight 

of the Thumb Rural Health Network as originally planned. The Network has received progress reports 

and information concerning the project by the Principal Investigator quarterly. Once the HIE becomes 

operational, the TRHN is expected to exercise management of the Network including: (a) provide 

Network oversight; (b) communicate with other health care providers in the region; and (c) facilitate 

subsequent expansion of the HIE Network to other CAHs in the Thumb as well as other area tertiary 

facilities. 
164

Goals: The goal of the project was measurable improvements on the health status of local communities 

including improvements in: (a) the safety and quality of care delivery and patient’s care; (b) the effective 

delivery of health services and continuity of patient care; and (c) operating efficiencies in terms of time 

saved, increases in revenue, increased tracking and or reporting of patient’s quality and health 

outcomes.

Objectives:

• Use HIT as a tool to improve the safety, quality, efficiency, and effectiveness of health care 

delivery in the Thumb region of Michigan

• Adopt the effective use of a clinical information system through an integrated system with a 

common architecture

• Create sustainable business model for deploying HIT in the Thumb region

• Improve the quality and performance of our organizations both individually and jointly

• Improve healthcare quality through the elimination of handwritten clinical data

• Successfully deploy or enable the deployment of hospital HIT practices that reduce medical 

errors and improve overall patient safety

• Increase the identifications and reporting of medical errors and adverse events

• Develop HIT systems that support the regional collection and assessment of patient care data as 

part of the state-wide Michigan Critical Access Hospital Quality Network for the measurement of 

health care quality

• Enhance the Network’s ability to collect data regarding the impact of HIT on healthcare 

outcomes, improving patient safety and quality of care

• Increase the number of non-hospital providers in our local communities utilizing CPOE with 

CDSS and HER Systems and identify and support local and regional HIT collaborative projects 

that lead to standards-based data sharing across healthcare delivery sites

• Share their HIT implementation experiences with other organizations and networks.

Expected Impact: The CAH HIT Network was expected to produce measurable improvements on the 

health status of local communities including:

• Improvements in the safety and quality of care delivery and patient’s care.

• Improvements in the effective delivery of health services and continuity of patient care.

• Improvements in operating efficiencies in terms of time saved, increases in revenue, increased 

tracking and/or reporting of patient’s quality and health outcomes.

The project will be used as a template for adoption and/or expansion to Michigan’s other CAHs, as well 

as those in other States.

Catchment Area and Target Population: The targeted service area for the project is located on the 

eastern Lake Huron shoreline of Michigan’s Thumb, and included Deckerville Community Hospital in 

Sanilac County, and Harbor Beach Community Hospital in Huron County. The service area is rural with 

no freeway access to more populous areas and primarily agricultural (approximately 80% of the land 

devoted to agriculture). The 2006 average income of residents in the service area is nearly 20% less than 
165

the State average. This lower-than-average income is also reflected by the region’s higher levels of 

population living at or below 200% of the poverty level. 

In addition, the target service area has a high number of persons aged over 65 years; a higher-than 

average number of residents both Medicare and Medicaid eligible, and a higher-than-average level of 

unemployment. The two target counties are also designated as Medically Underserved and Health 

Professional Shortage Areas. According to the Michigan Department of Community Health, the service 

area experiences high levels of hospitalizations for heart diseases, injuries, cancer, pneumonia, 

cerebrovascular diseases, and osteoarthrosis. The area has a higher-than-average rate of deaths due to 

diabetes. 

Evaluation:

• Surveys were completed at the 3 hospital sites by December 2008. 

• The onsite Technology Consultant distributed pre-surveys to clinical staff at the 3 hospital sites 

by December 2008.

• Surveys, initially scheduled for distribution in February 2009, have been delayed pending 

completion of interfaces between the HIE and Mercy Hospital in late August 2009.

• Post-surveys cannot be completed until patient data is transferred. A complete project 

evaluation cannot be completed until patient data is transferred after October 2009.

Data collected-to-date consists primarily of “process outcome” measures concerning the successful 

accomplishment of implementation schedules as planned. In addition, pre-surveys of physicians and 

clinicians were collected at each hospital site in December of 2008 for use in comparison with Postsurveys to be conducted after the HIE becomes operational. At that time, the planned electronic 

transfer of patient records between facilities will become possible, and an evaluation of how the HIE 

Network impacts the transfer of patients between facilities, and the care they receive, will ensue.

Project Status at End of Grant Period:

As of August 31, 2009, the following projected outcomes have been achieved:

• The sharing of electronic health records between separate and unique health care providers to 

improve the safety, quality, efficiency, and effectiveness of health care delivery in the Thumb 

region of Michigan; 

• Adoption of the effective use of a clinical information system through an integrated HIE system 

with a common architecture; 

• Development of HIT systems that support the regional collection and assessment of patient care 

data as part of the Michigan Critical Access Hospital Program’s Quality Committee for the 

measurement of healthcare quality; 

• The ability to share the project’s HIT implementation experience with other organizations and 

networks.

As of August 31, 2009, the following projected outcomes have not been achieved:

• Creation of a sustainable business model for deploying HIE in the Thumb region; 

• Improvement in the quality and performance of rural health care organizations both individually 

and jointly; 

• Improvement in healthcare quality through the elimination of handwritten clinical data; 
166

• Successful deployment of hospital HIT practices that reduce medical errors and improve overall 

patient safety; 

• An increase in the identification and reporting of medical errors and adverse events; 

• Enhancements to the Network’s ability to collect data regarding the impact of HIT on healthcare 

outcomes, improving patient safety and quality of care.

While the grantee anticipates that the network will perform as expected, as of August 31, 2009, the 

network is not fully operational. However, the grantee noted that the Michigan project is the initiation 

of a HIE network and its existence does make a difference in the outcome of the project.

Sustainability: The operation and use of EHRs at each hospital site is now an integral part of each 

organization’s patient care delivery system. The ongoing cost of the Covisent HIE is being built into the 

existing insurer costs of transferring a patient between the local CAH and the tertiary referral center. 

The electronic patient data will be made available seamlessly within each facility’s EHR system. 

As additional CAH facilities in the Thumb Rural Healthcare Network establish their EHR systems and seek 

to electronically transfer patient information with a tertiary facility, they only need to have their EHR

vendor create the data interface with the HIE to access the Regional HIE Network. Coordination of the 

HIE Network is currently an integral component of the TRHN. The TRHN will continue to: (a) provide 

communications about the benefits of the HIE with other health care providers in the region; (b) 

facilitate subsequent expansion of the HIE Network to other CAH in the Thumb as well as other Tertiary 

facilities; and (c) identify other related HIE services that can generate a revenue steam to help support 

long-term sustainability of the HIE Network.

Minnesota

Project Title: Minnesota Flex CAH HIT Network Implementation Project

Project Summary: The grantee successfully implemented an electronic health record system among its 

three facilities, Madison Lutheran Home, Johnson Memorial Health Services, and Appleton Area Health 

Services, in order to create an integrated community health information system to allow patients served 

to travel seamlessly through the continuum of care and permit reporting of quality measurement data. 

Project Background: In 2007, the LqPHN conducted an HIT strategic planning process that provided a 

comprehensive blueprint for the selection and implementation of a system-wide electronic health 

record (EHR) system. The three organizations identified a need for patients in the three communities to 

move seamlessly through the continuum of care from emergency services to acute hospital-based care 

to assisted living and/or nursing home and referral to Rice Memorial Hospital in Willmar or other 

referring hospitals, such as Avera McKennan in Sioux Falls, South Dakota, or Level I Trauma Hospitals in 

the Minneapolis-St. Paul metropolitan area. It also identified a need to continue improving patient 

quality of care and safety by integrating outcomes reporting into its EMR environment. 

Collaboration on HIT implementation would allow the facilities to share hardware and software 

purchases, share consulting fees for HIT implementation and rollout, share IT staff positions and 

support, and achieve efficiencies. Finally, an HIT collaboration would support the practitioners in those 

communities as they experience the challenges of a major environmental, procedural and cultural 

change.
167

Network Participants: The Minnesota Office of Rural Health and Primary Care selected Lac qui Parle 

Health Network, an incorporated non-profit network of three Critical Access Hospitals in two 

neighboring counties in western Minnesota, for participation in the Flex CAH HIT Network Grant. The 

Lac qui Parle Health Network members - Johnson Memorial Health Services (JMHS), Dawson; Madison 

Lutheran Home (MLH), Madison; and Appleton Area Health Services (AAHS), Appleton – each 1) 

represent a full service health care delivery system in their communities, 2) have a successful history of 

collaboration in a number of areas, 3) had completed a six-month Health Information Technology 

strategic planning process, and 4) were close to selecting products and vendors from an identified list 

and were ready to implement a fully electronic health records environment across the organizations. 

The Lac qui Parle Health Network (LqPHN), a non-profit tax-exempt hospital services cooperative, 

incorporated in 1998 to assist its members in building and maintaining quality health services, consists 

of:

• Johnson Memorial Health Services (JMHS), Dawson, MN (Lac qui Parle County) – a community 

based hospital district made up of the following providers: Critical Access Hospital, Rural Health 

Clinic, nursing home, home care agency, and ambulance service.

• Madison Lutheran Home (MLH), Madison, MN (Lac qui Parle County) –a non-profit community 

based organization made up of the following providers: Critical Access Hospital, Rural Health Clinic, 

nursing home, assisted living housing, and senior residential facility.

• Appleton Area Health Services (AAHS), Appleton, MN (Swift County) – owned by the city of 

Appleton made up of the following providers: Critical Access Hospital, clinic, nursing home, assisted 

living housing, and senior residential facility.

Project Management: The Minnesota Office of Rural Health and Primary Care provided coordination 

and program guidance to project and distributed funds to sub-grantees. Stratis Health, Minnesota’s 

Medicare Quality Improvement Organization, provided consultatative planning services to the Lac qui 

Parle Health Network and to each of the Critical Access Hospitals in Dawson, Appleton and Madison to 

assist with final planning, selection of an EHR product and adoption and integration of health 

information technology and electronic health records (EHR). Stratis was brought into the project to 

emphasize planning and address the cultural shift necessary to help the organization(s) move toward 

the adoption of an electronic health record. They did this by adapting the structured DOQ-IT program 

that has been shown to be effective in helping clinics successfully adopt and integrate electronic health 

record systems to a small hospital setting.

Goals: To implemented an electronic health record system among the LqPHN’s three facilities, Madison 

Lutheran Home, Johnson Memorial Health Services, and Appleton Area Health Services, in order to 

create an integrated community health information system to allow patients served to travel seamlessly 

through the continuum of care and permit reporting of quality measurement data. 

Objectives:

• Completing a collaborative long range health information technology (HIT) strategic plan, 

including development of change management strategies.

• Verifying organizational readiness for electronic health record implementation.

• Selecting a vendor and electronic health record product and entering into contract.

• Developing an implementation plan, to include:

 Application design
168

 Infrastructure build

 Testing, and 

 Staff training

 Implementation. 

Expected Impact: It was determined that a strong, integrated Critical Access Hospital-based community 

health information system would allow patients served by these community health systems to travel 

seamlessly through the continuum of care, would build efficiencies and cost-savings by sharing of 

resources, and would build upon already-existing quality improvement programs in each of the facilities, 

enhancing their ability to utilize patient data to improve care and report outcomes. In addition, the 

LqPHN CAH-HIT network project was seen as a potential model that could be replicated and in other 

CAH-based hospital systems in communities in Minnesota and other states.

Catchment Area and Target Population: Located in a remote region of Southwest Central Minnesota, 

LqPHN’s primary service area in Lac qui Parle and Swift Counties is a federally designated underserved 

area. West Central Minnesota is similar to the Great Plains states with which it borders; an agricultural 

economic base with declining and aging populations. Lac qui Parle and Swift counties are typical of its 

surrounding rural counties and face the same challenges. 

Over 25 % of the approximate 20,000 people living in these two counties are over the age of 65. 

According to 2005 Minnesota Demographer estimates, the percentage of persons over 65 years of age 

in the service area and adjacent counties of Minnesota is higher than the State average. Of note, Lac qui 

Parle and Swift Counties have greater than 20% of the population over 65 years of age. 

Projected changes in the distribution of persons over the next twenty years within the state indicate the 

population of the proposed network service area will continue to decline, a reflection both aging 

population and declining economies. This aging population will be high users of healthcare services, yet 

the ability to finance and staff technologies and services will become increasingly more difficult.

Evaluation: With assistance from Stratis Health, LqPHN defined and developed processes to collect point 

of care patient data at the clinic, hospital and nursing home settings through collection and analysis of 

primary and secondary patient data for the purposes of improving patient outcomes, to include:

• Establishing, monitoring and analyzing performance indicators for:

• Heart failure

• Pneumonia

• Diabetes

• Cardiovascular care

• Adverse events

• Participating in Hospital Compare Reporting

At the time of the final report, LqPHN was in the process of phasing in the EHR product and was to begin 

measurement and analysis of quality indicators as the EHR is implemented. Stratis Health assisted 

LqPHN to evaluate and assess their quality measures and to develop a plan to adapt the measures for 

the new EHR environment. The Health Information Management (HIM) Directors from each 

organization were guided to facilitate an internal analysis, working with physicians, nurses, and other 
169

staff members to review and evaluate clinical practice guidelines, determine discrete data requirements, 

and map process to ensure the EHR will capture key data to enable LqPHN to more efficiently and 

effectively measure quality improvement activities.

In addition, LqPHN management submitted monthly progress reports to the Office of Rural Health and 

Primary Care (ORHPC) along with a final report of progress. One site visit was conducted midway 

through the project to observe joint facility steering committee activity and the process for evaluation of 

vendors. In addition, regular informal communication occurred between the ORHPC and LqPHN 

leadership. Stratis Health submitted mid-term and final progress reports. 

Project Status at End of Grant Period: LqPHN successfully implemented an electronic health record 

system among its three facilities, Madison Lutheran Home, Johnson Memorial Health Services, and

Appleton Area Health Services. Implementation of the long term care modules were delayed because of 

software delays product development by the vendor, and a home care software interface was delayed 

due to a scheduled major upgrade on the part of the home care vendor. Both of these installs are 

anticipated for full completion by the end of 2009 or the first quarter of 2010 at the latest.

Sustainability: The project will be sustained after CAHHIT funding has ended. The prior existence of 

LqPHN’s organizational network and prior HIT strategic planning work positioned the LqPHN members 

to build sustainability into their planning. Resources for shared IT staff and ongoing costs are 

incorporated into both the network’s and the individual health system’s long range planning. 

Stratis Health assisted LqPHN in developing a five-year total cost-of-ownership plan. This gave the 

network an insight to the cost savings achieved through a group purchase through the LqPHN. Charles 

River Consulting assistance with contract negotiation resulted in contract terms that will lock 

maintenance fees at the current rate for five years. In addition, because maintenance fees are 

calculated from the initial purchase price, LqPHN was able to negotiate a group discount because three 

hospital systems purchased the software at the same time. With a fully implemented interoperable 

medical record system in place, LqPHN hospitals have also positioned themselves as meaningful EHR 

users and thus able to capture Medicare and Medicaid HIT stimulus incentive payments.

LqPHN is willing to expand membership into the network and has held discussions with regional 

facilities. They are also considering joining a regional effort to establish a data center. A long range plan 

is to consider this option when it is time to upgrade current servers approximately five years from now.
170

North Dakota

Project Title: Flex CAH HIT Network Implementation Grant

Project Summary: The overall project goal was to facilitate the exchange of health information by 

implementing an interoperable EMR along the continuum of care that is patient-centered, facilitating 

patient safety, efficiency, and effectiveness of health care services. Three CAHs, one tertiary referral 

hospital, and ancillary providers including a community health center, will serve as the network model.

Project Background: At the time of the application, North Dakota had 31 critical access 

hospitals (CAHs), none of which was involved in a health information technology (HIT) network 

with its tertiary. Statewide data indicated that CAHs were supportive of the need and benefits 

of HIT; however, they struggled with available resources (e.g. financial and IT staff). At the time 

only 16% of ND CAHs had a formal HIT plan; yet, 68% were starting to budget for HIT. 

Additionally, 89% reported not yet using any form of electronic medical records.

Currently, the participating tertiary facility utilizes a clinical portal to allow dissemination of patient 

information from their facility to the CAHs, clinics, and other providers. There is no electronic 

transmission of patient information from the CAHs to the tertiary facility, nor between each other. 

Northwood Deaconess (CAH) has limited patient information sharing capabilities between itself and 

Valley Community Health Centers—a limited pilot-project was tested with mixed results that involved 

the sharing of laboratory test results with clinic providers, using an intranet-based web portal. 

Network Participants:

• Northwood Deaconess Health Center: a 12 bed CAH and 77 bed skilled nursing facility 

in Northwood, ND. Included in the original grant application. 

• Valley Community Health Center: a CHC co-located with Northwood Deaconess Health 

Center. Included in the original grant application. 

• First Care Health System: a 14 bed CAH and clinic in Park River, ND. Selected through a 

competitive application process upon receiving the HRSA grant. 

• Pembina County Memorial Hospital: a 15 bed CAH, clinic and long term care center in 

Cavalier, ND. Selected through a competitive application process upon receiving the 

HRSA grant. 

• Altru Health System: a 277 bed integrated health system in Grand Forks, ND that serves 

as the region’s tertiary referral center for approximately 20 CAHs. Included in the 

original grant application. 

Project Management: The project was managed by the ND Flex and State Office of Rural Health (SORH) 

Program Directors which was beneficial due to each person’s relationships with the facilities and 

knowledge of quality improvement and health information technology. Coordination of the project was 

to be the responsibility of the executive director of the North Region Health Alliance, a health 

consortium consisting of 20 health facilities in northwestern MN and northeastern ND. However, within 

the first few months of the project the realization was made that there were trust issues between this 

individual and the hospitals within the network as well as the PI’s observations related to lack of follow 

through and inclusiveness. The individual was spoken to and advised that a change of the coordination 

needed to be made in order for the project to be successful. The PIs assumed additional responsibility 
171

for coordination of project activities and asked the project’s CIO to increase his involvement from an IT 

perspective and work with all three CAHs, not just his own. This was negotiated and supported by the 

network participants.

Goals:

• Access quality health services. ND Flex facilitates CAHs’ ability to develop systems of care that 

are responsive to improving health status, monitoring patient outcomes, improving patient 

safety, and other central facets of quality focused access. 

• Health communication is a second primary focus of the NRHA CAH HIT Network through the 

development of a network approach to HIT and EMR function. 

Objectives (as stated in the grant application:

• to assist CAHs with the implementation of electronic medical records and the sharing of patient 

data with at least one ancillary facility within their respective communities; 

• facilitate the exchange of patient information (diabetes) between the CAH facility/ancillary and 

the regional tertiary center; and

• strengthen regional network to address HIT needs and issues.

Expected Impact: There are no other tertiary-CAH HIT networks in place in ND, so this will be the first 

true HIT network of its type in the State. The partners believe that this pilot is a significant opportunity 

and incentive to begin further testing of the benefits of HIT exchange in ND, and will serve to add depth 

to the State Rural Health Plan. The timeliness of this pilot project will strengthen the future planning of 

the ND Flex Program as it looks to further support other CAHs. 

This proposal focuses on two significant chronic conditions, diabetes and cardiovascular disease, putting 

in place the technology to gather baseline data as part of an overall HIT plan with the building blocks for 

evidenced-based practices to guide care delivery. The project will also improve safety and quality of care 

in the service area.

Catchment Area and Target Population: The intent of the project was to eventually impact the 

northeast region of North Dakota and northwest Minnesota consisting of 18 CAHs, 1 rural nonCAH, 1 tertiary facility and 1 mental health agency. The potential regional impact covered 

20,000 square miles serving a population of over 207,000 residents in 17 counties. In rural ND 

areas, a simultaneously shrinking, aging population, coupled with low population density, 

presents challenges in terms of an adequate workforce, health service options, and

infrastructure. ND ranks highest in the proportion of residents 85 and older.

Evaluation: A portion of the evaluation plan submitted in the original proposal was geared to the 

implementation of an integration engine which allowed access to a bi-directional clinical portal. This did 

not come to fruition, therefore the PIs revised the original goals to support the redirection of the project 

(i.e. changes to consultants, coordinator, partner roles, budget, etc.). Therefore, process outcomes were 

used to evaluate this project as originally described. The following activities were part of the overall 

implementation process. 

Activity Date Outcome 

1) A “kick off” meeting was held 

with all project partners. 

October 2007 23 - attended 
172

2) Staff from each CAH completed a 

survey to assess computer skills, 

attitudes about EMR (via 

SurveyMonkey) 

Nov-Dec 2007 All three CAHs completed the 

survey and were provided a report 

of the survey results to be used for 

developing training. 

3) Each CAH was required to submit 

an updated work plan to the 

PIs/program directors every two 

months. 

Every two months 

December, 2007-

February, 2009 

Information provided was used to 

engage project partners in 

problem solving discussions on 

monthly conference calls or oneon-one meetings. 

4) Monthly conference calls were 

held with CAH HIT Steering 

Committee in order to provide 

updates from each facility; problem 

solve as needed. 

Monthly Meeting minutes and roster of 

attendance were emailed out to 

all partners and retained on file 

with PIs. 

5) Each CAH was asked to track their 

use of Clinical Workstation to assess 

how it was used; examples of 

improved care, saved costs, etc. 

Three month 

period 

CW was accessed 31 times, for 

various information, within 3 

months. Only one facility followed 

up with this. 

5) The PIs decided it would be more 

useful for the CAHs and best use of 

grant funds to engage John Snow 

Inc. to conduct site-visits at each 

CAH in order to evaluate the success 

of EHR implementation at it relates 

to workflow efficiencies and 

improved data flow. 

January/February 

2009 

A site visit was conducted at each 

CAH; an individual analysis was 

completed and a report given to 

each CAH. In addition an overall 

report was compiled of common 

successes, challenges and lessons 

learned and provided to PIs and 

each CAH. 

6) A face-to-face “wrap up” meeting 

was held with all partners 

presenting their lessons learned to 

an audience of their peers 

throughout the state (in-person and 

over videoconference). 

March 2009 75 participants from 18 CAHs, and 

others attended 

8) The grantee was selected as one 

of four CA HIT projects to take part 

in the evaluation conducted by 

Altarum Institute – contracted with 

ORHP. As a result, each of the three 

CAHs were asked to complete a 

comprehensive three part survey 

designed by Altarum; in addition 

they also participated in an onsite 

visit/interviews. 

May 2009 Surveys completed/compiled; site 

visit completed; report will be 

completed by Altarum. 
173

Process Measures: A work plan was developed upon receipt of the award and provided to each 

of the CAHs. The expectation was for each to update their respective work plans every two 

months which they did. Process outcomes were tracked utilizing this format. Workflow analysis 

information also lends to outcome measures. John Snow Inc. was contracted with to assess this 

and provide the findings and recommendations with each CAH and an develop an overall report 

of common themes and recommendations.

Health Outcome Measures: Due to the limitation of the few months of actual EMR 

implementation for this grant project and the difficulties encountered with establishing a portal 

to share diabetes data across the continuum of care, therefore these measures were not 

collected at the end of the project. However, examples of adjustment/clarification of 

medications and treatment plans, reduced duplication of services, and potential cost savings 

were tracked and shared. Altru Diabetes Center staff worked in collaboration with rural clinical 

staff to discuss and agree upon “treat-to-target” indicators which were included on the 

electronic work sheet created and made available by clicking on a Regional Diabetes Project -

Tab on clinical workstation (CW). 

Average HbA1c for diabetic patients in the electronic patient registry (required measure) 

Patients with blood pressure < 130/80 (required measure) 

Patients with LDL < 100 mg/dl (required measure) 

Dilated eye exam in last year. (selected measure) 

Foot exam in last year (selected measure) 

Are you a smoker? Y/N 

Do you take aspirin? Y/N Start date 

Have you seen the nurse educator in the past 6 months? 

Have you seen the dietitian in the past 6 months? Y/N 

Use of Clinical Workstation (CW) tracking: All CAHs were asked to track their use of CW. (Only one CAH, 

First Care Health Center, Park River did so.) Over three months – CW was accessed 31(reported) times. 

Project Status at End of Grant Period:

The grant provided funding to support a portion of the cost to implement an EMR in three CAHs 

and their respective ancillary facility which in turn allowed providers access to health 

information for the rural residents of each service area. Patient information is now shared 

electronically within each rural facility; between the rural facilities (e.g. local clinics and nursing 

homes) and from the tertiary referral center back to the rural facility via a one-directional 

portal.

The network functioned as originally planned and the benefits of shared knowledge by working 

collaboratively made a difference in the successful outcome. The collaboration between the rural-torural and rural-to-urban improved as a result of the project and continues. The only network component 

change from the original plan was that of having the Executive Director of the network coordinate the 

project. This change did not impede the outcome of the project.
174

Sustainability: Each of health facilities involved in this project is committed to sustaining their 

EMRs into the future. The anticipated costs associated with maintaining their respective HIT 

systems has resulted in higher than expected HIT budget with one estimating an average cost of 

$50,000 per annum. Products are continually being updated and the needed upgrades are 

costly. Additional modules that enhance the EMR systems implemented through this project 

will be needed for CAHs to achieve the recommended HIMSS level 4 necessary for the exchange 

of health information which will be required to be eligible for the enhanced Medicare/Medicaid 

reimbursement to be made available through the HITECH ACT portion of the economic stimulus 

package.

Each of the CAHs continues its membership with North Region Health Alliance (NRHA) and their 

relationship with the tertiary facility, Altru Health System. The benefits of networking is well 

understood and recognized by each, and going forward the health facilities will negotiate what 

they need from their partners as the need and “fit” arises. 

Each of the CAHs have chosen a different server solution; one has a service agreement with 

Altru Health System which is very cost effective and involves 3-4 other CAH members of NRHA; 

a second originally worked with North Region Health Alliance to host their server but had 

difficulties with the maintenance, service, capacity, and compatibility with their EMR system 

which resulted in changing to their own server at additional unplanned expense; the third 

maintained their own server and continues to do so. 

Three different work groups resulted from this project: 1) HIT personnel from the region began 

meeting over the course of this project and continue to meet sharing information and 

brainstorming solutions from a regional perspective; 2) representatives from a number of CAHs 

throughout the state, including this project’s participants, have held calls (and continue to work 

together) to work on policies and procedures that address changes with the implementation of 

HIT and the electronic exchange of health information (this group is working with a 

representative from the ND Health Information Management Association (HIMA)); and 3) a 

diabetes work group was convened between the Altru Diabetes Center and the health care 

professionals from the rural facilities for the purpose of developing shared outcome indictors 

for tracking on CW. The first two groups have continued to meet. 

The UND Center for Rural Health continues to leverage funds (SORH/Flex, SHIP) to support 

planning and adoption of EMR in other facilities throughout the state which often requires 

networking as a framework to obtain those funds. The lessons learned from this particular 

project network (3 CAHs and one tertiary) has already expanded the “network” by sharing 

lessons learned with other CAHs within NRHA and outside of this network (i.e. state legislators 

during the 2009 Legislative Session, the ND HIT Steering Committee has and will use the 

information as the state HIT/HIE strategic plan is developed). 

Lastly, the SORH and Flex programs are currently working on the development of a technical 

assistance (TA) project for other CAHs using the lessons learned from this project. The TA 

project will provide the necessary tools to assist facilities to work through the necessary 

steps/activities to assess their readiness to implement EMR. This project intends to accelerate 

the readiness of ND healthcare facilities. 
175

The ND Legislature recently passed Senate Bill 2332 which will make available funds for a low 

interest loan program to purchase and upgrade EMR. However a certain level of readiness will 

be required in order to be eligible to access the loan funds. 
176

Nebraska

Project Title: NE Flex CAHHITN Implementation Grant

Project Summary: The main focus of the Nebraska Project was to share and exchange clinical health 

information along a continuum of care. This continuum involved the Thayer County Health Services (a 

Critical Access Hospital and physician’s clinic), five additional rural health clinics, one assisted living 

facility, one nursing home, three EMS ambulance units, an independent pharmacy, a hospital-owned 

retail pharmacy, a home health agency, and the network hospital (St. Elizabeth Regional Medical Center 

in Lincoln). The desired outcome of the project was to improve the quality, safety, efficiency, and 

effectiveness of the Thayer County Health Care system.

Project Background: This project was designed to meet the health needs of Thayer County and the 

surrounding communities in Nebraska and Kansas. The population of Thayer County is older and the 

mortality rates for many of the major health problems are considerably above the state average. For 

example, the heart disease death rate was 22 percent higher than the Nebraska average. The 

unintentional injury rate was 2.1 times higher and the motor vehicle death rate was 3.2 times above the 

Nebraska rate. The hospitalization rate of the county residents exceeded the Nebraska rate by 19 

percent. Almost 26 percent of the residents of Thayer County are over the age of 65, and about nine 

percent live in nursing homes. There are also access to care issues. A major area of concern is the 

growing number of uninsured and underinsured, which directly translates into increasing levels of 

uncompensated care. Thayer County has been a federally-designated health professional shortage area 

(HPSA) for several years. 

In terms of existing technology, the CAH has been successful in obtaining additional grants to support 

HIT projects. Recent investments include: online nurse charting for inpatient and outpatient services, 

patient scheduling for clinics and hospital departments, order entry for lab, radiology, and survey, online 

transcription, Pyxis Medication and supplies, computerized radiology, and financial packages for 

automated time clocks, payroll, accounts payable, general ledger, and materials management. 

Telehealth videoconferencing is available for interactive medical consultations. The Medi-Notes 

Electronic Medical Records program interface was recently implemented in the Hebron Rural Health 

Clinic, and will eventually link all rural health clinics, CAH, nursing homes, and pharmacy. 

Network Participants: The network included a critical access hospital (Thayer County Health Services), 

and ambulance services in Hebron, Carleton, and Alexandria, a nursing home in Hebron, assisted living 

facilities in Hebron and Deshler, five rural health clinics in Milligan, Deshler, Davenport, Bruning, and 

Chester, two pharmacies in Hebron, and the network hospital in Lincoln (St. Elizabeth Regional Medical 

Center).

Project Management: The project was led the administrator of TCHS. One of the keys to their success 

was the formation of a 9-member project committee. This committee met at least weekly to assure that 

the project was moving forward in a timely manner and to work with the vendors and other health 

professionals to lay out a vision, objectives, and action steps to complete each of the modules.

Goals: The goal of the project was to create an electronic linkage among health care providers that will 

transmit clinical information so that each provider can make informed treatment decisions based upon 

current patient data and information. 

Objectives:

• Improve the connection to share information with network tertiary medical center
177

• Build a bridge with EMS to improve communication and share information

• Improve communication and share clinical information with long-term care and assisted living 

facilities

• Improve connection to share information with satellite rural health clinics

• Improve electronic connection to pharmacies to strengthen patient safety, quality, 

effectiveness, and efficiency

• Improve efficiency and effectiveness of critical access hospital operations to strengthen patient 

quality and market position

• Address the safety and security of clinical health information

Expected Impact: The proposed project was expected to have a significant impact on the clinical and 

business operations of Thayer County Health Services. Clinically, it was anticipated that there would be: 

a reduction in medication errors and improved medication reconciliation for patients and residents 

through the implementation of electronic prescribing; improvement in the coordination of care and 

efficiency through access to electronic transmission of patient information among tertiary medical 

center and CAH and nursing homes; reduction in duplication and cost regarding radiology tests with the 

ability to transmit them electronically; and a decline in patient transfers and medical complications due 

to ability to discuss patient data in real time. It was also expected that the adoption of HIT will facilitate 

improvement of health outcomes for patients with diabetes, heart failure, and stroke and improve 

prevention by increasing and documenting influenza and pneumococcal immunizations. From a business 

perspective, the grantee expected a ten percent increase in patient days (at the CAH) because fewer 

patients will need to be transferred to the tertiary medical center. 

Catchment Area and Target Population: Hebron (Thayer County, a designated HPSA) and surrounding 

communities and Lincoln (Lancaster County). The population of the targeted area has higher than 

average heart disease mortality and unintentional injury rates, as well as higher hospitalization rates. 

The proportion of adults that are uninsured and underinsured continues to be a major problem as well. 

Also, about 26 percent of Thayer county residents are over the age of 65 and almost nine percent of 

them live in nursing homes.

Evaluation: The project was evaluated by the Nebraska Center for Rural Health Research at the 

University of Nebraska Medical Center. The evaluation focused on three main areas: (1) the 

effectiveness of the planning process between TCHS and its partners (i.e., a common vision, an 

agreement on the strategies, and the action steps in the work plan), (2) the accomplishments of the 

project (i.e., were all of the goals achieved?), and (3) the outcomes of the project. 

The Center concluded that the planning process produced strong collaborative partnerships and a 

common vision between them. All of the major decisions between the partners were made jointly, and 

all of the partners worked together to resolve the problems. 

As previously documented, all of the major goals of the project were achieved and clinical patient 

information can be exchanged along the continuum of care. While there were some challenges, such as 

the slow speed of the CPOE and resistance from some staff, all of the major challenges were eventually 

overcome. As the system matures, nearly all of the technical issues should be resolved and the changes 

in staff beliefs and behaviors concerning the system have already created a more positive change in 

culture.

Because this project was primarily implemented during an 18-month period, it was more difficult to 

evaluate the performance measures. However, there have been some very encouraging early results in 
178

the patient safety area. Since the project began, medication errors have declined from 48 per quarter to 

9 per quarter. Also, medication reconciliation has been 100 percent for those physicians who are using 

e-prescribing but it has ranged from 17 percent to 50 percent for those who do not. There is also 

evidence that this project has greatly contributed to a culture of high quality and performance. One of 

the goals of the project was to improve communication within TCHS. Results from the AHRQ Hospital 

Survey on Patient Safety Culture found that major improvements had been made between 2005 and 

2009. For example, in 2005, 46 percent of the employees that were surveyed responded positively to 

the question, “There are systems in place to ensure patient safety.” By 2009, 70 percent of the 

employees indicated a favorable response. Major changes were also observed in the following 

categories: (1) Manage actions promoting safety, (2) Non-punitive response to errors, (3) Support for 

safety by hospital management, and (4) Teamwork across hospital departments. 

The 2009 survey also revealed several areas of strength. Some of these include: 

Indicator Positive Response Rate

• Patient safety is never sacrificed to get more work done. 77%

• Our procedures and systems are good at preventing errors 

from happening. 

77%

• When a lot of work needs to be done quickly, we work 

together as a team to get the work done. 

92%

• Hospital management provides a work climate that 

promotes patient safety. 

90%

• The actions of hospital management show that patient 

safety is a top priority.

89%

• Hospital departments work well together to provide the 

best care for patients. 

85%

These results and other positive findings from the survey clearly show a positive patient safety culture, 

and their culture provides a solid foundation for making improvements in the quality of patient care.

This changing culture has also helped to reduce the resistance to change, which translates into better 

patient care. This resistance was greatly reduced by selecting the best leaders to serve on the project 

committee, allowing end users to make the decisions on the hardware and software they would be 

using, creating a separate room for training, and having nurses assisting the physicians in completing the 

electronic medical records and the CPOE. Reducing resistance to change minimized the “down time” and 

allowed the medical and hospital staff to spend more time with patients. 

There were also other changes which improved patient care. For example, the sharing of patient 

information with the physician hospitalists at the network hospital and using the 12-Lead EKG has made 
179

it easier to determine if a transfer is really needed. Using telehealth consultations has also helped to 

reduce transfers as well as improve the quality of care.

Major changes have occurred in the workflow. For example, ordering tests and medications online has 

led to a faster turnaround on lab tests, which should result in more positive patient satisfaction scores 

and greater efficiency in the billing of services.

With the electronic medical record, physicians are more likely to follow the evidence-based clinical 

guidelines. As a result, children and adults are likely to receive more timely immunizations. Women 

should receive more timely mammograms, and both men and women over 50 will be tested for colon 

cancer. 

Finally, the Nebraska Center for Rural Health Research also developed an analytical framework to 

calculate a return on investment (ROI) for CAH health information technology projects. They identified 

both quantifiable and nonquantifiable benefits. The quantifiable benefits included potential revenue 

increases such as increased patient encounters and coding and billing improvements. Other savings may 

be generated from reduced transcription time, a decrease in nurse documentation time, a reduction in 

lab tests and drug utilization, and increased staff efficiency. The nonquantifiable costs included (1) 

reducing medical errors, (2) increasing patient satisfaction (3) improving organizational efficiency, (4) 

improving clinical decision making, and (5) reducing malpractice and litigation. 

Several performance measures have been incorporated into the Clinical Quality Improvement Scorecard 

to measure improvements in the quality of care. This scorecard consists of over 150 measures for the 

following categories: (1) Diabetes, (2) Heart Failure, (3) Stroke, (4) Immunizations, (5) Medication Safety, 

and (6) Effectiveness of Communication. Within each category, there are indicators to measure the 

quality of care for the CAH, the rural health clinics, the nursing home, the home health agency, and the 

assisted living facility. Because of the short time frame, it was difficult to demonstrate progress in most 

of these areas. Also, the page limitations of this report prevented displaying all of these indicators.

However, the indicators for medication safety are shown in Table 1 for the CAH, the rural health clinics, 

the nursing home, and the pharmacy. 

Project Status at End of Grant Period: According to the grantee, this project “was extremely successful 

and achieved all of its major goals.” The components of the system and the network were implemented 

as planned, although extra time was needed to complete certain tasks (e.g., the CPOE system). Despite 

several changes that were needed to modify the software and hardware, the project committee was 

very successful in engaging staff, modifying the work plan, and keeping the project on target. The 

grantee reports that the project would not have been successful without the existence of the network. 

The network made it possible to improve patient care along the continuum. 

Sustainability: The grantee reports that the program will be sustainable because of the leadership and 

commitment that has been demonstrated both before and after the project began. TCHS made 

significant investments in both hardware and software prior to this project. In order to be sustainable, 

new investments are needed for both maintenance and upgrades. These costs will be reflected on their 

Medicare Cost Reports and depreciated over a period of time. By setting aside the money for 

depreciation, these funds can be invested in new technology. It is also possible to use some of the 

depreciation funds from their building projects. 

The network will continue to be expanded as other hospitals in the area become connected. The 

networks will also be expanded to include the BryanLGH hospital and the Nebraska Heart Institute in 

Lincoln. Eventually, the reference lab and mental health practitioners will be connected as well. 
180

Oklahoma

Project Title: Northeastern Oklahoma CAH EHR Network

Project Summary: The project was designed to create a Critical Access Hospital (CAH) Electronic Health 

Record (EHR) Network in Northeastern Oklahoma. The Network members’ vision for the EHR network 

was to ensure that patient clinical information would be easily accessible to providers within a 

healthcare organization and to other providers as patients migrate from ambulatory care to acute 

service delivery sites within the region. The improved access to patient information would be integrated 

into a coordinated system of care that would result in more effective and efficient health care delivery 

that would ultimately lead to improved safety and quality of care for patients.

Project Background: The partners were not part of a pre-existing network, but they have created one 

(the "Northeastern Oklahoma CAH EHR Network") for the purposes of this project. The project was new 

at the time of the application.

Network Participants: Bristow Medical Center, Drumright Regional Hospital, Fairfax Memorial Hospital, 

Holdenville General Hospital, and the OSU Medical Center (all located in northeastern Oklahoma). 

Project Management: OSU managed the project, which included forming committees and conducting 

system selections, as well as travel to other locations for site visits. After the selection process OSU 

worked closely with both implementation teams as well as the vendor’s project managers to ensure the 

hospitals were implementing the systems correctly, on time, and within budget. OSU-CHS helped 

provide guidance to the CAH’s to ensure their goals and expectations were met.

Goals: The goal of the project was the creation of an EHR network that ensures that patient clinical 

information is easily accessible to providers within a healthcare organization and to other providers as 

patients migrate from ambulatory care to acute service delivery sites within the region.

Objectives: 

• Create strategic plan to direct the activities of the Network and member hospitals

• Purchase an EHR system for each hospital that will meet its specific needs

• Train staff in preparation for implementation based on their identified needs

• Follow planning process to implement EHR at hospitals

• Provide technical support to ensure system is running smoothly

• Perform evaluation plan

Expected Impact: The network members expected improvement in the following areas:

Effectiveness

• Electronic records would allow for improved billing operations as it would be easier to identify 

billing errors and unbilled charges.

• Electronic records would enable users to improve coding for their patients visits, thereby 

decreasing time spent on correcting billing codes, which would improve revenue.

Efficiency
181

• Electronic records would save staff time on pulling charts and locating them when they are 

difficult to find.

• Electronic records would make it easier to file notes and medication refills in a patient’s chart 

without having to track down the chart each time there was an entry.

• Network members would save money on transcription fees and reduce transcription errors.

• Providers would be able to access information across systems of care within a network.

• Providers would be able to access patient information when they were off-site.

• The time saved on performing administrative tasks would lead to more time with patients.

Safety

• Electronic records would allow users to verify accurate scripting. As a result, users would be 

able to reduce medical errors and increase safety.

• Providers would receive point-of-care decision support when providing care to patients and 

utilize evidence-based clinical guidelines.

• It would be easier to track and deliver patient education materials.

• Providers would be able to limit the number of people who have access to personal health 

information by restricting access according to job function.

Quality

• Providers would be able to retrieve patient records and information quickly.

• Electronic records would allow users to verify formulary compliance. As a result, users would be 

able to reduce costs for prescribing non-formulary scripts.

• Electronic records would make it easier to administer preventive care to patients and be 

proactive with disease management.

• It is easier to target interventions to patients with specific diseases

Catchment Area and Target Population: The original service area of the project was northeastern 

Oklahoma. The participating hospitals in the project changed during the course of the project, but it 

would eventually include Bristow Medical Center, Drumright Regional Hospital, Fairfax Memorial 

Hospital, Holdenville General Hospital, and the OSU Medical Center (all located in northeastern 

Oklahoma). Because the project included the purchase of a Health Information Exchange (HIE), 

however, the potential scope of the project has increased significantly. Other hospitals within the state 

and region may elect to connect to the HIE and exchange information with one another, making the 

scope of the project at least statewide.

Evaluation: Data collection sheets were distributed to each participating hospital in an effort to evaluate 

how the project affected the following categories: Heart Failure, Pneumonia, Financial Measures, 

Diabetes, and Cardiovascular disease. The data were requested monthly from a period of 1 year prior to 

implementation of the HIT project to the most current data available. As of August 2009, this data 

request has only been completed by one hospital. The remaining hospitals, while aware of the need for 

evaluation, have not been able to provide the requested data to the grant administrators in a timely 

manner. In particular, recent financial problems with the management firm associated with Bristow and 

Fairfax have put this data gathering effort low on their list of priorities. Short term success will be 

viewed as improvements in any collected measures less than 1 year after implementation. Long-term 
182

performance measures will involve continuing to gather data after a full year of project implementation 

and reviewing trends in each of the observed categories. 

According to the grantee, the evaluation of the project hinges on having usable data from the hospitals.

The plan is to observe levels of the captured variables (Heart Failure, Pneumonia, Financial Measures, 

Diabetes, and Cardiovascular disease) prior to implementation of the HIT system, and compare these 

baseline measures with those taken as the project moved forward and finally post-implementation. For 

example, Drumright started their implementation in May 2008, and had the majority of their systems 

“live” by January 2009. Prior to May 2008, the grant administration obtained monthly averages for the 

various measures, and compared them to the same measures from June – December 2008 

(implementation) and Jan – July 2009 (post-implementation). Results to date are mixed, including 

significant improvements in performance for some measures (most pneumonia, outpatient volume, net 

revenue) but drop-offs in others (some heart failure, profit margin). These comparisons, along with 

discussions with hospital personnel about other factors that might have influenced the results during 

the time frames in question, will provide measures of project evaluation from both a quality and 

financial perspective. Data and discussions are needed from each hospital in order to complete this 

evaluation.

Project Status at End of Grant Period: Drumright Regional Hospital and Holdenville General Hospital 

both began the Flex project with no electronic clinical system, but did have electronic systems for billing. 

As a result of the Flex CAH HIT project, Drumright has now implemented their EMR system, including all 

acute care modules except for the emergency department and ambulatory systems. Holdenville has 

currently implemented all modules including clinical, billing and registration, except for the emergency 

department, prescription drugs and ambulatory modules. Holdenville plans to continue documenting 

emergency department visits on paper and will scan the note into the EMR, and are in the process of 

trying to interface their current prescription drug system with their EMR. The associated clinics were 

expected to go live with the clinical modules before July 2009. 

Fairfax Memorial Hospital and Bristow Memorial Hospital are both owned by the same management 

company – Community Partners. Both hospitals joined the Flex project part way through the period of 

performance (when Cleveland Area Hospital left the project). Fairfax and Bristow came into the project 

with some EMR modules in place, and decided to expand that functionality under the same vendor 

rather than start over, and both are in process, as of August 2009, of implementing the new EMR 

modules. 

As of August 2009, all hospital sites (CAH and ancillary) are in the process of determining what data will 

be shared within the HIE. Oklahoma State University Medical Center and OSU physician clinics currently 

do not have an EMR system that will directly connect to the HIE. One portion of the purchased HIE 

system is a portal that will query information from all the disparate systems, utilizing HL7 messaging, 

and allow physicians as well as clinical staff to view, update and send requests for information from 

other locations. OSU is in the process of negotiating the integration of other third party systems such as 

labs, e-prescribing, and payers into the HIE. This portal will allow physicians and staff real time access to 

patient data.

Sustainability: According to the grantee, one of the participating sites has increased its revenue by 

$10,000-$15,000 per month, which is “more than enough” to sustain an EMR system such as one of 

these in a CAH. The grantee further states that “as for the HIE it should be [sustainable] as well 

depending on the system chosen.” The goal of the grantee is to continue the expansion of the network. 

Vendors were chosen based on additional functionality that could be offered to rural facilities that do 

not currently have an EMR system. This offering will help the grantee sustain this network in the future.
183
184

South Carolina

Project Title: Lakelands Rural Health Network HIE

Project Summary: The purpose of this project was to acquire the necessary funding and technical 

resources to implement a regional health information exchange and quality improvement initiative in 

the Lakelands area of western South Carolina. 

Project Background: The Lakelands Rural Health Network (LRHN) is a nonprofit, multi-county vertical 

network that was developed in 2004 with the guidance and financial assistance of the South Carolina 

Office of Rural Health, the state FLEX grantee, to achieve efficiencies, expand access, coordinate and 

improve the quality of essential health care services, and strengthen the rural health care system as a 

whole.

Network Participants: The Lakelands Rural Health Network (LRHN) is a vertical network of health care 

providers LRHN was established in January 2004 and consists of a variety of partner organizations: a 

regional referral hospital, two Critical Access Hospitals (CAH), a Federally Qualified Health Center (FQHC) 

organization with nine medical practices, several Rural Health Clinics (RHC), a private foundation, a 

family practice residency program, the local public health agency, and the South Carolina Office of Rural 

Health. 

LRHN partners specific to this project are identified in the table below.

Name of Tertiary Hospital for the CAH Network Location Number of Beds

Self Regional Healthcare Greenwood, SC 421

Name of CAHs Location Number of Beds

Abbeville Area Medical Center Abbeville, SC 25

Edgefield County Hospital Edgefield, SC 25

Ancillary Providers Location Provider Type

Laurens County Health Care System Laurens, SC Acute care hospital

Carolina Health Centers

1. Calhoun Falls Family Practice

2. Lakelands Family Practice

3. Uptown Family Practice

4. The Children’s Center

5. McCormick Family Practice 

6. Saluda Family Practice

7. Ridge Spring Family Practice

Calhoun Falls, SC

Laurens, SC

Greenwood, SC

Greenwood, SC

FQHC

“ Family Practice

“ Family Practice

“ Family Practice

“ Pediatrics
185

8. Ware Shoals Family Practice

9. Ware Shoals Center for Family Medicine

McCormick, SC

Saluda,SC

Ridge Springs, SC

Ware Shoals, SC

Ware Shoals, SC

“ Family Practice

“ Family Practice

“ Family Practice

“ Family Practice

“ Family Practice

Montgomery Center for Family Medicine Greenwood, SC Family Practice and 

Residency Program

Riley Family Practice Saluda, SC RHC

Due West Family Medicine Abbeville, SC RHC

Family Healthcare Center Laurens, SC Family Practice 

Family Health Care Greenwood, SC Family Practice

# Annual Common Patient encounters 

expected between CAH-HIT Network 

Providers 250,000 (Shared Service Area)

# Total Network Provider FTEs 77.5 FTE

Software Licenses required for the 

project period

- CareEvolution Adapter License (one per facility)

- SQL Database General Use Licenses (one per processor)

- SSL Certificate License (one grouping per facility)

- Business Objects Query Reporting Tool (one per 

Project Management: South Carolina Office of Rural Health provided financial oversight to the project 

as well as served on its board. The LRHN network director helped coordinate meetings and organized 

policy and procedures that were developed.

Goals: The goals for this grant were: 1) To serve as the SC rural pilot site for an already developed health 

information exchange technology platform, that includes a personal health record; and 2) to establish a 

regional quality improvement program and reporting function within the health information exchange.

Objectives:

Expected Impact: 

• Establish a platform that will meet the growing needs of area physicians, non-profit and CHA, 

health care organizations, and empower service area residents to be active participants in their 

own health care

• LRHN Health Information Exchange (HIE), an ambulatory Electronic Medical Records expansion, 

or Community Electronic Health Record (CEHR) for area physicians

• $850,000 cost avoidance associated with purchase of an Enterprise Master Patient Index
186

• All authorized LRHN caregivers will have real-time access to dynamic health history for the 

target population

• Timely results and notification, patient reminders, etc. in clinical operations and improved 

patient satisfaction

• Dissemination of project progress: newsletters and websites of SC Office of Rural Health 

(SCORH) and South Carolina Rural Health Association, rural health, physician, and HIT-related 

conferences, through presentations to graduate public health students at the University of 

South Carolina, to other states through the National Organization of State Offices of Rural 

Health.

Catchment Area and Target Population: LRHN’s service area is Abbeville, Edgefield, Greenwood, 

Laurens, McCormick, and Saluda counties (in a rural, economically depressed region of western South 

Carolina). The LRHN target population is aging, increasingly diverse, under-educated, poor, and underemployed. The impact of chronic conditions on the health of the target population is significant and 

contributes to the leading causes of death and morbidity.

Evaluation:

According to the grantee, the project had many components and therefore any outcome achieved has 

multiple ways to evaluate:

• Edgefield Hospital had no significant IT structure so the purchase of a complete system with an 

implementation of a practice management system in 2009 electronic medical record next year 

would be considered a success, especially since they do not have a full time IT staff member.

• Abbeville Hospital had practice management software through CPSI but older servers. Their 

implementation of an electronic medical record from a practice management system to a full 

EMR system and then connection of that system to the HIE would be considered a success.

• Self Regional Healthcare and 3 of their outpatient clinics fully connected to the HIE was 

considered a success.

• Carolina Health Centers and the Self Regional Healthcare outpatient centers were involved in 

the diabetes collaborative. Their data from 2007 to 2008 showed significant improvement in 

NCQA standards for diabetes and early 2009 data shows that they reached the 75 point 

standard.

Project Status at End of Grant Period: According to the grantee, the network was able to function 

extremely well as planned. The network was designed to use as little hardware as possible – allowing for 

a much lower cost. All four initial adapters used only a set of redundant routers, switches, firewalls and 

IPS sensors. The de-identified server worked well to gather information. However, data from Misys 

was difficult to map and is still being worked on. Without the network, practices would not have been 

willing to combine and share quality improvement results.

Sustainability: According to the grantee, the LRHN does not create any revenue itself. If it is not 

expanded, it may be sustainable. There are yearly cost involved such as internet connectivity, personnel, 

hardware and network support, additional tapes, storage medium that would have to be absorbed by 

the participants in the program currently. Sustainability depends on proving to insurance companies 

that the network is saving them money by avoiding duplication of tests, improving healthcare through 

quality improvement efforts, and decreasing expensive workups by all physicians in the network due to 

better information concerning the patient.
187

Washington

Project Title: Bridging the Gap—Building HIT Links in Rural Washington State

Project Summary: This project set out to connect three CAHs and their respective ancillary facilities, all 

with disparate information systems, to each other and to the tertiary referral hospital in Seattle, 

Harborview Hospital, which is the only Level 1 Trauma Center in the state, through a health information 

exchange. This HIT project aimed to develop a universal data exchange/interfacing capability on multiple 

levels, and has the potential to serve as the universal EHR exchange platform that can be integrated into 

the statewide telehealth pipeline, thus making the platform scalable to CAHs and other providers 

statewide.

Project Background: The three CAHs are members of the HIT Implementation Network of Western 

Washington Rural Healthcare Collaborative (WWRHCC), and have a long history of collaboration, 

particularly in the areas of HIT and telemedicine. Since 2003, all WWRHCC hospitals have been members 

of the University of Washington’s Telehealth Network, and have been working together to develop HIT 

systems to support the delivery of specific specialty services through telemedicine. This is not their first 

technology endeavor. During 2005-2007 the WWRHCC implemented a teleradiology network.

Network Participants: The three CAHs in this project are Forks Community Hospital (FCH), Jefferson 

Healthcare (JH) and Morton General Hospital (MGH), all of which are Public Hospital Districts (PHDs) and 

are the sole healthcare providers in their districts (note: two of the three originally proposed CAHs had 

to be replaced). Each maintains at least one rural health center and JH has four. Harborview Medical 

Center (HMC) served as the urban referral center for this project. HMC is the only Level I trauma facility 

in the state.

Project Management: At the onset of the project, the grant administrator set up a steering committee 

that consisted of one member for each participating entity: or one per CAH, HMC, the Washington State 

Department of Health, and two from WWRHCC. The actual vendor selection occurred with WWRHCC 

since the other steering committee members (HMC WA DOH) while invited, did not participate in the 

site visits or the final decision-making process.

Goals:

• Develop a seamless patient information exchange capability among local rural providers for 

each CAH participant.

• Develop a seamless patient information exchange capacity to support rural trauma and 

inpatient consultations and transfers to Harborview Medical Center.

• After the grant project period, support the expansion of the project’s HIT system to other 

WWRHCC members, and to other rural hospitals statewide.

Objectives: 

• The FLEX CAH HIT project will give health professionals the ability to immediately access a single 

record containing all pertinent and current information belonging to an individual patient, and 

to make changes to that record.

• Reduce negative health outcomes of the ‘hand offs’ associated with non-HIT by providing an 

easily accessible and comprehensive vehicle for all information relating to a patient’s care.

• Eliminate delays in and duplication of care, and thereby reduce the costs associated with 

transferring and assembling information from disparate sources.

• Improve safety and reduce medical errors.

• Establish a telepharmacy capability and increase the availability of CAH pharmacists.
188

Expected Impact: The project will employ HIT as a tool to support patient safety and quality by making 

clinical information available to prevent misdiagnosis and incorrect treatments. It is efficient as it 

prevents duplicative lab and imaging studies and other ancillary procedures, and reduces staff time 

currently spent managing paper-based patient records and data produced by the wide variety of 

disparate information systems currently used across all care settings.

Catchment Area and Target Population: The three CAHs in this project are Forks Community 

Hospital (FCH), Jefferson Healthcare (JH) and Morton General Hospital (MGH), all of which are 

Public Hospital Districts (PHDs) and are the sole healthcare providers in their districts. Each 

maintains at least one RHC and JH has four. Harborview Medical Center is the only Level I 

trauma facility in the State.

Forks is the northwestern-most city in the contiguous United States and experiences on average 

12-14 feet of rain per year as part of the Olympic rainforest. There are two Native American 

tribes located on either side of Forks: the Quileute and Quinault. Neither has its own hospital 

and so they seek care from FCH. Forks is often “socked in” with fog and not accessible by air. 

Additionally, mud slides have isolated the area in the recent past, making it nearly impossible to 

transfer patients out of the hospital. Jefferson Healthcare is located in Port Townsend, a scenic 

port and maritime city. Its main facility also serves outlying areas as diverse in terrain as they 

are in population demographics. For both FCH and JH the fastest and easiest route by land to 

any urban facilities requires crossing the Hood Canal Bridge, which has been closed for 

retrofitting for more than three months in recent years. A ferry ride is required for land 

transportation to Seattle and HMC. Morton is located between Mt. St. Helens and Mt. Rainer. 

While picturesque, this location does pose inherent risks for the community. Like Forks, Morton 

experiences a great deal of fog and has been isolated due to flooding several times within the 

last two years. 

This project will serve approximately 166,500 people (FCH, JH and MGH). When the project is 

expanded to include all nine of the WWRHCC members this number will increase to over 

510,000.

Evaluation: According to the grantee, their project evaluation was the completion of the HIE; 

having all three CAHs sending information into the HIE and the training of HMC on the use of 

the system. Their intent was to have the system up and running by the end of March, which 

would have provided three months to evaluate the system. Unfortunately, the delays we 

encountered with MGH and the time it took to overcome those problems prevented them from 

achieving that goal. 

Performance measures for the long-term use of the HIE will include:

• The number of providers using the system at each site.

• A decrease in adverse drug events with the inception of 24/7 telepharmacy coverage.

• Decreased transfer time when transferring trauma patients from the three CAHs to HMC (no longer 

having to wait for documents to be copied).

• An increase in the total number of providers utilizing the HIE and (CAHs and eventually other urban 

facilities) joining the HIE.
189

• A decrease in medical errors related to “hand-off” from outpatient (RHC) to CAH; CAH to RHC; from 

CAH to HMC or other urban provider, and lastly, from the urban providers back to the RHCs.

Project Status at End of Grant Period: All three CAHs are sending information into the HIE and 

HMC staff have been trained on the use of the system; grant administrators are still 

determining how to provide access to the multitude of users (residents and attending 

physicians) at HMC without using all of their user licenses and overburdening those responsible 

for the management of users, which has thus far prevented the use of the HIE when 

transferring trauma patients.

While some feeds flow freely into the HIE from MGH, the transcription feed continues to be a problem. 

The grantee also was not able work out all of the issues related to telepharmacy prior to the end of this 

project, and are currently working on determining needs from all WWRHCC members to develop an 

absolute cost for this project. 

Sustainability: The three CAHs have agreed to pay for the costs of the T1 line when WWRHCC is unable 

to do so. There are maintenance fees that will be covered by a federal earmark, after which time the 

cost will have to be divided amongst those hospitals using the system. If other non-WWRHCC providers 

join the network they will be charged maintenance and professional fees.

In addition, the grantee is for ARRA funds through the state to expand the project. They also have a 

federal earmark which will provide some funds to assist with adding other WWRHCC members onto the 

HIE. The HIE vendor Orion is also helping the grantee look for additional funding.
190

Tennessee

Project Title: Middle Tennessee Rural Health Information Network (MTRHIN)

Project Summary: This project aimed to enhanced collaboration among health care providers within the 

Middle Tennessee Rural Health Information Network (MTRHIN), including three Critical Access Hospitals

(CAHs), one Federally Qualified Health Center (FQHC) and their referral hospital by supporting the 

electronic exchange of health information. The purpose of this project was to support the use of health 

information technology (HIT) as a tool to improve the quality of patient care and minimize health access 

issues, which in turn improves patient outcomes, decreases medical costs and hospital admissions. The 

goals included the adoption and effective use of HIT; the creation of sustainable business models for 

deploying HIT in Medicare Rural Hospital Flexibility (Flex) CAH networks; enhancing the ability of safety 

net providers to leverage initiatives and resources as well as improving quality and promoting 

performance improvement within the Flex program.

Project Background: At the time of application, Tennessee already had three regional HIE networks and 

four statewide HIE networks in operation, from which MTRHIN planned to utilize lessons learned and 

related experience. Tennessee is the only state that has electronically connected all local health 

departments into a central database of patient data, and offers an affordable and secured private 

broadband network already in place that connects all 95 counties. This network was to be used to 

economically connect all health information exchange participants in the MTRHIN network. 

Network Participants: The following organizations are the participants of the MTRHIN:

• Sumner Regional Medical Center, a tertiary hospital with 155 beds

• Macon County General Hospital, a CAH with 25 beds

• Riverview Regional Medical Center – South, a CAH with 25 beds

• Trousdale Medical Center, a CAH with 25 beds

• United Neighborhood Health Services, a Federally Qualified Health Center

Project Management: Community Health Network, (a group of ten Community Health Centers (CHC) in 

Tennessee and Mississippi which established CHN with the goal of creating shared information 

technology and other services for member organizations) has provided complete project management 

throughout the implementation phases of this project to include a director, assistant director, network 

administration, purchasing agent, financial oversight, training and daily administration of the HIE 

system. CHN worked with participating hospitals to develop a customized plan to fully deploy the 

MTRHIN and link those sites to a managed statewide network. CHN provides strategic planning support 

for hospital staff, and obtained the most competitive group purchase prices possible from software 

vendors. The network collaborated with hospital project teams to install and test the selected software, 

and train staff in its use. 

Goals: The goal of this project was to connect three rural non-profit Critical Access Hospitals (Trousdale 

Medical Center, Macon County General Hospital, and Riverview Regional Medical Center South) with the 
191

non-profit tertiary hospital, Sumner Regional Medical Center and a federally qualified health Clinic, 

United Neighborhood Health Services, Hartsville Family Clinic.

Objectives:

• Implement an integrated EHR network system at 3 CAH and one larger community hospital sites by 

June 30, 2010.

• Establish an EHR network system that provides for health record portability and that facilitates 

continuity of care for RHN patients across health providers and geographic settings (home, 

outpatient, inpatient, referrals, urgent, emergency), while at the same time protecting and ensuring 

patient privacy.

• To achieve outcomes at each hospital that result in:

- Improved patient satisfaction

- Improvement in patient safety and measurable quality

- Improvements in the ability of patients and providers to monitor health conditions

- Advanced care processes which include additional value of information to emergency 

department visits, ambulatory care, hospital discharge and admissions, and

- Impact on provider and statewide health delivery organizational effectiveness through 

productivity gains including changes in work patterns, modification of quality improvement 

processes and lower cost of date aggregation, analysis, and reporting.

Expected Impact:

• Hospitals will be able to track and report on all five performance indicators.

• The quality of patient care will improve as demonstrated by performance indicator improvements 

over the baseline data for each hospital.

• Improved safety and quality of patient care is expected as a result of the improved provider 

communications capability.

• Patient satisfaction scores with respect to each hospital’s service efficiency and patient-provider 

communication will increase.

Catchment Area and Target Population: The majority of the counties in the service area have been 

designated as federal shortage areas in terms of mental health services, dental care, medical 

professionals, and as medically underserved geographical areas and populations in general. The average 

rate of unemployment in the service area is 5.7 percent, while the average rate of unemployment 

nationally is 4.7 percent. The rural counties served by RHN are beleaguered by lack of industry which 

contributes to unemployment thereby increasing the numbers of low-income residents who are 

uninsured. Without income or insurance, these individuals are less likely to seek medical services until 

their health issues become life threatening, which then, sadly, requires extensive costs for care. In 

addition, communication between health professionals, service providers, and primary care facilities is 

limited because few of the hospitals in the more rural counties have updated electronic archiving and 

communication systems.

Evaluation: Base Camp online project management website was used as a reporting and communication 

resource to provide a one stop snapshot of the project progress to all participants at any one time. 

Milestones were created within base camp and assigned target completion dates and in turn actual 
192

completion dates. Bi-weekly conference calls among project participants were conducted to share 

information and provide a status report on milestones. Monthly in-person meeting were also conducted 

with the UMTRHN members to provide project updates. Additional evaluation included the meeting of 

HRSA-established milestones for the grant reports. 

The outcomes achieved include further development of the UMTRHN, implementation of a fully 

functioning HIE system and networking infrastructure in a rural setting; the collection of valid patient 

clinical information for use in rendering quality health care; an HIE system that provides the ability to 

effectively track quality health measures across a board area.

Project Status at End of Grant Period: According to the grantee, the network functioned as planned 

which is to provide better cooperation between respective service areas which, in turn, also assisted in 

the overall success of the project. As noted above, the outcomes achieved include further development 

of the UMTRHN, implementation of a fully functioning HIE system and networking infrastructure in a 

rural setting; the collection of valid patient clinical information for use in rendering quality health care; 

and an HIE system that provides the ability to effectively track quality health measures across a board 

area.

Sustainability: This program is sustainable due to plans to expand to other provider groups in the region 

and to duplicate the project in other rural areas of Tennessee. Funding is being sought for continuation 

and expansion. Participants in this pilot will be assessed appropriate fees for continued and future 

services.

Hardware and software for this project was purchased with the ability to allow expansion easily with 

little cost. Software vendor selection was made keeping in mind their stability and commitment to 

continued development of the product. A Memorandum of Agreement was developed and signed by 

each participant to provide funding for up to 2 years after the grant period ends. Opportunities for 

funding sources that will provide continuation and expansion are continuously evaluated. 
193

Texas

Project Title: Improving Texas Rural Community Healthcare through HIT Implementation

Project Summary: The goal and primary need for this project has been to improve health care delivery 

and quality of life through Health Information Technology (HIT) implementation within two rural 

communities as a demonstration model for all rural Texas communities. The objectives are to improve 

the safety, quality, efficiency and effectiveness of healthcare delivery through implementation of health 

information technology.

Project Background: The long distances in rural Texas make access to healthcare extremely difficult. It is 

felt that interoperable connectivity can serve as a means to decreasing some access barriers. 

Additionally connectivity has the ability to improve continuity of care for emergency situations and 

chronic diseases that is otherwise interrupted when patients move from one community to another. 

Without the ability to transport patient information from one location to another electronically many 

tests are repeated resulting in increased healthcare cost. Piecemeal health care, which is the norm 

without community connectivity, also impacts negatively on patient safety and quality. Electronic 

exchange of patient information would enhance disaster preparedness and the ability to provide care in 

crisis situations.

There wasn’t a formal network between the CAHs prior to this project. However, all the partners have 

signed MOUs among themselves. This project was their second collaborative effort. All the partners feel 

strongly committed to work together because they realize that the current climate indicates that no 

community will have adequate healthcare without network connectivity.

Network Participants: The two CAHs are Collingsworth General Hospital (16 beds) in Wellington and the 

Parmer County Community Hospital (25 beds) in Fiona, both located in the Texas panhandle. Northwest 

Texas Hospital in Amarillo is the tertiary referral partner. The network partners were picked as a result 

of geographic location, need, existing infrastructure, and referral structure.

Project Management: This grant was awarded to the Texas Office of Rural Community Affairs 

(ORCA). The Director of Rural Health for the Texas Office of Rural Community affairs served as 

the Principle Investigator. A member of the ORCA staff served as a project liaison and member 

of the project management team. An overarching management team was developed. This 

team composition included both ORCA staff members, the Texas A&M Health Science Center 

Project Manager, and the Regional Coordinator for Preferred Management Corporation. Both 

Hospitals selected for this project are affiliated with Preferred Management. The purpose of 

this team was to coordinate project oversight, ensure the Electronic Health Record (EHR) 

selection process was in alignment with the grant requirements and meet the goals of 

Preferred Management. In addition, the Rural and Community Health Institute (RCHI) of the 

Texas A & M Health Science Center was subcontracted to manage the project. The Texas A&M 

Research Foundation administered the funds and adhered to its policies and procedures for 

letting bids and subcontracts for the EHR, hardware and interfaces. 
194

Goals: The goal and primary need for this project has been to improve health care delivery and quality 

of life through Health Information Technology (HIT) implementation within two rural communities as a 

demonstration model for all rural Texas communities. 

Objectives: The objectives are to improve the safety, quality, efficiency and effectiveness of healthcare 

delivery through implementation of health information technology.

Expected Impact: ORCA hoped to show that this project addresses the service areas, creates 

opportunities for cost-sharing, and generates economies of scale. The partner centers and their clients 

expected to show the benefit of network activities and enhance the possibility of other CAHs modeling 

the same system. The network partners share a common vision of improving the health of the 

communities that they serve by providing vital health services and serving as agents for change in health 

care delivery.

Catchment Area and Target Population: This project impacts the two frontier communities of 

Wellington and Friona, located in the Texas panhandle. This region of west Texas is one of the largest 

medically underserved areas (MUA) in the U.S. In fact, the majority of Texas counties are designated as 

MUA’s. The two CAHs in the project are the Collingsworth General Hospital in Wellington and the 

Parmer County Community Hospital (PCCH) in Friona. Both of these community hospitals and local 

providers offer limited specialty care, forcing local residents to drive more than 70 miles from one 

community and more than 100 miles from the other, to access specialty care in Amarillo. 

The combined area of both counties served through this grant is 1,729 square miles. With a combined 

population of about 13,000, the population density is seven persons per square mile. These are both 

poor counties with a higher than average population over the age of 65. Transportation to medical 

facilities is a problem for many of the people. This area of Texas is more vulnerable to tornados than 

the average area of the U.S, 

Evaluation: The first phase (vendor selection) of the project was evaluated using data collected by an 

independent evaluator through document study and semi-structured interviews. The evaluation utilized 

mixed methods: quantitative and qualitative. Qualitative data include: semi-structured telephone 

interviews with ten participants in the project and document study. The Project Director composed a 

quantitative Information Technology Assessment survey that was administered to the two communities 

at the beginning of the project. The ten interviews captured process evaluation. The interviews 

included semi-structured questions tailored to the interviewee’s function, which were sent by email to 

each interviewee prior to the scheduled interview. The telephone interviews were audio-taped. The 

Evaluator utilized pattern coding to analyze and interpret the interview data. Essential documents were 

studied and compared. 

After the go live two members of the management team conducted an onsite evaluation. Due to the 

remote location of both of these hospitals the team chose to visit when the go live was complete at both 

sites. The Wellington site had been live for approximately one month at the time of the visit. Staff 

reported they were very pleased with the EHR and the implementation process. The grantee noted that 

the staff openly discussed how they had changed some processes, for the better, as a result of the tool. 

They also noted that while they were still learning they could see how beneficial automation will be. In 

particular the nurse responsible for abstracting discreet data elements for Core Measure reporting was 

very excited that the data would be pull electronically. 
195

The second location, Friona, went live four days prior to the grant completion date. They reported the 

go live went very well and that some of the lessons learned at Wellington were helpful in making their 

implementation smooth. 

Project Status at End of Grant Period: The grant provided both communities with the same EHR 

provided by OPUS Healthcare Solutions. The hospitals’ connectivity includes the billing, laboratory and 

patient scheduling functions. The clinic in Wellington is transmitting patient records to and from the 

hospital. The clinic in Friona is capable of transmitting patient records to and from the hospital, but was 

not engaged in that function at the expiration of funding because the administrator chose to implement 

the billing function first. The health record function will be utilized after the billing function is fully 

utilized and the staff is comfortable with the electronic tool. The telemedicine equipment has been 

installed and training for the use of telemedicine has been completed in both communities. In both 

communities, the system is currently capable of retrieving the CMS Core Measures. Information needed 

for public health can be retrieved, and the system can be used in times of disaster such as tornadoes. 

It should be noted that all the activities originally proposed were too numerous to be accomplished 

within the 18 months of funding. In particular the pharmacy component and extracting measures with 

data-set that is large enough to be meaningful. Accordingly, this change in plans was reported to HRSA. 

These activities will be completed after the ending grant date in Phases II and III of the grantee’s 

negotiated agreement with OPUS and the rural hospitals. 

Sustainability: According to the grantee, much thought was given to sustainability of this project in both 

communities at the time of planning and writing the grant proposal. At this time there was strong 

evidence that both communities would be able to sustain the project if the start-up expenses were 

covered such as they were with the grant funds. Sustainability was also a part of the vendor selection 

process. The vendor selected, OPUS, negotiated an agreement that included three years of 

sustainability. It is felt the communities should be able to plan accordingly and be prepared to assume 

cost upon the completion of this timeframe. 

According to the grantee, there is considerable evidence that the healthcare agencies in Wellington will 

continue connecting to and will utilize the EHR implemented during this project. This is predicated on 

the two young family physicians in this community who based their decision to practice in the 

community if electronic capabilities were provided. The community EMS has raised money to purchase 

equipment to connect to the EHR. Additional evidence is in the willingness of the hospital to pay more 

the $15,000 in overtime pay to employees to complete the implementation on schedule. The pharmacy 

and physicians has given evidence that they are excited about implementing e-prescribing. Utilization 

will ensure sustainability of the EHR given the financial viability of the community that was investigated 

during the planning of the grant proposal. 

Prospects for utilization of the EHR in Friona are less certain. The physicians in this community were a 

little more apprehensive about using electronic patient records. However, the hospital administrator 

has stated there is a great potential for using the EHR for quality improvement efforts. 
196

The future of telemedicine is also uncertain. While the physicians in Wellington have used the 

equipment the referral network by the telemedicine provider is different from the community practice 

and referral patterns. This has made using the equipment less favorable than it could be. The physicians 

in the community of Friona have been trained to use the equipment; however, it does not appear that 

they are yet ready to adopt this technology. 

The grantee stated that the team worked very effectively together and is definitely interested in 

expanding the network. Their only challenge is that neither ORCA nor RCHI as a university component, 

have the funds to cover the start-up. As a team they will research and seek additional grants to allow 

them to expand the network.
197

Virginia

Project Title: Virginia’s Medicare Rural Hospital Flexibility (FLEX) Program Critical Access Hospital Health 

Information Technology (HIT) Network Implementation Grant

Project Summary: Using the American Stroke Association’s “Stroke Continuum of Care” as an organizing 

framework, the VA Critical Access Health HIT (VA CAH-HIT) Network served as a test bed for the 

implementation of HIT interventions across the full stroke continuum of care within CAH service areas of 

the Virginia Acute Stroke Telehealth (VAST) statewide network infrastructure. The CAH-HIT grant 

funding was utilized to set up the first regional network for VAST. The specific focus of the program was 

to implement, test and evaluate a variety of information and telecommunication technologies to 

determine how technology could be leveraged to address functional requirements--particularly at the 

critical access hospital (CAH) level in highly rural areas of the state. The outcome of the effort was 

intended to not only set up a network in this region, but to develop a model to be leveraged statewide.

Project Background: Virginia’s incidence and mortality rates for stroke are among the highest in the 

nation. Research has found that situations similar to those found in VA can be partially attributed to a 

fragmentation in the delivery of healthcare, particularly lack of HIT (especially tools that promote 

collaboration and coordination between health care providers). Rural areas are often hit hardest by 

problems of fragmentation and access to care. In order to help resolve this fragmentation and enhance

the prevention, treatment, and ongoing care of stroke patients, the ASA has developed the “Stroke 

Continuum of Care” blueprint. VHD has integrated this approach into its statewide HIT initiative. 

Network Participants:

• Bath Community Hospital (BCH)- CAH located in Hot Springs, Virginia 

• Augusta Medical Center (AMC) - community hospital (BCH sponsor hospital)

• University of Virginia Medical Center (UVA) - tertiary academic medical center and certified Primary 

Stroke Center

Project Management: The Virginia Department of Health contracted with the Virginia Telehealth 

Network (VTN) for management and implementation of the CAH-HIT grant. VTN then established 

subcontracts with designated network and industry partners. 

Goals: The mission of the VA CAH-HIT Network is to reduce stroke risk and maximize the functional 

status and quality of life of Virginians with stroke, by systematically and comprehensively introducing 

targeted HIT applications into routine clinical practice in rural areas. The goal is to successfully deploy 

HIT solutions for each of the five areas of the Stroke Continuum of Care. 

Objectives: One primary objective for improving each area of the stroke continuum of care in the 

targeted service area had been identified for this project:

• Prevention: increase public awareness of the signs and symptoms of stroke
198

• EMS Notification and Response: Strengthen and expand the stroke training delivered to EMS 

providers

• Acute Treatment: Accelerate time to diagnosis and treatment for strokes

• Sub-acute care and secondary prevention: enhance capacity of health care professionals to 

recognize and treat deterioration after stroke, prevent complications occurring in the early poststroke period, and address relevant modifiable risk factors for heart disease and other 

cardiovascular diseases

• Rehabilitation: develop enhanced capacity for data sharing between healthcare providers to 

facilitate patient care after the patient is returned to the community for rehabilitation poststroke.

Expected Impact: Coordinated efforts amongst rural and urban health care systems can make 

considerable impact through the design and implementation of innovative interventions leveraging 

telecommunications and HIT capabilities to strengthen stroke systems of care.

Catchment Area and Target Population: The service area for the VAST test-bed spanned five counties in 

the Central Shenandoah region of Virginia. Bath Community Hospital (BCH) is one of seven CAHs in 

Virginia. BCH is a 25 bed, not for profit, CAH located in Bath County. Bath County is a rural, sparsely 

populated county of 4,964 people nestled in the Allegheny Mountains on the western central border of 

Virginia. BCH has several unique characteristics as compared to other CAHs in Virginia. BCH is the 

smallest and most isolated CAH in Virginia. BCH is located 90 minutes from the closest community 

hospital (Augusta Medical Center) and two hours from the University of Virginia Medical Center (a large 

academic medical center). Along with its primary population, BCH cares for a seasonal workforce and 

influx of out-of-town guests who visit a neighboring resort. It also benefits from foundation funding that 

offers opportunities to acquire technology and pursue innovative services.

Evaluation: An evaluation plan was established for the VAST demonstration. It covered the 

three main focus areas within the stroke continuum of care: community outreach, EMS 

response, and acute stroke care. Each area included at least one intervention and a 

combination of process and/or short-term outcome measures.
199

Component Intervention Evaluation Objectives

Community Outreach Development of a web 

portal for stroke 

professionals

• Document the deployment and 

usage of the web portal for stroke

• Document increased rural 

community education 

opportunities

EMS Response Deployment of a webbased learning 

management system (LMS) 

with modules on stroke 

recognition and treatment 

in the field

• Document baseline EMS stroke 

care in the region

• Assess the success of the LMS 

intervention (deployment, usage, 

satisfaction, stroke knowledge)

Acute Care Deployment of ITH’s 

Remote Presence - 7 (RP-7) 

mobile robotic 

telemedicine device

• Assess clinician satisfaction with 

RP-7 training, usability, user 

interface

• Assess regional impact on care

• Description of RP-7 usage 

• Assess the generalizability of this 

approach to other regions 

/institutions in Virginia 

Deployment of PACS 

system at BCH and 

interfaces with UVA

• Technical assessments of system 

performance (system logs, 

networking tests)

• Clinician acceptance/satisfaction 

with usability, image quality, etc. 

• Impact on workflow and 

communication

Project Status at End of Grant Period: 

Community Outreach: The stroke portal site was officially launched February 9th, 2009. The site features 

a comprehensive set of resources that span the entire continuum of stroke care. Site ownership 

transferred to the VSS Task Force at the end of February 2009. A subset of Task Force members is also in 

the process of developing protocols and policies for site modifications, content authoring, etc, and has 

received training in use of the CMS. A fully functional, comprehensive stroke portal was successfully 

delivered in accordance with the project goals, and the site is now available. 

EMS Response: 

• A new LMS system was deployed and operational, but has not yet officially launched, pending 

communication between the vendor and VDH to ensure that course participants are able to receive 

continuing education credit. 

• A new online stroke course was developed by CSEMS with input from domain experts. 
200

• Evaluation instruments were developed to measure pre-post stroke knowledge and user satisfaction 

with the LMS.

Acute Stroke-Remote Consultation with RP-7: 

• Description of the BCH and UVA experience with the RP-7 technology, including usage reports, 

assessments of satisfaction, usability, and impact on care by physicians

• Commitment to continued use of the RP-7 at both UVA and BCH, based on experience to date

• Description of specific challenges and lessons learned to assist other organizations in making 

decisions about deploying this kind of technology

Acute Stroke – PACS and Interfaces:

• Deployment of fully functional PACS system with substantial storage capacity and interfaces with 

UVA

• Descriptions of baseline workflows completed

• Post-deployment workflows described (but not yet fully tested)

The original goals for the evaluation of PACS implementation with interoperability with UVA systems 

included technical assessments of image transfer speeds, clinical satisfaction with the systems and 

image quality, and a comparison of pre- and post- workflows. The evaluation had to be modified as a 

result of delays in deploying the PACS and related interfaces and limited availability of specific technical 

data within the time frame required. From a process perspective, as noted above, the initial interface 

testing is near completion. 

Sustainability: All technology implemented by the CAH will be sustained beyond the grant. The 

VAST pilot was able to sufficiently demonstrate data and results that could be used for decisionmaking relative to long-term investments. At BCH, information was presented to the Board of 

Directors regarding outcomes and anticipated benefits to come for the residents of Bath County 

and the surrounding region it supports. The Board agreed to include in its budget the funding to 

maintain and sustain both the PACS and RP-7 for another year. Plans are underway to expand 

services delivered over the RP-7. The ability to utilize it more broadly will play a role in its 

longer term sustainability. 

Based on the success of the pilot using the RP-7 system at BCH, the UVA team submitted an application 

for competitive institutional support for a larger regional stroke telehealth program. This was exactly 

what was hoped for in terms of using the CAH-HIT grant to start the first regional network for VAST. At 

UVA, the telestroke program was based on research on various established business models, including 

that at Massachusetts General Hospital and the Michigan Stroke Network. The Stroke Telemedicine And 

Tele-education (STAT) program was awarded a competitive Buchanan Endowment Award effective on 

July, 1st 2009, with funding for three years. Financial independence is expected by the termination of 

the award. 
201

Two technical platforms will be offered to spoke hospital systems, so that they can choose either the 

standard (for example Polycom, or Tandberg) or the RP-7 videoconferencing system. In addition the UVA 

Neuroscience Initiative has funds budgeted too continue the RP-7 system in the institution. Broader uses 

will include placement at the new Long Term Care Facility, in the UVA ER, and potentially at the HS 

Rehabilitation Hospital, for remote care and staff education. 

At a statewide level, all data gathered from this pilot project will be provided to the Virginia Stroke 

Systems Task Force Committee members in their June 2009 quarterly meeting for full discussion. In 

addition, a separate workgroup within VSS TF on Stroke Telehealth has been designated to more 

critically analyze the experiences from this pilot CAH project in order to replicate the successes at other 

CAHs, improve upon the experiences, or to help in decision-making of how to best integrate RP-7, 

standard platform, and PACS system technology into the stroke systems of care within the state of 

Virginia.
202

Wisconsin

Project Title: Rural Wisconsin Health Cooperative Information Technology Network

Project Summary: The goal of the project was to implement an electronic health record in four CAHs , using a 

collaborative network model. Each of the sites implemented various EHR related applications The RWHC ITN 

Network is one of several networks in which multiple independent hospitals use the same HIS/EHR vendor, 

access their applications from a shared datacenter, and use a pooled staff to achieve common goals. However, 

the grantee believes that this project is the only such network comprised solely of critical access hospitals. 

Project Background: The RWHC ITN Project was designed to address a variety of critical access hospital (CAH) 

needs and problems. As has been discussed in several reports and studies, CAHs face a number of HIT adoption 

challenges, including lack of financing, lack of sufficient HIT professionals, and limited expertise to facilitate 

project management and workflow redesign. These challenges are especially pronounced for small-volume 

CAHs, since they generally have tighter margins and smaller proportions of staff to devote to HIT and EHR 

related activities. In many cases the result of these challenges is that CAHs are simply not structurally positioned 

to move forward with the technology implementations that can benefit their patients and that will be required 

to meet the demands of the changing healthcare landscape (including ARRA, value-based purchasing, and health 

reform). 

The Rural Wisconsin Health Cooperative Information Technology Network (RWHC ITN) was designed to make it 

possible for all CAHs, including the smallest and most disadvantaged, to mitigate these structural challenges. By 

pooling resources and working collaboratively to meet common goals through a network approach, the four 

CAHs that are members of the RWHC ITN have worked to: (1) reduce datacenter, hardware, software, 

implementation, support, and operating costs due to group volume purchasing and a shared data center model, 

(2) improve HIT expertise and support quality with a network HIT staff, (3) better engage the high-level 

challenges of information exchange and quality reporting requirements through collaboration, and (4) 

implement advanced EHR-related systems that may have been out of their reach without the network model. 

The RWHC ITN has also worked with members to help them implement systems to address a variety of more 

specific challenges and needs. 

Network Participants:

Boscobel Area Health Care is a 25 bed Critical Access Hospital that provides inpatient and outpatient services to 

approximately 17,000 rural residents in the following Wisconsin counties: Grant, Crawford, and Richland. The 

hospital’s service area is concentrated in the most southwest portion of the state, which is almost entirely 

agricultural. Demographic data reflect poverty and unemployment rates that exceed state averages. BAHC is 

located in a Health Professional Shortage Area. In 2008 BAHC had 1015 inpatient admissions and 10,395 

outpatient visits. 

Boscobel Area Health Care (BAHC): 

Memorial Hospital of Lafayette County (Darlington, WI) is a 25-bed Critical Access Hospital that provides 

inpatient and outpatient services to approximately 18,500 rural residents in the following Wisconsin counties: 

Memorial Hospital of Lafayette County (MHLC): 
203

Lafayette, Iowa and Green. Built in 1952, MHLC offers inpatient services in acute care, surgery, obstetrics, 

orthopedics, and cardiac care. In 2008, MHLC had 573 inpatient admissions and 2807 outpatient visits.

St. Joseph’s Community Health Services is a 25-bed Critical Access Hospital that provides inpatient and 

outpatient services to approximately 18,000 rural residents in the following Wisconsin counties: Vernon; 

Monroe; Juneau; Sauk; and Richland. Vernon County (including the city of Hillsboro and area surrounding St. 

Joseph’s Community Health Services) is designated a Health Professional Shortage Area. In addition to the 

hospital, SJCHS also operates three of the physician practice clinics that service the community: one contiguous 

to the hospital, one in Wonewoc, and one in Elroy, which has been designated as a medically underserved area. 

In 2008, SJCHS had 485 inpatient admissions and 26,275 outpatient visits. 

St. Joseph’s Community Health Services (SJCHS): 

Tomah Memorial Hospital is a 25-bed, Critical Access Hospital that provides inpatient and outpatient services for 

a population of approximately 20,000 rural residents in the following Wisconsin counties: Monroe and Juneau. 

TMH has a hospice program that covers a five county territory. The two closest hospitals are 20 miles and 34 

miles respectively, making their emergency room services especially vital to the community. In 2008, Tomah 

had 1062 inpatient admissions and 21,538 outpatient visits. 

Tomah Memorial Hospital (TMH): 

Project Management: The Rural Wisconsin Health Cooperative Information Technology Network (RWHC ITN) is 

the 501(c) 3 organization that provides the legal, collaborative structure for the member hospital participants, 

and it was established expressly for this purpose. The RWHC ITN Board is comprised of the CEOs of the member 

hospitals. The Board reviews and approves the annual budget, determines the course of action regarding 

financial and legal issues, and sets the strategic direction of the Network. The organization contracts with 

vendors for software and hardware, and hosts the hardware (such as servers and storage devices), which is 

shared between participants from datacenters in Madison and Sauk City. The RWHC ITN also contracts for the 

collaborative staff that helps implement and then supports the HIT systems with a 24/7 helpdesk and intensive 

onsite assistance. Through the facilitation of various workgroups, the RWHC ITN staff facilitates facility 

collaboration on best practices, goal setting, and decision-making. 

Goals: The goal of the project was to implement an electronic health record in four CAHs , using a 

collaborative network model.

Objectives:

• Drive improvements to patient safety and the quality of care and service

• Increase secure access to healthcare information

• Improve patient-provider relationships

• Increase healthcare cost effectiveness

• Eventually provide all Wisconsin CAHs with an integrated, cost-effective option to meet 

their EHR needs
204

Expected Impact: According to the grant application, the grantee expected business and clinical 

improvement to be facilitated in the following ways:

• Information fragmentation and unavailability will be significantly reduced

• Quality of patient care will be significantly improved through implementation of applications 

that facilitate the practice of evidence-based medicine and provide decision support tools

• Cost of HIT system investment will be significantly reduced due to shared datacenter model and 

negotiated discounts

• Implementation and support quality and appropriate use of HIT systems will be significantly 

enhanced due to shared staffing model

Catchment Area and Target Population: The project serves four CAHs (Boscobel Area Health Care, Memorial 

Hospital of Lafayette County, St Joseph’s Community Health Services, and Tomah Memorial Hospital) and their 

service communities in Crawford, Grant, Green, Iowa, Juneau, Lafayette, Monroe, Richland, Sauk, and Vernon 

counties of Wisconsin. 

Evaluation: Evaluation of the project occurred/occurs in three primary ways: (1) a balanced scorecard of 

measures is used to evaluate the effectiveness of the Network; (2) a separate QI evaluation process is used to 

measure the long term effectiveness of the implementations at the sites; and (3) short term success of the 

project was measured using a value-based ROI process conducted prior to and during the Altarum site visit.

1. ITN Balanced Scorecard: This scorecard is intended to provide a multifaceted view of how the Network 

is performing. It is divided it three categories: Financial, Customer, and Internal. Financial measures 

are: (1) Profit Margin Variance, and (2) % of Monthly Service Fees Current. Customer measures are: (1) 

Service Satisfaction; (2) Helpdesk Ticket Resolution Time; (3) Wide Area Network Uptime; and (4) System 

Availability. Internal measures are: (1) Staff Training; and (2) Strategic and Operational Objectives Met. 

Data has been collected on these measures for three quarters through 6/30/2009. 

2. QI Evaluation Metrics: These metrics were designed with input from the QI Directors of the 

participating facilities, and are intended to measure the long term quality ramifications of the 

implemented applications. The metrics include: (1) CVD Discharge Instruction Core Measure Scores; (2) 

Medication Error Rates; (3) Patient Satisfaction Relating to HIT; (4) RWHC ITN Network User Satisfaction 

Rates (same as in Balanced Scorecard); and (5) Progress toward Integrating HMS EHR Database with 

Core Measure Submission Process. Data has been collected on these measures for 2-4 quarters 

(depending on measure) through 6/30/2009.

3. Altarum Value Based ROI Evaluation: This evaluation tool collected responses from facility 

representatives relating to the following issues: (1) How did grant activities change and/or improve 

workflow? (2) Were systems implemented easy to use? (3) Was training and support effective? (4) What 

was the level of user commitment? (5) How was user feedback sought and used? (6) How did the project 

impact patient satisfaction? (7) How did the project make information more available? (8) Were the 

systems utilized? (9) What non-financial values were derived from the new systems? (10) What were 

the efficiency benefits of the new systems? (11) How did the new systems support positive clinical 

outcomes? The project PI and eventually the Altarum evaluators interviewed stakeholders at each of 

the 4 sites (first separately and then all in one group) to solicit answers to these questions. 
205

Project Status at End of Grant Period: The Network was able to function as initially planned. The one major 

adjustment from initial planning was that four, rather than two, CAHs decided they wanted to participate in the 

project. Since additional participation creates additional value for all (as infrastructure costs are proportionately 

reduced), the grantee expanded the scope of the project to include all four CAHs. 

Sustainability: The project will be sustained once CAHHIT funding has ended. The grantee developed a formal 

business plan, facility service fee schedules to pay for the associated network costs over five years, and legal 

service agreements to ensure payment. After five years, facility fees are reduced, since Network loans will be 

paid off. The services provided by the Network are required for the facilities to operate, so the Network is not 

something that facilities would “turn off” unless after five years they decided to pay significantly more to selfprovide the services.

At the time of the submission of the final report, four new facilities were considering the ITN Network as an 

option to help them achieve “meaningful use” as required by ARRA. The ITN is working with these facilities to 

provide vendor demos, onsite visits to existing facilities, and cost modeling and network orientation sessions. 

The ITN “welcomes and anticipates expansion.” Each added facility would reduce the cost of Network operation 

by 5-10 percent.