Electronic Health Records: Outcome-Oriented Metrics and Goals Needed to Gauge DOD's and VA's Progress in Achieving Interoperability
Released: 2015-08-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: DOD and VA operate two of the nation's largest health care systems, serving approximately 16 million veterans and active duty service members and their beneficiaries, at a cost of more than $100 billion a year. For almost two decades, the departments have been engaged in various efforts to advance DOD and VA electronic health record interoperability. Among their most recent efforts, the DOD and VA Secretaries have committed the departments to achieving interoperability between their separate electronic health record systems.

The Consolidated Appropriations Act , 2014, and accompanying Joint Explanatory Statement, included a provision for GAO to review the departments' efforts. GAO evaluated the actions taken by DOD, VA, and the IPO to plan for and measure the progress toward achieving interoperability between the departments' electronic health record systems. GAO reviewed relevant program documents and interviewed agency officials.

section_title: What GAO Found
paragraphs: The Departments of Defense (DOD) and Veterans Affairs (VA), with guidance from the Interagency Program Office (IPO) that is tasked with facilitating the departments' efforts to share health information, have taken actions to increase interoperability between their electronic health record systems. Among other things, DOD and VA have initiated work focused on near-term objectives, including standardizing their existing health data and making them viewable by both departments' clinicians in an integrated format. The departments also have developed longer-term plans to modernize their respective electronic health record systems. For its part, the IPO has issued guidance outlining the technical approach for achieving interoperability between the departments' systems.

Even with the actions taken, DOD and VA did not, by the October 1, 2014, deadline established in the National Defense Authorization Act (NDAA) for Fiscal Year 2014 for compliance with national data standards, certify that all health care data in their systems complied with national standards and were computable in real time. Both departments stated that they intend to do so later in calendar year 2015. Further, the departments' system modernization plans identify a number of key activities to be implemented beyond December 31, 2016—the deadline established in the NDAA for the two departments to deploy modernized electronic health record software to support clinicians while ensuring full standards-based interoperability. Specifically, DOD has issued plans and announced the contract award for acquiring a modernized system to include interoperability capabilities across military operations. In addition, VA has issued plans describing an incremental approach to modernizing its existing electronic health records system. These plans—if implemented as currently described—indicate that deployment of the new systems with interoperability capabilities will not be completed across the departments until after 2018.

The IPO has taken steps to develop process metrics intended to monitor progress related to the data standardization and exchange of health information consistent with its responsibilities. For example, it has issued guidance that calls for tracking metrics, such as the percentage of data domains within the departments' current health information systems that are mapped to national standards. However, the office has not yet specified outcome-oriented metrics and established related goals that are important to gauging the impact that interoperability capabilities have on improving health care services for shared patients. IPO officials said this work is ongoing and that a team is working with DOD, VA, and subject matter experts to identify metrics that would provide more meaningful measures of the impact of increased interoperability. However, the IPO has not identified a time frame for when this team will report its results and when the IPO plans to incorporate these metrics and goals into its guidance. Without ensuring that outcome-oriented metrics and related goals are defined and incorporated into the current approach, the departments and the IPO will not be positioned to assess and report on the status of interoperability-related activities and determine areas that need improvement.

section_title: What GAO Recommends
paragraphs: GAO recommends that DOD and VA, working with the IPO, establish a time frame for identifying outcome-oriented metrics; define related goals to provide a basis for assessing and reporting on the status of interoperability; and update IPO guidance to reflect the metrics and goals identified. DOD and VA concurred with GAO's recommendations.

section_title: Letter
paragraphs: The Department of Defense (DOD) and the Department of Veterans  Affairs (VA) operate two of the nation’s largest health care systems,  providing care to approximately 6.3 million veterans and 9.6 million active  duty service members and their beneficiaries at estimated annual costs of  about $53 billion and $49 billion, respectively. Among these are more  than 5 million shared patients who receive health care and services from  both departments.

DOD and VA have long recognized the importance of electronic health  records as an essential part of delivering quality care to veterans and  service members. Thus, for almost two decades, the departments have  been engaged in various efforts to advance electronic health record  interoperability—intended to provide seamless sharing of health care data  and make patient information more readily available to health care  providers, reduce medical errors, and streamline administrative  functions.

Among their most recent efforts, the departments have stated that they  plan to rely on separate systems to achieve an interoperable electronic  health record. Toward this end, DOD is undertaking the procurement of a  new electronic health record system as part of the Defense Healthcare  Management System Modernization (DHMSM) program, whereas VA  plans to evolve and modernize its existing system as part of its Veterans  Health Information Systems and Technology Architecture (VistA)  Evolution program. The DOD/VA Interagency Program Office (IPO) currently chartered and tasked to facilitate the departments’ efforts by  identifying interoperability standards and coordinating activities to  increase the sharing of health information between the two systems.

The fiscal year 2008 NDAA also called for DOD and VA to set up an interagency program  office, which was to be a single point of accountability for their efforts to implement these  systems or capabilities.

In light of the departments’ not having yet implemented a solution that  allows for seamless electronic sharing of medical health care data, the  National Defense Authorization Act (NDAA) for Fiscal Year 2014  included requirements pertaining to the implementation, design, and  planning for interoperability between DOD’s and VA’s electronic health  record systems. Among other things, the departments were each directed  to (1) ensure that all health care data contained in DOD’s Armed Forces  Health Longitudinal Technology Application (AHLTA) and VA’s VistA  systems complied with national standards and were computable in real  time by October 1, 2014, and (2) deploy modernized electronic health  record software to support clinicians while ensuring full standards-based  interoperability by December 31, 2016.

In addition, the Consolidated Appropriations Act, 2014 accompanying Joint Explanatory Statement included a provision for GAO  to conduct a study of the two departments’ pursuit of interoperable  electronic health record systems. Our objective was to evaluate the  actions taken by DOD, VA, and the IPO to plan for and measure the  progress toward achieving interoperability between the departments’  electronic health record systems.

Pub. L. No. 113-76, 128 Stat. 5, Div. C, Title VI, 128 Stat. 101; Div. J, Title II, 128 Stat.  453 (Jan. 17, 2014). measurement.IPO Acting Director, Deputy Director, and other DOD and VA program  officials to understand current efforts related to improving interoperability  between the departments. Appendix I describes our objectives, scope,  and methodology in greater detail.

Further, we interviewed cognizant officials, such as the  We conducted this performance audit from September 2014 to August  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective.
subsections: section_title: Background
paragraphs: Historically, patient health information has been scattered across paper  records kept by many different caregivers in many different locations,  making it difficult for a clinician to access all of a patient’s health  information at the time of care. Lacking access to these critical data, a  clinician may be challenged in making the most informed decisions on  treatment options, potentially putting the patient’s health at risk. The use  of technology to electronically collect, store, retrieve, and transfer clinical,  administrative, and financial health information has the potential to  improve the quality and efficiency of health care.

Electronic health records are particularly crucial for optimizing the health  care provided to military personnel and veterans. While in active military  status and later as veterans, many DOD and VA personnel, along with  their family members, tend to be highly mobile and may have health  records residing at multiple medical facilities within and outside the United  States.

Each department operates separate electronic health record systems that  they rely on to create and manage patient health information. In  particular, DOD currently relies on AHLTA, which comprises multiple  legacy medical information systems that were developed from  commercial software products and customized for specific uses. For  example, the Composite Health Care System (CHCS), which was  formerly DOD’s primary health information system, is used to capture  information related to pharmacy, radiology, and laboratory order  management. In addition, the department uses Essentris (also called the  Clinical Information System), a commercial health information system  customized to support inpatient treatment at military medical facilities.

For its part, VA currently uses its integrated medical information system— VistA—which was developed in-house by VA clinicians and information  technology (IT) personnel. The system consists of 104 separate computer  applications, including 56 health provider applications; 19 management  and financial applications; 8 registration, enrollment, and eligibility  applications; 5 health data applications; and 3 information and education  applications.
subsections: section_title: Interoperability: An Overview
paragraphs: The sharing of health information among organizations is especially  important because the health care system is highly fragmented, with care  and services provided in multiple settings, such as physician offices and  hospitals, that may not be able to coordinate patient medical care  records. Thus, a means for sharing information among providers, such as  between DOD’s and VA’s health care systems, is achieving  interoperability.

The Office of the National Coordinator for Health IT (ONC), within the  Department of Health and Human Services, has issued draft guidance,describing interoperability as:  1.  the ability of systems to exchange electronic health information and  2.  the ability to use the electronic health information that has been  exchanged from other systems without special effort on the part of the  user.

Similarly, the fiscal year 2014 NDAA defines interoperability, as used in  the provision governing the departments’ electronic health records, as  “the ability of different electronic health records systems or software to  meaningfully exchange information in real time and provide useful results  to one or more systems.” Thus, in these contexts, interoperability allows  patients’ electronic health information to be available from provider to  provider, regardless of where the information originated.

Achieving interoperability depends on, among other things, the use of  agreed-upon health data standards to ensure that information can be  shared and used. If electronic health records conform to interoperability  standards, they potentially can be created, managed, and consulted by  authorized clinicians and staff across more than one health care  organization, thus providing patients and their caregivers the information  needed for optimal care. Information that is electronically exchanged from  one provider to another must adhere to the same standards in order to be  interpreted and used in electronic health records, thereby permitting  interoperability. In the health IT field, standards may govern areas  ranging from technical issues, such as file types and interchange  systems, to content issues, such as medical terminology. On a national  level, ONC has been assigned responsibility for identifying health data  standards and technical specifications for electronic health record  technology and overseeing the certification of this technology.

In addition to exchanging the information, systems must be able to use  the information that is exchanged. Thus, electronic health record  technology has the potential to improve the quality of care that patients  receive and to reduce health care costs, if the technology is used in a way  that improves providers’ and patients’ access to critical information. For  example, with interoperability, medical providers have the ability to query  data from other sources while managing chronically ill patients,  regardless of geography or the network on which the data resides.

section_title: DOD and VA Have a Long History of Efforts to Achieve Electronic Health Record Interoperability
paragraphs: Since 1998, DOD and VA have relied on a patchwork of initiatives  involving their health information systems to exchange information and to  increase electronic health record interoperability. These have included  initiatives to share viewable data in existing (legacy) systems; link and  share computable data between the departments’ updated health data  repositories; develop a virtual lifetime electronic health record to enable  private sector interoperability; implement IT capabilities for the first joint  federal health care center; and jointly develop a single integrated system.  Table 1 provides a brief description of the history of these various  initiatives.

In addition to the initiatives mentioned in table 1, DOD and VA previously  responded to provisions in the fiscal year 2008 NDAA directing the  departments to jointly develop and implement fully interoperable  electronic health record systems or capabilities in 2009.called for the departments to set up an interagency program office to be a  single point of accountability for their efforts to implement these systems  or capabilities by the September 30, 2009, deadline. In January 2009, the  IPO completed its charter, articulating, among other things, its mission  and functions with respect to attaining interoperable electronic health  data.

The act also  Further, the departments’ Interagency Clinical Informatics Boardestablished the following six interoperability objectives for meeting the  departments’ data-sharing needs and facilitating compliance with the  fiscal year 2008 NDAA: demonstrate initial network gateway operation, expand questionnaires and self-assessment tools, expand Essentris in DOD, and demonstrate initial document scanning between the departments.  refine social history data,  share physical exam data,  The departments’ officials, including the co-chairs of the group  responsible for representing the clinician user community, asserted that  they had met the priorities established by the Interagency Clinical  Informatics Board and, in conjunction with capabilities previously attained  (e.g., the Federal Health Information Exchange and the Bidirectional  Health Information Exchange), had met the deadline for achieving full  interoperability as required by the act. Nonetheless, in prior reviews, we  have identified a number of challenges the departments have faced in  managing their efforts in response to the fiscal year 2008 NDAA and to  address their common health IT needs.

While these initiatives, collectively, have yielded increased data sharing in  various capacities, we previously reported that they nonetheless  experienced persistent management challenges and did not result in the  fully interoperable electronic health record capabilities that the  departments had long sought. We have also noted that the manner in  which DOD and VA reported progress toward achieving interoperability  lacked results-oriented (i.e., objective, quantifiable, and measurable)  goals. Specifically, we noted that the departmental plans lacked  associated performance goals and measures that are a necessary basis  to provide the departments and their stakeholders with a comprehensive  picture to effectively manage their progress toward realizing increased  interoperability.
subsections: section_title: DOD and VA Committed to a Joint System to Address Common Business Needs
paragraphs: In March 2011, the Secretaries of Defense and Veterans Affairs  committed the two departments to the development of a new, joint  integrated electronic health record (iEHR) system. Further, in May 2012,  they announced their goal of implementing the integrated health record  across both departments by 2017. According to program documentation,  pursuing iEHR was expected to enable the departments to align  resources and investments with common business needs and programs,  resulting in a platform that would replace the two departments’ separate  electronic health record systems with a common system. In addition,  because it would involve both departments using the same system, this  approach was expected to largely sidestep the challenges they had  historically encountered in trying to achieve interoperability between  separate systems.

Toward this end, initial development plans called for the single, joint iEHR  system to consist of 54 clinical capabilities that would be delivered in six  increments between 2014 and 2017, with all existing applications in VistA  and AHLTA continuing uninterrupted until full delivery of the new  capabilities. The initiative was to deliver several common infrastructure  components—an enterprise architecture;user interface; data centers; and system interface and data exchange  standards. The system was to be primarily built by purchasing  commercially available solutions for joint use, with noncommercial  solutions developed or adopted only when a commercial alternative was  unavailable.

However, in February 2013, about 2 years after initiating iEHR, the  departments’ Secretaries announced changes to their approach— essentially abandoning their effort to develop a single, integrated  electronic health record system for both departments. This decision  resulted from an assessment of the iEHR program that the Secretaries  had requested in December 2012 because of their concerns about the  program facing challenges in meeting deadlines, costing too much, and  taking too long to deliver capabilities. The IPO reported spending about  $564 million on iEHR between October 2011 and June 2013.

section_title: Departments Return to Two- System Approach with a Need to Ensure Interoperability
paragraphs: In place of the iEHR initiative, DOD determined that it would buy a  commercially available system to replace its existing AHLTA system and  VA decided that it would modernize its existing VistA health information  system. In this regard, DOD is pursuing the acquisition of a replacement  system for its multiple legacy electronic health record systems under a  new program—the DHMSM program. For its part, VA intends to enhance  and modernize its existing VistA system under a new program called  VistA Evolution. The departments indicated that they would ensure  interoperability between their two new systems, and with other public and  private health care providers.

In December 2013, the IPO was re-chartered and recognized as the  single point of accountability in the development and implementation of  electronic health records systems or capabilities that allow for full  interoperability of health care information between DOD and VA.  According to the IPO charter, the office is responsible for, among other  things, establishing technical and clinical standards and processes to  ensure integration of health data between the two departments and other  public and private health care providers. Further, it is to monitor and  report on the departments’ progress in implementing the technical  standards during the development of their respective systems. It is also to  coordinate with the departments to ensure that advances in interoperable  capabilities enhance the quality, safety, efficiency, and effectiveness of  health care services.

While the departments have chosen their current approach to modernize  two separate systems, we have previously reported that they did not  substantiate their claims that the current approach would be less  expensive and faster than the single-system approach. Further, we have  noted that the departments’ efforts to modernize their two separate  systems were duplicative. We stressed that major investment  decisions—including terminating or significantly restructuring an ongoing  program—should be justified using analyses that compare the costs and  schedules of alternative proposals.

Accordingly, we recommended that DOD and VA develop a cost and  schedule estimate for their current approach from the perspective of both  departments that includes the estimated cost and schedule of VA’s VistA  Evolution program, DOD’s DHMSM program, and the departments’ joint  efforts to achieve interoperability between the two systems, and then  compare the cost and schedule estimates of the current and previous  (i.e., single-system) approaches and, if applicable, provide a rationale for  pursuing a more costly or time-consuming approach. The departments  agreed with our recommendation and stated that, while initial  comparisons indicated that their current approach would be more cost  effective, they would continue to refine cost estimates as part of both  departments’ acquisition programs.

section_title: DOD, VA, and the IPO Have Taken Steps toward Achieving Interoperability, but All Key Activities Will Not Be Completed until Future Years, and Plans Lack Outcome- Oriented Metrics and Goals
paragraphs: DOD, VA, and the IPO have undertaken various activities aimed at  achieving interoperability between the two departments’ electronic health  record systems. In this regard, DOD and VA have initiated work focused  on near-term objectives including standardizing their existing health data  and making them viewable by both departments’ clinicians in an  integrated format. The departments have also developed plans related to  their efforts to modernize their respective electronic health record  systems. Further, the IPO has issued guidance outlining the technical  approach for achieving interoperable capabilities between the  departments’ systems. Even with the actions taken, however, the two  departments did not, by the October 2014 deadline established in the  fiscal year 2014 NDAA for compliance with national data standards,  certify that all health care data in their systems complied with national  standards and were computable in real time. Further, the departments’  system modernization plans identify a number of key activities to be  implemented beyond the 2016 deadline established in the act, suggesting  that deployment of the new systems with interoperability capabilities will  not be completed across the departments until years later. In addition,  while the IPO has begun steps to measure and report on the progress of  the exchange of health data, its efforts have not included the use of  outcome-oriented metrics and established goals essential to gauging the  extent to which interoperability is being delivered and having an impact on  improving health outcomes.
subsections: section_title: DOD, VA, and the IPO have Taken Actions to Achieve Interoperability
paragraphs: As part of the approach toward achieving interoperability between their  electronic health record systems, both DOD and VA, along with the IPO,  have taken actions focused on near-term objectives including  standardizing data and expanding the functionality and availability of  patient health information. Specifically, both departments have taken  actions to:

Analyze data related to 25 health data domains that were identified  and prioritized by the Interagency Clinical Informatics Board and map  the data in their respective electronic health record systems—AHLTA  and VistA—to health data standards identified by the IPO. Such  standards include RxNorm and the Systematized Nomenclature of  Medicine Clinical Terms (SNOMED CT).

Expand functionality of the Joint Legacy Viewer—a tool that provides  a real-time, integrated, categorized, and chronological view of  electronic health record information contained in existing DOD and VA  systems. For example, it allows both departments to share certain  healthcare data (e.g., patient demographics, allergies, medications) in  a viewable interface that is available to clinicians. In expanding its  functionality, the departments took steps to make more computable  (mapped) data from both departments available via the Joint Legacy  Viewer and to increase user access to this tool.

Issue plans for their respective electronic health records  modernization or acquisition programs consistent with IPO guidance  that describes the technical approach to interoperability.

With regard to its specific actions, DOD’s recent work on interoperability  has been implemented by the Defense Medical Information Exchange  program. This program is focused on increasing the amount of data  shared with VA and with the private sector. The program has also been  working to make more data viewable through the Joint Legacy Viewer,  consolidate DOD data flow through a single exchange mechanism to  achieve the near-term interoperability objectives, and prepare for the  deployment of the department’s new electronic health record.

According to the program manager, the program has performed the  infrastructure testing intended to ensure that DOD and VA have the  necessary capacity to accommodate the increasing number of users of  the Joint Legacy Viewer. According to DOD documentation, during fiscal  years 2015 and 2016, the program’s plans include continuing to enhance  the Joint Legacy Viewer by adding additional private sector data and  incorporating capabilities for viewing data from existing tools, such as the  Bidirectional Health Information Exchange.

DOD also issued a request for proposals for the DHMSM program in  August 2014 that described the department’s plans to replace its legacy  systems and acquire a modernized electronic health record system with  interoperable capabilities across the military operations. The department  has developed a series of planning documents, including an acquisition  strategy that called for awarding the DHMSM contract by the end of fiscal  year 2015. The department noted in a January 2015 briefing to Congress  that it plans to reach initial operational capability for the modernized  system by December 2016. This is expected to include the deployment of  modernized electronic health record software at eight locations. The time  frame for reaching full operating capability for the new system is to be  determined after contract award.

For its part, VA has developed plans, such as the VA Interoperability Plan  and the VistA 4 Roadmap. Both documents describe the department’s  approach for modernizing its existing electronic health record system  through the VistA Evolution program, while helping to facilitate  interoperability with DOD’s system and the private sector. For example,  the VA Interoperability Plan, issued in June 2014, describes activities  intended to improve VistA’s technical interoperability, such as standardizing the VistA software across the department to simplify  sharing data.

In addition, the VistA 4 Roadmap, last revised in February 2015,  describes four sets of functional capabilities that are expected to be  incrementally deployed during fiscal years 2014 through 2018 to  modernize the VistA system and enhance interoperability. According to  the road map, the first set of capabilities was delivered by the end of  September 2014 and included access to the Joint Legacy Viewer and a  foundation for future functionality, such as an enhanced graphical user  interface and enterprise messaging infrastructure. Another interoperable  capability that is expected to be incrementally delivered over the course  of the VistA modernization program is the enterprise health management  platform. This platform is intended to provide clinicians with a  customizable view of a longitudinal health record that can integrate data  from DOD, VA, and third-party providers. Also, when fully deployed, VA  expects the enterprise health management platform to replace the Joint  Legacy Viewer.

Additionally, with regard to its actions to facilitate department  interoperability efforts, the IPO has developed guidance that describes a  technical approach for standardizing health data, to include related roles  and responsibilities and near-term actions intended to increase  interoperability. Specifically,

The IPO’s Healthcare Information Interoperability Technical Package,  first issued in July 2014 and subsequently updated twice, describes a  standards-based approach and technical objectives for  interoperability. This guidance addresses issues such as how the  DOD and VA systems are to exchange information consistent with  national and international standards. It also identifies the standards  that the IPO has selected for the 25 prioritized health data domains.

The Joint Interoperability Plan, most recently updated in January  2015, is characterized by the IPO as a working document to be  regularly updated. It summarizes the departments’ actions to increase  interoperability, including discussions of completed actions as well as  near-term goals and deliverables through 2016, such as  improvements to the Joint Legacy Viewer and VistA Evolution  capabilities and the DHMSM contract award, among others. In  addition, the plan identifies challenges to achieving interoperability,  such as the evolving nature of health care data standards and the  complexity of integrating interoperable data with clinicians’ current  workflows. Further, the plan includes four approved “use cases,”  which are scenarios that help describe areas where increased  interoperability would be most valuable. The use cases are intended  to help identify additional requirements for the development and  testing of interoperability capabilities beyond 2016.

The IPO’s Health Data Interoperability Management Plan, issued in  September 2014, outlines a high-level approach and roles and  responsibilities for achieving electronic health data exchange and  terminology standardization for DOD, VA, other government entities,  and private sector healthcare partners. The plan also establishes the  Health Data Interoperability Standards Lifecycle Model, which  describes the process by which data standards are expected to be  selected—consistent with national and international health data  standards as they evolve and are implemented by the departments.

Overall, the recent actions taken by DOD, VA, and the IPO have focused  on ensuring that health care data used by the two departments’ existing  systems, AHLTA and VistA, are compliant with national standards. In  particular, following the technical approach outlined by the IPO, the  departments have increased the amount of data from these systems that  are mapped to national standards; they also have made that data  available in the Joint Legacy Viewer.

Nonetheless, DOD and VA program officials acknowledged that these  actions did not result in the two departments meeting the October 1,  2014, deadline established in section 713 of the fiscal year 2014 NDAA  for certifying that all health care data in their systems complied with  national standards and were computable in real time. While the  departments did provide Congress with an update on their progress, DOD  officials stated that the department plans to certify that it has met the  requirement in the next several months. VA officials stated that the  department plans to certify that it has met the requirement later in  calendar year 2015.

section_title: Additional Actions Planned Beyond 2016 to Meet Interoperability Requirements
paragraphs: While important actions are being taken, the departments have indicated  that they do not intend to complete a number of key activities related to  the deployment of their modernized systems and interoperability until  after the December 31, 2016, statutory deadline for deploying modernized  electronic health record software. DHMSM program officials have  acknowledged that additional project details to guide the department’s  efforts towards achieving full operating capability for the modernized  system have yet to be determined. The officials said that they expect  these details to be developed after the contract for the DHMSM service  provider integrator and system solution has been awarded. The  department also currently estimates full operational capability to occur at  the end of fiscal year 2022. Thus, while initial deployment of an  interoperable electronic health record system is expected at eight  locations by the statutory deadline, additional work beyond 2016, which  DOD has yet to fully define, will be required to extend access to the  modernized software, provide interoperable capabilities throughout the  department, and include all users who would benefit from access.

In addition, deployment of VA’s modernized VistA system at all locations  and for all necessary users is not scheduled until 2018. The department  plans to deliver functionality in a phased approach in four product  releases over 5 years to improve performance and increase interoperable  capabilities in additional clinical areas. For example, according to its  plans, the department intends to deliver additional features through fiscal  year 2018 to improve the functionality of the enterprise health  management platform. It also plans to increase interoperable capabilities  in additional clinical areas as it replaces its legacy scheduling system that  is intended to reduce VA patient wait times. Thus, additional actions  beyond the 2016 statutory deadline are still planned for deploying a  modernized electronic health record system.

section_title: DOD, VA, and the IPO Lack Outcome-Oriented Metrics and Goals for Defining and Measuring Interoperability Progress
paragraphs: Prior work and guidance that we have issued stress the importance of  measuring program performance,reporting of accomplishments. This guidance further states that  performance measurement should evaluate both processes and  outcomes related to program activities. Specifically, process metrics  address the type or level of program activities conducted and the direct  products or services delivered by a program, such as the number of  electronic health records queried in an hour or day. Outcome metrics  address the results of products and services, such as improvements on  the quality of health care services or clinician satisfaction. Outcome   which is the ongoing monitoring and  metrics can help in assessing the status of program operations,  identifying areas that need improvement, and ensuring accountability for  end results. Further, measuring program performance is essential for  monitoring progress toward pre-established goals and should be tied to  program goals that allow organizations to demonstrate and report the  degree to which desired results are achieved.

The IPO’s responsibilities include monitoring and reporting on progress  made by the departments to standardize their health care data and  coordinating with the departments to ensure that interoperable  capabilities enhance health care services. With DOD and VA continuing  their activities to increase the sharing of health care data, the IPO has  begun taking steps to measure and report on the progress of the two  departments’ efforts.

Toward this end, the office has issued guidance describing process  metrics that are to be tracked and formally reported to the Health  Executive Committee and congressional stakeholders. For example,  among these metrics, the Health Data Interoperability Management Plan  calls for tracking the percentage of data domains within the departments’  current health information systems that are mapped to selected national  standards.

The plan also identifies metrics to be collected and reported that relate to  tracking health information exchanges through the departments’ existing  initiatives. These metrics include, for example, the number of laboratory  reports and the number of consultation reports exchanged from DOD to  VA through the Federal Health Information Exchange for separated  service members and the number of patient queries by providers from  both departments through the Bidirectional Health Information Exchange.  The measurements are included in a DOD/VA quarterly data sharing  report that the departments prepare and send to Congress. The report is  intended to provide a snapshot of the amount of data shared between the  departments.

While the IPO has developed process metrics and begun reporting the  departments’ progress related to standardizing and exchanging health  data, the office has not specified outcome-oriented metrics and  established goals that are important to gauging the impact that increased  interoperability has on improving health care services. The IPO’s Health  Data Interoperability Management Plan indicates that the DHMSM and  VistA Evolution programs are to develop outcome metrics related to their  respective acquisition and modernization programs. However, the  guidance does not identify outcome metrics or establish goals that the  IPO expects to use to measure progress toward improving health care  services resulting from the departments’ interoperable capabilities.

The IPO Acting Director said that he has tasked a team with working to  identify better metrics to capture both the technical and clinical progress  resulting from interoperability efforts between the departments. According  to the official, this team is working with DOD, VA, and ONC subject matter  experts to identify metrics that would be more meaningful for determining  the impact of increased interoperability, such as metrics on the quality of  a user’s experience and improvements in health outcomes.

However, as of late May 2015, the IPO had not established a time frame  for when the metrics would be completed and incorporated into their  guidance. Officials of the departments and the IPO explained that defining  appropriate outcome metrics for interoperability is not just a DOD and VA  issue; rather, it is a national challenge to identify how to measure  interoperability and what data are needed.

Using an effective outcome-based metrics approach could provide DOD  and VA a more accurate, ongoing picture of their progress toward  achieving interoperability and the value and benefits generated. Doing so  would also better position them to assess and report on the status of  interoperability-related activities in terms of results, and to determine  areas that need improvement. Until they establish a time frame, complete  steps to define outcome metrics and goals, and incorporate these into  IPO guidance, the departments and the IPO risk not knowing the status of  program operations and areas that need improvement, and ensuring  accountability for end results.

section_title: Conclusions
paragraphs: DOD and VA, with guidance from the IPO, have taken actions to increase  interoperability between their electronic health record systems, as called  for in the fiscal year 2014 NDAA. However, the departments have  indicated that they do not intend to complete a number of key activities  related to the deployment of their modernized systems and  interoperability until after the December 31, 2016, statutory deadline for  deploying modernized electronic health record software. To address the  2016 requirement, DOD has issued plans and announced the award of a  contract for acquiring a modernized system to include interoperability  capabilities across military operations. VA, for its part, has issued plans  describing an incremental approach to modernizing its existing electronic  health records system. However, these plans—if implemented as  currently described—show that interoperability delivered by the new  systems is not expected to be completely deployed until after 2018, which  is beyond the statutory deadline. To date, the departments have kept  Congress informed of their efforts, and we believe it is critical that they  continue to do so.

Further, the IPO has taken steps to develop guidance that includes  process-oriented metrics for monitoring and reporting on the increasing  exchange of health information between the departments. However, it has  yet to develop outcome-oriented metrics that are important to gauging the  impact that increased interoperability has on improving health care  services. While IPO officials have said that a team has been tasked to  identify metrics that would be more meaningful for determining the impact  of increased interoperability, no time frame has been identified for when  this team will report its results and when the IPO plans to incorporate  these metrics into its guidance. Further, the office has yet to identify goals  that can be used to indicate the status of interoperability-related activities  and the extent to which progress is being made to achieve full  interoperability of health care information by the departments. Without  defining outcome-oriented metrics and related goals and incorporating  these into the current approach, the departments and the IPO will not be  positioned to assess and report on the status of interoperability-related  activities and determine areas that need improvement.

section_title: Recommendations for Executive Action
paragraphs: To facilitate oversight and inform decision making regarding their  respective department’s interoperability-related activities, we recommend  that the Secretaries of Defense and Veterans Affairs, working with the  Interagency Program Office, take the following three actions: establish a time frame for identifying outcome-oriented metrics, ensure related goals are defined to provide a basis for assessing and  reporting on the status of interoperability-related activities and the  extent to which interoperability is being achieved by the departments’  modernized electronic health record systems, and update IPO guidance to reflect the metrics and goals identified.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to VA and DOD and received written  comments, which are reprinted in appendixes II and III, respectively. In  addition, VA provided technical comments, which we incorporated, as  appropriate.

In its comments, VA generally agreed with our conclusions and concurred  with our recommendations. With regard to our recommendation to  establish a time frame for identifying outcome-oriented metrics, the  department described recent actions it has taken toward the development  of interoperability milestones and metrics, which are intended to serve as  a blueprint for the IPO’s efforts to synchronize outcome-oriented metrics  between DOD and VA. In addition, VA noted that it has begun to develop  standardized metrics related to VistA Evolution that are tied to desired  business goals. VA also described its collaborative efforts with DOD and  the IPO to mature interoperability metrics into more meaningful, outcome- oriented metrics and to establish timelines for formal reporting through  IPO guidance and data sharing reports.

In its comments, DOD also concurred with our recommendations and  stated that the department continues to work with the IPO and VA  stakeholders to develop baseline interoperability metrics. The department  added that it plans to meet with the IPO and VA on a regular basis to  mature these metrics into more meaningful, outcome-oriented measures.

Nevertheless, DOD took issue with selected aspects of our discussion  related to requirements in section 713 of the fiscal year 2014 NDAA. For  example, the department contended that its limited initial deployment of  the new system, combined with the pending documentation of  interoperability, will satisfy the statutory requirement to “deploy  modernized electronic health record software supporting clinicians of the  departments by no later than December 31, 2016, while ensuring  continued support and compatibility with the interoperability platform and  full standards-based interoperability.” We disagree with DOD’s position  and reaffirm our finding that DOD’s and VA’s plans—if implemented as  currently described—indicate that deployment of the new systems will not  be completed across the departments until after 2016 and that much  additional work is needed to extend access to the modernized software to  all relevant users and department locations.

The history and framework of statutory requirements has long called for  the departments to take steps to achieve interoperable health record  capabilities. In this regard, the fiscal year 2008 NDAA included a mandate  to achieve fully interoperable health record capabilities. Further, in section  713 of the fiscal year 2014 NDAA, Congress stated that the departments  “…have failed to implement a solution that allows for seamless electronic  sharing of medical health care data…” and that “most of the information  shared…is not standardized or available in real time to support all clinical  decisions.” We recognize that section 713 of the fiscal year 2014 NDAA  does not qualify the degree of deployment. Nevertheless, we believe it is  reasonable to expect that electronic health records interoperability  between the departments should be demonstrated by more than a limited  number of users and locations and, more importantly, should be made  available to all relevant clinicians at all relevant locations as expeditiously  as possible.

DOD also disagreed that the fiscal year 2014 NDAA specified a date by  which certification of compliance with existing national data standards  was required and stated that the timetable for certification is distinct from  the October 1, 2014 deadline for compliance with national standards. We  assert that the October 1, 2014 deadline in section 713(g)(1) of the fiscal  year 2014 NDAA is linked to the departments’ certification that is required  in section 713(g)(2). As DOD points out, the certification requirement  depends on achieving the capability described in section 713(b)(1). This  capability is to be “interoperable with an integrated display of data…by  complying with the national standards…”. The national standards referred  to here are those national data standards described in section 713(g)’s  certification provision. Section 713(g)(2) requires a certification that DOD  and VA have complied with data standards referred to in section  713(g)(1). These data standards are described in section 713(g)(1) as the  existing national data standards to which all health data in the DOD and  VA systems must comply with by October 1, 2014. DOD and VA’s  required certification involves certification of compliance with existing  national data standards and that compliance was required by October 1,  2014. Thus, we stand by our statement in the report on this matter.

Further, the department expressed concern about our use of the terms  “full” deployment and “enhanced” interoperability, stating that these terms  are not cited in the act. We agree that section 713 of the fiscal year 2014  NDAA does not use the term “full” deployment and we have revised our  report accordingly to remove references to this term. On the other hand,  “interoperability enhancements” is a term that VA has used when  describing features to be delivered throughout its phased approach to the  VistA Evolution modernization. We used the term “enhanced”  interoperability to describe the improvements planned as the modernized  systems mature beyond October 1, 2014. However, to resolve any  ambiguity, we have revised our report to remove the term.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, the Secretary of Defense,  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.

If you or your staffs have questions about this report, please contact me  at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix IV.

section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to evaluate the actions taken by the Department of  Defense (DOD), the Department of Veterans Affairs (VA), and the  Interagency Program Office (IPO) to plan and measure the progress  toward achieving interoperability between the departments’ electronic  health record systems.

To evaluate the actions taken with regard to planning for electronic health  record interoperability between the departments’ systems, we reviewed  our previous work related to electronic health records and DOD and VA  efforts to develop health information systems, interoperable health  records, and interoperability standards to be implemented in federal  health care programs. We obtained and analyzed DOD, VA, and IPO  documentation to evaluate the departments’ plans and identify recent  actions taken toward achieving interoperability consistent with the  requirements specified in the National Defense Authorization Act (NDAA)  for Fiscal Year 2014. Specifically, we analyzed DOD, VA, and IPO plans  to evaluate how they relate to NDAA requirements that direct the DOD  and VA to each (1) ensure that all health care data contained in DOD’s  Armed Forces Health Longitudinal Technology Application (AHLTA) and  VA’s Veterans Health Information Systems and Technology Architecture  (VistA) systems complied with national standards and were computable in  real time by October 1, 2014, and (2) deploy modernized electronic health  record software supporting clinicians by no later than December 31, 2016,  while ensuring continued support and compatibility with the  interoperability platform and full standards-based interoperability.

Further, we reviewed and analyzed DOD’s request for proposals, the  Defense Healthcare Management System Modernization Program  Acquisition Strategy, VA’s VistA 4 Roadmap, the VistA Evolution Program  Plan, VA’s Interoperability Plan, and the IPO guidance including the  Healthcare Information Interoperability Technical Package, the Health  Data Information Management Plan, and the Joint Interoperability Plan.  Further, we identified what the departments plan to deliver by the 2014  and 2016 deadlines and compared planned activities to the statutory  requirements.

To evaluate the actions taken by DOD, VA, and IPO to measure the  progress toward achieving interoperability between the departments’  electronic health record systems, we reviewed the December 2013 IPO  Charter. Our review determined that DOD and VA have assigned the IPO  responsibility for, among other things, monitoring and reporting on the  progress of the departments’ use of national and international health data  standards; compliance with the implementation of IPO’s technical  standards; and coordinating and communicating with the departments to  ensure advances in interoperability capabilities enhance the quality,  safety, efficiency, and effectiveness of health care services. Accordingly,  we reviewed IPO documentation, such as the Healthcare Information  Interoperability Technical Package, the Health Data Interoperability  Management Plan, the Joint Interoperability Plan, the IPO Executive  Committee quarterly reports, and DOD/VA quarterly data sharing reports  to identify performance measures related to measuring and reporting  progress toward achieving interoperability. We compared the  performance measures identified in program documentation and reported  to Congress on our guidance related to process and outcome-oriented  metrics and goals reported in our prior work.

We supplemented our analyses with interviews of DOD, VA, and IPO  officials with knowledge of the interoperability efforts, including the IPO  Acting Director (also the current Program Executive Officer for the  Defense Healthcare Management Systems), IPO Deputy Director, DOD  officials from the Defense Medical Information Exchange program, VA  officials from the Office of Information and Technology and the Veterans  Health Administration with knowledge of the VistA Evolution program, and  members of the Interagency Clinical Informatics Board.

We conducted this performance audit from September 2014 to August  2015, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective.

section_title: Appendix II: Comments from the Department of Veterans Affairs

section_title: Appendix III: Comments from the Department of Defense

section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Mark Bird (Assistant Director),  Neelaxi Lakhmani (Assistant Director), Kami Brown, Nancy Glover,  Jennifer Stavros-Turner, and Marshall Williams, Jr., made key  contributions to this report.


Question-summary annotations: questions: question: What have DOD and VA been doing regarding their electronic health record systems?
answer: The Departments of Defense (DOD) and Veterans Affairs (VA), with guidance from the Interagency Program Office (IPO) that is tasked with facilitating the departments' efforts to share health information, have taken actions to increase interoperability between their electronic health record systems.
child_questions: question: What near-term work have DOD and VA begun?
answer: Among other things, DOD and VA have initiated work focused on near-term objectives, including standardizing their existing health data and making them viewable by both departments' clinicians in an integrated format.

question: What longer-term plans have they developed?
answer: The departments also have developed longer-term plans to modernize their respective electronic health record systems.

question: How has the IPO been involved in this process?
answer: For its part, the IPO has issued guidance outlining the technical approach for achieving interoperability between the departments' systems.

questions: question: To what extent were DOD and VA successful in meeting their deadline to comply with national data standards?
answer: Even with the actions taken, DOD and VA did not, by the October 1, 2014, deadline established in the National Defense Authorization Act (NDAA) for Fiscal Year 2014 for compliance with national data standards, certify that all health care data in their systems complied with national standards and were computable in real time.
child_questions: question: What is the agencies' expected timeline for meeting these standards?
answer: Both departments stated that they intend to do so later in calendar year 2015. Further, the departments' system modernization plans identify a number of key activities to be implemented beyond December 31, 2016—the deadline established in the NDAA for the two departments to deploy modernized electronic health record software to support clinicians while ensuring full standards-based interoperability.
child_questions: question: What have DOD and VA done in an effort to meet this deadline?
answer: Specifically, DOD has issued plans and announced the contract award for acquiring a modernized system to include interoperability capabilities across military operations. In addition, VA has issued plans describing an incremental approach to modernizing its existing electronic health records system.
child_questions: question: What do these plans indicate about the timeline?
answer: These plans—if implemented as currently described—indicate that deployment of the new systems with interoperability capabilities will not be completed across the departments until after 2018.

questions: question: What steps has the IPO taken to monitor progress?
answer: The IPO has taken steps to develop process metrics intended to monitor progress related to the data standardization and exchange of health information consistent with its responsibilities.
child_questions: question: How are tracking metrics part of the IPO's guidance?
answer: For example, it has issued guidance that calls for tracking metrics, such as the percentage of data domains within the departments' current health information systems that are mapped to national standards.

question: What is lacking in these efforts?
answer: However, the office has not yet specified outcome-oriented metrics and established related goals that are important to gauging the impact that interoperability capabilities have on improving health care services for shared patients.
child_questions: question: To what extent have IPO officials responded to this concern?
answer: IPO officials said this work is ongoing and that a team is working with DOD, VA, and subject matter experts to identify metrics that would provide more meaningful measures of the impact of increased interoperability.

question: What gaps exist in this reponse?
answer: However, the IPO has not identified a time frame for when this team will report its results and when the IPO plans to incorporate these metrics and goals into its guidance.
child_questions: question: Why is it necessary to improve this process?
answer: Without ensuring that outcome-oriented metrics and related goals are defined and incorporated into the current approach, the departments and the IPO will not be positioned to assess and report on the status of interoperability-related activities and determine areas that need improvement.

questions: question: What is the scale of DOD and VA's services?
answer: DOD and VA operate two of the nation's largest health care systems, serving approximately 16 million veterans and active duty service members and their beneficiaries, at a cost of more than $100 billion a year.

question: How have these agencies attempted to improve their operations?
answer: For almost two decades, the departments have been engaged in various efforts to advance DOD and VA electronic health record interoperability.
child_questions: question: Most recently, what is the DOD and VA Secretaries' mission regarding their electronic health record systems?
answer: Among their most recent efforts, the DOD and VA Secretaries have committed the departments to achieving interoperability between their separate electronic health record systems.