Information Technology: Opportunities Exist to Improve Management of DOD's Electronic Health Record Initiative
Published: 2010-10-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) provides medical care to 9.6 million active duty service members, their families, and other eligible beneficiaries worldwide. DOD's Military Health System has long been engaged in efforts to acquire and deploy an electronic health record system. The latest version of this initiative--the Armed Forces Health Longitudinal Technology Application (AHLTA)--was expected to give health care providers real-time access to individual and military population health information and facilitate clinical support. However, the system's early performance was problematic, and DOD recently stated that it intended to acquire a new electronic health record system. GAO was asked to (1) determine the status of AHLTA, (2) determine DOD's plans for acquiring its new system, and (3) evaluate DOD's acquisition management of the initiative. To do this, GAO reviewed program plans, reports, and other documentation and interviewed DOD officials.
section_title: What GAO Found
paragraphs: After obligating approximately $2 billion over the 13-year life of its initiative to acquire an electronic health record system, as of September 2010, DOD had delivered various capabilities for outpatient care and dental care documentation. DOD had scaled back other capabilities it had originally planned to deliver, such as replacement of legacy systems and inpatient care management. In addition, users continued to experience significant problems with the performance (speed, usability, and availability) of the portions of the system that have been deployed. DOD has initiated efforts to improve system performance and enhance functionality and plans to continue its efforts to stabilize the AHLTA system through 2015, as a "bridge" to the new electronic health record system it intends to acquire. According to DOD, the planned new electronic health record system--known as the EHR Way Ahead--is to be a comprehensive, real-time health record for service members and their families and beneficiaries. The system is expected to address performance problems, provide unaddressed capabilities such as comprehensive medical documentation, capture and share medical data electronically within DOD, and improve existing information sharing with the Department of Veterans Affairs. As of September 2010, the department had established a planning office, and this office had begun an analysis of alternatives for meeting the new system requirements. Completion of this analysis is currently scheduled for December 2010. Following its completion, DOD expects to select a technical solution for the system and release a delivery schedule. DOD's fiscal year 2011 budget request included $302 million for the EHR Way Ahead initiative. Weaknesses in key acquisition management and planning processes contributed to AHLTA having fewer capabilities than originally expected, experiencing persistent performance problems, and not fully meeting the needs of users. (1) A comprehensive project management plan was not established to guide the department's execution of the system acquisition. (2) A tailored systems engineering plan did not exist to guide the technical development of the system, an effort that was characterized by significant complexity. (3) Requirements were incomplete and did not sufficiently reflect user and operational needs. (4) An effective plan was not used to improve users' satisfaction with the system. DOD has initiated efforts to bring its processes into alignment with industry best practices. However, it has not carried out a planned independent evaluation to ensure it has made these improvements. Until it ensures that these weaknesses are addressed, DOD risks undermining the success of further efforts to acquire electronic health record system capabilities.
section_title: What GAO Recommends
paragraphs: GAO is recommending that DOD take six actions to help ensure that it has disciplined and effective processes in place to manage the acquisition of further electronic health record system capabilities. In written comments on a draft of this report, DOD concurred with GAO's recommendations and described actions planned to address them.

section_title: Letter
paragraphs: This report responds to your request that we examine the Department of  Defense’s (DOD) efforts to implement its military electronic health record  system known as the Armed Forces Health Longitudinal Technology  Application (AHLTA). When fully deployed, AHLTA was envisioned to  provide the department with a modernized health information system that  would generate and maintain a comprehensive, lifelong, computer-based  patient record for every soldier, sailor, airman, and marine; their family  members; and others entitled to DOD military health care. The electronic  health record was expected to give health care providers real-time access  to individual and military population health care information, thus  facilitating clinical decision support and rationale for care rendered to U.S.  service members worldwide. However, after more than a decade of effort  to deliver this system, the department has recently begun planning for a  new electronic health record system.
At your request, we conducted a study of DOD’s efforts to acquire and  implement its electronic health record system. Specifically, our objectives  were to (1) determine DOD’s status in implementing AHLTA, (2)  determine the department’s plans for acquiring a new system, and (3)  evaluate the department’s acquisition management for its electronic health  record system.
To accomplish the objectives, we reviewed relevant program  documentation and interviewed appropriate DOD officials. Specifically, to  determine the status of the AHLTA project, we reviewed project plans and  status reports. To determine the department’s plans for acquiring a new  electronic health record system, we reviewed relevant planning  documents, including an initial capabilities document. To evaluate the  department’s management of its electronic health record acquisition, we  compared the department’s activities for project management planning,  systems engineering management, requirements development and  management, user satisfaction feedback, and acquisition management  with DOD guidelines and industry best practices.
We conducted this performance audit from September 2009 to October  2010 at DOD offices in Falls Church, Virginia, and Bethesda, Maryland, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. A more complete description of our  objectives, scope, and methodology is provided in appendix I.
subsections: section_title: Background
paragraphs: DOD operates a worldwide health care program, through which it provides  medical care and assistance to 9.6 million active duty service members,  their families, and other eligible beneficiaries. Its health care operations  are significant, involving approximately 135,000 personnel in  approximately 700 Army, Navy, and Air Force medical facilities in 12  domestic regions, as well as European, Pacific, and Latin American  regions. The department’s fiscal year 2010 budget for providing health care  services was about $49 billion.
DOD’s health care program is a responsibility of the Office of the  Undersecretary of Defense for Personnel and Readiness. Within the Office  of the Undersecretary is the Office of the Assistant Secretary of Defense  for Health Affairs, which is responsible for the department’s Military  Health System (MHS) program.
MHS has two missions: wartime readiness (maintaining the health of  service members and treating wartime casualties) and peacetime care  (providing for the health care needs of the families of active-duty  members, retirees and their families, and survivors). The Assistant  Secretary of Defense for Health Affairs establishes policy regarding health  care for all DOD beneficiaries and also plans and budgets for health care  operations and maintenance. At the same time, each military service has  its own medical department that operates medical facilities (referred to as  military treatment facilities) and recruits and funds military medical  personnel. Currently, the military treatment facilities include 59 military  hospitals and 650 medical and dental clinics. DOD provides about half of  MHS services through these military facilities, supplementing this by  contracting for health services with civilian contract providers. Active- duty members are required to obtain care at military treatment facilities if  such care is available; in contrast, retirees and dependents may obtain  care at either military facilities or through civilian contract providers.
subsections: section_title: History of DOD’s Electronic Health Record System
paragraphs: To facilitate the delivery of medical services, in 1988, DOD initiated the  acquisition of an electronic health record system to support all of its  hospitals and clinics. This system, the Composite Health Care System  (CHCS), was intended to be the primary medical information system  deployed worldwide to support the department’s hospitals and clinics.  DOD envisioned that it would provide automated support for patient  administrative functions (such as registrations, admission, and  disposition); ordering and retrieving results of laboratory and radiology  procedures; ordering and recording prescriptions; and patient  appointment scheduling.
CHCS was deployed in 1993; however, it was supported by numerous  stand-alone medical information systems, such as the department’s  Ambulatory Data System, Preventive Health Care Application, and  Nutrition Management Information System, and was not designed to  facilitate the exchange of information from one system or military  treatment facility to the next. Specifically, CHCS was facility-centric, in  which each facility stored only its own medical information for patients  using different data standards. Therefore, if a medical provider wanted to  obtain complete information about a patient, a query would have to be  made to each of the CHCS locations—a time- and resource-intensive  activity. Additionally, when a patient moved to another region, the  electronic records did not transfer across the CHCS locations because of  the different data standards at each location. The lack of an integrated  system perpetuated the reliance on paper-based records, leading DOD to  pursue a comprehensive electronic health care record.
To this end, in 1997, the department initiated the CHCS II program to  address the need for a comprehensive, lifelong, computer-based health  care record for every service member and their beneficiaries. The vision  for CHCS II was to provide access to a patient’s health care information  with a single query by providers in military treatment facilities.  Specifically, with this system, DOD planned to provide worldwide access  to outpatient, inpatient, dental, and vision records, and to make them  available 24 hours a day, 7 days a week. This new system was to be  accomplished with the use of a centralized repository of all health care  information derived using common data standards. The system was to  build on capabilities of existing systems, subsuming their functionality  over time, while adding new functionality to meet mission needs.
CHCS II’s architecture was to be an open system, client-server design of  three levels: the user (client) workstation at various DOD locations, the  DOD computers’ (servers’) operating system and storage hardware and  software, and a clinical data repository at a remote computing center  where the information would be stored. The department had planned to  connect all workstations at an installation’s hospital or clinic to the  servers through the installation’s local or wide area network. It had  planned to divide the system acquisition into seven software releases to be  delivered incrementally by June 2006 at an estimated cost of $4.3 billion  (in 1998 dollars).
The department’s original plan had called for deploying a prototype  system in October 1998 and beginning deployment of the initial version in  about April 1999. However, the department did not meet its schedule to  deliver initial CHCS II system capabilities and associated mission benefits  by April 1999; it reported that the initial deployment was delayed by 6  months because of a failure to meet initial performance requirements and  changes in system requirements.
In July 2000, the department redefined its plans for the system to include  adopting a new technical architecture, establishing a means for controlling  changes to requirements, and committing to the incremental release of  system capabilities. It also delayed the decision date for deploying the  initial system capabilities (for outpatient documentation) to January  2001—21 months later than its original commitment for the system.
However, the department did not meet this commitment, and subsequently  established a new plan that called for incrementally deploying  functionality to achieve the system’s full operational capability. Delivery of  the system was to commence in July 2003 and was to be completed by  September 2007, yielding four blocks of capabilities that would  incrementally populate the system’s electronic health record at a revised  estimated life-cycle cost of $3.8 billion through 2017.
Block 1 was to make outpatient information available worldwide on a  continuous basis through the electronic health record system (as opposed  to CHCS legacy functionality which only made records available at a single  location), provide encounter documentation, aid in order entry/results  retrieval, assist in encounter coding support, provide alerts and reminders  (such as drug interaction alerts and special duty status), facilitate role- based security, and establish a health data dictionary and a master patient  index.
Block 2 was to provide automated clinical practice guidelines, optometric  documentation, and dental documentation.
Block 3 was to replace CHCS ancillary functionality for results retrieval  and order entry for outpatient encounters such as laboratory and  automatic pathology, pharmacy, and radiology.
Block 4 was to provide for inpatient order entry and management,  including inpatient clinical and critical care documentation.
When delivered, the system was to allow users to create and store  computer-based patient records using workstation- and computer-based  software packages. Each facility’s workstations and servers were to be  connected via each installation’s local or wide area networks. Further,  each installation was to be connected through a wide area network to a  defense computing center where the patient records would be stored in a  database known as the clinical data repository. DOD intended that medical  providers would ultimately be able to access a patient’s computer-based  record from any military treatment facility, no matter where the patient  was being or had been treated.
According to program documentation, the department began worldwide  deployment of Block 1 in January 2004. It completed the deployment of  this block in December 2006. However, program officials stated that users  experienced numerous performance problems with the capabilities that  were delivered, which impacted its usability, speed, and availability.  Specifically, the department reported experiencing the following problems  with the delivery of Block 1:    Usability. The system did not support varied clinical workflow to meet  the needs of various types of practitioners, had missing or incomplete  clinical capabilities (e.g., consult and referrals management, ancillaries,  specialty workflow support), did not support fully unified or user- customizable patient data, and did not have a user-friendly interface.
Speed. The system did not have the speed or performance to efficiently  support the clinicians’ workflow in certain environments and was affected  by problems such as coding and infrastructure which impacted its speed.
Availability. The system was not reliable on a 24-hour-a-day, 7-day-a- week basis; it had no backup for disaster recovery; and the data repository  experienced system shutdowns and functional interruptions.
As a result of the system problems associated with Block 1, DOD set a new  date for system completion—September 2011—and increased the  projected life-cycle cost of the system to approximately $5 billion, which it  attributed primarily to the need for increased operations and maintenance  for Block 1.
The department also took a number of other steps with regard to the  initiative. Specifically, in May 2005, it terminated plans for deploying the  Block 4 inpatient functionality with the intent of moving this functionality  into Block 3. However, due to continuing performance problems with the  functionality that had been delivered, and because the Block 3 deployment  had exceeded the department’s 5-year limit for achieving initial  operational capability by January 2008, DOD terminated Block 3  (laboratory, radiology, and pharmacy) as well. This action left only one of  the four planned blocks—Block 2—for implementation. Although the  department reduced the scope of the initiative to only two blocks, the  estimated life-cycle costs were revised back to the original $3.8 billion  (through 2021). However, the department encountered performance  problems with the Block 2 dental module as well and, in December 2009,  MHS senior leadership implemented a strategic pause in its further  deployment.
Beyond these actions, the department took other steps over the course of  the initiative. Specifically, in November 2005, the Assistant Secretary of  Defense for Health Affairs announced a change in the name of the system  from CHCS II to AHLTA, but did not give a specific reason for doing so.  Further, as part of its attempt to improve the system, DOD awarded  several contracts between fiscal year 2006 and fiscal year 2009 for a total  of approximately $40 million to address performance problems and  implement software enhancements. The contractors began deployment of  these software enhancements (which DOD referred to as AHLTA 3.3) in  December 2008.
section_title: DOD’s Acquisition Process for Its Electronic Health Record
paragraphs: To acquire its electronic health record system, DOD used several  contractors and types of contracts. These included fixed-price, time-and- materials, and cost-plus-fixed-fee contracts, each of which involved a  different level of cost or performance risk for the government. The prime  developer and lead integrator for CHCS II, Integic (acquired by Northrop  Grumman in 2005), was awarded a time-and-materials contract for about  $65.4 million in 1997 and was tasked to perform systems engineering,  requirements analysis, architecture evaluation, software design and  development, engineering and development testing, test and evaluation,  maintenance, site installation and implementation, and training. Contracts  for system development and integration continued through fiscal year  2009.
DOD also used noncompetitive contracts for the development of the  system. According to the program office, 11 noncompetitive contracts and  task or delivery orders, totaling approximately $44.6 million, were  awarded for the system from fiscal year 2004 through fiscal year 2012.  Program officials stated that the noncompetitive contracts were awarded  on the basis that (1) DOD’s need for the supplies or service was so urgent  that providing each awardee under a multiple award contact a fair  opportunity would have resulted in unacceptable delays; (2) only one  awardee was capable of providing the supplies or services required at the  level of quality required because the supplies or services ordered were  unique or highly specialized; or (3) an order was a logical follow-on to an  order already issued under the contract.
According to AHLTA program documentation, the system acquisition was  guided by the defense acquisition system, which is documented in the  department’s DOD 5000.02 Instructions. The defense acquisition system  consists of five key program life-cycle phases and three related milestone  decision points that major acquisitions must meet in order to proceed to  the next phase of the acquisition. At each milestone point, the program is  reviewed by a milestone decision authority to determine whether it can  move to the next life-cycle phase.
The five phases of the defense acquisition are as follows:  1.  Materiel solution analysis: The purpose of this phase is to assess,  through an analysis of alternatives, potential solutions to satisfy an  approved capability need. 2.  Technology development: The purpose of this phase is to determine  and mature the appropriate set of technologies to be integrated into  the investment solution by iteratively assessing the viability of the  various technologies while simultaneously refining user requirements.  To enter this phase, a program must have an approved analysis of  alternatives and pass milestone A. To exit this phase, the acquisition  must demonstrate affordable technology. 3.  Engineering and manufacturing development: The purpose of this  phase is to develop a system or an increment of capability, and  demonstrate integrated system design through developer testing to  show that the system can function in its target environment. To enter  this phase, a program must have approved requirements and pass  milestone B. To exit this phase, the acquisition must meet performance  requirements in the intended environment. 4.  Production and deployment: The purpose of this phase is to achieve  an operational capability that satisfies the mission needs, as verified  through independent operational test and evaluation, and to  implement the system at all applicable locations. To enter this phase, a  program must have completed development testing and pass milestone  C. To exit this phase, the system must be deployed and ready to  operate for all users. 5.  Operations and support: The purpose of this phase is to  operationally sustain the system in the most cost-effective manner  over its life cycle. DOD criteria do not require that the milestone  decision authority conduct milestone reviews during the period after a  system has been deployed and stabilized.
For the purpose of conducting milestone reviews, AHLTA was assigned  the highest level of oversight for DOD information system acquisitions.  As such, oversight was provided within the Office of the Secretary of  Defense.
section_title: Management Structure for AHLTA
paragraphs: Various DOD units were involved in acquiring and deploying AHLTA. As  the principal advisor to the Assistant Secretary of Defense for Health  Affairs and to the DOD medical leaders on all matters related to  information management and information technology, the MHS chief  information officer (CIO) has primary responsibility for overseeing the  acquisition, development, testing, and deployment of AHLTA to the  military treatment facilities. Key offices within the Office of the MHS CIO  perform critical information management and information technology  functions to support AHLTA, including the Joint Medical Information  Systems Office, which is responsible for the testing, implementation,  training, fielding of system components, operations, maintenance, and  ultimate disposal of system components.
Also within MHS, the Composite Health Care System (CHCS) II Program  Office was established in January 1997 to provide direct management of  the project; it had operational responsibility for the acquisition and  deployment of the electronic health record, as well as the migration of the  numerous standalone clinical information systems. In fiscal year 2000, the  CHCS II program office was renamed the Clinical Information Technology  Program Office (CITPO). In 2008, with the merger of CITPO and the MHS  Theater Medical Information Program Office—Joint, the office is now  called the Defense Health Information Management System (DHIMS).
To provide oversight in accordance with DOD’s defense acquisition  system, the Assistant Secretary of Defense for Networks and Information  Integration, within the Office of the Secretary of Defense, was designated  the milestone decision authority responsible for deciding at each  acquisition cycle milestone whether the project could proceed to the next  milestone. The project also received oversight from several other bodies,  including the Human Resources Management Investment Review Board,  headed by the MHS CIO, and the Overarching Integrated Project Team,  which evaluated project performance in accordance with DOD 5000 and  approved acquisition program baselines and acquisition decision  memorandums.
Table 1 summarizes the assignment of responsibilities for AHLTA among  the various DOD units.
section_title: Previous Reviews of DOD’s Electronic Health Record Initiatives Highlighted Management Deficiencies and Risks
paragraphs: DOD’s Inspector General and we have previously reported on the  department’s actions toward acquiring its new health care information  system and have noted the need for improvement in key management  areas, such as project management, contract management, and risk  management.
In reporting on the department’s efforts in January l999, the Inspector  General noted that the project management system for the acquisition  (called CHCS II at the time of the report) was not complete. While finding  that DOD had taken positive actions to manage the acquisition, the report  noted that the department had not established a project management  control system to evaluate and measure the program’s performance. In  addition, the report stated that the program’s funding visibility was limited  because DOD was combining funding for sustaining the system with  modernization funding for CHCS and other clinical business area  automated systems. The Inspector General made recommendations  related to designing and implementing a project management control  system, the reporting of funding for the system, and providing milestone  exit criteria that demonstrated the level of performance, accomplishments,  and progression.
Further, in May 2006, the Inspector General conducted an evaluation of  the project’s program requirements, the related acquisition strategy, and  system testing to determine whether the system was being implemented to  meet cost, schedule, and performance requirements. While the report  found that the program management office was using risk mitigation  techniques, such as risk management, lessons learned, and performance  monitoring, the program remained at high risk because of the complexities  of integrating commercial, off-the-shelf software into the existing program.  In particular, the report noted that the program office had not identified  any mitigation strategies to reduce and control program risk related to  integration of commercial, off-the-shelf software for the third block of  functionality. As a result, the Inspector General concluded that the  program was vulnerable to continued increases in cost, extended  schedules for implementation, and unrealized goals in performance from  underestimating the difficulties of integrating commercial, off-the-shelf  products. Subsequently, the program office developed mitigation  strategies, but the Inspector General reported that they were inadequate  and did not follow risk management guidance, including identifying  significant activities and milestones. Accordingly, the Inspector General  recommended that the program office develop more robust mitigation  strategies in accordance with the program office’s risk management plan.
We have also reported on DOD’s management of the system acquisition,  noting the need for improvements. For example, in 2002, we reported that,  because the department had not estimated the cost of delivering the initial  system capabilities, it had lacked a cost commitment against which to  measure progress. In addition, we noted that program benefits were in  question since measurements had not yet begun and that costs were about  two-and-a-half times the l998 estimate. Further, DOD had initially  identified a single economic justification for the entire project, which had  been used as the basis for its system releases, and had not treated the  releases as separate investment decisions. Finally, DOD had not followed  performance-based contracting practices, resulting in the risk that the  system would take longer to acquire and cost more than necessary.
Accordingly, we recommended that DOD expand its use of best practices  in managing the system by (1) modifying the project’s investment strategy  to justify investment in each system release before beginning development  and measuring return on investment and (2) employing performance- based contracting practices where possible on all future delivery orders.  The department agreed with these recommendations and took actions to  update and validate its life-cycle cost estimate in September 2002. This  was used by the department to approve the deployment of the system  release. Also, the department employed performance-based contracting  practices, such as using performance standards, quality assurance plans,  and contractor incentives on CHCS II delivery orders.
section_title: AHLTA Has Limited Capabilities and Continues to Experience Performance Problems
paragraphs: Despite having obligated approximately $2 billion over the 13-year life of  its initiatives to acquire and operate an electronic health record system, as  of September 2010, DOD continued to experience performance problems  with the one block of AHLTA functionality (Block 1) that it had fully  deployed and with a second block of functionality (Block 2) that it had  partially deployed. Further, after having terminated its plans for deploying  the two other blocks of functionality (Block 3 and Block 4) that were  intended to be part of the system, the department has identified April 2011  as the date by which it now expects to achieve full operational capability  of the scaled-backed AHLTA system. Program officials told us they are  taking steps to stabilize the existing system capabilities through 2015, as  the department proceeds with plans to pursue yet another new electronic  health record system.
In deploying Block 1, the department reported that it achieved all of the  planned outpatient capabilities for direct patient care, including encounter  documentation, order entry and results retrieval, encounter coding  support, consult tracking, and alerts and reminders. According to the  department, it deployed the AHLTA outpatient documentation capability  worldwide, providing 77,000 clinicians with the ability to document over  148,000 outpatient encounters daily. The department stated that medical  providers can access the patient’s computer-based record from any  military treatment facility. Also, DOD currently shares a significant  amount of patient information with the Department of Veterans Affairs,  including outpatient pharmacy data, laboratory results, and radiology  results on shared and separated service members.
In addition, with the deployment of Block 2, including enhancements to  Block 1, dental capabilities were provided to 73 of 375 dental treatment  facilities, allowing graphical dental charting, order and entry results  retrieval, and automated dental readiness classification. In this regard, the  capabilities were deployed to 46 Air Force dental medical facilities, 25  Navy facilities, and 2 Army facilities. Further, program officials stated that  in October 2009, because of technical and functionality upgrades made  over time to the legacy Spectacle Request Transmission System, funding  was ceased for optometric capabilities for Block 2. The department stated  that it plans to achieve full operational capabilities by April 2011. Table 2  shows the capabilities planned and delivered for Blocks 1 and 2.
Nonetheless, program officials, as well as users of the system,  acknowledged that problems with the system’s performance have  persisted. During a demonstration of the system’s operation in April 2010,  medical providers discussed problems with AHLTA, including limitations  in its availability and usability. For example, the providers participating in  the demonstration stated that it is time-consuming to document  encounters using AHLTA because of the time required to enter  information and navigate through the application screens. Thus, they  sometimes must document portions of an outpatient encounter after the  patient leaves. In their experience, using the system at the time of the  encounter would take attention away from the patient for unacceptable  periods of time. Also, they stated that when system downtime occurs,  providers can neither access patient data nor electronically document  care; in these instances, medical notes are recorded manually and later  entered in the system after it returns to operation—an inefficient process.
As noted in the earlier discussion, since fiscal year 2006 the department  has been taking steps to address performance problems and enhance  existing system capabilities. DOD is proceeding with what it refers to as  a “stabilization effort” to continue making improvements to the system and provide ongoing capabilities until a new system is acquired. According to  DOD officials, the estimated cost of this effort for fiscal year 2010 through  fiscal year 2015 is $826.3 million. The stabilization effort is expected to  improve the speed, availability, and usability of the system; moreover,  according to officials in the Office of the Deputy Secretary of Defense, the  stabilization effort is expected to allow the department to meet its near- term needs and implement additional enhancements to support its future  system.
section_title: DOD Has Initiated Planning Activities for the EHR Way Ahead
paragraphs: Because AHLTA has consistently experienced performance problems and  has not delivered the full operational capabilities intended, DOD has  initiated plans to develop a new electronic health record system. This new  initiative is called the Electronic Health Record (EHR) Way Ahead. As  with AHLTA, department officials stated that the new electronic health  record system is expected to be a comprehensive, real-time health record  for active and retired service members, their families, and other eligible  beneficiaries. They added that the new system is being planned to address  the capability gaps and performance problems of previous iterations, and  to improve existing information sharing between DOD and the Department  of Veterans Affairs and expand information sharing to include private  sector providers.
Thus far, the department has taken several steps to launch its acquisition  of the new system. Specifically, in February 2010 it established the EHR  Way Ahead Planning Office to identify options for the future electronic  health record system. The planning office currently resides within the  MHS Joint Medical Information Systems Program Executive Office under  the Office of the CIO.
In May 2010, the department approved plans to assess solutions for the  new electronic health record system. In this regard, the planning office  began conducting an analysis of alternatives to provide guidance on  selecting a technical solution. According to planning officials, efforts to  develop the analysis of alternatives are being supervised by the Office of  the Assistant Secretary of Defense for Health Affairs, and this analysis is  expected to define and evaluate reasonable alternatives for meeting the  capability requirements. The analysis is currently scheduled to be  completed by December 2010.
To facilitate the analysis of alternatives, planning officials stated that they  had identified system capabilities needed to meet the department’s  medical mission. They added that a list of the “top 10” priority capabilities  for a new system had been developed based on the gaps identified in prior  iterations of their electronic health systems. (These priorities are  summarized in table 3.)
According to planning documents, following completion of the analysis,  DOD expects to select a technical solution and to develop and release a  delivery schedule.
DOD’s fiscal year 2011 budget request includes $302 million for the EHR  Way Ahead initiative. For fiscal year 2012, the department intends to  submit an updated budget request and the schedule for delivery of the  EHR Way Ahead based on the results of the analysis of alternatives.
section_title: AHLTA Performance Was Hindered by Weaknesses in Key Acquisition Management and Planning Processes
paragraphs: The success of a large information technology project such as AHLTA is  dependent on an agency possessing capabilities to effectively plan and  manage acquisitions, design the associated systems, define and manage  system requirements, and use effective measures to gauge user  satisfaction. In the case of AHLTA, weaknesses in these key management  areas contributed to DOD delivering a system that provided fewer  capabilities than originally expected, experienced persistent performance  problems, and ultimately, did not fully meet the needs of its intended  users. Alleviating these areas of weakness will be essential to the success  of further initiatives, including the AHLTA stabilization effort and the EHR  Way Ahead, that the department undertakes in pursuit of its electronic  health record system capabilities.
subsections: section_title: Project Plan Was Incomplete and Not Maintained
paragraphs: Program management principles and best practices emphasize the  importance of having a project management plan in place that, among  other things, establishes a complete description that ties together all  program activities and evolves over time to continuously reflect the  current status and desired end point of the project. An effective plan is  comprised of a description of the program’s scope, cost, lines of  responsibility and authority, management processes, and schedule. Such a  plan incorporates all the critical areas of system development and is to be  used as a means of determining what needs to be done, by whom, and  when.
Other guidance, such as our Information Technology Investment  Management framework, states that effective program oversight of IT  projects and systems, including those in operation and maintenance,  involves maintaining approved project management plans that include  expected cost and schedule milestones and measurable benefit and risk  expectations.
However, officials did not follow best practices in developing a project  management plan to guide the department’s electronic health record  system. Although the department established a project management plan,  it did not include several standard components such as the project’s  scope, a requirements management plan, cost estimates and baseline, a  schedule, and a staffing management plan. In addition, although DOD  identified the plan as a keystone document for guiding the project, the  plan was last revised in 2005 and was not updated during subsequent  development work and the operations and maintenance phase to reflect  significant changes to the program. These changes included termination  and postponement of planned capabilities, and revisions to the acquisition  processes used to guide the AHLTA program. As a result, a plan was not in  place to effectively guide the program throughout these changes.  Moreover, there is no such plan to guide current activities associated with  the stabilization effort, which, as discussed previously, involves attempts  to address system performance problems and enhance functionality.
According to program officials, the project management plan was last  revised in 2005 before their focus shifted to addressing the system  performance problems that occurred as a result of completing Block 1  deployment in December 2006. Nevertheless, significant changes occurred  to the program’s scope, cost, and schedule after Block 1 deployment, and  the agency lacked a current and complete plan to guide activities and  measure program progress. Going forward, developing and maintaining a  comprehensive project plan will be an essential tool for overseeing the  AHLTA stabilization effort, which is to provide crucial improvements to  the system and act as a bridge over the next 5 years to the deployment of  the EHR Way Ahead system. Further, having a comprehensive and current  project plan for the EHR Way Ahead program will help to guide the project  and provide oversight of the project’s progress. Without a project  management plan that reflects the status and goals of the project, DOD  increases the risk that stakeholders will not have the insight into program  status that is needed to exercise effective oversight of both the AHLTA  stabilization effort and the EHR Way Ahead acquisition.
section_title: DOD Lacked a Systems Engineering Plan to Guide the Electronic Health Record System’s Design
paragraphs: According to industry best practices, systems engineering governs the  total technical and managerial effort required to transform a set of user  requirements and expectations into specific capabilities and, ultimately,  into a system design that will meet users’ needs. Systems engineering  practices include developing solutions for achieving system performance  requirements such as system availability, and ensuring compatibility when  integrating multiple systems and their components. Further, DOD  guidance states that a tailored and detailed systems engineering plan is a  critical tool for guiding systems engineering practices throughout the life  of an acquisition program. Having such a plan is particularly important for  a system characterized by significant technical complexities.
DOD’s electronic health record system design reflected numerous  technical complexities, such as the need to capture, manage, and share  health information across a worldwide network that must be available 24  hours a day, 7 days a week, and that is to serve a transient patient  population. In addition, the system design involved a network that had to  be integrated with a central patient database and multiple nonstandard  hardware and software platforms, such as commercial, off-the-shelf  products at over 800 military treatment facilities.
Nonetheless, although the program office recognized these types of system  complexities as being part of the electronic health record system design,  the office never established a tailored systems engineering plan to guide  the acquisition, or to facilitate the resolution of the many performance  problems that have plagued the system since its initial deployment.
In this regard, a particularly troublesome area for the department has been  in deploying enhancements to the system. For example, following Block 1  deployment in 2006, the department implemented local cache servers in an  attempt to improve the system’s operational availability. According to the  department, the specific purpose of the local cache servers had been to  mitigate the need to access patient medical information in the central data  repository during system outages. However, after the servers were  deployed, DOD realized that the placement of the servers within the  system architecture did not resolve the problem and created a single point  of failure. Rather than yield operational improvements, department  officials acknowledged that these actions resulted in additional challenges,  including the need for a costly local cache server redesign, which was  begun in fiscal year 2009. Program documentation noted that the local  cache server effort was probably one of the most difficult engineering  challenges that the program office had faced so far. Further, as various  issues were faced, it became increasingly clear that detailed planning in  the earlier stages was not what it could have been. In April 2010, clinicians  demonstrating the system at the Bethesda Naval Medical Center stated  that the servers continued to be a major contributing factor to system  availability issues.
The lack of a systems engineering plan to guide the program office  through this type of complexity is particularly notable in light of the DOD  Inspector General’s report of 2006, which stated that inadequate planning  for technical complexities significantly impacts the cost, schedule, and  performance of a program. The report further stated that the AHLTA  program office had underestimated the technical complexity of integrating  products with the electronic health record system and, as a result,  remained at high risk for continued cost increases, schedule overruns, and  unrealized performance goals.
In discussing this matter, agency officials stated that a tailored systems  engineering plan had not been developed to guide the design of AHLTA  because such a plan was not required when the system was originally  planned. Specifically, the officials stated that, it was not until February  2004 that DOD issued a policy requiring that a systems engineering plan be  in place for acquisition programs’ milestone reviews; but all milestone  reviews for AHLTA had been completed prior to this time.
However, current DOD guidance emphasizes the need for a tailored  systems engineering plan to guide all systems engineering practices,  including those that occur after the completion of milestone reviews.  Without a tailored systems engineering plan to guide the program’s efforts  to address long-standing system performance problems as part of the  AHLTA stabilization efforts, the department may continue to be challenged  in achieving the desired results. Further, in planning for the acquisition of  the new EHR Way Ahead system, it will be essential that the department  establish early in the process and have in place a detailed and tailored plan  to avoid encountering technical challenges similar to those of the AHLTA  program, and thus again failing to meet users’ needs.
section_title: Weaknesses in DOD’s Requirements Processes Impacted AHLTA’s Usability
paragraphs: According to recognized guidance, using disciplined processes for  developing and managing requirements can help reduce the risks of  developing a system that does not meet user and operational needs.  Requirements should serve as the basis for establishing agreement  between users and developers and a shared understanding of the system  to be developed. Effective requirements development practices include,  among other things, involving users in identifying requirements throughout  the project’s life cycle to ensure system requirements are complete and  accurately reflect their needs. Effective requirements management  practices include maintaining bidirectional traceability of requirements to  ensure that system-level requirements can be traced both backward to  high-level operational requirements, and forward to low-level system  design specifications.
For the AHLTA acquisition, program documentation revealed that users  were not adequately involved throughout the requirements development  process. According to the documentation, users did not seek involvement  in the requirements development process and system developers did not  seek user input when making changes to requirements. As a result,  requirements were neither complete nor sufficiently detailed to guide  system development, and did not adequately provide a shared  understanding between the users and developers of how the system was to  be developed. Program documentation noted that requirements often were  not adequately specified and did not adequately reflect user needs. In  particular, the program documentation revealed that, while users were  involved in developing an initial set of requirements used to make system  acquisition decisions, they were largely not involved in identifying new  requirements and making changes to existing ones while the system was  being developed and deployed.
In certain instances, because users were involved only at the beginning  and end of the requirements development process, they were only able to  determine that capabilities would not meet their needs after those  capabilities had already been deployed. For example, when the dental  application was in the process of being deployed to Army, Navy, and Air  Force sites, the MHS senior leadership voted to halt further training and  implementation because users reported that the capabilities were not  complete and did not address their needs. Consequently, alternate dental  solutions will be explored as part of the analysis of alternatives for the  EHR Way Ahead, resulting in additional costs and delays in deploying  dental capabilities that will meet users’ requirements.
Since the initial deployment, the department has taken steps to increase  user involvement in defining requirements. For example, to better involve  users in the requirements process and identify issues with system  usability, the program office held conferences in 2006 at which users  identified over 200 new requirements for inclusion in the system. Program  officials stated that the requirements identified during the conference  were used to develop the AHLTA 3.3 software release. However, our  evaluation of the requirements traceability matrix used to develop the  AHLTA 3.3 release showed that bidirectional traceability had not been  fully established; thus, the requirements were not always linked to high- level operational requirements or to more detailed design specifications.  Without adequate traceability, the department cannot ensure that all  agreed-upon requirements will be developed, fully tested, and work as  intended.
In addition, the department has plans for making improvements in the  requirements management process in its MHS Information  Management/Information Technology Strategic Plan 2010–2015 and  includes a goal to improve the requirements management process to  enable greater participation of system users. According to the plan, this  will improve the value, quality, timeliness, and stakeholder ownership of  the resulting system. However, because the department is in the early  stages of implementing improvements for greater user participation, it is  too early to determine their effectiveness.
As the department proceeds with the AHLTA stabilization effort and the  new EHR Way Ahead system, ensuring that user needs are met will be  essential to effective and cost-efficient delivery of system capabilities.  Until the department ensures that a requirements development process  with adequate user involvement is in place, it will continue to lack a vital  tool for ensuring the efficient and effective delivery of electronic health  record system capabilities that will meet the needs of its users.
section_title: Efforts to Improve User Satisfaction Were Not Guided by Effective Planning
paragraphs: DOD has stated that the success of AHLTA can be gauged by  improvements in user satisfaction and user acceptance, among other  things. In this regard, effectively managing program improvement  activities to improve user satisfaction requires planning and executing  such activities in a disciplined fashion. The Software Engineering  Institute’s IDEALSM21 model is a recognized approach for managing efforts  to make system improvements. According to this model, user satisfaction  improvement efforts should include a written plan that serves as the  foundation and basis for guiding improvement activities, including  obtaining management commitment to and funding for the activities,  establishing a baseline of commitments and expectations against which to  measure progress, prioritizing and executing activities and initiatives,  determining success, and identifying and applying lessons learned.  Through such a structured and disciplined approach, improvement  resources can be invested in a manner that produces optimal results.
The Software Engineering Institute is a federally funded research and development center  established at Carnegie Mellon University to address software engineering practices.  IDEALSM is a service mark of Carnegie Mellon University and stands for initiating,  diagnosing, establishing, acting, and leveraging. For more information on this model, see  IDEALSM: A User’s Guide for Software Process Improvement (CMU/SEI-96-HB-001). satisfaction levels in July 2007 after overall user satisfaction had declined  to its lowest point in more than 2 years. Between 2005 and 2007 the  program office collected user satisfaction feedback through online user  surveys, and used the data to identify areas for system improvements and  to measure progress toward improving satisfaction. The results of the  surveys showed not only that users rated their overall satisfaction level  with the system between below average and average, but that user  satisfaction levels had declined to a low point with the results of the final  survey report of July 2007. Thus, as shown in figure 1, the program office  was not able to improve user satisfaction during this time period.
According to program officials, they have implemented a major effort  toward improving user satisfaction with the AHLTA 3.3 software release.  The improvements associated with this software release began as early as  2006 and include features such as improved medical coding support and  increased speed of the order entry connection, as well as other changes to  improve users’ satisfaction with the system’s performance and  capabilities. Yet, program officials did not provide evidence of a plan to  guide these efforts or a schedule for implementing these improvements,  and it is unclear how specific capabilities of the software release will be  used to address specific user concerns. The lack of a documented plan to  guide user satisfaction improvement activities is of particular significance  because users have continued to express their dissatisfaction with the  system. Program officials stated that additional online user satisfaction  surveys were not conducted after 2007 because users had grown weary of  the surveys and efforts to address user feedback from the existing survey  results are ongoing. The next online survey is expected to be conducted  after full deployment of AHLTA 3.3, but a schedule has not yet been set.
Given the history of system performance problems and the extent to  which users have not been able to effectively and efficiently use AHLTA, it  is critical that the department identify and implement system  improvements in a disciplined and structured fashion. Without a  documented improvement plan, efforts to improve user satisfaction,  including those associated with the ongoing AHLTA stabilization effort,  may be reduced to trial and error, and the office cannot adequately ensure  that it is effectively investing program resources on improvement efforts  that will result in a system that satisfies users. Further, since increasing  user satisfaction is a key goal for the EHR Way Ahead, it is critical that a  disciplined approach is established and maintained throughout the  program’s life cycle.
section_title: MHS Lacks Assurance of a Disciplined Acquisition Management Process to Guide Its Electronic Health Record Initiative
paragraphs: The use of disciplined processes to guide the effort of acquiring and  implementing a major system has been shown to increase the likelihood of  achieving intended results and reduce the risks associated with an  acquisition to acceptable levels. Although there is no standard set of  practices that will ever guarantee success, several organizations, such as  Carnegie Mellon University’s Software Engineering Institute and the  Institute of Electrical and Electronics Engineers (IEEE), as well as  individual experts, have identified and developed the types of policies,  procedures, and practices that have been demonstrated to reduce  development time and enhance effectiveness. The key to having a  disciplined system development effort is to have disciplined processes in  multiple areas, including project planning, requirements management,  systems engineering, system testing, and risk management. Because  change in a program is constant, effective processes should be  implemented in each of these throughout the project life cycle. Effectively  implementing the disciplined processes necessary to reduce project risks  to acceptable levels is difficult because a project must effectively  implement several best practices, and inadequate implementation of any  one may significantly reduce or even eliminate the positive benefits of the  others.
Recognizing weaknesses in its acquisition of systems such as AHLTA, MHS  has been taking steps to institutionalize more disciplined management  processes across all of its programs. In March 2008 the MHS CIO identified  an approach for improving its management processes that included  aligning MHS processes with best practices outlined in the Software  Engineering Institute’s Capability Maturity Model Integration (CMMI) for  Acquisition. In support of the approach, certain program offices, including  DHIMS (the program office responsible for the AHLTA acquisition), were  selected for an internal evaluation to identify areas for improvement in the  existing MHS processes. The assessment, which was conducted in May  2008, identified weaknesses in processes such as project management,  requirements development, and project monitoring and control, among  others. It also identified weaknesses in MHS’s oversight of the  implementation of these processes within program offices. Specifically,  the assessment identified weaknesses in the area of Process and Product  Quality Assurance, which is supposed to provide staff and management  with objective insight into processes associated with work products. The  assessment found little evidence that process evaluations were performed  across the organization, quality assurance audits were conducted, and  noncompliance issues were tracked and reported.
In response to the assessment, officials stated that they established a plan  for addressing the identified weaknesses. Specifically, their goal was to  achieve CMMI’s “maturity level 2” for processes such as project planning  and acquisition requirements development. Level 2 processes are  “managed” processes, or processes that are planned and executed in  accordance with policy; employ skilled people who have adequate  resources to produce controlled outputs; involve relevant stakeholders;  are monitored, controlled, and reviewed; and are evaluated for adherence  to their process description. The department planned to conduct a formal  external assessment of the maturity of its processes by December 2008.
Program officials stated that they provided guidance and assistance for  program offices to adopt practices associated with CMMI maturity level 2  processes. However, they have yet to perform the planned external  assessment of their processes, and there is therefore little assurance that  improvements have been carried out. As the department proceeds with the  AHLTA stabilization effort, it is critical that it have disciplined processes in  place to avoid past problems with not delivering system improvements as  planned. Further, as the department is allocating resources to and  planning for the EHR Way Ahead acquisition, it is critical that it have  disciplined management processes in place to avoid repeating the  mistakes of the past. Until the department ensures that these disciplined  and managed processes are in place, it risks delivering another system  with limited functionality and performance problems and that does not  meet the needs of its users.
section_title: Conclusions
paragraphs: After over a decade of effort, DOD has not accomplished what it set out to  achieve in acquiring a comprehensive electronic health record system.  While it has delivered a number of outpatient capabilities, weaknesses in  key management areas hindered its ability to deliver the full complement  of intended capabilities and to ensure that the capabilities it has delivered  meet required performance parameters. The program office did not  maintain a comprehensive and current project management plan, a critical  document that provides stakeholders insight into the project’s plans and  status. Also, despite the department’s need to deliver a complex,  worldwide system, it did not develop a systems engineering plan to help  address the technical aspects of the project, and it continues to experience  problems with system availability, speed, and usability. Further, the  system requirements were too general and did not adequately reflect user  needs. Although the department has collected user feedback, it did not  establish a comprehensive plan for improving user satisfaction with the  system. Recognizing weaknesses in acquisition management areas, the  MHS CIO issued guidance for improving its management processes, but it  has not performed the planned external assessment that it needs to certify  that these improvements have been made or established a date for doing  so.
As DOD continues to invest significant resources in a stabilization effort to  address shortcomings of AHLTA and plan for the acquisition of a new  electronic health record system, it is imperative that the department take  immediate steps to improve its management of the initiative. Until it does  so, it risks a continuation of the problems it has already experienced,  which could again prevent DOD from delivering a comprehensive health  record system for serving its service members and their families.
section_title: Recommendations for Executive Action
paragraphs: To help guide and ensure the successful completion of the AHLTA  stabilization effort, we recommend that the Secretary of Defense, through  the Assistant Secretary of Defense for Health Affairs, direct the MHS CIO  to take the following six actions:    Develop and maintain a comprehensive project plan that includes key  elements, such as the project’s scope, cost, schedule, and risks and update  the plan to provide key information for stakeholders on the project’s plans  and status.
Develop a systems engineering plan in accordance with DOD guidance to  address the technical complexities of delivering a worldwide electronic  health record system.
Ensure that its requirements development process involves system users  throughout the development process, to obtain an understanding of what  will satisfy their needs.
Ensure the establishment of bidirectional traceability for all system  requirements.
Develop and document a plan for improving user satisfaction that  prioritizes improvement projects; identifies needed resources; includes  schedules for improvement efforts, including future user feedback  surveys; and links efforts to measurable outcomes and specific user needs.
Establish acquisition management processes in accordance with industry  best practices, including identifying milestones and a completion date for  the external evaluation that MHS’s processes are at maturity level 2 of the  Capability Maturity Model Integration for Acquisition.
Further, to help ensure that the EHR Way Ahead does not have shortfalls  similar to those experienced with AHLTA, we recommend that the above  six management practices be implemented as part of the planning for this  important initiative.
section_title: Agency Comments and Our Evaluation
paragraphs: The Deputy Assistant Secretary of Defense (Force Health Protection and  Readiness), performing the duties of the Assistant Secretary of Defense  (Health Affairs), provided written comments on a draft of this report. In its  comments, the department agreed with our six recommendations and  described actions planned to address them.
Specifically, to help guide and ensure the successful completion of the  AHLTA stabilization effort, DOD stated that it will develop and maintain a  comprehensive project plan in accordance with our recommendation and  DOD acquisition program guidelines. It also stated that it plans to develop  a systems engineering plan to address the technical complexities of the  project in accordance with current DOD requirements. Further, to obtain  an understanding of system users’ needs, the department stated that it  plans to engage users and manage the requirements development process  in accordance with our recommendation. The department stated that it  will ensure that bidirectional traceability is performed for all system  requirements. Regarding its intent to develop and document a plan for  improving user satisfaction, including identifying needed resources and a  schedule for improvement, the department stated that it will augment its  current user feedback plan to include these and other key elements, such  as measurable outcomes. Further, in response to the need to establish  acquisition management processes in accordance with industry best  practices, at maturity level 2, the department said it plans to establish a  milestone for completing the external review in accordance with  Capability Maturity Model guidelines.
Finally, the department stated that it will ensure that the six  recommendations are implemented as part of future EHR Way Ahead  initiative. To the extent that the department follows through on  implementing the recommendations, it should be better positioned to  deliver a comprehensive electronic health care record for serving its  service members and others entitled to military health care.
DOD also provided technical comments on our draft report. In these  comments, DOD said it took exception to several inaccurate, misleading,  and subjective statements provided in the report. The department said that  GAO’s statements conflicted with the extensive volume of programmatic  documentation, written responses, and consistent interview feedback  provided during the course of the audit. In particular, the department  believed that the report did not sufficiently reflect AHLTA’s operational  capabilities and its benefit to DOD’s worldwide health care operations.  While we agree that the department provided substantial documentation,  we believe that our analysis of the information received supports our  findings. Where appropriate, however, we have made revisions to  statements in the report to update our discussions of AHLTA’s operational  capabilities and the program’s management.
The department’s written comments are reproduced in appendix II. The  department also provided technical comments, which we have  incorporated in the report as appropriate  We are sending copies of the report to appropriate congressional  committees, the Secretary of Defense, and other interested parties. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) determine the Department of Defense’s (DOD)  status in implementing the Armed Forces Health Longitudinal Technology  Application (AHLTA) system, (2) determine the department’s plans for  acquiring a new electronic health record system, and (3) evaluate the  department’s acquisition management for its electronic health record  system.
To determine the department’s status in implementing the AHLTA system,  we reviewed project status reports, acquisition decision memorandums,  quarterly defense acquisition executive summaries, monthly in-progress  review reports, monthly contractor performance reports, and overarching  integrated project team meeting minutes. We supplemented these reviews  with interviews of DOD officials in the Defense Health Information  Management System (DHIMS) Program Office, including the DHIMS  Program Manager, Deputy Program Manager, and Director of Products  Branch officials with whom we discussed the project’s cost and schedule,  as well as the planning, development, and deployment of the original and  current release of AHLTA. We also attended two demonstrations of  AHLTA: at the program office located in Falls Church, Virginia, and at the  National Naval Medical Center in Bethesda, Maryland. We observed  demonstrations of AHLTA system functionality and held discussions with  system users. We also observed a daily technical review meeting with  technical staff from the Army, Navy, and Air Force in which the discussion  largely focused on the reporting of issues that caused the system to be  unavailable to users at various locations for up to 24 hours. The discussion  also included identification of known root causes of the availability  problems (e.g., incorrectly configured firewalls, tripped network circuits,  and problems with virtual private networks) and planned actions to  address the issues.
To determine the department’s plans for acquiring a new system, we  reviewed Electronic Health Record (EHR) Way Ahead planning  documents. Specifically, we reviewed the acquisition decision  memorandum issued by the milestone decision authority, the Joint  Requirements Oversight Council-approved Initial Capabilities Document  to identify EHR needs, and the Capabilities-Based Assessment. We also  reviewed the analysis of alternatives procedures for guidance on  determining a technology solution for the new EHR. We also reviewed  department briefings issued between 2008 and 2010, as well as a prepared  statement to Congress from 2009 on preliminary plans for the EHR Way  Ahead. These documents provided a high-level overview of the need and  the goals for the new system, as well as plans for the system’s enterprise  architecture and expected capabilities. We supplemented our review by  interviewing officials from the EHR Way Ahead planning office, including  the department’s Acting Chief Information Officer, the DHIMS Program  Manager, and the DHIMS Deputy Program Manager.
To evaluate the department’s acquisition management for its electronic  health record system initiative, we evaluated key practices used by the  agency against best practices. In this regard, we examined practices  related to project management planning, systems engineering planning,  system requirements development and management, and user satisfaction  improvement planning and compared the agency’s work with agency  policy, guidance, and recognized best practices. Specifically:    To assess DOD’s project planning for AHLTA, we compared the program’s  project management plan against relevant guidance, including the Military  Health System’s project management process area description and our  Information Technology Investment Management framework for assessing  and improving process maturity.
We assessed the agency’s approach to systems engineering by comparing  program documentation such as acquisition strategies and the AHLTA  project management plan to systems engineering guidance from the  Defense Acquisition University on systems engineering. We also reviewed  relevant agency policies, such as DOD Instruction 5000.02 which discusses  the use of systems engineering across the acquisition life cycle and  memorandums from the Office of the Under Secretary of Defense on a  2004 revision to the policy regarding use of a systems engineering plan, to  determine whether the AHLTA program was guided by appropriate  systems engineering planning documents such as a systems engineering  plan.
Regarding requirements development, we reviewed program procedures  describing the processes for developing requirements and reviewed  relevant external evaluations of the effectiveness of those processes  against recognized guidance. Specifically, we reviewed an external  evaluation of the requirements development processes including the 2002  Carnegie Mellon External Assessment of the AHLTA program office and  the process area description or requirements management. We also  reviewed the 2008 internal assessment of requirements management; a  2009 concept of operations document for a more integrated,  departmentwide requirements development process; and the 2010 Joint  Requirements Oversight Council-approved Initial Capabilities Document,  which identifies past challenges with the department’s requirements  processes. In addition, we analyzed the requirements traceability matrix  for the most recent version of AHLTA to determine the extent to which  bidirectional traceability had been performed. We also reviewed program  documentation relative to requirements development and user community  participation. In addition, we interviewed process improvement officials,  including the cognizant official from the Office of the Chief Information  Officer (CIO) about internal acquisition process evaluations and their  results and the status of plans for improving acquisition management  processes. We then compared the department’s current approach to  requirements development and management with best practices identified  in the Software Engineering Institute’s Capability Maturity Model  Integration for Acquisition.
To assess the department’s approach to improving user satisfaction, we  reviewed and analyzed program documentation pertaining to the  collection, analysis, and utilization of AHLTA user satisfaction feedback  such as seven survey reports and a postimplementation review that were  produced between 2005 and 2007 and compared the agency’s approach to  best practices such as the Office of Management and Budget’s Capital  Programming Guide and Standards and Guidelines for Statistical Surveys.  We also reviewed lessons learned reports from 2006 through 2008 and a  user conference briefing from 2006 that identified areas of user  dissatisfaction. In addition, we reviewed program office documents that  identified improvement initiatives such as the AHLTA 3.3 software release  and the deployment of local cache servers, which were intended to  improve user satisfaction. We supplemented our review by interviewing  program officials, including the DHIMS Program Manager and Deputy  Program manager, to determine the extent to which user satisfaction  improvement efforts and initiatives have been guided by documented  plans. We then compared the department’s approach to improving user  satisfaction with the Software Engineering Institute’s IDEALSM1 model,  which is a recognized approach for managing process improvement efforts  such as managing improvements to user satisfaction.
The Software Engineering Institute is a federally funded research and development center  established at Carnegie Mellon University to address software engineering practices.  IDEAL is a service mark of Carnegie Mellon University and stands for initiating, diagnosing,  establishing, acting, and leveraging. For more information on this model, see IDEALSM: A  User’s Guide for Software Process Improvement (CMU/SEI-96-HB-001).
DOD’s 2008 internal assessment related to acquisition management  processes, action plans, and tasks planned for process improvement.
We supplemented our analysis with interviews with officials in the DHIMS  Office, including, the Program Manager, Deputy Program Manager,  Director of Products Branch and Engineering and Resources offices. We  also obtained written responses from the responsible program manager or  subject matter expert for areas of our review. These responses were  approved by the MHS CIO or the Program Executive Officer, Joint Medical  Information Systems/Deputy MHS CIO.
We did not conduct an independent validation of the life-cycle costs and  obligations provided to us by DOD.
We conducted this performance audit at the DHIMS Program Office in  Falls Church, Virginia, and the National Naval Medical Center, in  Bethesda, Maryland, from September 2009 through October 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Cynthia J. Scott (Assistant  Director); Harold Brumm, Jr.; Neil Doherty; Ronalynn Espedido; Rebecca  Eyler; Nancy Glover; Joel Grossman; Linda Kochersberger; Lee  McCracken; Madhav Panwar; Donald Sebers; Sylvia Shanks; Adam  Vodraska; Daniel Wexler; and Robert Williams, Jr. made key contributions  to this report.