Information Technology: VA and DOD Face Challenges in Completing Key Efforts
Released: 2006-06-22T00:00:00

Letter

Background

VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring they receive medical care, benefits, social support, and lasting memorials. The information technology programs that I will be discussing today are primary concerns of two of VA’s major components: the Veterans Health Administration, which manages one of the largest health care systems in the United States, with 157 hospitals nationwide, and the Veterans Benefits Administration, which provides benefits and services to veterans and their dependents that include compensation and pension, educat guaranty, and insurance.

VA and DOD Have Been Working on Electronic Medical Records Since 1998

In 1998, following a presidential call for VA and DOD to start developing a “comprehensive, life-long medical record for eac service member,” the two departments began a joint course of action aimed at achieving the capability to share patient health h information for active duty military personnel and veterans. Their first initiative, undertaken in that year, was known as the Government Computer-Based Patient Record (GCPR) project; the goal of this project was an electronic interface that would allow physicians and other authorized users at VA and DOD health facilities to access data from any of the other agency’s health information systems. The interface was expected to compile requested patient information in a virtual record that could be displayed on a user’s computer screen.

In our reviews of the GCPR project, we determined that the lack of a lead entity, clear mission, and detailed planning to achieve that mission made it difficult to monitor progress, identify project risks, and develop appropriate contingency plans. In April 2001 and in June 2002, we made recommendations to help strengthen the management and oversight of the project. In 2001, we recommended that the participating agencies (1) designate a lead entity with final decision-making authority and establish a clear line of authority for the GCPR project and (2) create comprehensive and coordinated plans that included an agreed-upon mission and clear goals, objectives, and performance measures, to ensure that the agencies could share comprehensive, meaningful, accurate, and secure patient health care data. In 2002, we recommended that the participating agencies revise the original goals and objectives of the project to align with their current strategy, commit the executive support necessary to adequately manage the project, and ensure that it followed sound project management principles.

VA and DOD took specific measures in response to our recommendations for enhancing overall management and accountability of the project. By July 2002, VA and DOD had revised their strategy and had made progress toward being able to electronically share patient health data. The two departments had refocused the project and named it the Federal Health Information Exchange (FHIE) program and, consistent with our prior recommendation, had finalized a memorandum of agreement designating VA as the lead entity for implementing the program. This agreement also established FHIE as a joint activity that would allow the transfer from DOD to VA of health care information in two phases: ● The first phase, completed in mid-July 2002, enabled the one-way transfer of data from DOD’s existing health information system (the Composite Health Care System or CHCS) to a separate database that VA clinicians could access. ● A second phase, finalized in March 2004, completed VA’s and DOD’s efforts to add to the base of patient health information available to VA clinicians via this one-way sharing capability.

According to the December 2004 VA/DOD Joint Executive Council Annual Report, FHIE was fully operational, and providers at all VA medical centers and clinics nationwide had access to data on separated service members. According to the report, the FHIE data repository at that time contained historical clinical health data on 2.3 million unique patients from 1989 on, and the repository made a significant contribution to the delivery and continuity of care and adjudication of disability claims of separated service members as they transitioned to veteran status. The departments reported total GCPR/FHIE costs of about $85 million through fiscal year 2003.

In addition, officials stated that in December 2004, the departments began to plan for using the FHIE framework to transfer pre- and postdeployment health assessment data from DOD to VA. According to these officials, transferring of this information began in July 2005, and VA has now received about 1.3 million of these records on more than 560,000 separated service members.

However, not all DOD medical information is captured in CHCS. For example, according to DOD officials, as of September 2005, 1.7 million patient stay records were stored in the Clinical Information System (a commercial product customized for DOD). In addition, many Air Force facilities use a system called the Integrated Clinical Database for their medical information.

The revised DOD/VA strategy also envisioned achieving a longer term, two-way exchange of health information between DOD and VA, which may also address systems outside of CHCS. Known as HealthePeople (Federal), this initiative is premised on the departments’ development of a common health information architecture comprising standardized data, communications, security, and high-performance health information systems. The joint effort is expected to result in the secured sharing of hea between the new systems that each department is currently developing and beginning to implement—DOD’s AHLTA and VA’s HealtheVet VistA.

DOD began developing AHLTA in 1997. component for the planned electronic interface—its Clinical Data Repository, and it expects to complete deployment of all of its major system capabilities by 2011. (When we reported in June 2004, this deployment was expected in September 2008.) DOD expects to spend about $783 million for the system through fiscal year 2006.

At that time it was known as CHCS II. In November 2005, DOD renamed CHCS II the Armed Forces Health Longitudinal Technology Application (AHLTA). million on initiatives that comprise HealtheVet VistA through fiscal year 2005.

Under the HealthePeople (Federal) initiative, VA and DOD envision that, on entering military service, a health record for the service member would be created and stored in DOD’s Clinical Data Repository. The record would be updated as the service member receives medical care. When the individual separated from activeduty and, if eligible, sought medical care at a VA facility, VA w then create a medical record for the individual, which would be stored in its Health Data Repository. On viewing the medical recor the VA clinician would be alerted and provided with access to the individual’s clinical information residing in DOD’s repository. In same manner, when a veteran sought medical care at a military treatment facility, the attending DOD clinician would be alerted an provided with access to the health information in VA’s repository. According to the departments, this planned approach would mak e virtual medical records displaying all available patient health information from the two repositories accessible to both departments’ clinicians.

December 2004 VA and DOD Joint Strategic Plan.

Developing the two repositories, populating them with data, a linking them through the CHDR interface would be important steps toward the two departments’ long-term goals as envisioned in HealthePeople (Federal). Achieving these goals would then depend on completing the development and deployment of the associated health information systems—HealtheVet VistA and AHLTA.

In a review of the CHDR program in June 2004, we reported that the efforts of DOD and VA in this area demonstrated a number of management weaknesses. Among these were the lack of a well- defined architecture for describing the interface for a common health information exchange; an established project management lead entity and structure to guide the investment in the interface a its implementation; and a project management plan defining thetechnical and managerial processes necessary to satisfy project requirements. With these critical components missing, VA and DOD increased the risk that they would not achieve their goals. Accordingly, we recommended that the departments develop an architecture for the electronic interface between their health systems that includes system requirements, design specifications, and software descriptions; ● select a lead entity with final decision-making authority for the establish a project management structure to provide day-to guidance of and accountability for their inv estments in and implementation of the interface capability; and create and management plan for the electronic interface that defines the technical and managerial processes necessary to satisfy project requirements and includes (1) the authority and responsibility of each organizational unit; (2) a work breakdown structure for all o the tasks to be performed in developing, testing, and implemen the software, along with schedules associated with the tasks; and (3) a security policy. implement a comprehensive and coordinated project f ting In September 2005, we testified that VA and DOD had made progress in the electronic sharing of patient health data in their near-term demonstration projects. We noted that with regard to their long-term goals, the departments had improved the management of the CHDR program, but that this program continued to face significant challenges—in particular, developing a project management plan of sufficient specificity to be an effective guide for the program.

Besides pursuing their long-term goals for future systems through the HealthePeople (Federal) strategy, the departments are working on two demonstration projects that focus on exchanging information between existing systems: (1) Bidirectional Health Information Exchange, a project to exchange health information on shared patients, and (2) Laboratory Data Sharing Interface, an application used to transfer laboratory work orders and results. These demonstration projects were planned in response to provisions of the Bob Stump National Defense Authorization Act of 2003, which mandated that VA and DOD conduct demonstration projects that included medical information and information technology systems to be used as a test for evaluating the feasibi advantages, and disadvantages of measures and programs designed to improve the sharing and coordination of health care and health care resources between the departments.

Figure 1 is a time line showing initiation points for the VA and DOD efforts discussed here, including strategies, major programs, and the recent demonstration projects.

Work on VETSNET Dates to 1986

The VETSNET effort grew out of an initiative begun by the Veterans Benefits Administration (VBA) in 1986 to replace its outdated Benefits Delivery Network. The Benefits Delivery Network, parts of which were developed in the 1960s, contains over 3 million veterans benefits records, including compensation and pension, education, and vocational rehabilitation and employment. Originally, the plan was to modernize all of these systems and in so doing provide a rich source for answering questions about veterans’ benefits and enable faster processing of benefits. As envisioned in the 1980s, the modernization would produce a faster, more flexible, higher capacity system that would be both an information system and a payment system. In 1996, after experiencing numerous false starts and spending approximately $300 million on the overall modernization of BDN, VBA revised its strategy and narrowed its focus to modernizing the compensation and pension payment system.

At that time, we undertook an assessment of the department’s software development capability and determined that it was immature. In our assessment, we specifically examined the VETSNET effort and concluded that VBA could not reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints. VBA showed significant weaknesses in requirements management, software project planning, and software subcontract management, with no identifiable strengths. We also testified that (1) VBA did not follow sound systems development practices on VETSNET, such as validation and verification of systems requirements; (2) it employed for the project a new systems development methodology and software development language not previously used; and (3) it did not develop the cost-benefit information necessary to track progress or assess return on investment (for example, total software to be developed and cost estimates). As a result, we concluded that VBA’s modernization efforts had inherent risks.

Between 1996 and 2002 we reported several more times on VETSNET, highlighting concerns in several areas. (See attachment 1 for a description of the conclusions and findings of our products on this topic.) In these products, we made several recommendations aimed at improving VA’s software development capabilities, including that the department take steps to achieve greater maturity in its software development processes and that it delay any major investment in software development (beyond that needed to sustain critical day-to-day operations) until it had done so. In addition, we made recommendations aimed specifically at VETSNET development, including that VBA assess and validate users’ requirements for the new system; complete testing of the system’s functional business capability, as well as end-to-end testing to ensure that payments are made accurately; and establish an integrated project plan to guide its transition from the old to the new system.

Although VBA took various actions in response to these recommendations, we continued to identify the department’s weak software development capability as a significant factor contributing to VBA’s persistent problems in developing and implementing the system—the same condition that we identified in 1996. We also reported that VBA continued to work on VETSNET without an integrated project plan. As a result, the development of VETSNET continued to suffer from problems in several areas, including project management, requirements development, and testing.

VA and DOD Are Working to Share Medical Information

VA and DOD have made progress in sharing patient health data by implementing applications developed under two demonstration projects that focus on the exchange of electronic medical information. The first—the Bidirectional Health Information Exchange—has been implemented at 16 VA/DOD locations, and the second—Laboratory Data Sharing Interface—has been implemented at 6 VA/DOD locations.

Bidirectional Health Information Exchange. According to a VA/DOD annual report and program officials, Bidirectional Health Information Exchange (BHIE) is an interim step in the departments’ overall strategy to create a two-way exchange of electronic medical records. BHIE builds on the architecture and framework of FHIE, the application used to transfer health data on separated service members from DOD to VA. As discussed earlier, FHIE provides an interface between VA’s and DOD’s existing health information systems that allows one-way transfers only, which do not occur in real time: VA clinicians do not have access to transferred information until about 6 weeks after separation. In contrast, BHIE focuses on the two-way, near-real-time exchange of information (text only) on shared patients (such as those at sites jointly occupied by VA and DOD facilities). This application exchanges data between VA’s VistA system and DOD’s CHCS system (and AHLTA where implemented). As of September 2005, the departments reported having spent $2.6 million on BHIE.

The primary benefit of BHIE is near-real-time access to patient medical information for both VA and DOD, which is not available through FHIE. During a site visit to a VA and DOD location in Puget Sound in 2005, we viewed a demonstration of this capability and were told by a VA clinician that the near-real-time access to medical information was very beneficial in treating shared patients.

As of June 2006, BHIE was deployed at VA and DOD facilities at 16 sites, where the exchange of demographic, outpatient pharmacy, radiology, laboratory, and allergy data (text only) has been achieved. In addition, according to officials, over 120 outpatient military clinics associated with these sites also have access to this information through BHIE. According to VA and DOD, BHIE will be implemented at two more sites in July 2006. Table 1 presents a schedule for implementation of BHIE; the sites listed are all DOD sites with nearby VA facilities.

Additionally, because DOD stores electronic medical information in systems other than CHCS (such as the Clinical Information System and the Integrated Clinical Database), work is currently under way to allow BHIE to have the ability to exchange information with those systems. Currently, one site is testing the use of BHIE as an interface allowing both departments’ staff to view discharge summaries stored in the Clinical Information System. DOD and VA plan to perform a side-by-side comparison to ensure that this capability maintains data quality. When they are satisfied, the capability will be provided to those DOD locations that currently use the Clinical Information System and have BHIE implemented. Doing so will permit all VA sites access to the information in the Clinical Information System on shared patients at DOD sites running BHIE.

In addition, at the VA/DOD site in El Paso, a prototype is being designed for exchanging radiological images using the BHIE/FHIE infrastructure. If the prototype is successful, this capability will be extended to the rest of the sites.

Laboratory Data Sharing Interface. The Laboratory Data Sharing Interface (LDSI) initiative enables the two departments to share laboratory resources. Through LDSI, a VA provider can use VA’s health information system to write an order for laboratory tests, and that order is electronically transferred to DOD, which performs the test. The results of the laboratory tests are electronically transferred back to VA and included in the patient’s medical record. Similarly, a DOD provider can choose to use a VA lab for testing and receive the results electronically. Once LDSI is fully implemented at a facility, the only nonautomated action in performing laboratory tests is the transport of the specimens.

Among the benefits of LDSI are increased speed in receiving laboratory results and decreased errors from manual entry of orders. However, according to the LDSI project manager in San Antonio, a primary benefit of the project will be the time saved by eliminating the need to rekey orders at processing labs to input the information into the laboratories’ systems. Additionally, the San Antonio VA facility will no longer have to contract out some of its laboratory work to private companies, but instead use the DOD laboratory. As of September 2005, the departments reported having spent about $3.3 million on LDSI.

An early version of what is now LDSI was originally tested and implemented at a joint VA and DOD medical facility in Hawaii in May 2003. The demonstration project built on this application and enhanced it; the resulting application was tested in San Antonio and El Paso. It has now been deployed to six sites. According to the departments, a plan to export LDSI to two additional locations has been approved. Table 2 shows the locations at which it has been or is to be implemented.

VA and DOD Are Taking Action to Achieve a Virtual Medical Record, but Much Work Remains

Besides the near-term initiatives just discussed, VA and DOD continue their efforts on the longer term goal: to achieve a virtual medical record based on the two-way exchange of computable data between the health information systems that each is currently developing. The cornerstone for this exchange is CHDR, the planned electronic interface between the data repositories for the new systems.

The departments have taken important actions on the CHDR initiative. As we testified in September 2005, they successfully completed Phase I of CHDR in September 2004 by demonstrating the two-way exchange of pharmacy information with a prototype in a controlled laboratory environment. According to department officials, the pharmacy prototype provided invaluable insight into each other’s data repository systems, architecture, and the work that is necessary to support the exchange of computable information. These officials stated that lessons learned from the development of the prototype were documented and being applied to Phase II of CHDR, the production phase, which is to implement the two-way exchange of patient health records between the departments’ data repositories. Further, the same DOD and VA teams that developed the prototype were developing the production version.

In addition, the departments developed an architecture for the CHDR electronic interface, as we recommended in June 2004. The architecture for CHDR includes major elements required in a complete architecture. For example, it defines system requirements and allows these to be traced to the functional requirements, it includes the design and control specifications for the interface design, and it includes design descriptions for the software.

Also in response to our recommendations, the departments established project accountability and implemented a joint project management structure. Specifically, the Health Executive Council was established as the lead entity for the project. The joint project management structure consists of a Program Manager from VA and a Deputy Program Manager from DOD to provide day-to-day guidance for this initiative. Additionally, the Health Executive Council established the DOD/VA Information Management/Information Technology Working Group and the DOD/VA Health Architecture Interagency Group, to provide programmatic oversight and to facilitate interagency collaboration on sharing initiatives between DOD and VA.

To build on these actions and successfully carry out the CHDR initiative, however, the departments still have a number of challenges to overcome. The success of CHDR will depend on the departments’ instituting a highly disciplined approach to the project’s management. Industry best practices and information technology project management principles stress the importance of accountability and sound planning for any project, particularly an interagency effort of the magnitude and complexity of this one.

Accordingly, in 2004 we recommended that the departments develop a clearly defined project management plan that describes the technical and managerial processes necessary to satisfy project requirements and includes (1) the authority and responsibility of each organizational unit; (2) a work breakdown structure for all of the tasks to be performed in developing, testing, and implementing the software, along with schedules associated with the tasks; and (3) a security policy. As of September 2005, the departments had an interagency project management plan that provided the program management principles and procedures to be followed by the project. However, this plan did not specify the authority and responsibility of organizational units for particular tasks; the work breakdown structure was at a high level and lacked detail on specific tasks and time frames; and security policy was still being drafted. No more recent plan has yet been provided. Without a plan of sufficient detail, VA and DOD increase the risk that the CHDR project will not deliver the planned capabilities in the time and at the cost expected.

In addition, officials did not meet a previously established milestone: by October 2005, the departments had planned to be able to exchange outpatient pharmacy data, laboratory results, allergy information, and patient demographic information on a limited basis. However, according to officials, the work required to implement standards for pharmacy and medication allergy data was more complex than originally anticipated and would result in a delay. The new target date for the limited exchange of medication allergy, outpatient pharmacy, and patient demographic data has been postponed from February to June 2006.

Currently, the departments report that they are close to finishing the development of a pilot to perform this data exchange at their joint facility in El Paso. They expect to be able to begin the pilot by the end of this month, which will allow them to share outpatient pharmacy and medication allergy information that can support drug- drug interaction checking and drug-allergy alerts. If the pilot is successful, it will enable for the first time the exchange of computable information between the departments’ two data repositories.

Finally, the health information currently in the data repositories has various limitations. ● Although DOD’s Clinical Data Repository includes data in the categories that were to be exchanged at the missed milestone described above (outpatient pharmacy data, laboratory results, allergy information, and patient demographic information), these data are not yet complete. First, the information in the Clinical Data Repository is limited to those locations that have implemented the first increment of AHLTA, DOD’s new health information system. As of June 15, 2006, according to DOD officials, 115 of 138 medical treatment facilities worldwide have implemented this increment, and officials expect that the remaining facilities will receive the increment by the end of this year. Second, at present, health information in systems other than CHCS (such as the Clinical Information System and the Integrated Clinical Database) is not yet being captured in the Clinical Data Repository. However, work is currently under way to allow BHIE to have the ability to exchange information with those systems. ● The information in VA’s Health Data Repository is also limited: although all VA medical records are currently electronic, VA has to convert these into the interoperable format appropriate for the Health Data Repository. So far, the data in the Health Data Repository consist of patient demographics, vital signs records, allergy data, and outpatient pharmacy data for the 6 million veterans who have electronic medical records in VA’s current system, VistA (this system contains all the department’s medical records in electronic form). VA officials told us that they are currently converting lab results data.

VA Has Been Severely Challenged by VETSNET Project

Since its inception, the VETSNET program has been plagued by problems. In 2002, we offered a number of recommendations regarding the ongoing compensation and pension (C&P) replacement program. We testified that VBA should assess and validate users’ requirements for the new system and complete testing of the system’s functional business capability, including end- to-end testing. We also recommended that VA appoint a project manager, thoroughly analyze its current initiative, and develop a number of plans, including a revised C&P replacement strategy and an integrated project plan. We also noted that VBA had much work to do before it could fully implement the VETSNET C&P system by its target date (at that time) of 2005, and thus it would have to ensure that the aging Benefits Delivery Network (BDN) would be available to continue accurately processing benefits payments until a new system could be deployed. Accordingly, we recommended that VBA develop action plans to move from the current to the replacement system and to ensure the availability of BDN to provide the more than 3.5 million payments made to veterans each month.

VA concurred with our recommendations and took several actions to address them. For example, it appointed a full-time project manager. Also, the project team reported that to ensure that business needs were met, certification had been completed of users’ requirements for the system’s applications.

In addition, VA reported that a revised strategy for the replacement system was completed. This revised strategy included the business case, described the methodology used to identify system development alternatives, displayed the cost/benefit analysis results of the viable alternatives that could be used to develop the system, and provided a description of the recommended development plan. Based on this strategy, the Secretary of Veterans Affairs, Assistant Secretary for Information and Technology, the Under Secretary for Benefits, and the Deputy Chief Information Officer for Benefits approved continuation of the VETSNET development in September 2002.

Further, to ensure that the benefits delivery network would be able to continue accurately processing benefits payment until the new system was deployed, VBA purchased additional BDN hardware, hired 11 new staff members to support BDN operations, successfully tested a contingency plan in the event of disruption of the system, and provided retention bonuses to staff familiar with BDN operations.

However, VBA did not develop an integrated project plan for VETSNET, which is a basic requirement of sound project management. In addition, it did not develop an action plan for transitioning from the current to the replacement system. Thus, although the actions taken addressed some of our specific concerns, they were not sufficient to establish the program on a sound footing.

In 2005, the VA CIO became concerned by continuing problems with VETSNET: the project continued to postpone target dates, and costs continued to increase (VA indicated that by 2005 these costs exceeded $69 million). Accordingly, he arranged to contract for an independent assessment of the department’s options for the VETSNET project, including an evaluation of whether the program should be terminated. This assessment, conducted by the Carnegie Mellon Software Engineering Institute (SEI), concluded that the program faced many risks arising from management, organizational, and program issues, but no technical barriers that could not be overcome. According to SEI, terminating the program would not solve the underlying management and organizational problems, which would continue to hamper any new or revised effort.

SEI recommended that the department not terminate the program but take an aggressive approach to dealing with the issues SEI described while continuing to work on the program at a reduced pace. According to SEI, this approach would allow VA to make necessary improvements to its system and software engineering and program management capabilities while making gradual progress on the system. SEI also discussed specific concerns about the system’s management and the organization’s capabilities, presenting areas that required focus regardless of the particular course that VA chose for the system. For example: ● Setting realistic deadlines. SEI commented that there was no credible evidence that VETSNET would be complete by the target date, which at the time of the SEI review was December 2006. Because this deadline was unrealistic, VBA needed to plan and budget for supporting BDN so that its ability to pay veterans benefits would not be disrupted. ● Establishing an effective requirements process. ● Implementing effective program measurements in order to assess progress. ● Establishing sound program management. According to SEI, different organizational components had independent schedules and priorities, which caused confusion and deprived the department of a program perspective.

These observations are consistent with our long-standing concerns regarding fundamental deficiencies in VBA’s management of the project.

In the wake of the SEI assessment and recommendations, VA is in the process of creating, with contract help, an integrated master plan that is to cover the C&P replacement project. Because this plan is in process, no cost or schedule milestones have yet been finalized. According to VA, the integrated master plan is to be completed by the end of August 2006.

VA officials told us that they intend to complete this plan before beginning to plan for modernizing the systems for paying education benefits or for paying vocational rehabilitation and employment benefits. Plans for making the transition to VETSNET and ending VBA’s dependence on BDN are also on hold.

Thus, VA still lacks an integrated project plan or a plan to move from the current to the replacement system. Until it has an integrated project plan and schedule incorporating all the critical areas of the system development effort, VBA will lack the means of determining what needs to be done and when, and of measuring progress. Without plans to move from the current to the replacement system, VBA will lack assurance that it can c pay beneficiaries accurately and on time through the transition period.

In summary, developing an electronic interface that will enable VA and DOD to exchange computable patient medical records is a highly complex undertaking that could lead to substantial benef its— improving the quality of health care and disability claims processing for the nation’s service members and veterans. VA and DOD have made progress in the electronic sharing of patient health data in their limited, near-term demonstration projects, and have taken a important step toward their long-term goals by improving the management of the CHDR program. However, the departments considerable work and significant challenges before they can achieve these long-term goals. While the departments have ma progress in developing a project management plan, it is not yet complete. Having a project management plan of sufficient speci to guide the program—including establishing accountability and addressing security—would help the departments avoid further delays in their schedule and ensure that they produce a capabilit that meets their expectations.

VA has also been working to modernize the delivery of benefits through its development of VETSNET, but the pace of progress has been discouraging. Much work remains in accomplishing the original comprehensive goal of modernizing the aging system that VBA currently depends on to pay veterans benefits. Until VBA develops an integrated project plan that addresses the long-stan management weaknesses that we and others have identified, it will be uncertain when and at what cost VETSNET will be delivered.

Mr. Chairman, this concludes my statement. I would be pleased to respond to any questions that you or other members of the Subcommittee may have at this time.

Contacts and Acknowledgments

For information about this testimony, please contact Linda D. Koontz, Director, Information Management Issues, at (202) 512-6240 or at koontzl@gao.gov. Other individuals making key contributions to this testimony include Barbara S. Collier, Martin Katz, Barbara S. Oliver, Eric L. Trout, Robert Williams Jr., and Charles Youman.

Attachment 1. Past GAO Products Highlighting VETSNET Concerns

We previously performed several reviews addressing VETSNET and made numerous recommendations aimed at strengthening the program and VA’s software development and management capabilities. The table summarizes the results of these reviews.

GAO Products Highlighting Concerns with VETSNET Project to Replace Compensation and Pension (C&P) Payment System VETSNET had inherent risks in that (1) it did not follow sound systems development practices, such as validation and verification of systems requirements; (2) it employed a new systems development methodology and software development language not previously used; and (3) VBA did not develop the cost-benefit information necessary to track progress or assess return on investment (for example, total software to be developed and cost estimates).

VBA’s software development capability was immature and it could not reliably develop and maintain high- quality software on any major project within existing cost and schedule constraints, placing its software development projects at significant risk. VBA showed significant weaknesses in requirements management, software project planning, and software subcontract management, with no identifiable strengths.

VETSNET experienced schedule delays and missed deadlines because (1) it employed a new software development language not previously used by the development team, one that was inconsistent with the agency’s other systems development efforts; (2) the department’s software development capability was immature and it had lost critical systems control and quality assurance personnel, and (3) VBA lacked a complete systems architecture; for example, neither a security architecture nor performance characteristics had been defined for the project.

VBA’s software development capability remained ad hoc and chaotic, subjecting the agency to continuing risk of cost overruns, poor quality software, and schedule delays in software development. $11 million had reportedly been spent on VETSNET C&P; neither the May 1998 completion date nor the revised completion date of December 1998 were met. Contributing factors included lack of an integrated architecture defining the business processes, information flows and relationships, business requirements, and data descriptions, and VBA’s immature software development capability.

VBA’s software development capability remained ad hoc and chaotic. The VETSNET implementation approach lacked key elements, including a strategy for data conversion and an integrated project plan and schedule incorporating all critical systems development areas. Further, data exchange issues had not been fully addressed.

The project’s viability was still a concern. It continued to lack an integrated project plan and schedule addressing all critical systems development areas, to be used as a means of determining what needs to be done and when. A pilot test of 10 original claims that did not require significant development work may not have been sufficient to demonstrate that the product was capable of working as intended in an organizationwide operational setting.

VBA still had fundamental tasks to accomplish before it could successfully complete development and implementation. It still had to assess and validate users’ requirements for the new system to ensure that business needs were met. It needed to complete testing of the system’s functional business capability, as well as end-to-end testing to ensure that payments would be made accurately. Finally, it needed to establish an integrated project plan to guide its transition from the old to the new system.

VA still needed to address long-standing concerns regarding development and implementation. VA needed to appoint a project manager, undertake a complete analysis of the initiative, and develop plans, including a revised C&P replacement system strategy and an integrated project plan. It also needed to develop and implement action plans to move VBA from the current to the replacement system and to ensure that the Benefits Delivery Network would be able to continue accurately processing benefits payments until the new system was deployed.

Much work remained before VBA could fully implement the VETSNET C&P system, and complete implementation was not expected until 2005. This meant that VBA had to continue relying on its aging Benefits Delivery Network to provide the more than 3.5 million payments that VA had to make to veterans each month. In late March, a VETSNET executive board and a project control board were established to provide decision support and oversee implementation, and VBA expected to hire a full-time project manager by the end of September. VBA also began revalidating functional business requirements for the new system, with completion planned by January 2003, and it identified actions needed to transition VBA from the current to the replacement system. VBA also hired a contractor and tasked the contractor with conducting functional, integration, and linkage testing, as well as software quality assurance for each release of the system applications. Despite these actions, completing implementation of the new system could take several years. All but one of the software applications for the new system still needed to be fully deployed or developed. Specifically, a rating board automation tool (RBA 2000) was deployed, although VBA did not plan to require all its regional offices to use it until July 2003. In addition, two others had not been completely deployed: one of these (Share, used to establish a new claim) was in use by only 6 of the 57 regional offices. The other (Modern Award Processing–Development, used to develop information on claims) was in pilot testing at two regional offices—Salt Lake and Little Rock—but was not expected to be implemented at the other 55 regional offices until October 2003. The remaining three software applications (Award Processing, Finance and Accounting System, and Correspondence) were still in development.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.