Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist
Released: 2007-04-27T00:00:00

Letter

Background

VBA provides benefits for veterans and their families through five programs: (1) compensation and pension, (2) education, (3) vocational rehabilitation and employment (VRE) services, (4) loan guaranty, and (5) life insurance. It relies on the BDN to administer benefit programs for three of VBA’s five programs: compensation and pension, education, and VRE services.

Replacing the aging BDN has been a focus of systems development efforts at VBA since 1986. Originally, the administration planned to modernize the entire system, but after experiencing numerous false starts and spending approximately $300 million on the overall modernization of the BDN, VBA revised its strategy in 1996. It narrowed its focus to replacing only those functionalities that support the compensation and pension program, and began developing a replacement system, which it called VETSNET.

As reported by the department in its fiscal year 2008 budget submission, the compensation and pension program is the largest of the three programs that the BDN supports: The compensation and pension program paid about $35 billion in benefits in fiscal year 2006 to about 3.6 million veterans or veterans’ family members.

Of this amount, compensation programs paid benefits of about $31 billion to about 3.1 million recipients.

Pension programs paid benefits of about $3.5 billion to about 535,000 recipients.

The education program paid about $2.8 billion to about 498,000 veterans or their dependents in fiscal year 2006.

The VRE services program paid about $574 million for VRE services in 2006 and provides rehabilitation services to approximately 65,700 disabled veteran participants per year.

The Processes Supporting the Variety of Compensation and Pensions Benefits Are Complex

One of the challenges of developing the replacement system is that it must include processes to support the administration of a complex set of benefits. Different categories of veterans and their families are eligible for a number of different types of benefits and payments, some of which are based on financial need. Compensation programs, which are based on service-connected disability or death, provide direct payments to veterans and/or veterans’ dependents and survivors. These programs are not based on income. Pension benefits programs, on the other hand, are income based; these are designed to provide income support to eligible veterans and their families who experience financial hardship. Eligible veterans are those who served in wartime and are permanently and totally disabled for reasons that are not service-connected (or who are age 65 or older). Veterans are also eligible for burial benefits.

Survivor benefits may be paid to eligible survivors of veterans, depending on the circumstances. Some of these benefits are based on financial need, such as death pensions for some surviving spouses and children of deceased wartime veterans, and Dependency and Indemnity Compensation to some surviving parents.

Finally, certain benefits may be paid to third parties, such as individuals to whom a veteran has given power of attorney or medical service providers designated to receive payments on the veteran’s behalf.

Generally, VBA administers benefit programs through 57 veterans benefits regional offices in a process that requires a number of steps, depending on the type of claim. When a veteran submits, for example, a compensation claim to any of the regional offices, a veterans service representative must obtain the relevant evidence to evaluate the claim (such as the veteran’s military service records, medical examinations, and treatment records from VA medical facilities or private medical service providers). In the case of pension claims, income information would also be collected.

Once all the necessary evidence has been compiled, a rating specialist evaluates the claim and determines whether the claimant is eligible for benefits. If the veteran is determined to be eligible for disability compensation, the Rating Veterans Service Representative assigns a percentage rating based on the veteran’s degree of disability. This percentage is used in calculating the amount of payment.

Benefits received by veterans are subject to change depending on changing circumstances. More than half of VBA’s workload consists of dealing with such changes. If a veteran believes that a service-connected condition has worsened, for example, the veteran may ask for additional benefits by submitting another claim. The first claim submitted by a veteran is referred to as the original claim, and a subsequent change is referred to as a reopened claim.

Our Prior Products Identified Weaknesses in the Development and Implementation of the Replacement System

Since its inception, VETSNET has been plagued by problems. Over the years, we have reported on the project, highlighting concerns about VBA’s software development capabilities.

In 1996, our assessment of the department’s software development capability determined that it was immature. In our assessment, we specifically examined VETSNET and concluded that VBA could not reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints. The department showed significant weaknesses in requirements management, software project planning, and software subcontract management, with no identifiable strengths. We also testified that VBA did not follow sound systems development practices on VETSNET, and we concluded that its modernization efforts had inherent risks.

Between 1996 and 2002, we continued to identify the department’s weak software development capability as a significant factor contributing to persistent problems in developing and implementing the system. We also reported that VBA continued to work on VETSNET without an integrated project plan. As a result, the development of the system continued to suffer from problems in several areas, including project management, requirements development, and testing.

Over the years, we made several recommendations aimed at improving VA’s software development capabilities. Among our recommendations was that the department take actions to achieve greater maturity in its software development processes and that it delay any major investment in software development (beyond that needed to sustain critical day-to-day operations) until it had done so. In addition, we made specific recommendations aimed at improving VETSNET development. For example, we recommended that VA appoint a project manager, thoroughly analyze its current initiative, and develop a number of plans, including a revised compensation and pension replacement strategy and an integrated project plan.

VA concurred with our recommendations and took several actions to address them. For example, it appointed a full-time project manager and ensured that business needs were met by certification of user requirements for the system applications. The actions taken addressed some of our specific concerns; however, they were not sufficient to fully implement our recommendations or to establish the program on a sound footing.

SEI Assessed System Replacement Initiative and Recommended a Changed Approach

As a result of continuing concerns about the replacement project, in 2005 VA’s CIO and its Under Secretary for Benefits contracted for an independent assessment of the department’s options for the initiative. The chosen contractor, SEI, is a federally funded research and development center operated by Carnegie Mellon University. Its mission is to advance software engineering and related disciplines to ensure the development and operation of systems with predictable and improved cost, schedule, and quality.

SEI recommended that the department reduce the pace of development while at the same time taking an aggressive approach to dealing with management and organizational weaknesses hampering VBA’s ability to complete the replacement system. According to SEI, these management and organizational concerns needed to be addressed before the replacement initiative or any similar project could deliver a full, workable solution.

For example, the contractor stressed the importance of setting realistic deadlines and commented that there was no credible evidence that VETSNET would be complete by the target date, which at the time of the review had slipped to December 2006. According to the assessment, because this deadline was unrealistic, VBA needed to plan and budget for supporting the BDN so that its ability to pay veterans’ benefits would not be disrupted. SEI also noted that different organizational components had independent schedules and priorities, which caused confusion and deprived the department of a program perspective. Further, the contractor concluded that VBA needed to give priority to establishing sound program management to ensure that the project could meet targeted dates. These and other observations were consistent with our long-standing concerns regarding fundamental deficiencies in VBA’s management of the project.

To help VBA implement the overall recommendation, the contractor’s assessment included numerous discussions of activities needed to address these areas of concern, which can be generally categorized as falling into two major types: Overall management concerns with regard to the initiative included governance structure, including assigning ownership for the project project planning, including the development schedule and capacity conversion of records currently on the BDN to the replacement system.

Software development process improvements were needed in the program measures.

VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be Addressed

As recommended by SEI, VBA is continuing to work on the replacement initiative at a reduced pace and taking action to address identified weaknesses in the project’s overall management and software development processes. For example, VBA has established a new governance structure and has developed an integrated master schedule that provides additional time and includes the full range of project activities. However, additional effort is needed to complete a number of the corrective actions, such as improving project accountability through monitoring and reporting all project costs. Further, VBA has not yet institutionalized many of the improvements that it has undertaken for the initiative. In particular, process improvements remain in draft and have not been established through documented policies and procedures. According to the VETSNET management team, it gave priority to other activities, such as establishing appropriate governance and organizational structures, and it is still gathering information to assist in prioritizing the activities that remain. Nonetheless, if VBA does not institutionalize these improvements, it increases the risk that these process improvements may not be maintained through the life of the project or be available for application to other development initiatives.

VBA Is Taking Action on Certain Overall Management Concerns

SEI concluded that VBA’s management issues would need to be addressed as part of the implementation of its overall recommendation. SEI’s overall management concerns focused on the project governance, project planning, and conversion of records currently on the BDN to the replacement system.

Top Management Included in Revised Governance Structure

SEI guidance for software development stresses the need for organizational commitment and the involvement of senior management in overall project governance. In its assessment, SEI noted that because management of the VETSNET project had been assigned to VBA’s information technology (IT) group, certain activities critical to the veterans’ benefits program, but not traditionally managed by the IT group, had not been visible to the project’s management. The contractor pointed out that the IT group, business lines, and regional offices needed to share ownership and management of the replacement project through an established governance process and that the project management office should include business representatives. According to SEI, the project needed to establish ownership responsibility, including addressing total system and process operating costs.

In response to the assessment, VBA developed a new governance structure for the initiative, which the Under Secretary for Benefits approved in March 2006. In the new structure, the VETSNET Executive Board that had been in place was expanded and reorganized to serve as a focal point and major governance mechanism for the replacement initiative. A Special Assistant (reporting directly to the Under Secretary) was appointed to coordinate and oversee the initiative as the head of the VETSNET Executive Team, which was established to provide day-to-day operational control and oversight of the replacement initiative. Implementation Teams were also established to conduct the day-to-day activities associated with implementing the initiative. This governance structure established a process for IT, business lines, and regional offices to share ownership and management, as SEI advised.

The roles and membership of each of the organizational elements in the new governance structure are described in table 1.

When the new governance structure was approved in March 2006, the Under Secretary ensured that those involved in the project gave it high priority, directing certain key personnel (such as members of the executive and implementation teams) to make the initiative their primary responsibility, and other personnel (technical staff that provide support to other systems) with collateral (non-VETSNET) duties to make the project their first priority. He also placed limitations on the transfer of personnel away from the project, recognizing the importance of staff continuity in successfully completing the initiative. Staff members assigned project responsibilities could be reassigned (i.e., given nonpromotion, lateral reassignments) only with approval from the Under Secretary or his deputy.

By implementing the new governance and organizational structure and ensuring that the project has priority, VBA partially responded to SEI’s concerns in this area; however, VBA has not yet taken action with regard to ownership responsibility for total system and process operating costs, as SEI advised. According to administration officials, the replacement initiative is an in-house, contractor-assisted development effort, in which three different contractors provide support for program management, system development, and testing and validation of requirements. VA reported VETSNET system costs to the Congress totaling about $89 million for fiscal years 1996 through 2006, with additional estimated costs for completion of the initiative in 2009 of about $62.4 million. However, according to project management officials, these costs do not include expenditures for in-house development work. This in-house work involves many VA personnel, as well as travel to various locations for testing and other project related activities. Thus, considerable costs other than contract cost have been incurred, which have not been tracked and reported as costs for the replacement initiative. Without comprehensive tracking and reporting of costs incurred by the replacement project, the ability of VBA and the Congress to effectively monitor progress could be impaired.

VBA Took Action to Address Project Planning and Management

A second major area of overall management concern was project planning. In particular, the lack of an integrated master schedule for the VETSNET project was a major concern articulated by SEI, as well as in our prior work. An integrated project plan and schedule should incorporate all the critical areas of system development and be used as a means of determining what needs to be done and when, as well as measuring progress. Such an integrated schedule should consider all dependencies and include subtasks so that deadlines are realistic, and it should incorporate review activities to allow oversight and approval by high-level managers. Among other things, the program plan should also include capacity requirements for resources and technical facilities to support development, testing, user validation, and production.

SEI was specifically concerned that releases with overlapping functionality were being developed at the same time, with insufficient time to document or test requirements; this approach constrained resources and added complexity because of the need to integrate completed applications and newly developed functionality. In addition, SEI observed that the VETSNET program suffered from lack of sufficient test facilities because it did not have enough information to plan for adequate capacity.

In response to these project planning concerns, VETSNET management, with contractor support, developed an integrated master schedule to guide development and implementation of the remaining functionalities for the replacement system. The VETSNET Integrated Master Schedule, finalized in September 2006, includes an end-to-end plan and a master schedule. According to VBA, the end-to-end plan documents the end state of the project from a business perspective, which had not previously been done. The master schedule identifies the necessary activities to manage and control the replacement project through completion. The schedule also describes a new software release process that provides more time to work on requirements definition and testing, and allows for more cross- organizational communications to lessen the possibility of not meeting requirements.

In addition, the new release process includes a series of management reviews to help control the software development process and ensure that top management has continuous visibility of project related activities. These reviews occur at major steps in the system development life cycle (as described in fig. 1: initiation, preliminary design, and so on). Such reviews are intended to ensure that the VETSNET Executive Team and the VETSNET Executive Board agree and accept that the major tasks of each step have been properly performed.

Nonetheless, while the Integrated Master Schedule is an important accomplishment, it may not ensure that the project sufficiently addresses capacity planning, one of SEI’s areas of concern. According to its assessment, capacity requirements for the fully functional production system were unclear. Capacity planning is important because program progress depends on the availability of necessary system capacity to perform development and testing; adjustments to such capacity take time and must be planned. If systems do not have adequate capacity to accommodate workload, interruptions or slowdowns could occur. According to SEI, capacity adjustments cannot be made instantly, and program progress will suffer without sufficient attention to resource requirements.

However, the VETSNET Integrated Master Schedule does not identify activities or resources devoted to capacity planning. According to officials, the capacity of the corporate environment (that is, corporate information systems, applications, and networks) is being monitored by operational teams with responsibility for maintaining this environment. According to project officials, VETSNET representatives participate in daily conference calls in which the performance of corporate applications is discussed, and changes in application performance are reported to the VETSNET developers for investigation and corrective action. Project officials reported that when a performance degradation occurred in some transactions during performance testing, it was determined that additional computing capacity was needed and would be acquired. One reason why the occurrence of degradation had not been anticipated by the VETSNET project was that capacity planning had not taken place. Unless it ensures that capacity planning and activities are included in the Integrated Master Schedule, the replacement project may face other unanticipated degradations that it must react to after the fact, thus jeopardizing the project’s cost, schedule, and performance.

Conversion Efforts Were Suspended as Advised, but Issues Remain

In its assessment, SEI questioned VBA’s approach to developing functionality while concurrently converting records from the BDN to the replacement system. It noted that VBA had chosen to complete software development according to location rather than according to the type of functionality. Specifically, in 2004, VBA began an effort to remove all claims activity (both new and existing claims) at one regional office (Lincoln, Nebraska) from the BDN to the replacement system, developing the software as necessary to accommodate processing the types of claims encountered at that site. The intention was to address each regional office in turn until all sites were converted. According to SEI, this approach had resulted in the development being stalled by obstacles arising from the variety of existing claims. The contractor advised VBA to focus first on developing functionality to process original claims and discontinue efforts to convert existing claims until all the necessary functionality had been developed, and the replacement system’s ability to handle new cases of any complexity had been proven by actual experience.

In accordance with this advice, VBA stopped converting existing records from the BDN and changed its focus to developing the necessary functionality to process all new compensation claims. According to the integrated master schedule, conversion activities are now timed to follow the release of the needed functionality. That is, according to the schedule, VBA plans to begin converting each type of record from the BDN only after the necessary functionality for the replacement system has been developed and deployed to process that type of record.

In addition, the project is mitigating risk by resuming conversions beginning with a test phase. Its strategy is first to convert records for terminated claims—claims that are no longer being paid. Conversion of the terminated records will be followed by additional conversions of records for claims receiving payment at Lincoln and Nashville (these two sites are being used to test system functionality during development). The VETSNET leadership will consider testing complete with the successful conversion at these two sites.

However, SEI raised three additional issues with regard to the conversion of records that VBA has not fully addressed: First, SEI expressed concerns that conversion failures could lead to substantial numbers of records being returned to the BDN. Because of differences in the database technologies used for the old system and the replacement system, certain types of errors in BDN records cause conversion to fail (according to SEI, approximately 15 percent of all these records are estimated to have such errors). If records fail to convert correctly, they may need to be returned to the BDN so that benefits can continue to be paid. However, this process is not simple and may involve manually reentering the records.

Second, SEI observed that VBA was also depending on manual processes for determining that records were converted successfully, including the use of statistically random samples, and that it was aiming to ensure correctness to a confidence level of 95 percent. However, in the absence of a straightforward method for automatically returning records to the BDN, SEI considered the 5 percent risk of error unacceptable for conversions of large numbers of records.

Finally, SEI observed that the lack of automated methods and the complexity of the processes meant that conversions required careful planning and assurance that adequate staff would be available to validate records when the conversions took place.

However, the VETSNET leadership has not developed any strategy to address the possibility that a large number of cases might need to be returned to the BDN during the testing phase. For example, it has not included this possibility as a risk in its risk management plan. The absence of a strategy to address this possibility could lead to delays in program execution.

Further, VBA has not yet decided whether a possible 5 percent error rate is acceptable or developed a plan for addressing the resulting erroneous records. If VBA does not address these issues in its planning, it increases the risk that veterans may not receive accurate or timely payments.

Finally, the VETSNET leadership has not yet developed detailed plans that include the scheduled conversions for each regional office and identified staff to perform the necessary validation. Having such plans would reduce the risk that the conversion process could be delayed or fail.

VBA Is Improving Software Development Processes, but Improvements Are Not Yet Institutionalized

In addition to actions addressing the overall management concerns identified by SEI, VBA has steps action to improve its software development processes in risk management, requirements management, defect/change management, and performance measures. SEI described weaknesses in all of these areas. The steps taken have generally been effective in addressing the identified weaknesses, but VBA has not yet institutionalized many of these improvements. According to the VETSNET management team, it made a conscious decision first to establish the governance, build the organization, implement processes to gain control, and gather additional information about the project to assist in prioritizing the remaining activities. The team also stated that some of the processes are no longer VBA’s responsibility but are now that of the newly realigned Office of Information and Technology. Nonetheless, if VA does not develop and establish documented policies and procedures to institutionalize these improvements, they may not be maintained through the life of the project or available to be applied to other development initiatives.

Risk Management Plan Has Been Revised

Risk management is a process for identifying and assessing risks, their impact and status, the probability of their occurrence, and mitigation strategies. Effective risk management includes the development of a risk management plan and tracking and reporting progress against the plan. According to SEI, to the extent that risk management existed at all in the replacement program, it was conducted on a pro forma basis without real effect on program decisions. SEI said that risks and risk mitigation activities needed to be incorporated into all aspects of program planning, budgeting, scheduling, execution, and review.

In response to these concerns, VBA has instituted risk management activities that, if properly implemented, should mitigate the risks associated with the project. Specifically, the VETSNET team, with contractor support, developed a risk management plan that was adopted in January 2007. The plan includes procedures for identifying, validating, analyzing, assessing, developing mitigation strategies for, controlling and tracking, reporting, and closing risks. It also establishes criteria for assessing the severity of the risks and their impact.

The VETSNET leadership also developed a Risk Registry database, and its contractor reviewed and prioritized the open risks. Each open risk was evaluated, and a proposed disposition of the risk was submitted to VETSNET management. Of the 39 open risks, all but 3 had been addressed as of January 2007.

The development documentation for each planned software release also includes sections on risk. In accordance with these plans, the VETSNET leadership is currently capturing potential risks and tracking action items and issues. At weekly status meetings, VETSNET leadership reviews Risk Registry reports of open risks. According to the contractor, the reports identify each risk and provide information on its age, ownership, and severity.

However, these risk management activities have not yet been institutionalized through the definition and establishment of associated policies and procedures. If it does not institutionalize these improvements, VBA increases the possibility that the VETSNET project’s improvements in risk management may not be maintained through the life of the project.

Requirements Management Has Improved

Requirements management is a process for establishing and maintaining a common understanding between the business owners and the developers of the requirements to be addressed, as well as verifying that the system meets the agreed requirements. SEI’s report commented that the VETSNET project requirements were not stable, and that the business owners (including subject-matter experts) and developers were separated by many organizational layers, resulting in confusion and delays in development of the system. SEI suggested that VA restructure project activities to focus on defining an effective requirements process. According to SEI, the project needed to ensure that subject-matter experts were included in developing requirements and that evaluation criteria were established for prioritizing requests for changes to requirements. Finally, business owners should confirm that the system is meeting organizational needs.

VBA has instituted requirements management activities that, if properly implemented, should help avoid the instability and other requirements problems identified by SEI. Specifically, VBA took steps to establish a requirements management process and to stabilize the requirements. For example, the development release process in the Integrated Master Schedule includes a phase for requirements identification. In addition, the project has established and begun applying evaluation criteria to prioritize change requests for its development releases. Further, until all claims are completely migrated from the BDN to the replacement system, in July 2006, the Under Secretary directed that any additional requirements would have to have his approval.

Responding to SEI’s advice regarding the involvement of subject-matter experts and business owners, VBA designed the new release process to directly involve subject-matter experts in requirements workshops. Further, the business teams participate in user-acceptance testing.

However, these requirements management activities have not yet been institutionalized through the definition and establishment of policies and procedures. Until they are established, VBA runs the risk that the improved processes will not be maintained through the life of the VETSNET project or used in other software development projects.

Management Attention Is Being Focused on Major Software Defects

SEI raised numerous concerns regarding the defect process for the replacement system. These concerns for defect management included (1) identification of defects, (2) determination of cause, and (3) disposition of defects—either by correction or workaround. According to SEI guidance, defect management prevents known defects from hampering the progress of the program. The management process should include clearly identifying and tracking defects, analyzing defects to establish their cause, tracking their disposition, clearly identifying the rationale for not addressing any defects (as well as proposing workarounds), and making information on defects and their resolution broadly available. SEI’s report stated that VBA needed to distinguish defects from changes to requirements and develop a process for defect management.

To respond to these concerns and focus program management attention on major defects, the VETSNET Executive Team, with contractor support, conducted an audit of existing defects and revised the defect management process. The audit of the defect database determined that the VETSNET database used to capture software defects also included change requests; as a result, work required to address processes that did not work properly was not distinguished from requests for added or changed functionality, which would require review and approval before being addressed. To address this issue, the team separated defects from change requests, and a new severity rating scale was developed. All open defects were recategorized to ensure the major defects would receive appropriate program management attention. Also, all defect categorizations must meet the approval of the VETSNET Business Architect and are scheduled for action as dictated by the severity level.

Although these steps address many of SEI’s concerns regarding VBA’s defect management process, more remains to be done before the process is institutionalized. The Program Management Office has reported that actions to revise the defect management process are complete, but the process description is still in draft, and policies and procedures have not been fully established. Without institutionalized policies and procedures for the defect management process, it may not be maintained consistently through the life of the project.

The Replacement Initiative Is Tracking Certain Performance Measures

According to SEI, performance measures are the only effective mechanism that can provide credible evidence of a program’s progress. The chosen measures must link directly to the expected accomplishments and goals of the system, and they must be applied across all activities of the program. In its report, SEI stated that although VBA was reporting certain types of performance measures, it was not relating these to progress in system development. For example, VBA reported the total number of veterans paid, but did not provide estimates of how many additional veterans would be paid when the system incorporated specific functionalities that were under development. SEI suggested several measures that would provide more evidence of progress, such as increases in the percentage of original claims being paid by the replacement system, as well as user satisfaction and productivity gains resulting from use of the replacement system applications at regional offices.

In response to these concerns, the replacement project has begun tracking a number of the measures suggested by SEI, including increases in the percentage of original claims being paid by the replacement system, increases in the percentage of veterans’ service representatives using the new system, decreases in the percentage of original claims being entered in the BDN rather than the replacement system.

Although these measures provide indications of VA’s progress, other measures that could demonstrate the effectiveness of the replacement system have not been developed. For example, VBA has not developed results-oriented measures to capture user satisfaction or productivity gains from the system. Without measuring user satisfaction, VBA has reduced assurance that the replacement system will be accepted by the users. In addition, measures of productivity would provide VBA with another indication of progress toward meeting business needs.

VETSNET Is Currently Processing a Portion of Compensation Claims, but Much Work Remains

After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has achieved critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. For example, the replacement system is currently being used to process a portion of the original claims that veterans file for compensation. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents. In addition, VBA still faces the substantial task of moving approximately 3.5 million beneficiaries who are currently being served by the BDN to the replacement system.

As designed, VETSNET consists of five major system applications that are used in processing benefits: Share—used to establish claims; it records and updates basic information about veterans and dependents both in the BDN and the replacement system.

Modern Award Processing–Development (MAP-D)—used to manage the claims development process, including the collection of data to support the claims and the tracking of claims.

Rating Board Automation 2000 (RBA 2000)—provides laws and regulations pertaining to disabilities, which are used by rating specialists in evaluating and rating disability claims.

Award Processing (Awards)—used to prepare and calculate the benefit award based on the rating specialist’s determination of the claimant’s percentage of disability. It is also used to authorize the claim for payment.

Finance and Accounting System (FAS)—used to develop the actual payment record. FAS generates various accounting reports and supports generation and audit of benefit payments.

According to VBA officials, all five of the software applications that make up the new system are now being used in VA’s 57 regional offices to establish and process new compensation claims for veterans. As of March 2007, VBA leadership reported that the replacement system was providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments). In addition, the replacement system has been processing a steadily increasing percentage of all new compensation claims completed: this measure was 47 percent in January 2007, increasing to 60 percent in February and 83 percent in March.

Nonetheless, considerable work must be accomplished before VBA will be able to rely on the replacement system to make payments to all compensation and pension beneficiaries. Specifically, while all five software applications can now be used to process original compensation claims for veterans, two of the applications—Awards and FAS—require further development before the system will be able to process claims for the full range of benefits available to veterans and their dependents. Table 2 shows the status of development of all five applications.

According to VBA officials, Awards and FAS do not yet have the capability to process original claims for payment to recipients other than veterans: that is, the applications do not have the functionality to process claims for survivor benefits and third-party/nonveteran payee claims. In addition, further development of these applications is needed to process pension benefits for qualified veterans and their survivors. Until enhancements are made to Awards and FAS, these claims must continue to be processed and paid through the BDN. Also, according to VBA, FAS does not yet have the capability to generate all the necessary accounting reports that support the development of benefits payments to claimants.

As described earlier, VBA now has an Integrated Master Schedule that incorporates the activities that VBA needs to manage in order to complete the replacement project. According to the schedule, the remaining capabilities necessary to process compensation and pension claims are to be developed and deployed in three software releases, as shown in table 3.

As table 3 shows, VBA does not expect to complete the development of the functionalities needed to process all new compensation and pension claims until August 2008. However, according to VBA, the estimated completion date is the planned date for completing all development and testing, but it is not necessarily the date when users will be able to begin using the new system. Before such use can begin, other activities need to occur. For example, users must receive training, and VETSNET program management must authorize the use of the system at each regional office.

In addition to its remaining software development activities, VBA also faces the challenge of converting records for claims currently paid by the BDN to the replacement system. Existing compensation and pension cases on the BDN number about 3 million and about 535,000, respectively. Table 4 shows the phases in which VA is planning to perform conversions, according to its Integrated Master Schedule.

As the table shows, VBA conversion efforts began in March 2007. VBA first performed conversion testing on 310,000 terminated (that is, inactive) compensation cases so that it could develop and apply lessons learned to the conversion of live records. According to VETSNET officials, VBA planned to continue testing by converting live cases at two regional offices (Lincoln and Nashville) that were used as testing sites during development. It then plans to perform the conversion of all live compensation cases. After the compensation conversion is complete, VBA plans to begin efforts to convert pension benefits cases.

Based on VETSNET project documentation, activities supporting the releases have so far been performed on time, consistent with the milestones in the recently finalized Integrated Master Schedule. For example, VA completed the Project Initiation and Review Authorization for Release 1 on September 7, 2006, as scheduled (see fig. 1, shown earlier in the report, for the phases of system development and the required milestone reviews). It also completed the Preliminary Design Review and the Critical Design Review as scheduled (on November 20 and December 22, respectively). Planning for Release 2 is also on schedule: a kickoff meeting was held on January 24, 2007, which established the scope of the release, and the Project Initiation and Review Authorization was conducted on February 8.

Conclusions

VA has responded to SEI’s assessment by making significant changes in its approach to the project and its overall management, including slowing the pace of development, establishing a new governance structure, and ensuring staff resources. However, VBA has not yet addressed all the issues raised by the SEI assessment. That is, it has not ensured ownership responsibility for total system and process operating costs, because it is not currently monitoring and reporting in-house expenditures for the project. It has not defined processes and resources for capacity planning for the project. In addition, VBA has not yet addressed issues related to the conversion of records now on the BDN to the replacement system. Specifically, it has not addressed the risk that large numbers of records may need to be returned to the BDN, decided on the degree of confidence it will require that records are converted accurately, or developed complete plans for converting and validating records. In addition, although VBA has improved key processes for managing the software development, these processes have not yet been institutionalized in defined policies and procedures, and performance measures of productivity and user satisfaction have not been developed. VETSNET management has stated that it gave priority to other activities, such as establishing appropriate governance and organizational structures, and that it is still gathering information to assist in prioritizing the activities that remain.

Much work remains to be done to complete the VETSNET initiative. Although VBA has substantially increased the number of claims being paid by the replacement system, it must not only finish the development and deployment of the software, it must also convert the over 3.5 million records now on the BDN to the replacement system. Addressing the remaining issues identified by SEI would improve VBA’s chances of successfully completing the replacement system and ending reliance on the aged BDN to pay compensation and pension benefits.

Recommendations for Executive Action

To enhance the likelihood that the replacement system will be successfully completed and implemented, we are recommending that the Secretary of Veterans Affairs take the following five actions: Direct the CIO to institute measures to track in-house expenditures for the project.

Direct the VETSNET project to include activities for capacity planning in the VETSNET Integrated Master Schedule and ensure that resources are available for these activities.

Direct VBA to (1) develop a strategy to address the risk that large numbers of records may need to be returned to the BDN; (2) determine whether a greater confidence level for accuracy should be required in the conversion process; and (3) develop a detailed validation plan that includes the scheduled conversions for each regional office and the validation team members needed for that specific conversion.

Direct the CIO to document and incorporate the improved processes for managing risks, requirements, and defects into specific policy and guidance for the replacement initiative and for future use throughout VBA.

Direct the replacement project to develop effective results-oriented performance measures that show changes in efficiency, economy, or improvements in mission performance, as well as measures of user satisfaction, and to monitor and report on the progress of the initiative according to these measures.

Agency Comments and Our Evaluation

In providing written comments on a draft of this report, the Secretary of Veterans Affairs agreed with our conclusions and concurred with the report’s recommendations. (The department’s comments are reproduced in app. II.) The comments described actions planned that respond to our recommendations, such as incorporating processes developed for the VETSNET project in standard project management policies, processes, and procedures that would be used for all IT projects in the department. In addition, the comments provided further information on actions already taken, such as details of the records conversion process. If the planned actions are properly implemented, they could help strengthen the department’s management of the replacement system project and improve the chances that the system will be successfully completed.

We are sending copies of this report to the Chairman and Ranking Minority Member of Committee on Veterans’ Affairs. We are also sending copies to the Secretary of Veterans Affairs and appropriate congressional committees. We will make copies available to other interested parties upon request. Copies of this report will also be made available at no charge on GAO’s Web site at http://www.gao.gov.

Should you or your staff have any questions about this report, please contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) to what extent the Department of Veterans Affairs (VA) has followed the course of action recommended by the Carnegie Mellon Software Engineering Institute (SEI) and addressed the concerns that it raised and (2) the current status of the replacement project, the Veterans Service Network (VETSNET).

To determine the actions taken to implement SEI’s recommended approach and address the concerns it raised, we determined the recommended actions by analyzing the report; compared the concerns identified in the assessment to actions planned, actions undertaken but not completed, and actions implemented by VA officials or contractors; interviewed contractor, VA, and VETSNET program office officials to gain an understanding about processes developed and procedures implemented; and obtained and reviewed relevant VA and contractor documents that disclosed or validated VA responses to SEI’s concerns.

To determine the status of system development efforts and the extent that tasks planned for the initiative were completed, we analyzed VA and contractor documentation regarding system operations and development, time frames, and activities planned. We analyzed VA documents that disclosed costs to date and costs planned for completion of the initiative. We did not assess the accuracy of the cost data provided to us. We supplemented our analyses with interviews of VA and contractor personnel involved in the replacement initiative.

We visited the Nashville and St. Petersburg regional offices to observe the replacement system in operation and the processes and procedures used to test and validate the replacement system as it was being developed and implemented. We analyzed VA documentation and relevant evidence from contractors involved in the replacement effort to establish the work remaining to complete the project. Finally, we interviewed cognizant VA and contractor officials, responsible for developing, testing, and implementing the replacement system.

We performed our work at VA offices in Washington, D.C., and at VA regional offices in Nashville, Tennessee, and St. Petersburg, Florida, from April 2006 to April 2007 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributions were made to this report by Barbara Oliver, Assistant Director; Nabajyoti Barkakati; Barbara Collier; Neil Doherty; Matt Grote; Robert Williams; and Charles Youman.