Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist
Published: 2007-04-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1996, the Veterans Benefit Administration (VBA) has been working on an initiative to replace its aging system for paying compensation and pension benefits. In 2005, concerned about the slow pace of development, VBA contracted with the Software Engineering Institute (SEI) for an independent evaluation of the project, known as the Veterans Service Network (VETSNET). SEI advised VBA to continue working on the project at a reduced pace while addressing management and organization weaknesses that it determined had hampered the project's progress. GAO was requested to determine to what extent the VETSNET project has followed the course of action recommended by SEI and describe the project's current status. To perform its review, GAO analyzed project documentation, conducted site visits, and interviewed key program officials.
section_title: What GAO Found
paragraphs: VBA is generally following the course of action recommended by SEI by continuing to work on the replacement initiative at a reduced pace, while taking action to address identified weaknesses in overall management and software development processes. For example, VBA established a new governance structure for the initiative that included senior management and involved all stakeholders, and it incorporated all critical areas of system development in an integrated master schedule. However, not all of SEI's management concerns have been addressed. For example, SEI advised VBA to ensure that stakeholders take ownership responsibility for the project, including the total system and process operating costs; however, although VBA is tracking costs incurred by contractors, it is not yet tracking and reporting in-house costs incurred by the project. Further, although the project has improved its management processes, such as establishing a process to manage and stabilize system requirements, it has not yet developed processes for capacity planning and management. This will be important for ensuring that further VETSNET development does not lead to delays and slowdowns in processing of benefits. In addition, although the project has established certain performance measures, it has not yet established results-oriented measures for productivity and user satisfaction, both of which will be important for measuring progress. Finally, the process improvements that VBA has incorporated in the replacement initiative remain in draft and have not been established through documented policies and procedures. If VBA does not institutionalize these improvements, it increases the risk that they may not be maintained through the life of the project or be available for application to other development initiatives. After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has developed critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. According to VBA officials, all five of the major software applications that make up the new system are now being used to establish and process new compensation claims for veterans. In total, the replacement system is currently providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments); the system was used to process about 83 percent of all new compensation claims completed in March 2007. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents.

section_title: Letter
paragraphs: Since 1996, the Veterans Benefits Administration (VBA), a major  component of the Department of Veterans Affairs (VA), has been  undertaking a project to replace its compensation and pension benefits  payment system. VBA’s existing system, the Benefits Delivery Network  (BDN), is essential to ensuring the accurate processing of compensation  and pension benefits payments to over 3.5 million veterans and their  dependents each month. However, this system, which has been in  operation for more than 40 years, is based on antiquated software  programs, which have become increasingly difficult and costly to  maintain. VBA is in the process of replacing the aging BDN with a faster,  more flexible, and higher capacity system. When it began this project in  1996, which it generally refers to as the Veterans Service Network  (VETSNET), VBA had planned to complete the replacement system in  May 1998 at an estimated cost of $8 million.
Over the years, we have reported on numerous problems that VBA has  encountered in completing the replacement system. Our prior work  found, for example, that the project was begun before VBA had fully  developed its business requirements, resulting in confusion over the  requirements to be addressed; in addition, VBA’s software development  capability was too immature to ensure that the agency could reliably  develop and maintain high-quality software on any major project within  cost and schedule constraints. In 2002, we offered a number of  recommendations to improve managerial and program weaknesses,  including that VBA appoint a project manager; thoroughly analyze its  initiative; and develop a number of plans, including a revised  compensation and pension replacement strategy and an integrated project  plan. VA concurred with our recommendations, and as we last reported in  June 2006, it took several actions to address them. For example, it  appointed a full-time project manager, and the project team reported that  it had completed certifications of users’ requirements for the system’s  applications. Nonetheless, these actions did not implement all our  recommendations and were not sufficient to establish the program on a  solid footing: certain basic requirements of sound project management,  such as an integrated project plan for the replacement system, continued  to be lacking.
In 2005, the VA Chief Information Officer (CIO) and Under Secretary for  Benefits became concerned about continuing problems with the  replacement project and contracted for an independent assessment of the  department’s options for the project, including whether the project should  be terminated. This assessment, conducted by the Carnegie Mellon  Software Engineering Institute (SEI), concluded that the replacement  project faced many risks arising from management and organizational  issues, but no technical barriers that could not be overcome. According to  SEI, a new system was still needed, and VBA would not be able to  successfully deliver a full, workable solution unless it addressed its  management and organizational weaknesses. SEI recommended that VBA  continue to work on the project at a reduced pace, while taking an  aggressive approach to addressing the identified weaknesses.
Given the importance of ensuring the effective and efficient delivery of  veterans’ benefits, we were requested to review the department’s  continuing efforts to develop and implement the compensation and  pension replacement system. Specifically, our objectives were to  determine (1) to what extent VA has followed the course of action  recommended by SEI and addressed the concerns that it raised and (2) the  current status of the replacement project.
In conducting this study, we visited VBA’s Nashville Regional Office to  observe processing of new benefits claims in the compensation and  pension replacement system, and we visited the St. Petersburg Regional  Office to observe processes and procedures used to test and validate key  functionalities of the replacement system. We obtained and analyzed  documents related to the replacement initiative and to SEI’s review of the  initiative. We supplemented our analysis with interviews of VBA and  contractor personnel at headquarters and the two regional offices.  Appendix I contains a more detailed discussion of our objectives, scope,  and methodology. We conducted our study between April 2006 and April  2007 in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: VBA provides benefits for veterans and their families through five  programs: (1) compensation and pension, (2) education, (3) vocational  rehabilitation and employment (VRE) services, (4) loan guaranty, and   (5) life insurance. It relies on the BDN to administer benefit programs for  three of VBA’s five programs: compensation and pension, education, and  VRE services.
Replacing the aging BDN has been a focus of systems development efforts  at VBA since 1986. Originally, the administration planned to modernize  the entire system, but after experiencing numerous false starts and  spending approximately $300 million on the overall modernization of the  BDN, VBA revised its strategy in 1996. It narrowed its focus to replacing  only those functionalities that support the compensation and pension  program, and began developing a replacement system, which it called  VETSNET.
As reported by the department in its fiscal year 2008 budget submission,  the compensation and pension program is the largest of the three  programs that the BDN supports:    The compensation and pension program paid about $35 billion in benefits  in fiscal year 2006 to about 3.6 million veterans or veterans’ family  members.
Of this amount, compensation programs paid benefits of about $31  billion to about 3.1 million recipients.
Pension programs paid benefits of about $3.5 billion to about 535,000  recipients.
The education program paid about $2.8 billion to about 498,000 veterans or  their dependents in fiscal year 2006.
The VRE services program paid about $574 million for VRE services in  2006 and provides rehabilitation services to approximately 65,700 disabled  veteran participants per year.
subsections: section_title: The Processes Supporting the Variety of Compensation and Pensions Benefits Are Complex
paragraphs: One of the challenges of developing the replacement system is that it must  include processes to support the administration of a complex set of  benefits. Different categories of veterans and their families are eligible for  a number of different types of benefits and payments, some of which are  based on financial need. Compensation programs, which are based on  service-connected disability or death, provide direct payments to veterans  and/or veterans’ dependents and survivors. These programs are not based  on income. Pension benefits programs, on the other hand, are income  based; these are designed to provide income support to eligible veterans  and their families who experience financial hardship. Eligible veterans are  those who served in wartime and are permanently and totally disabled for  reasons that are not service-connected (or who are age 65 or older).  Veterans are also eligible for burial benefits.
Survivor benefits may be paid to eligible survivors of veterans, depending  on the circumstances. Some of these benefits are based on financial need,  such as death pensions for some surviving spouses and children of  deceased wartime veterans, and Dependency and Indemnity  Compensation to some surviving parents.
Finally, certain benefits may be paid to third parties, such as individuals to  whom a veteran has given power of attorney or medical service providers  designated to receive payments on the veteran’s behalf.
Generally, VBA administers benefit programs through 57 veterans benefits  regional offices in a process that requires a number of steps, depending  on the type of claim. When a veteran submits, for example, a  compensation claim to any of the regional offices, a veterans service  representative must obtain the relevant evidence to evaluate the claim  (such as the veteran’s military service records, medical examinations, and  treatment records from VA medical facilities or private medical service  providers). In the case of pension claims, income information would also  be collected.
Once all the necessary evidence has been compiled, a rating specialist  evaluates the claim and determines whether the claimant is eligible for  benefits. If the veteran is determined to be eligible for disability  compensation, the Rating Veterans Service Representative assigns a  percentage rating based on the veteran’s degree of disability. This  percentage is used in calculating the amount of payment.
Benefits received by veterans are subject to change depending on  changing circumstances. More than half of VBA’s workload consists of  dealing with such changes. If a veteran believes that a service-connected  condition has worsened, for example, the veteran may ask for additional  benefits by submitting another claim. The first claim submitted by a  veteran is referred to as the original claim, and a subsequent change is  referred to as a reopened claim.
section_title: Our Prior Products Identified Weaknesses in the Development and Implementation of the Replacement System
paragraphs: Since its inception, VETSNET has been plagued by problems. Over the  years, we have reported on the project, highlighting concerns about VBA’s  software development capabilities.
In 1996, our assessment of the department’s software development  capability determined that it was immature. In our assessment, we  specifically examined VETSNET and concluded that VBA could not  reliably develop and maintain high-quality software on any major project  within existing cost and schedule constraints. The department showed  significant weaknesses in requirements management, software project  planning, and software subcontract management, with no identifiable  strengths. We also testified that VBA did not follow sound systems  development practices on VETSNET, and we concluded that its  modernization efforts had inherent risks.
Between 1996 and 2002, we continued to identify the department’s weak  software development capability as a significant factor contributing to  persistent problems in developing and implementing the system. We also  reported that VBA continued to work on VETSNET without an integrated  project plan. As a result, the development of the system continued to  suffer from problems in several areas, including project management,  requirements development, and testing.
Over the years, we made several recommendations aimed at improving  VA’s software development capabilities. Among our recommendations was  that the department take actions to achieve greater maturity in its  software development processes and that it delay any major investment  in software development (beyond that needed to sustain critical day-to-day  operations) until it had done so. In addition, we made specific  recommendations aimed at improving VETSNET development. For  example, we recommended that VA appoint a project manager, thoroughly  analyze its current initiative, and develop a number of plans, including a  revised compensation and pension replacement strategy and an integrated  project plan.
VA concurred with our recommendations and took several actions to  address them. For example, it appointed a full-time project manager and  ensured that business needs were met by certification of user  requirements for the system applications. The actions taken addressed  some of our specific concerns; however, they were not sufficient to fully  implement our recommendations or to establish the program on a sound  footing.
section_title: SEI Assessed System Replacement Initiative and Recommended a Changed Approach
paragraphs: As a result of continuing concerns about the replacement project, in 2005  VA’s CIO and its Under Secretary for Benefits contracted for an  independent assessment of the department’s options for the initiative.  The chosen contractor, SEI, is a federally funded research and  development center operated by Carnegie Mellon University. Its mission is  to advance software engineering and related disciplines to ensure the  development and operation of systems with predictable and improved  cost, schedule, and quality.
SEI recommended that the department reduce the pace of development  while at the same time taking an aggressive approach to dealing with  management and organizational weaknesses hampering VBA’s ability to  complete the replacement system. According to SEI, these management  and organizational concerns needed to be addressed before the  replacement initiative or any similar project could deliver a full, workable  solution.
For example, the contractor stressed the importance of setting realistic  deadlines and commented that there was no credible evidence that  VETSNET would be complete by the target date, which at the time of the  review had slipped to December 2006. According to the assessment,  because this deadline was unrealistic, VBA needed to plan and budget for  supporting the BDN so that its ability to pay veterans’ benefits would not  be disrupted. SEI also noted that different organizational components had  independent schedules and priorities, which caused confusion and  deprived the department of a program perspective. Further, the contractor  concluded that VBA needed to give priority to establishing sound program  management to ensure that the project could meet targeted dates. These  and other observations were consistent with our long-standing concerns  regarding fundamental deficiencies in VBA’s management of the project.
To help VBA implement the overall recommendation, the contractor’s  assessment included numerous discussions of activities needed to address  these areas of concern, which can be generally categorized as falling into  two major types:    Overall management concerns with regard to the initiative included    governance structure, including assigning ownership for the project    project planning, including the development schedule and capacity    conversion of records currently on the BDN to the replacement system.
Software development process improvements were needed in the    program measures.
section_title: VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be Addressed
paragraphs: As recommended by SEI, VBA is continuing to work on the replacement  initiative at a reduced pace and taking action to address identified  weaknesses in the project’s overall management and software  development processes. For example, VBA has established a new  governance structure and has developed an integrated master schedule  that provides additional time and includes the full range of project  activities. However, additional effort is needed to complete a number of  the corrective actions, such as improving project accountability through  monitoring and reporting all project costs. Further, VBA has not yet  institutionalized many of the improvements that it has undertaken for the  initiative. In particular, process improvements remain in draft and have  not been established through documented policies and procedures.  According to the VETSNET management team, it gave priority to other  activities, such as establishing appropriate governance and organizational  structures, and it is still gathering information to assist in prioritizing the  activities that remain. Nonetheless, if VBA does not institutionalize these  improvements, it increases the risk that these process improvements may  not be maintained through the life of the project or be available for  application to other development initiatives.
subsections: section_title: VBA Is Taking Action on Certain Overall Management Concerns
paragraphs: SEI concluded that VBA’s management issues would need to be addressed  as part of the implementation of its overall recommendation. SEI’s overall  management concerns focused on the project governance, project  planning, and conversion of records currently on the BDN to the  replacement system.
subsections: section_title: Top Management Included in Revised Governance Structure
paragraphs: SEI guidance for software development stresses the need for  organizational commitment and the involvement of senior management in  overall project governance. In its assessment, SEI noted that because  management of the VETSNET project had been assigned to VBA’s  information technology (IT) group, certain activities critical to the  veterans’ benefits program, but not traditionally managed by the IT group,  had not been visible to the project’s management. The contractor pointed  out that the IT group, business lines, and regional offices needed to share  ownership and management of the replacement project through an  established governance process and that the project management office  should include business representatives. According to SEI, the project  needed to establish ownership responsibility, including addressing total  system and process operating costs.
In response to the assessment, VBA developed a new governance structure  for the initiative, which the Under Secretary for Benefits approved in  March 2006. In the new structure, the VETSNET Executive Board that had  been in place was expanded and reorganized to serve as a focal point and  major governance mechanism for the replacement initiative. A Special  Assistant (reporting directly to the Under Secretary) was appointed to  coordinate and oversee the initiative as the head of the VETSNET  Executive Team, which was established to provide day-to-day operational  control and oversight of the replacement initiative. Implementation Teams  were also established to conduct the day-to-day activities associated with  implementing the initiative. This governance structure established a  process for IT, business lines, and regional offices to share ownership and  management, as SEI advised.
The roles and membership of each of the organizational elements in the  new governance structure are described in table 1.
When the new governance structure was approved in March 2006, the  Under Secretary ensured that those involved in the project gave it high  priority, directing certain key personnel (such as members of the  executive and implementation teams) to make the initiative their primary  responsibility, and other personnel (technical staff that provide support to  other systems) with collateral (non-VETSNET) duties to make the project  their first priority. He also placed limitations on the transfer of personnel  away from the project, recognizing the importance of staff continuity in  successfully completing the initiative. Staff members assigned project  responsibilities could be reassigned (i.e., given nonpromotion, lateral  reassignments) only with approval from the Under Secretary or his deputy.
By implementing the new governance and organizational structure and  ensuring that the project has priority, VBA partially responded to SEI’s  concerns in this area; however, VBA has not yet taken action with regard  to ownership responsibility for total system and process operating costs,  as SEI advised. According to administration officials, the replacement  initiative is an in-house, contractor-assisted development effort, in which  three different contractors provide support for program management,  system development, and testing and validation of requirements. VA  reported VETSNET system costs to the Congress totaling about   $89 million for fiscal years 1996 through 2006, with additional estimated  costs for completion of the initiative in 2009 of about $62.4 million.  However, according to project management officials, these costs do not  include expenditures for in-house development work. This in-house work  involves many VA personnel, as well as travel to various locations for  testing and other project related activities. Thus, considerable costs other  than contract cost have been incurred, which have not been tracked and  reported as costs for the replacement initiative. Without comprehensive  tracking and reporting of costs incurred by the replacement project, the  ability of VBA and the Congress to effectively monitor progress could be  impaired.
section_title: VBA Took Action to Address Project Planning and Management
paragraphs: A second major area of overall management concern was project planning.  In particular, the lack of an integrated master schedule for the VETSNET  project was a major concern articulated by SEI, as well as in our prior  work. An integrated project plan and schedule should incorporate all the  critical areas of system development and be used as a means of  determining what needs to be done and when, as well as measuring  progress. Such an integrated schedule should consider all dependencies  and include subtasks so that deadlines are realistic, and it should  incorporate review activities to allow oversight and approval by high-level  managers. Among other things, the program plan should also include  capacity requirements for resources and technical facilities to support  development, testing, user validation, and production.
SEI was specifically concerned that releases with overlapping  functionality were being developed at the same time, with insufficient time  to document or test requirements; this approach constrained resources  and added complexity because of the need to integrate completed  applications and newly developed functionality. In addition, SEI observed  that the VETSNET program suffered from lack of sufficient test facilities  because it did not have enough information to plan for adequate capacity.
In response to these project planning concerns, VETSNET management,  with contractor support, developed an integrated master schedule to guide  development and implementation of the remaining functionalities for the  replacement system. The VETSNET Integrated Master Schedule, finalized  in September 2006, includes an end-to-end plan and a master schedule.  According to VBA, the end-to-end plan documents the end state of the  project from a business perspective, which had not previously been done.  The master schedule identifies the necessary activities to manage and  control the replacement project through completion. The schedule also  describes a new software release process that provides more time to work  on requirements definition and testing, and allows for more cross- organizational communications to lessen the possibility of not meeting  requirements.
In addition, the new release process includes a series of management  reviews to help control the software development process and ensure that  top management has continuous visibility of project related activities.  These reviews occur at major steps in the system development life cycle  (as described in fig. 1: initiation, preliminary design, and so on). Such  reviews are intended to ensure that the VETSNET Executive Team and the  VETSNET Executive Board agree and accept that the major tasks of each  step have been properly performed.
Nonetheless, while the Integrated Master Schedule is an important  accomplishment, it may not ensure that the project sufficiently addresses  capacity planning, one of SEI’s areas of concern. According to its  assessment, capacity requirements for the fully functional production  system were unclear. Capacity planning is important because program  progress depends on the availability of necessary system capacity to  perform development and testing; adjustments to such capacity take time  and must be planned. If systems do not have adequate capacity to  accommodate workload, interruptions or slowdowns could occur.  According to SEI, capacity adjustments cannot be made instantly, and  program progress will suffer without sufficient attention to resource  requirements.
However, the VETSNET Integrated Master Schedule does not identify  activities or resources devoted to capacity planning. According to officials,  the capacity of the corporate environment (that is, corporate information  systems, applications, and networks) is being monitored by operational  teams with responsibility for maintaining this environment. According to  project officials, VETSNET representatives participate in daily conference  calls in which the performance of corporate applications is discussed, and  changes in application performance are reported to the VETSNET  developers for investigation and corrective action. Project officials  reported that when a performance degradation occurred in some  transactions during performance testing, it was determined that additional  computing capacity was needed and would be acquired. One reason why  the occurrence of degradation had not been anticipated by the VETSNET  project was that capacity planning had not taken place. Unless it ensures  that capacity planning and activities are included in the Integrated Master  Schedule, the replacement project may face other unanticipated  degradations that it must react to after the fact, thus jeopardizing the  project’s cost, schedule, and performance.
section_title: Conversion Efforts Were Suspended as Advised, but Issues Remain
paragraphs: In its assessment, SEI questioned VBA’s approach to developing  functionality while concurrently converting records from the BDN to the  replacement system. It noted that VBA had chosen to complete software  development according to location rather than according to the type of  functionality. Specifically, in 2004, VBA began an effort to remove all  claims activity (both new and existing claims) at one regional office  (Lincoln, Nebraska) from the BDN to the replacement system, developing  the software as necessary to accommodate processing the types of claims  encountered at that site. The intention was to address each regional office  in turn until all sites were converted. According to SEI, this approach had  resulted in the development being stalled by obstacles arising from the  variety of existing claims. The contractor advised VBA to focus first on  developing functionality to process original claims and discontinue efforts  to convert existing claims until all the necessary functionality had been  developed, and the replacement system’s ability to handle new cases of  any complexity had been proven by actual experience.
In accordance with this advice, VBA stopped converting existing records  from the BDN and changed its focus to developing the necessary  functionality to process all new compensation claims. According to the  integrated master schedule, conversion activities are now timed to follow  the release of the needed functionality. That is, according to the schedule,  VBA plans to begin converting each type of record from the BDN only  after the necessary functionality for the replacement system has been  developed and deployed to process that type of record.
In addition, the project is mitigating risk by resuming conversions  beginning with a test phase. Its strategy is first to convert records for  terminated claims—claims that are no longer being paid. Conversion of  the terminated records will be followed by additional conversions of  records for claims receiving payment at Lincoln and Nashville (these two  sites are being used to test system functionality during development). The  VETSNET leadership will consider testing complete with the successful  conversion at these two sites.
However, SEI raised three additional issues with regard to the conversion  of records that VBA has not fully addressed:    First, SEI expressed concerns that conversion failures could lead to  substantial numbers of records being returned to the BDN. Because of  differences in the database technologies used for the old system and the  replacement system, certain types of errors in BDN records cause  conversion to fail (according to SEI, approximately 15 percent of all these  records are estimated to have such errors). If records fail to convert  correctly, they may need to be returned to the BDN so that benefits can  continue to be paid. However, this process is not simple and may involve  manually reentering the records.
Second, SEI observed that VBA was also depending on manual processes  for determining that records were converted successfully, including the  use of statistically random samples, and that it was aiming to ensure  correctness to a confidence level of 95 percent. However, in the absence of  a straightforward method for automatically returning records to the BDN,  SEI considered the 5 percent risk of error unacceptable for conversions of  large numbers of records.
Finally, SEI observed that the lack of automated methods and the  complexity of the processes meant that conversions required careful  planning and assurance that adequate staff would be available to validate  records when the conversions took place.
However, the VETSNET leadership has not developed any strategy to  address the possibility that a large number of cases might need to be  returned to the BDN during the testing phase. For example, it has not  included this possibility as a risk in its risk management plan. The absence  of a strategy to address this possibility could lead to delays in program  execution.
Further, VBA has not yet decided whether a possible 5 percent error rate  is acceptable or developed a plan for addressing the resulting erroneous  records. If VBA does not address these issues in its planning, it increases  the risk that veterans may not receive accurate or timely payments.
Finally, the VETSNET leadership has not yet developed detailed plans that  include the scheduled conversions for each regional office and identified  staff to perform the necessary validation. Having such plans would reduce  the risk that the conversion process could be delayed or fail.
section_title: VBA Is Improving Software Development Processes, but Improvements Are Not Yet Institutionalized
paragraphs: In addition to actions addressing the overall management concerns  identified by SEI, VBA has steps action to improve its software  development processes in risk management, requirements management,  defect/change management, and performance measures. SEI described  weaknesses in all of these areas. The steps taken have generally been  effective in addressing the identified weaknesses, but VBA has not yet  institutionalized many of these improvements. According to the VETSNET  management team, it made a conscious decision first to establish the  governance, build the organization, implement processes to gain control,  and gather additional information about the project to assist in prioritizing  the remaining activities. The team also stated that some of the processes  are no longer VBA’s responsibility but are now that of the newly realigned  Office of Information and Technology. Nonetheless, if VA does not  develop and establish documented policies and procedures to  institutionalize these improvements, they may not be maintained through  the life of the project or available to be applied to other development  initiatives.
subsections: section_title: Risk Management Plan Has Been Revised
paragraphs: Risk management is a process for identifying and assessing risks, their  impact and status, the probability of their occurrence, and mitigation  strategies. Effective risk management includes the development of a risk  management plan and tracking and reporting progress against the plan.  According to SEI, to the extent that risk management existed at all in the  replacement program, it was conducted on a pro forma basis without real  effect on program decisions. SEI said that risks and risk mitigation  activities needed to be incorporated into all aspects of program planning,  budgeting, scheduling, execution, and review.
In response to these concerns, VBA has instituted risk management  activities that, if properly implemented, should mitigate the risks  associated with the project. Specifically, the VETSNET team, with  contractor support, developed a risk management plan that was adopted  in January 2007. The plan includes procedures for identifying, validating,  analyzing, assessing, developing mitigation strategies for, controlling and  tracking, reporting, and closing risks. It also establishes criteria for  assessing the severity of the risks and their impact.
The VETSNET leadership also developed a Risk Registry database, and its  contractor reviewed and prioritized the open risks. Each open risk was  evaluated, and a proposed disposition of the risk was submitted to  VETSNET management. Of the 39 open risks, all but 3 had been addressed  as of January 2007.
The development documentation for each planned software release also  includes sections on risk. In accordance with these plans, the VETSNET  leadership is currently capturing potential risks and tracking action items  and issues. At weekly status meetings, VETSNET leadership reviews Risk  Registry reports of open risks. According to the contractor, the reports  identify each risk and provide information on its age, ownership, and  severity.
However, these risk management activities have not yet been  institutionalized through the definition and establishment of associated  policies and procedures. If it does not institutionalize these improvements,  VBA increases the possibility that the VETSNET project’s improvements in  risk management may not be maintained through the life of the project.
section_title: Requirements Management Has Improved
paragraphs: Requirements management is a process for establishing and maintaining a  common understanding between the business owners and the developers  of the requirements to be addressed, as well as verifying that the system  meets the agreed requirements. SEI’s report commented that the  VETSNET project requirements were not stable, and that the business  owners (including subject-matter experts) and developers were separated  by many organizational layers, resulting in confusion and delays in  development of the system. SEI suggested that VA restructure project  activities to focus on defining an effective requirements process.  According to SEI, the project needed to ensure that subject-matter experts  were included in developing requirements and that evaluation criteria  were established for prioritizing requests for changes to requirements.  Finally, business owners should confirm that the system is meeting  organizational needs.
VBA has instituted requirements management activities that, if properly  implemented, should help avoid the instability and other requirements  problems identified by SEI. Specifically, VBA took steps to establish a  requirements management process and to stabilize the requirements. For  example, the development release process in the Integrated Master  Schedule includes a phase for requirements identification. In addition, the  project has established and begun applying evaluation criteria to prioritize  change requests for its development releases. Further, until all claims are  completely migrated from the BDN to the replacement system, in July  2006, the Under Secretary directed that any additional requirements would  have to have his approval.
Responding to SEI’s advice regarding the involvement of subject-matter  experts and business owners, VBA designed the new release process to  directly involve subject-matter experts in requirements workshops.  Further, the business teams participate in user-acceptance testing.
However, these requirements management activities have not yet been  institutionalized through the definition and establishment of policies and  procedures. Until they are established, VBA runs the risk that the  improved processes will not be maintained through the life of the  VETSNET project or used in other software development projects.
section_title: Management Attention Is Being Focused on Major Software Defects
paragraphs: SEI raised numerous concerns regarding the defect process for the  replacement system. These concerns for defect management included   (1) identification of defects, (2) determination of cause, and   (3) disposition of defects—either by correction or workaround. According  to SEI guidance, defect management prevents known defects from  hampering the progress of the program. The management process should  include clearly identifying and tracking defects, analyzing defects to  establish their cause, tracking their disposition, clearly identifying the  rationale for not addressing any defects (as well as proposing  workarounds), and making information on defects and their resolution  broadly available. SEI’s report stated that VBA needed to distinguish  defects from changes to requirements and develop a process for defect  management.
To respond to these concerns and focus program management attention  on major defects, the VETSNET Executive Team, with contractor support,  conducted an audit of existing defects and revised the defect management  process. The audit of the defect database determined that the VETSNET  database used to capture software defects also included change requests;  as a result, work required to address processes that did not work properly  was not distinguished from requests for added or changed functionality,  which would require review and approval before being addressed. To  address this issue, the team separated defects from change requests, and a  new severity rating scale was developed. All open defects were  recategorized to ensure the major defects would receive appropriate  program management attention. Also, all defect categorizations must meet  the approval of the VETSNET Business Architect and are scheduled for  action as dictated by the severity level.
Although these steps address many of SEI’s concerns regarding VBA’s  defect management process, more remains to be done before the process  is institutionalized. The Program Management Office has reported that  actions to revise the defect management process are complete, but the  process description is still in draft, and policies and procedures have not  been fully established. Without institutionalized policies and procedures  for the defect management process, it may not be maintained consistently  through the life of the project.
section_title: The Replacement Initiative Is Tracking Certain Performance Measures
paragraphs: According to SEI, performance measures are the only effective mechanism  that can provide credible evidence of a program’s progress. The chosen  measures must link directly to the expected accomplishments and goals of  the system, and they must be applied across all activities of the program.  In its report, SEI stated that although VBA was reporting certain types of  performance measures, it was not relating these to progress in system  development. For example, VBA reported the total number of veterans  paid, but did not provide estimates of how many additional veterans would  be paid when the system incorporated specific functionalities that were  under development. SEI suggested several measures that would provide  more evidence of progress, such as increases in the percentage of original  claims being paid by the replacement system, as well as user satisfaction  and productivity gains resulting from use of the replacement system  applications at regional offices.
In response to these concerns, the replacement project has begun tracking  a number of the measures suggested by SEI, including  increases in the percentage of original claims being paid by the  replacement system, increases in the percentage of veterans’ service representatives using the  new system,   decreases in the percentage of original claims being entered in the BDN  rather than the replacement system.
Although these measures provide indications of VA’s progress, other  measures that could demonstrate the effectiveness of the replacement  system have not been developed. For example, VBA has not developed  results-oriented measures to capture user satisfaction or productivity  gains from the system. Without measuring user satisfaction, VBA has  reduced assurance that the replacement system will be accepted by the  users. In addition, measures of productivity would provide VBA with  another indication of progress toward meeting business needs.
section_title: VETSNET Is Currently Processing a Portion of Compensation Claims, but Much Work Remains
paragraphs: After more than 10 years of effort, including the recent management,  organizational, and process improvements, VBA has achieved critical  functionalities needed to process and pay certain original compensation  claims using the replacement system, but it remains far from completing  the project. For example, the replacement system is currently being used  to process a portion of the original claims that veterans file for  compensation. Nonetheless, the system requires further development  before it can be used to process claims for the full range of compensation  and pension benefits available to veterans and their dependents. In  addition, VBA still faces the substantial task of moving approximately 3.5  million beneficiaries who are currently being served by the BDN to the  replacement system.
As designed, VETSNET consists of five major system applications that are  used in processing benefits:    Share—used to establish claims; it records and updates basic information  about veterans and dependents both in the BDN and the replacement  system.
Modern Award Processing–Development (MAP-D)—used to manage the  claims development process, including the collection of data to support  the claims and the tracking of claims.
Rating Board Automation 2000 (RBA 2000)—provides laws and  regulations pertaining to disabilities, which are used by rating specialists  in evaluating and rating disability claims.
Award Processing (Awards)—used to prepare and calculate the benefit  award based on the rating specialist’s determination of the claimant’s  percentage of disability. It is also used to authorize the claim for payment.
Finance and Accounting System (FAS)—used to develop the actual  payment record. FAS generates various accounting reports and supports  generation and audit of benefit payments.
According to VBA officials, all five of the software applications that make  up the new system are now being used in VA’s 57 regional offices to  establish and process new compensation claims for veterans. As of March  2007, VBA leadership reported that the replacement system was providing  monthly compensation payments to almost 50,000 veterans (out of about   3 million veterans who receive such payments). In addition, the  replacement system has been processing a steadily increasing percentage  of all new compensation claims completed: this measure was 47 percent in  January 2007, increasing to 60 percent in February and 83 percent in  March.
Nonetheless, considerable work must be accomplished before VBA will be  able to rely on the replacement system to make payments to all  compensation and pension beneficiaries. Specifically, while all five  software applications can now be used to process original compensation  claims for veterans, two of the applications—Awards and FAS—require  further development before the system will be able to process claims for  the full range of benefits available to veterans and their dependents.   Table 2 shows the status of development of all five applications.
According to VBA officials, Awards and FAS do not yet have the capability  to process original claims for payment to recipients other than veterans:  that is, the applications do not have the functionality to process claims for  survivor benefits and third-party/nonveteran payee claims. In addition,  further development of these applications is needed to process pension  benefits for qualified veterans and their survivors. Until enhancements are  made to Awards and FAS, these claims must continue to be processed and  paid through the BDN. Also, according to VBA, FAS does not yet have the  capability to generate all the necessary accounting reports that support the  development of benefits payments to claimants.
As described earlier, VBA now has an Integrated Master Schedule that  incorporates the activities that VBA needs to manage in order to complete  the replacement project. According to the schedule, the remaining  capabilities necessary to process compensation and pension claims are to  be developed and deployed in three software releases, as shown in table 3.
As table 3 shows, VBA does not expect to complete the development of the  functionalities needed to process all new compensation and pension  claims until August 2008. However, according to VBA, the estimated  completion date is the planned date for completing all development and  testing, but it is not necessarily the date when users will be able to begin  using the new system. Before such use can begin, other activities need to  occur. For example, users must receive training, and VETSNET program  management must authorize the use of the system at each regional office.
In addition to its remaining software development activities, VBA also  faces the challenge of converting records for claims currently paid by the  BDN to the replacement system. Existing compensation and pension cases  on the BDN number about 3 million and about 535,000, respectively.   Table 4 shows the phases in which VA is planning to perform conversions,  according to its Integrated Master Schedule.
As the table shows, VBA conversion efforts began in March 2007. VBA first  performed conversion testing on 310,000 terminated (that is, inactive)  compensation cases so that it could develop and apply lessons learned to  the conversion of live records. According to VETSNET officials, VBA  planned to continue testing by converting live cases at two regional offices  (Lincoln and Nashville) that were used as testing sites during  development. It then plans to perform the conversion of all live  compensation cases. After the compensation conversion is complete, VBA  plans to begin efforts to convert pension benefits cases.
Based on VETSNET project documentation, activities supporting the  releases have so far been performed on time, consistent with the  milestones in the recently finalized Integrated Master Schedule. For  example, VA completed the Project Initiation and Review Authorization  for Release 1 on September 7, 2006, as scheduled (see fig. 1, shown earlier  in the report, for the phases of system development and the required  milestone reviews). It also completed the Preliminary Design Review and  the Critical Design Review as scheduled (on November 20 and December  22, respectively).  Planning for Release 2 is also on schedule: a kickoff  meeting was held on January 24, 2007, which established the scope of the  release, and the Project Initiation and Review Authorization was  conducted on February 8.
section_title: Conclusions
paragraphs: VA has responded to SEI’s assessment by making significant changes in its  approach to the project and its overall management, including slowing the  pace of development, establishing a new governance structure, and  ensuring staff resources. However, VBA has not yet addressed all the  issues raised by the SEI assessment. That is, it has not ensured ownership  responsibility for total system and process operating costs, because it is  not currently monitoring and reporting in-house expenditures for the  project. It has not defined processes and resources for capacity planning  for the project. In addition, VBA has not yet addressed issues related to the  conversion of records now on the BDN to the replacement system.  Specifically, it has not addressed the risk that large numbers of records  may need to be returned to the BDN, decided on the degree of confidence  it will require that records are converted accurately, or developed  complete plans for converting and validating records. In addition, although  VBA has improved key processes for managing the software development,  these processes have not yet been institutionalized in defined policies and  procedures, and performance measures of productivity and user  satisfaction have not been developed. VETSNET management has stated  that it gave priority to other activities, such as establishing appropriate  governance and organizational structures, and that it is still gathering  information to assist in prioritizing the activities that remain.
Much work remains to be done to complete the VETSNET initiative.  Although VBA has substantially increased the number of claims being paid  by the replacement system, it must not only finish the development and  deployment of the software, it must also convert the over 3.5 million  records now on the BDN to the replacement system. Addressing the  remaining issues identified by SEI would improve VBA’s chances of  successfully completing the replacement system and ending reliance on  the aged BDN to pay compensation and pension benefits.
section_title: Recommendations for Executive Action
paragraphs: To enhance the likelihood that the replacement system will be successfully  completed and implemented, we are recommending that the Secretary of  Veterans Affairs take the following five actions:    Direct the CIO to institute measures to track in-house expenditures for the  project.
Direct the VETSNET project to include activities for capacity planning in  the VETSNET Integrated Master Schedule and ensure that resources are  available for these activities.
Direct VBA to (1) develop a strategy to address the risk that large numbers  of records may need to be returned to the BDN; (2) determine whether a  greater confidence level for accuracy should be required in the conversion  process; and (3) develop a detailed validation plan that includes the  scheduled conversions for each regional office and the validation team  members needed for that specific conversion.
Direct the CIO to document and incorporate the improved processes for  managing risks, requirements, and defects into specific policy and  guidance for the replacement initiative and for future use throughout VBA.
Direct the replacement project to develop effective results-oriented  performance measures that show changes in efficiency, economy, or  improvements in mission performance, as well as measures of user  satisfaction, and to monitor and report on the progress of the initiative  according to these measures.
section_title: Agency Comments and Our Evaluation
paragraphs: In providing written comments on a draft of this report, the Secretary of  Veterans Affairs agreed with our conclusions and concurred with the  report’s recommendations. (The department’s comments are reproduced  in app. II.) The comments described actions planned that respond to our  recommendations, such as incorporating processes developed for the  VETSNET project in standard project management policies, processes,  and procedures that would be used for all IT projects in the department. In  addition, the comments provided further information on actions already  taken, such as details of the records conversion process. If the planned  actions are properly implemented, they could help strengthen the  department’s management of the replacement system project and improve  the chances that the system will be successfully completed.
We are sending copies of this report to the Chairman and Ranking  Minority Member of Committee on Veterans’ Affairs. We are also sending  copies to the Secretary of Veterans Affairs and appropriate congressional  committees. We will make copies available to other interested parties  upon request. Copies of this report will also be made available at no  charge on GAO’s Web site at http://www.gao.gov.
Should you or your staff have any questions about this report, please  contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine (1) to what extent the Department of  Veterans Affairs (VA) has followed the course of action recommended by  the Carnegie Mellon Software Engineering Institute (SEI) and addressed  the concerns that it raised and (2) the current status of the replacement  project, the Veterans Service Network (VETSNET).
To determine the actions taken to implement SEI’s recommended  approach and address the concerns it raised, we    determined the recommended actions by analyzing the report;    compared the concerns identified in the assessment to actions planned,  actions undertaken but not completed, and actions implemented by VA  officials or contractors; interviewed contractor, VA, and VETSNET program office officials to gain  an understanding about processes developed and procedures  implemented; and   obtained and reviewed relevant VA and contractor documents that  disclosed or validated VA responses to SEI’s concerns.
To determine the status of system development efforts and the extent that  tasks planned for the initiative were completed, we analyzed VA and  contractor documentation regarding system operations and development,  time frames, and activities planned. We analyzed VA documents that  disclosed costs to date and costs planned for completion of the initiative.  We did not assess the accuracy of the cost data provided to us. We  supplemented our analyses with interviews of VA and contractor  personnel involved in the replacement initiative.
We visited the Nashville and St. Petersburg regional offices to observe the  replacement system in operation and the processes and procedures used  to test and validate the replacement system as it was being developed and  implemented. We analyzed VA documentation and relevant evidence from  contractors involved in the replacement effort to establish the work  remaining to complete the project. Finally, we interviewed cognizant VA  and contractor officials, responsible for developing, testing, and  implementing the replacement system.
We performed our work at VA offices in Washington, D.C., and at VA  regional offices in Nashville, Tennessee, and St. Petersburg, Florida, from  April 2006 to April 2007 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributions were made to  this report by Barbara Oliver, Assistant Director; Nabajyoti Barkakati;  Barbara Collier; Neil Doherty; Matt Grote; Robert Williams; and Charles  Youman.