Software Capability Evaluation: VA's Software Development Process Is Immature
Published: 1996-06-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed software development processes and practices at the Department of Veterans Affairs' Veterans Benefits Administration (VBA) and Austin Automation Center (AAC).
section_title: What GAO Found
paragraphs: GAO found that: (1) neither VBA nor AAC satisfy any of the criteria for a repeatable software development capability; (2) VBA and AAC do not adequately define systems requirements, train personnel, plan software development projects, estimate costs or schedules, track software project schedules or changes, manage software subcontractors, or maintain quality assurance and software configuration procedures; (3) VBA initiatives to improve its software development processes include developing and distributing interim configuration management procedures, identifying a library structure for all work products, and meeting with the Software Engineering Institute (SEI) to discuss software development; (4) VBA and AAC cannot reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints; and (5) VBA and AAC can use their strengths in software quality assurance and their improvement activities in software configuration management as a foundation for improving their software development processes.

section_title: Letter
paragraphs: You requested that we review the Department of Veterans Affairs’ (VA) capability for developing and maintaining software for its computer-based systems. The Department depends on these systems to support program operations and assist in managing the achievement of critical mission objectives. Our objective was to review software development processes at the Veterans Benefits Administration (VBA) and VA’s Office of Information Resources Management’s Austin Automation Center (AAC). The sites and projects were selected by VBA and VA, respectively, as those that represent their best software development processes and practices.
subsections: section_title: Background
paragraphs: VBA is in the process of modernizing many of its older, inefficient systems and has reportedly spent an estimated $294 million on these activities between October 1, 1986 and February 29, 1996. The modernization program can have a major impact on the efficiency and accuracy with which over $20 billion in benefits and other services is paid to our nation’s veterans and their dependents. However, in the last 6 years some aspects of VBA’s service to the veterans have not improved. For example, in the past 6 years, VBA’s reported processing time for an original compensation claim rose from 151 days in fiscal year 1990 to 212 days in fiscal year 1994. In March 1996 the average time was 156 days.
Software development is a critical component of this major modernization initiative. VBA, with the assistance of contractors, will be developing software for the Veterans Services Network (VETSNET) initiative, a replacement for the existing Benefit Delivery Network. For efforts like VETSNET to succeed, it is crucial that VBA have in place a disciplined set of software development processes to produce high quality software within budget and on schedule.
VBA relies upon its own staff and contractors to develop and maintain software that is crucial to its overall operations. In fiscal year 1995, VBA had 314 full-time equivalents, with payroll expenses of $20.8 million, devoted to developing and maintaining software throughout the organization. It also spent $17.7 million in contract services in these areas.
section_title: Scope and Methodology
paragraphs: To evaluate VA’s software development capability, version 2.0 of the Software Engineering Institute’s (SEI) software capability evaluation (SCE) method was used by an SEI-trained team of GAO specialists. The SCE is a method for evaluating agencies’ and contractors’ software development processes against SEI’s five-level software Capability Maturity Model (CMM), as shown in table 1. These levels and the key process areas (KPAs) described within each level define an organization’s ability to develop software, and can be used to improve its software development processes. The findings generated from an SCE identify (1) process strengths that mitigate risks, (2) process weaknesses that increase risks, and (3) improvement activities that indicate potential mitigation of risks.
We requested that VA identify for our evaluation those projects using the best software development processes implemented within VBA and AAC. VBA and AAC identified the following sites and projects. —Compensation & Pension/Financial Management System —Claims Processing System We evaluated the software development processes used on these projects, focusing on KPAs necessary to achieve a repeatable capability. Organizations that have a repeatable software development process have been able to significantly improve their productivity and return on investment. In contrast, organizations that have not developed the process discipline necessary to better manage and control their projects at the repeatable level incur greater risk of schedule delay, cost overruns, and poor quality software. These organizations rely solely upon the variable capabilities of individuals, rather than on institutionalized processes considered basic to software development.
According to SEI, processes for a repeatable capability (i.e., CMM level 2) are considered the most basic in establishing discipline and control in software development and are crucial steps for any project to mitigate risks associated with cost, schedule, and quality. As shown in table 2, these processes include (1) requirements management, (2) software project planning, (3) software project tracking and oversight, (4) software subcontract management, (5) software quality assurance, and (6) software configuration management.
We conducted our review between August 1995 and February 1996, in accordance with generally accepted government auditing standards.
section_title: Veterans Benefits Administration Software Practices
paragraphs: Highlights of our evaluation of VBA’s software practices using the SEI criteria outlined in appendix II follow.
Requirements Management - The purpose of requirements management is to establish a common understanding between the customer and the software project of the customer’s requirements that will be addressed by the software project. The first goal within this KPA states that, “system requirements allocated to software are controlled to establish a baseline for software engineering and management use.” VBA does not manage and control system requirements as required by this goal. Moreover, members of software-related groups are not trained in requirements management activities. Also, changes made to software plans, work products, and activities resulting from changes to the software requirements are not assessed for risk.
Software Project Planning - The purpose of software project planning is to establish reasonable plans for performing the software engineering and for managing the software project. VBA projects do not have software development plans, estimates for software project costs are not derived using conventional industry methods and tools, and VBA is unable to show the derivation of the estimates for the size (or changes to the size) of the software work products. Also, individuals involved in the software project planning are not trained in estimating and planning procedures applicable to their area of responsibility.
Software Project Tracking and Oversight - The purpose of software project tracking and oversight is to provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from software plans. VBA does track software project schedules against major milestones; however, as mentioned previously, these schedules and milestones are not derived using conventional industry methods nor is there a comprehensive software plan against which to track activities. Moreover, the size of software work products (or the size of changes to software work products) are not tracked, and the software risks associated with cost, resource, schedule, and technical aspects of the project are not tracked.
Software Subcontract Management - The purpose of software subcontract management is to select qualified software subcontractors and manage them effectively. VBA does not have a written organizational policy that describes the process for managing software contracts. Additionally, the software work to be contracted is neither defined nor planned according to a documented procedure. Finally, software managers and other individuals who are involved in developing, negotiating, and managing a software contract are not trained to perform these activities.
Software Quality Assurance - The purpose of software quality assurance is to provide management with appropriate visibility into the process being used by the software project and of the products being built. VBA has a software quality and control (SQ&C) group that has a reporting channel to senior management, independent of the project managers. The SQ&C group also performs testing of the software code. However, the SQ&C group does not participate in other software quality assurance (SQA) functions, such as the preparation, review, and audit of projects’ software development plans, standards, procedures, and other work products. Also, projects do not have SQA plans.
Software Configuration Management - The purpose of software configuration management is to establish and maintain the integrity of products of the software project throughout the project’s software life cycle. VBA has provided formal training to its staff in defining software processes. However, VBA cannot effectively control the integrity of its software work products because it has no software configuration control board, it does not identify software work products to be placed under configuration management, and it has no configuration management library system to serve as a repository for software work products. VBA has begun improvement activities in this area by (1) establishing a software configuration management group and (2) drafting a software configuration management procedure.
Following a presentation of GAO’s SCE results to the Chief Information Officer of VBA, the Director of VBA’s Office of Information Systems forwarded a letter to GAO citing a number of initiatives that are currently underway to address some of the stated deficiencies. Initiatives cited by the VBA include: development and distribution of interim configuration management procedures; identification of a library structure to hold all of the work products from the development process; and initiation of several meetings with SEI to discuss the Software CMM.
section_title: Austin Automation Center Software Practices
paragraphs: Similar to VBA, we compared the CMM criteria in appendix II to the software development practices at AAC. Summary results of this evaluation follow.
Requirements Management - AAC does not create or control a requirements baseline for software engineering. Also, AAC does not manage or control requirements. AAC does have a process for negotiating periodic contractual arrangements with customers, but this process does not include baselining and controlling software requirements.
Software Project Planning - Although AAC documents its schedule estimates for software development projects, there is (1) no defined methodology in use for estimating software costs, size, or schedule, (2) no derivation of estimates for the size (or changes to the size) of software products, and (3) no derivation of the estimates for software project costs. Similarly, AAC uses a project planning tool called “MultiTrak”. However, projects do not have software development plans.
Software Project Tracking and Oversight - AAC performs schedule tracking at major milestones. However, the goals for this KPA call for (1) the tracking of actual results and performances against software plans, (2) the management of corrective actions when deviations from the software plan occur, and (3) the affected parties to mutually agree to changes in commitments. AAC does not conform to these goals. For example, AAC does not track (1) the software risks associated with cost, resource, schedule, and technical aspects of the project and (2) the size of software work products (or size of changes to software work products).
Software Subcontract Management - Although the goals for this KPA emphasize the selection of qualified software subcontractors and managing them effectively, AAC does not (1) have a documented procedure that explains how the work to be contracted should be defined and planned and (2) ensure that software managers and other individuals who are involved in establishing a software contract are trained to perform this activity.
Software Quality Assurance - The goals within this KPA emphasize (1) the verification of the adherence of software products and activities to applicable standards, procedures, and requirements and (2) the reporting of noncompliance issues that cannot be resolved within the project to senior management. AAC has an automated data processing system integrity guideline and a systems integration service (SIS) group that has a reporting channel to senior management and is independent of the project managers. However, projects do not have SQA plans; the SIS group does not participate in certain SQA functions, such as the preparation, review, and audit of projects’ software development plans, standards, and procedures; and members of the SIS group are not trained to perform their SQA activities.
Software Configuration Management - AAC performs software (i.e., code only) change control using a tool called “ENDEVOR,” and its employees are trained in the use of this tool. However, the scope of the goals within this KPA cover all products in the entire software life cycle and not just the software code. AAC has not identified software work products (with the exception of software code) that need to be placed under configuration management, established a configuration management library system that can be used as a repository for software work products, or established a software configuration control board.
Unless both VBA and AAC initiate improvement activities within the various KPAs and accelerate those already underway, they are unlikely to produce and maintain high-quality software on time and within budget.
section_title: Conclusions and Recommendations
paragraphs: Because VBA and AAC do not satisfy any of the KPAs required for a level 2 (i.e., repeatable) capability, there is no assurance that (1) investments made in new software development will achieve their operational improvement objectives or (2) software will be delivered consistent with cost and schedule estimates. To better position VBA and AAC to develop and maintain their software successfully and to protect their software investments, we recommend that the Secretary of Veterans Affairs take the following actions:
Delay any major investment in software development beyond that which is needed to sustain critical day-to-day operations until the repeatable level of process maturity is attained.
Obtain expert advice to assist VBA and AAC in improving their ability to develop high-quality software, consistent with criteria promulgated by SEI.
Develop an action plan, within 6 months from the date of this letter, that describes a strategy to reach the repeatable level of process maturity. Implement the action plan expeditiously.
Ensure that any future contracts for software development require the contractor have a software development capability of at least CMM level 2.
section_title: Agency Comments and Our Evaluation
paragraphs: VBA comments responded to its SCE results, and VA comments responded to the SCE results for AAC.
subsections: section_title: VBA Comments on GAO Recommendations
paragraphs: In commenting on a draft of this report, the Veterans Benefits Administration (VBA) agreed with four of our recommendations and disagreed with one recommendation. VBA stated that while it agreed that a repeatable (i.e., level 2) level of process maturity is a goal that must be attained, it disagreed that “...all software development beyond that which is day-to-day critical must be curtailed...” VBA further stated that the payment system replacement projects, the migration of legacy systems, and other activities to address the change of century must continue. While we agree that the software conversion or development activities required to address issues such as the change of century or changes to legislation must continue, we would characterize these as sustaining critical day-to-day operations. However, major system development initiatives in support of major projects such as the system modernization effort, which involves several system replacement projects and the migration of legacy systems, and VETSNET, which includes several payment system replacement projects, should be reassessed for risk of potential schedule slippage, cost overrun, and shortfall in anticipated system functions and features. Shortcomings such as these are more likely from organizations with a software development maturity rating below level 2 (i.e., the repeatable level). Therefore, to minimize software development risks, we continue to believe that VBA should delay any major investment in software development unless it is required to sustain day-to-day operations, until a maturity rating of level 2 is reached.
Regarding the remaining four recommendations, we are pleased to see that VBA is already initiating positive actions, including acquiring the assistance of the Software Engineering Institute.
section_title: VA Comments on SCE Results for AAC
paragraphs: VA stated that we did not demonstrate a willingness or flexibility in relating AAC documentation products, activities, and terms to the SEI terms. We reviewed all documentation provided to us by VA including the documents listed in their comments on our draft report. As called for by the SCE methodology, we carefully compared all this documentation to the SEI CMM criteria. As stated throughout our report, we found some strengths but in many cases, VA’s documentation was not commensurate with that called for by the SCE methodology. Our comments on the specific key process areas follow.
subsections: section_title: Requirements Management
paragraphs: The VA comments stated that the OFM/IRM Business Agreement, dated September 1994, contains guidelines which mandate the management of software requirements. However, in our review of the documentation listed under requirements management (Enclosure 1: Documents Addressing Key Process Area), we found no evidence that these documents addressed any of the goals of this KPA. For example, (1) the allocated requirements are neither managed, controlled, nor baselined, and (2) no software development plans were developed based on the allocated requirements.
section_title: Software Project Planning
paragraphs: VA feels that the AAC Business Agreement and the negotiated quarterly contract satisfies this KPA; however, we found that AAC does not perform a majority of the activities required to meet the goals within this KPA. For example, AAC was not able to submit evidence for estimating software size and cost, nor did AAC demonstrate any methodology used for estimating schedules.
section_title: Software Project Tracking and Oversight
paragraphs: VA stated that project size and risk remain consistent throughout the development/implementation cycle. However, AAC did not provide our SCE team with any evidence validating this assertion and, as discussed on page 8, AAC does not track this information.
section_title: Software Subcontract Management
paragraphs: VA claims that specific written policies and procedures are followed when managing software contracts; however, AAC staff interviewed were unable to provide us with any specific policies or procedures used for software contracting. The AAC staff acknowledged that they do not track (1) software contractor performance at the coding level (i.e., track functionality only) or (2) contractor produced software documentation. Regarding training for software contract management, VA stated that its COTRs receive training in procurement, project management, and evaluating contractor performance. However, there is no indication that these courses are specific to software contracting. In addition, other individuals involved in establishing the software contract for the projects reviewed had not received contract management training related to software.
section_title: Software Quality Asurance
paragraphs: VA states that its ADP System Integrity Guide, dated September 1994, contains detailed procedures directing the SIS group in specific SQA functions. Although this is a good first step, the AAC is still deficient because it does not have project specific software quality assurance plans that are implemented for individual projects, as requuired by this KPA within the CMM. Furthermore, we were not provided with any evidence showing that the ADP System Integrity Guide has been officially issued or whether its use will be mandatory or discretionary.
section_title: Software Configuration Management
paragraphs: The VA comments do not present any additional evidence that would help to satisfy the criteria for this KPA. Specifically, communication between the SIS, AAC staff, and customer do not substitute for the rigor and discipline of a software configuration control board, which VA acknowledged they do not have. Furthermore, the placement of software code under configuration management is not sufficient to satisfy this KPA because other software work products—such as system design specifications, database specifications, and computer program specifications—are also required. Finally, although the AAC does maintain a library of those software work products that it does produce, the products are not maintained under a formal software configuration management discipline, which would include version control and rigorous requirements traceability.
We are sending copies of this report to the Chairmen and Ranking Minority Members of the House and Senate Committees on Veterans Affairs and the House and Senate Committees on Appropriations; the Secretary of Veterans Affairs; and the Director, Office of Management and Budget. Copies will also be made available to other interested parties upon request.
This work was performed under the direction of William S. Franklin, Director, Information Systems Methodology and Support, who can be reached at (202) 512-6234. Other major contributors are listed in appendix IV.
section_title: Comments From the Veterans Benefits Administration
section_title: Comments From the Department of Veterans Affairs
paragraphs: The following is GAO’s comment on the Department of Veterans Affairs’ May 24, 1996, letter.
subsections: section_title: GAO Comment
paragraphs: 1. This issue is not addressed in our report.
section_title: Level 2 Key Process Area Goals
paragraphs: To establish a common understanding between the customer and the software project of the customer’s requirements that will be addressed by the software project.
Goal 1 System requirements allocated to software are controlled to establish a baseline for software engineering and management use.
Goal 2 Software plans, products, and activities are kept consistent with the system requirements allocated to software.
To establish reasonable plans for performing the software engineering and for managing the software project.
Goal 1 Software estimates are documented for use in planning and tracking the software project.
Goal 2 Software project activities and commitments are planned and documented.
Goal 3 Affected groups and individuals agree to their commitments related to the software project.
To provide adequate visibility into actual progress so that management can take effective actions and when the software project’s performance deviates significantly from software plans.
Goal 1 Actual results and performances are tracked against the software plans.
Goal 2 Corrective actions are taken and managed to closure when actual results and performance deviate significantly from the software plans.
Goal 3 Changes to software commitments are agreed to by the affected groups and individuals.
To select qualified software subcontractors and manage them effectively.
Goal 1 The organization selects qualified software subcontractors.
Goal 2 The organization and the software subcontractor agree to their commitments to each other.
Goal 3 The organization and the software subcontractor maintain ongoing communications.
Goal 4 The organization tracks the software subcontractors’ actual results and performance against its commitments. (continued)
To provide management with appropriate visibility into the process being used by the software project and of the products being built.
Goal 1 Software quality assurance activities are planned.
Goal 2 Adherence of software products and activities to the applicable standards, procedures, and requirements is verified objectively.
Goal 3 Affected groups and individuals are informed of software quality assurance activities and results.
Goal 4 Noncompliance issues that cannot be resolved within the software project are addressed by senior management.
To establish and maintain the integrity of products of the software project throughout the project’s software life cycle.
Goal 1 Software configuration management activities are planned.
Goal 2 Selected software work products are identified, controlled, and available.
Goal 3 Changes to identified software work products are controlled.
Goal 4 Affected groups and individuals are informed of the status and content of software baselines.
section_title: Major Contributors to This Report
subsections: section_title: Accounting and Information Management Division, Washington, D.C.
paragraphs: David Chao, SCE Team Leader Gary R. Austin, SCE Team Member K. Alan Merrill, SCE Team Member Madhav S. Panwar, SCE Team Member Keith A. Rhodes, SCE Team Member Paul Silverman, SCE Team Member The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.