Test and Evaluation: DOD Has Been Slow In Improving Testing of Software-Intensive Systems
Published: 1993-09-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed the Department of Defense's (DOD) efforts to improve its software acquisition processes and test and evaluation of software-intensive systems, focusing on: (1) how software-related problems affect defense acquisition programs during operational test and evaluation (OT&E); and (2) barriers that limit OT&E effectiveness.
section_title: What GAO Found
paragraphs: GAO found that: (1) DOD software-intensive systems often fail to meet their performance expectations, experience cost overruns and schedule delays, and are not ready for OT&E because they are not subject to rigorous developmental testing and evaluation and software development is often not fully mature when OT&E begins; (2) DOD has not adequately addressed barriers that inhibit software acquisition and OT&E effectiveness; (3) DOD has not adequately addressed the importance of software development, lacks a consistent policy that defines and coordinates software maturity and operational testing readiness, has not adequately defined and managed software requirements, and lacks reliable cost, schedule, and performance data; (4) DOD has not implemented previous recommendations to ensure that quality software is adequately developed and tested and software-intensive systems are effective; (5) although the military services have tried to improve their software development processes, they lack a DOD-wide coordinated strategy; and (6) the creation of a single executive-level office for software development and testing would help to resolve long-standing software problems.

section_title: Letter
paragraphs: This report addresses test and evaluation of software-intensive systems and the Department of Defense’s efforts to improve the software process. It contains recommendations to you.
The head of a federal agency is required under 31 U.S.C. 720 to submit a written statement on actions taken on our recommendations to the House Committee on Government Operations and the Senate Committee on Governmental Affairs not later than 60 days after the date of the report. A written statement must also be submitted to the House and Senate Committees on Appropriations with the agency’s first request for appropriations made more than 60 days after the date of the report.
We are sending copies of this report to the Chairmen and Ranking Minority Members of the House Committee on Government Operations, Senate Committee on Governmental Affairs, and House and Senate Committees on Armed Services; the Secretaries of the Army, the Navy, and the Air Force; and the Director of the Office of Management and Budget. We will also make copies available to others upon request.
If you or your staff have any questions concerning this report, please call me at (202) 512-4587. Major contributors to this report are listed in appendix IV.
subsections: section_title: Introduction
paragraphs: Because computer software controls most functions of modern defense systems, the systems’ performance depends largely on the quality of that complex and increasingly costly software. In fact, many major weapon systems may be inoperable if software fails to function as required. Mission-critical computer software, which is integral to most military applications, tends to be more difficult to develop than software for other types of applications primarily because it must operate in real time under very unique environments. Accordingly, software quality has become a primary concern in emerging defense acquisition programs, including weapon systems; automated information systems; and command, control, communications, and intelligence systems.
subsections: section_title: Defense Software Costs
paragraphs: Software-intensive systems are, by nature, highly complex and often require millions of lines of code. These significant factors increase the overall costs of software. Although the Department of Defense (DOD) does not know precisely how much it spends on software, the Defense Systems Management College projected that DOD would spend about $36.2 billion for software in 1992. The Management College expects software costs to continue to rise at a rate proportionately higher than computer hardware costs. According to the DOD Inspector General, the costs of computer hardware components integral to weapon systems and other critical military and intelligence systems, are expected to remain stable at about $6 billion annually between 1990 and 1995, whereas corresponding software costs are expected to grow from about $30 billion to $42 billion.(See fig. 1.1.)
DOD estimates that about 30 percent of its software life-cycle expenditures are for initial development and 70 percent are for post-deployment software support, that is, maintaining, upgrading, and modifying existing software to correct deficiencies, respond to mission changes, or enhance technology. Up-front improvements in the quality of software development processes and more effective software test and evaluation may play a significant role in controlling these costs, which are incurred largely after systems have been fielded.
section_title: Overview of Test and Evaluation
paragraphs: The primary purpose of test and evaluation during the acquisition process is to reduce the risk that the system or equipment either will not meet performance specifications or cannot be effectively used in its intended operational environment. Test and evaluation is therefore designed to detect errors in both software and hardware before a system is fielded and to provide essential information to decisionmakers for assessing acquisition risk.
Early in the acquisition cycle, development test and evaluation (DT&E) primarily measures a system’s technical performance and compliance with contractual specifications. DT&E, which starts at the systems requirements phase and proceeds through the design phase, is designed to detect errors in software and hardware prior to operational test and evaluation (OT&E). Later, OT&E focuses on the system’s effectiveness and suitability. (See table 1.1.) Before a system is certified as ready for OT&E, any major deficiency is expected to be identified and corrected during DT&E.
Deficiencies discovered during developmental and operational testing affect a system’s cost, schedule, and performance. However, problems that are not identified and resolved until operational testing and production begins are generally more difficult and costly to correct.
Test and evaluation is the key internal control to ensure that decisionmakers have valid, credible information for making development and production decisions. OT&E results contribute to decisions not only on acquiring new systems but also on modifying systems deployed in the field and upgrading the software or hardware of systems already in production.
section_title: Objectives, Scope, and Methodology
paragraphs: The Congress and senior DOD officials have long been concerned with DOD’s inability to field software-intensive defense acquisition programs on time and within budget. Because of these concerns, we initiated this review to identify (1) the extent to which software-related problems affect the performance of defense acquisition programs during OT&E, (2) pervasive barriers in the acquisition process that limit the effectiveness of test and evaluation of software-intensive systems, and (3) DOD’s efforts to resolve software test and evaluation problems.
A wide range of technical and management challenges impact the development, testing, and fielding of software-intensive systems. However, this report is not intended to address the technical aspects of the software development process, which, at best, is a difficult and complex undertaking. Rather, the report focuses more directly on those barriers that require the attention of DOD acquisition and technology management officials and that DOD believes limit the effectiveness of OT&E of software-intensive systems.
To accomplish our objectives, we reviewed defense acquisition, software development, and test and evaluation policy documents. To determine the status of systems’ software during OT&E, we analyzed the OT&E results of  27 systems that represented the total population of major programs the services had identified as having undergone OT&E during the 2-year period from January 1990 to December 1992.
We also visited several prime contractors identified by DOD and service officials to obtain an overview of industry practices. The organizations we visited included the
Office of the Under Secretary of Defense for Acquisition, Washington,
Office of the Director, Defense Research and Engineering, Washington,
Office of the Director, Operational Test and Evaluation, Washington, D.C.;
Army’s Director of Information Systems for Command, Control, Communications, and Computers, Washington, D.C.;
Test and Evaluation Management Agency, Washington, D.C.;
Army Operational Test and Evaluation Command, Alexandria, Virginia;
U.S. Army Communications-Electronics Command, Fort Monmouth, New
Navy Operational Test and Evaluation Force, Norfolk, Virginia;
Fleet Combat Direction Support System Activity, Dam Neck, Virginia;
Marine Corps Operational Test and Evaluation Activity, Quantico, Virginia;
Air Force Operational Test and Evaluation Center, Albuquerque,
Sacramento Air Logistics Center, California;
Jet Propulsion Laboratory, Pasadena, California;
Hughes Aircraft Corporation, Ground Systems Group, Fullerton,
Hughes Aircraft Corporation, Radar Systems Group, Torrance, California;
General Dynamics Electronics Division, San Diego, California;
Science Applications International Corporation, San Diego, California; and
TRW Systems Integration Group, Carson, California.
We conducted our review between April 1992 and April 1993 in accordance with generally accepted government auditing standards.
section_title: Software Test and Evaluation Problems and Obstacles to Solving Them
paragraphs: Since the 1970s, software problems discovered during OT&E have adversely affected the cost, schedule, and performance of major defense acquisition systems. Because many systems do not undergo rigorous DT&E and therefore begin OT&E before their software is fully mature (i.e., the software is able to satisfy all documented user requirements), they often fall short of system performance expectations. The readiness of such systems for OT&E is therefore questionable.
Although DOD recognizes these problems, it has made only limited progress in adopting solutions. Fundamentally, DOD has not (1) acknowledged or adequately addressed the criticality of software to systems’ operational requirements early enough in the acquisition process; (2) developed, implemented, or standardized decision-making tools and processes (e.g., metrics) to help measure or project weapon system software cost, schedule, and performance risk; (3) developed test and evaluation policy that provides specific guidance regarding software maturity; and (4) adequately defined and managed requirements for its increasingly complex software (see ch. 3).
subsections: section_title: OT&E Often Identifies Immature Software-Intensive Systems
paragraphs: To ensure no surprises during OT&E, defense systems are expected to be subjected to rigorous DT&E. Formal operational test readiness reviews also address the readiness of systems for OT&E. However, software-intensive systems have repeatedly failed to meet users’ requirements during OT&E and, in some cases, during operations in the field. This has been recognized in DOD and industry as a major contributor to DOD’s “software crisis.” In general, the thoroughness of DT&E and the readiness of such systems for operational testing has been questionable.
According to a 1992 report by the Secretary of the Air Force, virtually all software-intensive defense systems suffer from difficulties in achieving cost, schedule, and performance objectives. Our prior reports, the comments of senior officials responsible for various aspects of software development, and the reports of the services’ operational test agencies  (27 such reports were analyzed during our review) corroborate the existence of significant software problems.
subsections: section_title: OT&E Results Reported by the Services’ Operational Test Agencies
paragraphs: Our review of the services’ OT&E reports from January 1990 to December 1992 showed that 23 of 27, or 85 percent, of the software-intensive systems tested were immature, ineffective in a threat environment, or difficult to maintain in field operations. Table 2.1 contains some typical examples of software problems found in these systems, and appendix I provides a more complete list of the problems.
section_title: Software Problems Had Been Previously Reported
paragraphs: Since the early 1970s, we have reported that defense systems have begun production without timely or realistic OT&E. More recently, we have reported on software shortfalls in individual systems (see table 2.2). For example, in December 1992 we reported that DOD’s mission-critical computer systems continued to have significant software problems due in part to a lack of management attention, ill-defined requirements, and inadequate testing.
section_title: Reasons Cited for Software Problems
paragraphs: DOD officials cited the following reasons for immature software during OT&E: In many cases, rigorous DT&E is not being done before systems begin OT&E; nonetheless, the systems are being certified as ready before they have achieved appropriate levels of maturity.
Problems are not identified until it is too late to address them effectively and economically because some program managers may not fully report program weaknesses that could lead decisionmakers (i.e., the Office of the Secretary of Defense, Congress, or the military services’ acquisition executive) to adjust program funding.
The career success of participants in the acquisition process is perceived to depend more on getting programs into production than on achieving successful program outcomes. Thus, program managers have incentives to delay testing and take chances that immature systems might succeed in OT&E.
The congressional appropriations process forces programs to be calendar-driven rather than event-driven, causing program managers to prematurely certify systems as ready for OT&E to avoid losing funding or slipping schedules.
Some program managers give priority to developing software that will support a system production decision and give less attention to the post-deployment support element of life-cycle costs.
section_title: Several Barriers Prevent Effective Software Test and Evaluation
paragraphs: Our review identified several pervasive barriers that need the attention of DOD acquisition and technology management officials and inhibit the solutions of DOD’s software test and evaluation problems. Eliminating these barriers will require the difficult process of changing the acquisition culture, a task that must be driven from the top and must be consistent. The barriers are (1) failure of the acquisition community to adequately address the critical nature of software; (2) lack of credible cost, schedule, and performance data as the basis for decision-making; (3) lack of specific software test and evaluation policy; and (4) ineffective definition and management of requirements for software.
subsections: section_title: The Acquisition Community Has Not Adequately Responded to the Critical Nature of Software
paragraphs: Although major defense acquisition systems depend largely on the quality of computer resources, software has been perceived as secondary to hardware and as a lower priority during development. Due to the traditional mind-set of the prevailing acquisition culture, the acquisition community has not appropriately focused on the criticality of software to cost, schedule, and performance. Also, software, unlike hardware, has lacked a disciplined, systems engineering approach to development.
Viewing software as something that can be fixed later, DOD’s acquisition community has been almost exclusively concerned with the cost and schedule of hardware early in the development life cycle. Historically, program managers have known little about software; have left software management to technical managers who are not always part of the decision-making process; and generally have not become involved in software development and testing until problems affected cost and schedule, by which time it was usually too late to resolve these problems cost-effectively.
Additionally, program managers have little incentive to alter these practices and to ensure that software is appropriately mature before systems are certified as ready for OT&E based on rigorous DT&E.
DOD officials generally believe that test and evaluation should focus on the total system—both software and hardware—rather than two separate systems, as in the past. They told us that the acquisition process is most effective when development problems are detected and corrected early in the acquisition life cycle, rather than during or after OT&E.
section_title: DOD Lacks Credible Data for Decision-Making
paragraphs: Software managers, developers, acquisition officials, and those charged with oversight responsibilities need dependable information and independent techniques for measuring the progress of development efforts and for monitoring the balance of cost, schedule, and performance objectives. However, the quality of data available to decisionmakers remains largely ad hoc and overly optimistic and may be too dependent on informal channels. As a result, the ability of decisionmakers to objectively or accurately estimate future costs and schedules of defense systems continues to be limited. Also, the ability to learn from the past has been impaired, as each software development effort has tended to start anew and independent of other, sometimes quite similar efforts.
DOD has yet to develop and implement the management processes and tools required to improve the reliability of its data. For example, the “best practices” in the private sector indicate the following benefits that can be achieved by using software management, quality, and process metrics:
Management metrics help determine progress against plans.
Quality metrics help assess product attributes, such as requirements stability, performance, user satisfaction, and supportability.
Process metrics provide indicators evaluating tools, techniques, organization, procedures, and so on.
DOD officials told us that software metrics have broad applications for defense acquisition programs because of their usefulness to government and private software developers, the test and evaluation community, and decisionmakers. Some officials believe that software metrics, in combination with prescribed work breakdown structure, are essential for managing cost, schedule, and performance risk in the defense systems acquisition process. Other officials told us that software metrics present a valuable input for independently monitoring the maturity of software and its readiness for OT&E. However, although they are useful, software metrics cannot substitute for actual test and evaluations.
subsections: section_title: The Private Sector Has Benefited From Software Metrics
paragraphs: All of the contractors we visited had established, with assistance from the Software Engineering Institute, similar quality improvement programs that incorporated software metrics into day-to-day decisions on software development projects. The contractors viewed software metrics as valuable decision-making and estimating tools. They generally credited the metrics for savings, higher productivity, and more credible information, resulting in better program and management decisions. Contractor officials believe that DOD could benefit similarly from implementing metrics-based software development approaches.
One division within one company we visited invested about $400,000 in software metrics-based process improvements. Company officials projected annual savings from this one-time investment at about  $2 million. Officials estimated that, due to the company’s lower software development costs, the government had saved millions of dollars in its development contracts. They added that increasing contractor productivity could translate into more competition and more affordable DOD acquisitions.
DOD has indicated that it expects the cost of maintaining software-intensive systems after deployment in the field, which typically accounts for 70 percent of DOD’s total software costs now, will decline when disciplined, measurable software development processes are used. According to defense officials we talked to, the additional cost of requiring the use of software metrics and related tools during development ranges from 0 to 6 percent of overall software development costs.
section_title: Software Metrics Are Not Widely Used in DOD
paragraphs: Although DOD has a written acquisition policy that requires the use of software metrics and other tools to improve the quality and the timeliness of data, to date DOD has not fully implemented it. The development and use of software metrics by the Office of the Secretary of Defense (OSD) and the services has not been consistent or fully coordinated. For example, each of the services is independently developing software metrics for its own use while OSD is sponsoring studies of a core set for DOD-wide implementation. However, coordination to avoid costly duplication of effort has been limited.
Although the potential benefits of software metrics have been recognized, program management offices have not fully adopted their use. Industry and DOD officials attributed this hesitancy to several basic attitudes. For example, the officials believe that software metrics may provide decisionmakers with too much direct access to a program’s status (i.e., they may enable decisionmakers to determine how problems began and who is responsible); may add to the overall cost of software development, even though the potential benefits may not be realized in the short term; may be used against managers who are perceived as not performing well; may not represent an improvement over “business as usual.”
section_title: DOD Lacks a Consistent Software Test and Evaluation Policy
paragraphs: OSD and service officials acknowledge that current OSD acquisition and life-cycle support policy does not adequately address planning for software test and evaluation in a logical, structured fashion and that a software policy void exists with respect to test and evaluation of software-intensive systems. Current policy is not definitive and does not adequately address the following critical questions:
When is software ready to test (i.e., maturity)?
When and how much should modifications or upgrades be retested (i.e., regression testing)?
What is the appropriate use of software “patches” (e.g., temporary software programming fixes) during testing?
However, OSD does not plan to issue guidelines specifically directing the services how to manage these complex issues. Rather, it plans to issue a series of “expectations” or “evaluation guidelines” for software development and test and evaluation for use by oversight action officers in addressing both maturity and regression testing. OSD is also considering developing an on-line data base for software issues to capture best practices, lessons learned, and some specific guidance from superseded software test and evaluation policy. Although these efforts may prove beneficial, they fall short of providing enforceable criteria for ensuring effective oversight and test and evaluation.
section_title: Requirements Definition and Management Lack Continuous User Involvement
paragraphs: Even though many factors contribute to DOD’s failure to produce systems that meet user needs in a timely, cost-effective manner, one critical factor is defining and managing users’ requirements. Effectively defining requirements, along with focused management and appropriate user involvement, is critical to the success of a program. As discussed earlier, the inability of software-intensive systems to meet users’ needs has consistently been demonstrated during OT&E and, in general, has been a pervasive problem in the acquisition process.
The definition of users’ requirements, particularly for large, complex systems and for unprecedented systems (i.e., those that are unlike any that have already been built), is subject to varying interpretations by many different groups. These groups include field users, users’ representatives, program managers, contracting officers, and contractors. Each group brings different expectations and agendas to the contracting, development, and acquisition processes. (See app. II for a list of DOD organizations responsible for test and evaluation.)
In the Army, the Training and Doctrine Command establishes the general user requirements for a new system in an operational requirements document. The Army Materiel Command then transforms these requirements into technical and administrative requests for proposals for contracting, after which the contractor proposes how to meet those requirements. Because this process often does not keep the user appropriately involved during this transformation of original requirements into procurement efforts, top-level operational requirements documents may result in systems that differ from those the user had envisioned.
Because these requirements often represent an area of great risk, DOD believes a comprehensive requirements management strategy is essential to reducing overall program risk. In addition to the fact that user requirements may not be well understood, formulated, or articulated at the start of a program, the requirements almost invariably change throughout a system’s life cycle. These changes are often directly related to the length of the development process and include changes in threat, technology, doctrine, unrealistic schedules, and perceived user opportunity.
DOD considers early and continuous user involvement in the requirements definition and management process and early testing programs to be essential to successful outcomes of software-intensive development efforts. To help ensure that a system will meet user requirements and expectations, DOD officials believe they need to formally involve the users in the total system development. They said that system developers should be communicating with the users from the beginning of an acquisition program and throughout its life cycle and allowing the users to influence the development effort. DOD and private sector officials cited the benefits of user involvement in several programs, particularly in automated information systems.
section_title: DOD Efforts to Resolve Long-Standing Software Testing Problems
paragraphs: Since the 1980s, DOD has studied and restudied its inability to field software-intensive systems on time, within cost projections, and in accordance with users’ performance requirements. Each succeeding study built upon earlier studies and consistently recommended key actions needed for successful acquisition strategies and effective OT&E. However, OSD has made only limited progress in adopting these long-standing recommendations. The individual military services have tried to improve their software development processes, but a DOD-wide, coordinated approach is lacking. Senior OSD officials told us that they believe the creation of a single OSD-level office for software would, in part, help to resolve long-standing software problems.
subsections: section_title: DOD Has Not Effectively Implemented Solutions to Software Problems
paragraphs: DOD’s 1983 Software Test and Evaluation Project report concluded that solutions to the software test and evaluation problems required more effective management, rather than technological breakthroughs. The report’s recommendations included integrating test and evaluation into software development; defining clearly testable software requirements and capabilities; assessing and identifying critical software risks and applying appropriate developing, recording, and using software metrics; developing and supporting the use of automated test tools and systematic developing and implementing triservice standards for unified software development, testing, and evaluation approaches.
More recently, Defense Science Board (1987) and Software Assessment Project (1990) studies have reached conclusions similar with those of the earlier study.
OSD has responded to these recommendations by issuing written policy and guidance manuals and instructions. For example, DOD Instruction 5000.2, Defense Acquisition Management Policies and Procedures, requires the use of a software work breakdown structure, software metrics, and a disciplined development process. However, according to a 1987 Defense Science Board study, most of the recommendations remained unimplemented. The Board stated that “if the military software problem is real, it is not perceived as urgent.” Our work demonstrates that many basic improvements to the DT&E of software-intensive systems remain unimplemented in 1993.
OSD responsibility for oversight of software-intensive systems is shared between the Under Secretary of Defense for Acquisition (computers embedded in weapon systems) and the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence Systems (also responsible for Automated Information Systems). Although the acquisition processes are essentially identical for all major defense programs, two different acquisition policy series are used to govern their development. In its draft Software Master Plan, DOD stated that this dual oversight has resulted in duplicative and artificially fragmented acquisition guidance, policies, and oversight for software-intensive systems.
We found that DOD officials had been unable to reconcile various test and evaluation resourcing issues that exist, in part, due to this organizational division of responsibility. According to DOD officials, for example, even though the services’ operational test agencies are responsible for conducting OT&E of automated information systems, they have not been funded for this testing and have generally not conducted such testing because their focus has been on the traditional testing of weapon systems. Further, DOD officials indicated that the OT&E of one automated system was delayed due to lack of test funding and disagreements between OSD and the services regarding OT&E policy. Additionally, senior defense officials specifically singled out the lack of dedicated government DT&E of automated information systems as a concern that needed to be addressed.
section_title: Services Are Working Independently to Improve Software
paragraphs: According to the Software Engineering Institute, all of the services have used ad hoc practices that have resulted in unpredictable costs and schedules and low-quality software products that do not meet users’ needs. To address these problems, the services have taken different approaches to improving software development and test and evaluation and are in various stages of implementing those improvements.
Among the services, the Army has implemented more of the recommended software development and testing processes in a servicewide, goal-oriented way. The Air Force’s operational test organization has used a well-documented and consistent metrics process to measure software maturity for OT&E. A Navy software support activity has also established a software development process using metrics similar to the practices in industry.
Although the services’ operational test agencies have agreed on implementing five common software metrics, OSD and the services are generally developing software metrics and making other improvements independently, rather than using the same ones to meet common needs. OSD recently established a Software Test and Evaluation Task Force to work toward implementing common policies and consensus. It is too soon, however, to determine if this will effectively address the fragmented, redundant approaches observed during our field work.
section_title: Army’s Servicewide, Proactive Approach
paragraphs: In September 1989, the Army convened the Software Test and Evaluation Panel to improve software test and evaluation practices and to prevent immature software from being deployed in operational systems. The Army Operational Test and Evaluation Command initiated the panel because it believed that software problems were the primary cause of delays in operational testing.
As a result of the panel’s 1992 report, the Army issued policy guidance on the procedures necessary for effective OT&E of software-intensive systems and on software requirements management. Unlike the other services, the Army has made substantial progress in developing enforceable policy guidance. The Army also implemented 12 servicewide requirements, management, and quality software metrics and is in the process of implementing a centralized metrics data base to enable decisionmakers to better monitor the progress made during a system’s life cycle. Other improvements include a standard framework for test and evaluation of software, a clear definition of responsibilities under the Army process known as continuous evaluation, the involvement of independent test and evaluation personnel from the start of a software development, increased user involvement in defining and refining requirements, and the early and frequent demonstration of software development progress.
In addition, Army post-deployment software support agencies have begun to provide such support throughout a system’s life cycle. For example, the Communications-Electronics Command provides (1) life-cycle software engineering for mission-critical defense systems; (2) support in developing and testing command, control, communications, and intelligence systems; and (3) hands-on experience and formal training in software engineering for Army Materiel Command interns.
The Communications-Electronics Command also developed the Army Interoperability Network to support software and interoperability throughout Army systems’ life cycles. This computer resources network, available for use by developers, testers, and evaluators of command, control, communications, and intelligence systems, is designed to provide for early integration and interoperability assurance, reduced development costs, and more efficient system scheduling for life-cycle software support. Additionally, the command developed (1) a software life-cycle engineering process, (2) software process metrics, and (3) automated tools to manage software support and the software development process. The overall effect of such tools on DT&E and OT&E is still to be seen.
subsections: section_title: Air Force Efforts to Increase the Effectiveness of Software OT&E
paragraphs: With the assistance of the Software Engineering Institute, the Air Force developed a process improvement program in December 1991 and directed its software development activities to implement the program in July 1992. The Air Force is now developing policy intended to encourage all bidders on Air Force software contracts to improve current operations so that the bids could be assessed at a high software maturity level. Also, the Air Force is developing standardized procedures for software test and evaluation that will be used in both development and operational tests.
Historically, the Air Force acquisition community has not required the use of metrics due to differences in opinions about which metrics have value and whether attention to the wrong metrics could lead to an incorrect focus and a skewed outcome. Through a limited set of software metrics, the Air Force Operational Test and Evaluation Center has had some success in measuring software maturity for OT&E. To assess the contribution of software to the system’s operational suitability, the Center uses software deficiency reports in evaluating software maturity and a structured questionnaire approach to determine software supportability. This approach has been less than optimal due to the absence of more formal metrics programs in Air Force acquisition programs. Also, this relatively informal process has focused on projecting weapon systems’ suitability but not effectiveness.
However, as part of its software process improvement plan, the Air Force is developing a software metrics policy on what data are required to support the metrics (such as contracting how the metrics will be reported, and how to interpret the results.
The policy will require all acquisition programs to use certain software metrics and to report the results of the metrics at every Air Force program review, thus providing decisionmakers with more current, reliable insights into the status of software development. The software metrics will include the OSD core set and the Air Force core set. Some Air Force guidance on using metrics was issued in 1991.
In September 1991, the Sacramento Air Logistics Center, with the assistance of the Software Engineering Institute, made its first self-assessment of software development and support processes. The assessment indicated that program results were unpredictable, systems were completed behind schedule and over cost, and customers were often dissatisfied. Also, because the Center lacked a unified, structured process for supporting software, it was difficult for management to gain the insight required to effectively plan and control software programs. To correct these deficiencies, the Center plans to establish a documented process for project planning, project management, requirements management, configuration management, and support organization management. According to Center officials, other Air Force logistics centers are similarly involved in these assessments.
Center officials also believe that program managers lack an effective process model and reliable historical data to estimate the cost, schedule, and resource requirements for software development. Such estimates are essential to effectively measure progress against planned performance. Recognizing that software metrics are key to oversight by all decisionmakers, the Center has established a Software Metrics Working Group. The group is expected to define the data to be collected and a way to present those data to project managers and first-line supervisors so that the software project management process can be stabilized and be repeated.
Also, to improve post-deployment software support, the Center has established a team to develop a process model that will provide a comprehensive, structured process for making software changes to improve the maintainability and supportability of software, make software support more cost-effective and more responsive to user be tailorable to all types of software applications, and make managers more responsible and accountable for resources used in software test and evaluation.
The process model is aimed at overcoming all the historical problems common to post-deployment software support, as well as improving (1) project management visibility, (2) user productivity through better understanding of responsibilities, and (3) day-to-day activities.
section_title: Navy Implementation of Software OT&E Process Improvements
paragraphs: The Navy’s software OT&E improvement efforts have been slow compared to the other services, and its primary OT&E focus has been on the total system. However, as part of its Total Quality Leadership Plan, the Navy is beginning to take some actions to improve its software development and testing processes. For example, in August 1992, the Navy tasked a Test and Evaluation Process Action Team to develop recommendations to improve the Navy’s test and evaluation process throughout all phases of the acquisition process and readiness of systems for OT&E.
On the basis of a 4-month review that included an analysis of Army and Air Force software management initiatives and policy guidance, the team concluded that the Navy needed to implement stronger fleet support for test and evaluation in Navy acquisition a formal working group to coordinate and resolve Navy test planning a clearer, more timely requirements definition and management process; a rigorous OT&E certification review process.
The team further concluded that certification and testing of software releases and of mission-critical systems needed additional study. With respect to metrics, the team concluded that a common set of metrics was needed as a tool to measure software maturity but that metrics alone should not be the determining factor in the certification of systems as ready for OT&E.
In addition, the team concluded that the Navy should issue revised policy guidance on the procedures necessary for effective test and evaluation of software-intensive systems, conduct rigorous operational test readiness reviews before systems are certified as ready for operational test, develop and recommend a common set of at least eight metrics to be used by decisionmakers as indicators of software maturity, develop additional metrics that can be used by program managers to measure the progress of development efforts, determine the appropriate level of testing for major and minor releases of develop and recommend methods to streamline the process for testing software releases.
The report concluded that the test and evaluation initiatives and recommendations would work if implemented by instructions, formal policy, and clear command guidance and accompanied with command involvement. Although it is far too soon to tell if these recommendations will be effective for Navy acquisition programs, they appear to be consistent with those made by the DOD Software Test and Evaluation Project report in 1983.
By contrast, the Navy Fleet Combat Direction Systems Support Activity, with the assistance of the Software Engineering Institute, has already developed an improvement program that mirrors the best practices in the private sector and may assist government software development activities in improving their processes. Begun in January 1992, the program was intended to improve the Activity’s software development and support processes. As a result, the Activity began using 13 software management metrics to make fact-based decisions about systems’ performance during software development. Activity officials said that the indicators provided a clear and concise reporting structure; improved planning and data collection for future projections; provided a degree of standardization among projects, contractors, and in-house personnel; fostered closer adherence to military standards; greatly improved communication and problem and process definition; acted as a reliable early-warning system if plans were not met; and highlighted small problems early so that they could be resolved before they grew.
The Activity’s software development process, in our view, is structured and measurable and includes relatively open, periodic reporting to management, which is a good foundation for decision-making.
section_title: Conclusions and Recommendations
paragraphs: Test and evaluation of software and software-intensive defense systems remains among the most difficult, most expensive, and least understood processes in the defense acquisition cycle. Although key software test and evaluation study recommendations have provided a starting point for DOD to address and resolve the software crisis, only limited progress has been made in improving the ability of software-intensive systems to meet users’ requirements.
Some OSD officials believe that the lack of a single OSD-level office for software has been the primary reason long-standing software problems have not been resolved. Other officials have concluded that OSD needs to take a more direct role in ensuring that software-intensive systems are ready for OT&E before this critical process begins.
In our view, consistent adoption across DOD of the recommendations in this report could greatly enhance the OT&E of software and better enable DOD to accomplish its objectives of developing software-intensive systems on schedule, within cost, and in accordance with required performance capabilities. DOD must go beyond simply reworking the prior studies on software test and evaluation. Moreover, promulgating more policy guidance without ensuring that the guidance is implemented is not the solution.
Overall, DOD has not (1) developed an overarching strategy that ensures development and implementation of key software test and evaluation policy throughout DOD; (2) issued definitive acquisition and life-cycle support policies that focus on software test and evaluation; and (3) adopted a focused, disciplined approach to software development and test and evaluation that recognizes the critical nature of software.
To achieve the potential of its mission-critical software and to accomplish its software improvement objectives, DOD must overcome the prevailing acquisition culture’s failure to react to the problems of modern defense systems; that is, DOD must understand that software is a critical path through which systems achieve performance objectives. Because this message has often been ignored, systems have proceeded into production when software had not yet achieved the appropriate level of maturity to yield valid test results. Because of the acquisition community’s bias toward hardware, DOD has not adequately ensured that software was fully mature and had undergone thorough and rigorous DT&E before systems were certified as ready for OT&E.
DOD does not currently have a high-quality decision-making data base to ensure that decisions concerning mission-critical software are made based on reliable, credible data. Further, DOD does not have reasonable assurance that unnecessary duplication and redundancy in software development are being avoided. DOD has not adequately (1) coordinated its efforts to develop and use software metrics for defense acquisition programs; (2) made maximum use of contractors’ software development processes that have been favorably assessed by credible, independent evaluation; (3) developed team-building efforts to ensure the early and continuous involvement of users with acquisition, post-deployment support and testing personnel; and (4) filled in the policy and oversight voids that have contributed to the shortfalls that we have addressed.
OSD has recently established a Software Test and Evaluation Task Force to work toward common policies. Further, some senior defense officials and software development and support personnel at the services’ working levels are working independently to resolve some of the pressing software issues. Overall, however, the agency has not adequately responded to the magnitude of the problem.
We are encouraged by the progress the services have made in developing policy guidance to improve the OT&E of its software-intensive systems and of its software requirements. We are particularly encouraged by the Army’s implementation of requirements, management, and quality software metrics, as well as a centralized metrics data base. Accordingly, we believe these Army initiatives should be watched closely by OSD, since these efforts may show potential for application throughout DOD.
We are similarly encouraged by senior defense acquisition and technology officials’ recognition of the critical need to improve management of the requirements process for software-intensive systems. Effectively defining requirements; building teams of appropriate users, logisticians, program management, and contractor personnel; and focusing appropriate management attention is critical to improving software maturity before OT&E begins.
subsections: section_title: Recommendations
paragraphs: To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, we recommend that the Secretary of Defense issue and implement a software test and evaluation policy that defines testing requirements for software maturity, regression testing, and the use of temporary software fixes during testing; strengthen controls to ensure that operational testing does not begin until results of development test and evaluation demonstrate an appropriate level of software maturity; require program management officials to define exit criteria for certifying a systems’ readiness for operational testing at the beginning of full-scale development (i.e., milestone II); and require the services to develop a common core set of management metrics for software (i.e., cost, schedule, and quality) for major defense programs early in the development cycle to be approved at milestone II.
section_title: Agency Comments and Our Evaluation
paragraphs: In its comments on a draft of this report, DOD generally agreed with our findings and recommendations that additional steps can be taken to improve the test and evaluation of software-intensive systems. Accordingly, DOD indicated that, during fiscal year 1994, it will issue revised software policy guidance to address these concerns. However, we believe that the issuance of revised policy guidance without incentives to change behavior or ensure effective implementation could have little effect in ensuring software maturity.
DOD pointed out that many of the reasons for immature software during OT&E were outside the control of the test and evaluation community. We agree with DOD’s comment and specifically address this fact in the report.
DOD indicated that programs reviewed as part of our analysis preceded DOD’s most recent acquisition guidance and that the potential benefits of such guidance were therefore not sufficiently acknowledged in the report. DOD indicated that current updates of its acquisition policy series provided improved guidance and stronger program oversight for development strategies, testing, and requirements. However, this policy has some voids and, more importantly, it remains to be seen whether and to what degree the policy updates will be implemented and whether they will address the long-standing problems.
DOD also indicated that the benefits of software metrics for OT&E were not supported. We did not attempt to quantify the direct benefits of software metrics for OT&E. We pointed out that experts in DOD and in the private sector believe that software metrics could improve the management of the software development process and thus contribute to greater software maturity before OT&E begins.
DOD’s comments appear in appendix III.