Test and Evaluation: Impact of DOD's Office of the Director of Operational Test and Evaluation
Published: 1997-10-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Department of Defense's (DOD) Office of the Director of Operational Test and Evaluation's (DOT&E) operations and organizational structure for overseeing operational testing, focusing on: (1) DOT&E's efforts and their impact on the quality of operational testing and evaluation in DOD; and (2) the strengths and weaknesses of the current organizational framework in DOD for operational testing.
section_title: What GAO Found
paragraphs: GAO noted that: (1) GAO's review of 13 case studies indicated that DOT&E oversight of operational testing and evaluation increased the probability that testing would be more realistic and more thorough; (2) specifically, DOT&E was influential in advocating increasing the reliability of the observed performance and reducing the risk of unknowns through more thorough testing, conducting more realistic testing, enhancing data collection and analysis, reporting independent findings, and recommending follow-on operational test and evaluation when suitability or effectiveness was not fully demonstrated prior to initiating full-rate production; (3) the independence of DOT&E--and its resulting authority to report directly to Congress--is the foundation of its effectiveness; (4) that independence, along with its legislative mandate, provides sufficient freedom and authority to exercise effective oversight of the operational testing and evaluation of new systems before a decision is made to begin full-rate production; (5) DOT&E can reduce the risk that systems are not adequately tested prior to the full-rate production decision but DOT&E cannot ensure that: (a) only systems whose operational effectiveness and suitability have been demonstrated through operational testing will proceed to the full-rate production decision; or (b) new fielded systems will accomplish their missions as intended or that the fielded systems are safe, survivable, and effective; (6) DOT&E management must balance its oversight responsibilities for operational testing with the broader acquisition priorities of program managers and service test agencies; (7) though supportive of DOT&E's mission and independence, program and service representatives frequently considered the time, expense, and resources expended to accommodate DOT&E concerns to be ill-advised; (8) several current trends may challenge DOT&E's ability to manage its workload and its ability to impact operational test and evaluation: (a) service challenges to DOT&E's authority to require and oversee follow-on operational testing and evaluation; (b) a decline in resources available for oversight; (c) an expansion of DOT&E involvement in activities other than oversight of major acquisition programs; (d) participation of DOT&E in the acquisition process as a member of working-level integrated product teams; and (e) greater integration of developmental and operational testing; and (9) these trends make it imperative that DOT&E prioritize its workload to achieve a balance between the oversight of major defense acquisition programs and other initiatives important to the quality of operational test and evaluation.

section_title: Letter
paragraphs: The Department of Defense (DOD) has proposed that the practices and policies of the Office of the Director of Operational Test and Evaluation (DOT&E) be modified to reduce the time and cost of developing and fielding new weapon systems. To help focus deliberations on DOD’s proposal, you asked us to review DOT&E’s operations and organizational structure for overseeing operational testing. Specifically, you asked us to assess (1) DOT&E’s efforts and their impact on the quality of operational testing and evaluation in DOD and (2) the strengths and weaknesses of the current organizational framework in DOD for operational testing. As part of our review, we conducted 13 case studies of the testing of individual weapon systems. (Our scope and methodology are described in app. I, and brief descriptions of the 13 weapon systems are provided in app. II.)
subsections: section_title: Background
paragraphs: In 1983, Congress established DOT&E to coordinate, monitor, and evaluate operational testing of major weapon systems. As part of the Office of the Secretary of Defense (OSD), DOT&E is separate from the acquisition community that conducts developmental and operational testing and therefore is in a position to provide the Secretary and Congress with an independent view. Congress created DOT&E in response to reports of conflicts of interest in the acquisition community’s oversight of operational testing leading to inadequate testing of operational suitabilityand effectiveness and the fielding of new systems that performed poorly. (DOD’s system acquisition process is described in app. III.)
By law, DOT&E serves as the principal adviser on operational test and evaluation in DOD and bears several key responsibilities, including monitoring and reviewing all operational test and evaluation in DOD, reporting to the Secretary of Defense and congressional committees whether the tests and evaluations of weapon systems were adequate and whether the results confirmed that the system is operationally suitable and effective for combat before a decision is made to proceed to full-rate production, and submitting to the Secretary of Defense and congressional decisionmakers an annual report summarizing operational test and evaluation activities during the preceding fiscal year.
In 1993, DOD’s advisory panel on streamlining and codifying acquisition laws concluded that DOT&E was impeding the goals of acquisition reform by (1) promoting unnecessary oversight, (2) requiring excessive reporting detail, (3) inhibiting the services’ discretion in testing, and (4) limiting participation of system contractors in operational tests where such involvement is deemed necessary by the services. The following year, DOD proposed legislative changes that would have reduced the scope and authority of DOT&E. In testimony, we opposed these changes because they were directed at perceived rather than documented problems and would undermine a key management control over the acquisition process—independent oversight of operational test and evaluation.
Although the legislative proposals were not adopted, in 1995 the Secretary of Defense implemented several operational test and evaluation initiatives in the Department to (1) involve operational testers earlier in the acquisition process, (2) use models and simulations effectively, (3) combine tests where possible, and (4) combine tests and training. The goals of these initiatives included saving time and money by identifying and addressing testing issues earlier in the acquisition process; merging or closely coordinating historically distinct phases, such as developmental and operational testing to avoid duplication; and using existing technologies and training exercises to create realistic and affordable test conditions.
section_title: DOT&E Advocates More Thorough Testing Than the Services
paragraphs: A frequent complaint among representatives of the services’ operational testing agencies was that DOT&E frequently demanded more tests than were proposed by the operational test agencies in draft master plans or test plans. Operational test agency representatives contended that the additional testing was either unnecessary for determining the operational effectiveness or suitability of a program or unrealistic in light of the limitations in the services’ testing resources. However, our review indicated that DOT&E urged more testing to reduce the level of risk and number of unknowns prior to the decision to begin full production, while program and service officials typically sought less testing and were willing to accept greater risk when making production decisions. The additional testing DOT&E advocated, often over the objections of service testers, served to meet the underlying objectives of operational testing—to reduce the uncertainty and risk that systems entering full-rate production would not fulfill their requirements.
The impact of DOT&E oversight varies with the system under development. Table 1 summarizes the types of impacts that DOT&E advocated or facilitated in operational testing among the 13 cases we studied. While the impacts vary, one consistent pattern in our case studies was a reduction in uncertainty regarding the weapon systems’ suitability or effectiveness prior to the full-rate production decision. Each of the impacts are discussed in more detail in tables 2-6 and in subsequent sections.
subsections: section_title: DOT&E Oversight Led to More Testing Than Proposed by the Operational Test Agencies
paragraphs: Two of DOT&E’s typical concerns in reviewing service test plans are that the proposed test methodologies enable (1) comparisons of a system’s effectiveness through side-by-side testing between the existing and modified systems and (2) assessments of a system’s reliability through a sufficient number of test repetitions. Table 2 illustrates examples of cases where additional testing was conducted at DOT&E’s insistence or with DOT&E’s support to alleviate these and other types of effectiveness and suitability concerns.
section_title: DOT&E Oversight Led to More Realistic Testing Than Proposed by the Operational Test Agencies
paragraphs: Table 3 illustrates examples where the design or conduct of operational testing was modified at DOT&E’s insistence or with DOT&E’s support to increase the realism of test conditions and reduce the uncertainty of system suitability or effectiveness.
section_title: DOT&E Oversight Led to Changes in the Data Analysis Plan
paragraphs: DOT&E can insist on or support changes in data analysis plans that provide more meaningful analyses for decisionmakers. Table 4 illustrates instances in which DOT&E altered the proposed data collection or analysis plans to enhance the reliability or utility of the test data.
section_title: DOT&E Interpreted the Results of Some Testing Less Favorably Than the Operational Test Agencies
paragraphs: DOT&E’s independent analysis of service test data may confirm or dispute the results and conclusions reported by the service. In the cases described in table 5, DOT&E’s analysis of service operational test and evaluation data resulted in divergent, often less favorable conclusions than those reached by the service.
section_title: DOT&E Recommended Follow-On Operational Test and Evaluation
paragraphs: When DOT&E concludes that a weapon system has not fully demonstrated operational suitability or effectiveness, or if new testing issues arise during initial operational test and evaluation, it may recommend that follow-on operational test and evaluation be done after the full-rate production decision. Table 6 identifies follow-on operational test and evaluation that DOT&E supported.
section_title: Strengths and Weaknesses of Current Organizational Framework
paragraphs: The existence of a healthy difference of opinion between DOT&E and the acquisition community is a viable sign of robust oversight. In nearly all of the cases we reviewed, the services and DOT&E cited at least one testing controversy. For example, services differ on how they view the relationship between operational testing and their development of tactics, techniques, and procedures. In addition, DOT&E’s ability to independently view the development and testing of new systems across the services brings value to the context of testing. However, several current trends have the potential to adversely affect DOT&E’s independence and its ability to affect operational test and evaluation, including (1) service challenges to DOT&E’s authority to require and oversee follow-on operational testing and evaluation, (2) declining resources available for oversight, (3) the management of limited resources to address competing priorities, (4) DOT&E’s participation in the acquisition process as a member of the program manager’s working-level integrated product teams, and (5) greater integration of developmental and operational testing. DOT&E’s impact on operational testing is dependent upon its ability to manage these divergent forces while maintaining its independence.
subsections: section_title: Independence Is the Key to DOT&E’s Effectiveness
paragraphs: Although the acquisition community has three central objectives—performance, cost, and schedule—DOT&E has but one: operational testing of performance. These distinct priorities lead to testing disputes. Characteristically, the disputes for each system we reviewed revolved around questions of how, how much, and when to conduct operational testing, not whether to conduct operational testing. Conflicts encompassed issues such as (1) how many and what types of tests to conduct; (2) when testing should occur; (3) what data to collect, how to collect it, and how best to analyze it; and (4) what conclusions were supportable, given the analysis and limitations of the test program. The foundation of most disputes lay in different notions of the costs and benefits of testing and the levels of risk that were acceptable when making full-rate production decisions. DOT&E consistently urged more testing (and consequently more time, resources, and cost) to reduce the level of risk and number of unknowns before the decision to proceed to full-rate production, while the services consistently sought less testing and accepted more risk when making production decisions. Among our case studies, these divergent dispositions frequently led to healthy debates about the optimal test program, and in a small number of cases, the differences led to contentious working relations.
In reviews of individual weapon systems, we have consistently found that testing and evaluation is generally viewed by the acquisition community as a requirement imposed by outsiders rather than a management tool to identify, evaluate, and reduce risks, and therefore a means to more successful programs. Developers are frustrated by the delays and expense imposed on their programs by what they perceive as overzealous testers. The program office strives to get the program into production despite uncertainties that the system will work as promised or intended. Therefore, reducing troublesome parts of the acquisition process—such as operational testing—is viewed as a means to reduce the time required to enter production.
Nonetheless, the commanders and action officers within the service operational test agencies were nearly unanimous in their support for an independent test and evaluation office within OSD. For example, the Commander of the Army’s Operational Test and Evaluation Command commended the style and orientation of the current DOT&E Director and affirmed the long-term importance of the office and its independent reporting responsibilities to Congress. The Commander of the Navy’s Operational Test and Evaluation Force stated that the independence of both DOT&E and the operational test agency was an essential element in achieving their common goal of ensuring that new programs pass sufficiently rigorous and realistic operational testing prior to fielding. The Commander of the Air Force’s Operational Test and Evaluation, while critical of DOT&E oversight of several major weapon systems, said that the services were well served by DOT&E’s potential to independently report to Congress. Moreover, nearly all the operational test agency action officers we interviewed participate in the integrated product teams with the DOT&E action officers and recognized the value of the Office’s independent oversight role. The action officers within the service testing organizations also have a degree of independence that enables them to represent the future users of systems developed in the acquisition community. These action officers stated that their ability to voice positions unpopular with the acquisition community was strengthened when DOT&E separately supported their views.
In discussions with over three dozen action officers and analysts responsible for the 13 cases we reviewed, the independence of DOT&E emerged as the fundamental condition to enable effective and efficient oversight. The foundation of interagency (i.e., DOT&E and service operational test agencies) relations is based on the independence of DOT&E, its legislative mandate, and its independent reporting to Congress. DOT&E is outside the chain of command of those responsible for developing and testing new systems. The services need to cooperate with DOT&E primarily because the Office must approve all test and evaluation master plans and operational test plans. Moreover, DOT&E independently reports on the operational suitability and effectiveness at a system’s full-rate production milestone, a report that is sent separately to Congress.
section_title: Unfavorable Reports on Operational Testing Do Not Always Inhibit Full-Rate Production
paragraphs: DOT&E’s report on a system’s operational suitability and effectiveness is only one of several inputs considered before the full-rate production decision is made. An unfavorable DOT&E report does not necessarily prevent full-rate production. In each of the cases cited below, an affirmative full-rate production decision was made despite a DOT&E report concluding that the system had not demonstrated during operational test and evaluation that it was both operationally suitable and operationally effective:
Full-rate production of the M1A2 tank was approved despite DOT&E’s report that found the system unsuitable.
Full-rate production of Joint STARS was approved, though the system demonstrated only limited effectiveness for “operations other than war” and found “as tested is unsuitable.” Only 18 of the 71 performance criteria were met; 53 others required more testing.
Full-rate production of the C-17 Airlifter was approved despite a number of operational test and evaluation deficiencies, including immature software and failure to meet combination and brigade airdrop objectives.
section_title: Services Contest DOT&E Oversight of Follow-on Operational Test and Evaluation
paragraphs: The services contend that DOT&E does not have authority to insist on, or independently approve the conduct of, follow-on operational test and evaluation. However, in several of the systems we reviewed, DOT&E overcame service opposition and monitored follow-on operational test and evaluation. It used several means to achieve success, such as (1) incorporating follow-on operational test and evaluation in test and evaluation master plans developed and approved prior to the full-rate production decision milestone; (2) persuading the Secretary of Defense to specify follow-on operational test and evaluation, and DOT&E’s oversight role, in the full-rate production acquisition decision memorandum; and (3) citing policy, based on title 10, that entitles DOT&E to oversee operational test and evaluation whenever it occurs in the acquisition process.
Nonetheless, DOT&E action officers stated that the service’s acceptance of DOT&E’s role in follow-on operational test and evaluation varies over time, by service and acquisition system, and is largely dependent upon the convictions of executives in both the services and DOT&E. Among the cases reviewed in this report, the services offered a variety of arguments against DOT&E’s having a role in follow-on operational test and evaluation. They specifically asserted the following:
DOT&E need not be involved because the scope of follow-on operational test and evaluation is frequently less encompassing than initial operational test and evaluation. Follow-on operational test and evaluation has been characterized as testing by the user to determine the strengths and weaknesses of the system and to determine ways to compensate for, or fix, shortcomings observed in initial operational test and evaluation.
Title 10 provides DOT&E with the authority to monitor and review—but not necessarily approve—service follow-on operational test and evaluation plans.
Follow-on operational test and evaluation is unnecessary when a system is found to be operationally effective and suitable during initial operational test and evaluation—even though DOT&E does not concur.
A clear distinction between DOT&E oversight in follow-on operational test and evaluation versus initial operational test and evaluation is that DOT&E is not required to report follow-on operational test and evaluation results to Congress in the detailed manner of the Beyond-LRIP report. Therefore, even if follow-on operational test and evaluation is conducted to assess modifications to correct effectiveness or suitability shortcomings reported to Congress in the Beyond-LRIP report, there is no requirement that Congress receive a detailed accounting of the impact of these modifications.
section_title: DOT&E’s Resources Are Declining
paragraphs: DOT&E’s primary asset to conduct oversight—its cadre of action officers—has decreased in size throughout the decade. This creates a management challenge for the Office because at the same time staff has decreased, the number of programs overseen by DOT&E has increased. As illustrated in table 7, authorized staffing declined from 48 in fiscal year 1990 to 41 in fiscal year 1997, as did funding (in constant dollars) from $12,725,000 in fiscal year 1990 to $11,437,000 in fiscal year 1997. The decline in DOT&E funding is consistent with the general decline in DOD appropriations during this period. However, since fiscal year 1990, while the authorized staffing to oversee operational test and evaluation has declined by 14.6 percent, the number of systems on the oversight list has increased by 17.7 percent.
section_title: DOT&E’s Limited Resources Must Address Competing Priorities
paragraphs: With declining resources, DOT&E must manage competing priorities related to its oversight, advisory, and coordination responsibilities. DOT&E must balance the continuing need to allocate resources to these different priorities while not being perceived as having lost any independence. DOT&E management has flexibility in defining some portion of the scope of its oversight and has continued to electively oversee a substantial number of nonmajor defense acquisition programs and assumed a leading role in advocating an examination of the modernization needs of the test and evaluation infrastructure.
subsections: section_title: DOT&E Continues to Oversee a Substantial Number of Nonmajor Programs
paragraphs: Between fiscal year 1990 and 1996, the number of nonmajor acquisition programs overseen annually by DOT&E ranged between 19 and 43. In fiscal year 1996, when the oversight list reached a peak of 219, 1 of every 8 programs was listed at the discretion of DOT&E. Thus, during this period when the resources to oversee operational testing declined and acquisition reforms have placed additional burdens on oversight staff, the directors of DOT&E continued to place extra responsibility on their staff by augmenting the required oversight of major acquisition programs with a substantial number of optional systems.
Despite a relative decline in resources for oversight, DOT&E management has also elected to assume “a larger role in test resource management planning and leadership in an attempt to achieve much-needed resource modernization.” Although the Director is designated as the principal adviser to the Secretary of Defense and the Under Secretary of Defense for Acquisition and Technology on operational test and evaluation, including operational test facilities and equipment, assuming the larger role defined by DOT&E may be at the expense of its testing oversight mission and perception of independence. The DOT&E Director is now an adviser to the Central Test and Evaluation Investment Program and previously served as Chairman of the Test and Evaluation Committee. The Committee is responsible for the investment program and presides over the planning, programming, and budgeting for development and operational test resources. When the Director served as chairman, we questioned whether these ties created the perception that the Director was not independent from developmental testing. This issue may resurface as DOT&E seeks a larger role in test resource management planning. Also, as the emphasis, cost, and time for operational test and evaluation are increasingly questioned in the drive to streamline acquisition, and as oversight assets are stretched, new DOT&E initiatives may stress the Office’s capacity to manage oversight effectively.
section_title: DOT&E Participation in Working-Level Integrated Product Teams Has the Potential to Complicate Independence
paragraphs: In May 1995, the Secretary of Defense directed DOD to apply the integrated product and process development concept—using integrated product teams—throughout the acquisition process. The revised DOD acquisition regulations (DOD 5000.2-R March 1996) also addressed the use of empowered integrated product teams at the program office level. DOT&E action officers participate as members of the working-level integrated product teams, and the DOT&E Director is a member of the overarching team. One objective of integrated product teams, and DOT&E participation in particular, is to expedite the approval process of test documents by reaching agreement on the strategy and plan through the identification and resolution of issues early, understanding the issues, and documenting a quality test and evaluation master plan that is acceptable to all organizational levels the first time. Integrated product teams are designed to replace a previously sequential test and evaluation master plan development and approval process and therefore enhance timeliness. While this management tool could increase communication between testers and the program managers, it also poses a challenge to DOT&E independence. The challenge was recognized by the Department of Defense Inspector General (DOD IG) when after reviewing the conduct of operational testing it subsequently recommended that “to meet the intent of 10 U.S.C. 139, DOT&E should be a nonvoting member [of the working-level integrated product team] so as to maintain his independence.” {emphasis added} Though integrated product teams were not used throughout the entire time period covered by this report, several action officers noted that this management tool created threats to their effectiveness other than having their positions out-voted. One DOT&E action officer reported having the lone dissenting opinion in a meeting of 30 participants seeking to reach consensus and resolve issues early. The pressure of maintaining independent, contrary positions in large working groups can be a test. Several DOT&E representatives also noted that the frequency of integrated product team meetings to cover the multiple systems for which they were responsible made it impossible for them to attend all, thereby lessening the possibility that testing issues can be identified and resolved as early as possible.
Moreover, program managers and DOT&E pursue different objectives through integrated product teams. The services and program managers view the teams as a way to facilitate their program objectives for cost, schedule, and performance; DOT&E’s objective is oversight of performance through operational testing. The program managers and DOT&E share a desire to identify testing issues as early as possible. However, the priority of the program manager to resolve these issues as early as possible through the teams may conflict with DOT&E’s mission. DOT&E must remain flexible and react to unknowns as they are disclosed during developmental testing, operational assessments, and initial operational test and evaluation. Thus, DOT&E’s participation on the teams is a natural source of tension and a potential impediment to the team’s decision-making. The challenge for DOT&E action officers is to maintain an independent and potentially contrary position in an ongoing working group during the life of a program, which may extend over several years.
section_title: Increased Integration of Developmental and Operational Testing May Attenuate Independent Oversight
paragraphs: The objectives of developmental and operational testing are distinct. Developmental testing determines whether a system meets its functional requirements and contractual technical performance criteria sufficiently to proceed with operational testing. Operational testing determines whether the system meets the operational requirements and will contribute to mission effectiveness in relevant operational environments sufficiently to justify proceeding with production. The integration of these two disparate test activities is proposed to save the time and resources required for testing and evaluation. The sentiment to more closely link developmental and operational testing dates from at least the 1986 Blue Ribbon Commission on Defense Management (Packard Commission), which found that “developmental and operational testing have been too divorced, the latter has been undertaken too late in the cycle, and prototypes have been used and tested far too little.” However, both we and the DOD IG have found that systems were regularly tested before they were ready for testing. In its 1996 report, the DOD IG reported that “4 of 15 systems we examined for operational testing were not ready for testing. This situation occurred because a calendar schedule rather than system readiness often drove the start of testing.” Similarly, we have observed numerous systems that have been pushed into low-rate initial production without sufficient testing to demonstrate that the system will work as promised or intended. Our reviews of major system development in recent years have found that because insufficient time was dedicated to initial testing, systems were produced that later experienced problems during operational testing and systems entered initial production despite experiencing problems during early operational testing.
In 1996 the Secretary of Defense also urged the closer integration of developmental and operational testing, and combined tests where possible, in part to enhance the objectives of acquisition reform. Combined developmental and operational testing is only one of many sources of test data that DOT&E has used to foster more timely and thorough operational test and evaluation. Other sources of information include contractor developmental testing, builder’s trials, component testing, production lot testing, stockpile reliability testing, and operational deployments. While DOT&E has some influence over the quality of operational testing, by independently reviewing the design, execution, analysis, and reporting of such tests, it has no direct involvement or oversight of these other sources of testing information. The use of alternative sources of test data as substitutes for operational test and evaluation will limit DOT&E’s oversight mission, which was created to improve the conduct and quality of testing.
section_title: Conclusions and Recommendations
paragraphs: DOT&E’s challenge is to manage an expansion in independent oversight while satisfying the efficiency goals of acquisition reform and undergoing the economic pressures of downsizing. DOT&E oversight is clearly affecting the operational testing of new defense systems. DOT&E actions (such as the insistence on additional testing, more realistic testing, more rigorous data analysis, and independent assessments) are resulting in more assurance that new systems fielded to our armed forces are safe, suitable, and effective. However, DOT&E is not, by design or practice, the guarantor of effective and suitable acquisitions. DOT&E oversight reduces, but does not eliminate, the risk that new systems will not be operationally effective and suitable. Affirmative full-rate production decisions are made for systems that have yet to demonstrate their operational effectiveness or suitability. Moreover, the services question DOT&E’s authority regarding follow-on test and evaluation of subsequent corrective actions by the program office.
We recommend that the Secretary of Defense revise DOD’s operational test and evaluation policies in the following ways:
Require the Under Secretary of Defense for Acquisition and Technology, in those cases where affirmative full-rate production decisions are made for major systems that have yet to demonstrate their operational effectiveness or suitability, to (1) take corrective actions to eliminate deficiencies in effectiveness or suitability and (2) conduct follow-on test and evaluation of corrective actions until the systems are determined to be operationally effective and suitable by the Director, Operational Test and Evaluation.
Require the Director, Operational Test and Evaluation, to (1) review and approve follow-on test and evaluation master plans and specific operational test plans for major systems before operational testing related to suitability and effectiveness issues left unresolved at the full-rate production decision and (2) upon the completion of follow-on operational test and evaluation, report to Congress, the Secretary of Defense, and the Under Secretary of Defense for Acquisition and Technology whether the testing was adequate and whether the results confirmed the system is operationally suitable and effective.
Further, in light of increasing operational testing oversight commitments and to accommodate oversight of follow-on operational testing and evaluation, we recommend that the Director, Operational Test and Evaluation, prioritize his Office’s workload to ensure sufficient attention is given to major defense acquisition programs.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, DOD concurred with our first and third recommendations and partially concurred with our second recommendation. Concerning the recommendation with which it partially concurred, DOD stated that system specific reports to the Secretary of Defense and Congress are not warranted for every system that requires follow-on operational test and evaluation. DOD pointed out that for specific programs designated for follow-on oversight, test plans are prepared to correct previously identified deficiencies by milestone III, and DOT&E includes the results of follow-on testing in its next annual report.
We continue to believe our recommendation has merit. We recommended that the Secretary require DOT&E approval of follow-on test and evaluation of corrective actions because during our review we found no consensus within the defense acquisition community concerning DOT&E’s role in follow-on operational test and evaluation. In its comments DOD did not indicate whether it intended to give DOT&E a role in follow-on operational test and evaluation that is comparable to its role in initial operational test and evaluation. Moreover, we continue to believe that if a major system goes into full-rate production (even though it was deemed by DOT&E not to be operationally suitable and effective) based on the premise that corrections will be made and some follow-on operational test and evaluation will be performed, DOT&E should report, as promptly as possible, whether or not the follow-on operational test and evaluation results show that the system in question had improved sufficiently to be characterized as both operationally suitable and effective.
DOD’s comments are reprinted in their entirety in appendix IV, along with our specific evaluation.
As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 15 days after its date of issue. We will then send copies to other congressional committees and the Secretary of Defense. We will also make copies available to others upon request.
If you have any questions or would like additional information, please do not hesitate to call me at (202) 512-3092 or the Evaluator-in-Charge, Jeff Harris, at (202) 512-3583.
section_title: Scope and Methodology
paragraphs: To develop information for this report, we selected a case study methodology—evaluating the conduct and practices of DOT&E through an analysis of 13 weapon systems. Recognizing that many test and evaluation issues are unique to individual systems, we determined that a case study methodology would offer the greatest probability of illuminating the variety of factors that impact the value or effectiveness of oversight at the level of the Office of the Secretary of Defense (OSD). Moreover, with nearly 200 systems subject to review of the Director, Operational Test and Evaluation (DOT&E) at any one time, we sought a sample that would enable us to determine if the Office had any impact as well as the ability to examine the variety of programs overseen. Therefore, we selected a judgmental sample of cases reflecting the breadth of program types. As illustrated in table I.1, we selected systems (1) from each of the primary services, (2) categorized as major defense systems, and (3) representing a wide array of acquisition and testing phases—from early operational assessments through and beyond the full-rate production decision. We studied both new and modified systems.
Table I.1: Characteristics of Weapon Systems Used for Case Studies Service(s)
MS III (1995);  IOT&E (1995)
FOT&E (1995-96); Bosnia (1995)
FOT&E (1996-98); MS IIIB (1995)
AFSARC III (1997);  IOT&E (1995-96)
MS III (2003); IOT&E (2002); LRIP (1999)
MS III (1997); LUT (1996); UE (1996) (continued)
Service(s)
FOT&E (1997); MS III (1996); Bosnia (1996)
MS III (1998); IOT&E (1997-98); Bosnia (1995)
MS II (1996); EOA-2 (1996); EOA-1 (1994-95)
FOT&E (1995-96); MS III (1994)
FOT&E-1 (1997-98); MS III (1996); IOT&E-2 (1995-96)
FOT&E (1997); MS III (1996); OPEVAL (1996)
MS III (1997); DT/IOT&E (1994)
MS III (1998); OPEVAL (1998); IOT&E (1997)
MS III (2000); OPEVAL (1999-00); IOT&E (1999)
OPEVAL (1999); OT-IIC (1996) (Table notes on next page)
DOT&E, the service operational test agencies, and the Institute for Defense Analyses (IDA) personnel agreed that DOT&E was influential in the testing done on these 13 systems. In several cases, the participating agencies vehemently differed on the value of DOT&E’s actions; however, whether DOT&E had an impact on testing (be it perceived as positive or negative) was not in dispute.
In conducting our 13 case studies, we assessed the strengths and weaknesses of the organizational framework in DOD for operational testing via test agency representatives, an assessment on the origins and implementation (exemplified by the 13 cases) of the title 10 amendments creating and empowering DOT&E, and a review of the literature.
To compile case study data, we interviewed current action officers in both DOT&E and the appropriate operational test agency and reviewed documentation provided by the operational test agencies, DOT&E, and IDA. Using structured questionnaires, we interviewed 12 DOT&E and 27 operational test agency action officers responsible for the 13 selected systems as well as managers and technical support personnel in each organization. In addition, we interviewed the commanders of each of the service testing agencies and DOT&E. When possible, we corroborated information obtained from interviews with documentation, including test and evaluation master plans, beyond low-rate initial production reports, defense acquisition executive summary status reports, defense acquisition memoranda, and interagency correspondence.
In Washington, D.C., we obtained data from or performed work at the Office of the Director of Operational Test and Evaluation, OSD; Deputy Under Secretary of Defense for Acquisition Reform; Directorate of Navy Test and Evaluation and Technology Requirements, Office of the Chief of Naval Operations; Test and Evaluation Management Agency, Director of Army Staff; Air Force Test and Evaluation Directorate; and the DOD Office of the Inspector General. We also reviewed data and interviewed officials from the Army Operational Test and Evaluation Command and the Institute for Defense Analyses, Alexandria, Virginia; the Navy Commander, Operational Test and Evaluation Force, Norfolk, Virginia; and the Air Force Operational Test and Evaluation Command, Kirtland Air Force Base, New Mexico.
The use of a systematic case study framework enabled us to identify and categorize the types of impacts attributable to DOT&E among the systems studied. In addition, this framework enabled us to identify trends among factors that correlate with DOT&E effectiveness. However, we were unable to generalize to all systems subject to OSD-level oversight. In light of this limitation, we included only major (high-cost) systems and systems identified by DOT&E and the lead operational test agency as having been affected by DOT&E initiatives. Moreover, while our methodology and data collection enabled us to qualitatively assess the impact of DOT&E, it was not sufficiently rigorous either to evaluate the cost-effectiveness of DOT&E actions or to determine the deterrent effects, if any, the Office exerts over the acquisition and testing process. Finally, our methodology did not enable an assessment of whether the additional testing requested by DOT&E was necessary to provide full-rate production decisionmakers the essential information on a system’s operational effectiveness and suitability or whether the additional data was worth the time, expense, and resources necessary to obtain it.
Our review was performed from June 1996 through March 1997 in accordance with generally accepted government auditing standards.
section_title: Description of 13 Case Study Systems
subsections: section_title: AH-64D Longbow Apache Helicopter
paragraphs: The AH-64D Longbow Apache is a remanufactured and upgraded version of the AH-64A Apache helicopter. This Army system is equipped with a mast-mounted fire control radar, fire-and-forget radio frequency Hellfire missile, and airframe improvements (i.e., integrated cockpit, improved engines, and global positioning system navigation).
section_title: Airborne Self-Protection Jammer
paragraphs: The Airborne Self-Protection Jammer is a defensive electronic countermeasures system using reprogrammable deceptive jamming techniques to protect tactical aircraft from radar-guided weapons. This Navy system is intended to protect Navy and Marine Corps F-18 and F-14 aircraft.
section_title: C-17A Airlifter
paragraphs: The C-17A Airlifter provides strategic/tactical transport of all cargo, including outsized cargo, mostly to main operational bases or to small, austere airfields, if needed. Its four-engine turbofan design enables the transport of large payloads over intercontinental ranges without refueling. This Air Force aircraft will replace the retiring C-141 aircraft and augment the C-130 and C-5 transport fleets.
section_title: E-3 AWACS Radar System Improvement Program
paragraphs: The Air Force’s E-3 AWACS consists of a Boeing 707 airframe modified to carry a radome housing a pulse-Doppler radar capable of detecting aircraft and cruise missiles, particularly at low altitudes. The Radar System Improvement Program replaces several components of the radar to improve detection capability and electronic countermeasures as well as reliability, availability, and maintainability.
section_title: F-22 Air Superiority Fighter
paragraphs: The F-22 is an air superiority aircraft with a capability to deliver air-to-ground weapons. The most significant features include supercruise, the ability to fly efficiently at supersonic speeds without using fuel-consuming afterburners, low observability to adversary systems with the goal to locate and shoot down the F-22, and integrated avionics to significantly improve the pilot’s battlefield awareness.
section_title: Javelin Missile
paragraphs: The Javelin is a man-portable, antiarmor weapon developed for the Army and the Marine Corp to replace the aging Dragon system. It is designed as a fire-and-forget system comprised of a missile and reusable command launch unit.
section_title: Joint Surveillance Target Attack Radar System
paragraphs: The Joint Surveillance Target Attack Radar System is designed to provide intelligence on moving and stationary targets to Air Force and Army command nodes in near real time. The system comprises a modified Boeing 707 aircraft frame equipped with radar, communications equipment, and the air component of the data link, computer workstations, and self-defense suite as well as ground station modules mounted on Army vehicles.
section_title: LPD-17 Amphibious Assault Ship
paragraphs: The LPD-17 will be an amphibious assault ship capable of launching (1) amphibious assault craft from a well deck and (2) helicopters or vertical takeoff and landing aircraft from an aft flight deck. It is intended to transport and deploy combat and support elements of Marine expeditionary brigades as a key component of amphibious task forces.
section_title: M1A2 Abrams Main Battle Tank
paragraphs: The M1A2 Abrams main battle tank is an upgrade of the M1A1 and is intended to improve target acquisition and engagement rates and survivability while sustaining equivalent operational suitability. Specifically, the modified tank incorporates a commander’s independent thermal viewer, a position navigation system, and an intervehicle command and control system.
section_title: Sensor Fuzed Weapon
paragraphs: The Sensor Fuzed Weapon is an antiarmor cluster munition to be employed by fighter, attack, or bomber aircraft to achieve multiple kills per pass against armored and support combat formations. Each munition contains a tactical munitions dispenser comprising 10 submunitions containing a total of 40 infrared sensing projectiles. High-altitude accuracy is to be improved through the incorporation of a wind-compensated munition dispenser upgrade.
section_title: Standard Missile-2
paragraphs: The Standard Missile-2 is a solid propellant-fueled, tail-controlled, surface-to-air missile fired by surface ships. It was originally designed to counter high-speed, high-altitude antiship missiles in an advanced electronic countermeasures environment. The block IIIA version provides improved capacity against low-altitude targets with an improved warhead. The block IIIB adds an infrared seeker to the block IIIA to enhance the missile’s capabilities against specific threats. These improvements are being made to provide capability against theater ballistic missiles while retaining its capabilities against antiair warfare threats.
section_title: Tomahawk Weapon System
paragraphs: The Tomahawk Weapon System is a long-range subsonic cruise missile for land and sea targets. The baseline IV upgrade is fitted with a terminal seeker, video data link, and two-way digital data link. The primary  baseline IV configuration is the Tomahawk multimission missile; a second variant is the Tomahawk hard target penetrator.
section_title: V-22 Osprey
paragraphs: The V-22 is a tilt rotor vertical/short takeoff and landing, multimission aircraft developed to fulfill operational combat requirements in the Marine Corps and Special Operations Forces.
section_title: DOT&E’s and DOD’s System Acquisition Process
paragraphs: DOT&E’s role in the system acquisition process does not become prominent until the latter stages. As weapon system programs progress through successive phases of the acquisition process, they are subject to major decision points called milestones. The milestone review process is predicated on the principle that systems advance to higher acquisition phases by demonstrating that they meet prescribed technical and performance thresholds. Figure III.1 illustrates DOD’s weapon system acquisition process.
Figure III.1: DOD’s Weapon System Acquisition Process (EMD)
Per DOD directive, test and evaluation planning begins in phase 0, Concept Exploration. Operational testers are to be involved early to ensure that the test program for the most promising alternative can support the acquisition strategy and to ensure the harmonization of objectives, thresholds, and measures of effectiveness in the operational readiness document and the test and evaluation master plan. Early testing of prototypes in phase I, Program Definition and Risk Reduction, and early operational assessments are to be emphasized to assist in identifying risks. A combined developmental and operational test approach is encouraged to save time and costs. Initial operational test and evaluation is to occur during phase II to evaluate operational effectiveness and suitability before the full-rate production decision, milestone III, on all acquisition category I and II programs. For all acquisition category I programs and other programs designated for OSD test and evaluation oversight, a test and evaluation master plan is prepared and submitted for approval prior to first milestone review (excluding milestone 0). The master plan is to be updated at milestones when the program has changed significantly. DOT&E must approve the test and evaluation master plan and the more specific operational test plans prior to their execution. This process and the required plan approvals provide DOT&E opportunities to affect the design and execution of operational testing throughout the acquisition process.
section_title: Comments From the Department of Defense
paragraphs: The following are GAO’s comments on the September 19, 1997, letter from the Department of Defense.
subsections: section_title: GAO Comments
paragraphs: 1. In prior reviews of individual weapon systems, we have found that operational testing and evaluation is generally viewed by the acquisition community as a costly and time-consuming requirement imposed by outsiders rather than a management tool for more successful programs. Efforts to enhance the efficiency of acquisition, in general—and in operational testing, in particular—need to be well balanced with the requirement to realistically and thoroughly test operational suitability and effectiveness prior to the full-rate production decision. We attempted to take a broader view of acquisition reform efficiency initiatives to anticipate how these departures from past ways of doing business could impact both the quality of operational testing and the independence of DOT&E. 2. We were asked to assess the impact of DOT&E on the quality and impact of testing and reported on the Secretary of Defense initiatives only to the extent they may pose a potential impact on DOT&E’s independence or effectiveness. Moreover, we did not recommend or suggest that testers wait until milestone III to discover problems that could have been learned and corrected earlier. Since its inception, DOT&E has been active in test integration and planning working groups and test and evaluation master plan development during the earliest phases of the acquisition process. In fact, we have long advocated more early testing to demonstrate positive system performance prior to the low-rate initial production decision. DOT&E’s early involvement in test planning is appropriate, necessary, and required by DOD regulations. In this report we do not advocate the elimination of DOT&E participation during the early stages of the acquisition process; rather, we merely observe that DOT&E participation through the vehicle of working-level program manager integrated product teams has the potential to complicate independence and may be increasingly difficult to implement with declining resources and increasing oversight responsibilities following milestone III. 3. We did not recommend or suggest that DOT&E ignore its statutory responsibility to review and make recommendations to the Secretary of Defense on budgetary and financial matters related to operational test facilities and equipment. We only observed that in an era of declining resources, earlier participation, and extended oversight responsibilities, a decision to assume a larger role in test resource management planning and leadership is likely to result in tradeoffs in other responsibilities—the largest being oversight. 4. We made this recommendation because DOT&E, the services, and the program offices did not necessarily agree on the degree to which system performance requirements have been met in initial operational test and evaluation. Furthermore, there was no consensus within the acquisition community concerning DOT&E’s authority to oversee follow-on operational test and evaluation conducted to ensure that proposed corrections to previously identified deficiencies were thoroughly tested and evaluated. 5. Under 10 U.S.C. 2399, DOT&E is required to independently report to Congress whether a major acquisition system has proven to be operationally suitable and effective prior to the full-rate production decision. When follow-on operational test and evaluation is necessary to test measures intended to correct deficiencies identified in initial operational test and evaluation, Congress does not receive an equivalent independent report from DOT&E that concludes, based on required follow-on operational test and evaluation, whether or not a major system has improved sufficiently to be considered both operationally suitable and effective.
section_title: Related GAO Products
paragraphs: Tactical Intelligence: Joint STARS Full-Rate Production Decision Was Premature and Risky (GAO/NSIAD-97-68, Apr. 25, 1997).
Weapons Acquisition: Better Use of Limited DOD Acquisition Funding Would Reduce Costs (GAO/NSIAD-97-23, Feb. 13, 1997).
Airborne Self-Protection Jammer (GAO/NSIAD-97-46R, Jan. 29, 1997).
Army Acquisition: Javelin Is Not Ready for Multiyear Procurement (GAO/NSIAD-96-199, Sept. 26, 1996).
Tactical Intelligence: Accelerated Joint STARS Ground Station Acquisition Strategy Is Risky (GAO/NSIAD-96-71, May 23, 1996).
Electronic Warfare (GAO/NSIAD-96-109R, Mar. 1, 1996).
Longbow Apache Helicopter: System Procurement Issues Need to Be Resolved (GAO/NSIAD-95-159, Aug. 24, 1995).
Electronic Warfare: Most Air Force ALQ-135 Jammers Procured Without Operational Testing (GAO/NSIAD-95-47, Nov. 22, 1994).
Weapons Acquisition: Low-Rate Initial Production Used to Buy Weapon Systems Prematurely (GAO/NSIAD-95-18, Nov. 21, 1994).
Acquisition Reform: Role of Test and Evaluation in System Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar. 22, 1994).
Test and Evaluation: The Director, Operational Test and Evaluation’s Role in Test Resources (GAO/NSIAD-90-128, Aug. 27, 1990).
Adequacy of Department of Defense Operational Test and Evaluation (GAO/T-NSIAD-89-39, June 16, 1989).
Weapons Testing: Quality of DOD Operational Testing and Reporting (GAO/PEMD-88-32BR, July 26, 1988).
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.