Defense Acquisitions: Actions to Improve Navy SPAWAR Low-Rate Initial Production Decisions
Published: 2001-08-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: During its review of the Navy's Space and Naval Warfare (SPAWAR) Systems Command's fiscal year 2001 budget request, GAO found that many information technology systems were being procured and fielded in relatively large quantities--sometimes exceeding 50 percent of the total--during low-rate initial production and before completion of operational testing. The primary purpose of low-rate initial production is to produce enough units for operational testing and evaluation and to establish production capabilities to prepare for full-rate production. Commercial and Department of Defense (DOD) best practices have shown that completing a system's testing before producing significant quantities substantially lowers the risk of costly fixes and retrofits. For major weapons systems, statutory provisions limit the quantities of systems produced during low-rate initial production to the minimum quantity necessary. These statutory provisions also require justification for quantities exceeding 10 percent of total production. Although these provisions do not apply to non-major systems, DOD and Navy acquisition regulations encourage these programs to make use of the low-rate initial production concept. This report reviews (1) information systems being procured and fielded for SPAWAR in large numbers before operational testing, (2) what effects this practice was having on SPAWAR and the fleet, and (3) what the Navy is doing to mitigate the risks associated with this practice.
section_title: What GAO Found
paragraphs: GAO found that the main reason for the high percentage of low-rate initial production quantities is to more quickly respond to fleet demands for information systems improvements. Many information technology systems purchased and fielded during low-rate initial production and prior to completing operational testing experienced problems that negatively impacted fleet operations and capabilities. SPAWAR has taken several steps to mitigate the risks of high percentage low-rate initial production procurements, such as requiring program managers to use a standardized checklist and establishing an Acquisition Reform Office to serve as a focal point and command-wide disseminator of lessons learned and process improvements.

section_title: Letter
paragraphs: During our review of the Navy’s Space and Naval Warfare (SPAWAR) Systems Command’s fiscal year 2001 budget request, we found that many information technology systems were being procured and fielded in relatively large quantities—sometimes exceeding 50 percent of the total— during low-rate initial production and before completing operational testing. The primary purpose of low-rate initial production is to produce enough units for operational testing and evaluation and to establish production capabilities to prepare for full-rate production. Limiting initial production affords the opportunity to confirm the stability and soundness of a new system before producing larger quantities. Commercial and Department of Defense (DOD) best practices have shown that completing a system’s testing prior to producing significant quantities will substantially lower the risk of costly fixes and retrofits. For major weapons systems, there are statutory provisions that limit the quantities of systems produced during low-rate initial production to the minimum quantity necessary and require justification for quantities exceeding 10 percent of total production. Although these provisions do not apply to non-major systems, DOD and Navy acquisition regulations encourage these programs to make use of the low-rate initial production concept where appropriate.
Given the risks associated with acquiring high percentages of production quantities before completing operational testing, we reviewed the acquisitions of information systems at SPAWAR to determine (1) why information systems were being procured and fielded in large numbers before operational testing, (2) what effects this practice was having on SPAWAR and the fleet, and (3) what the Navy is doing to mitigate the risks associated with this practice.
subsections: section_title: Background
paragraphs: SPAWAR is one of the Navy’s three major acquisition commands. SPAWAR provides information technology systems to naval forces on land, at sea, and in space and integrates all information products, including those developed by other systems commands and agencies outside the Navy. The SPAWAR workforce is over 7,300 personnel, and the fiscal year 2000 budget was $3.7 billion—about $2.7 billion to develop and procure systems and about $1.0 billion to operate and maintain them. Specifically, SPAWAR develops, acquires, and manages  battle management systems (for example, software applications and  undersea, terrestrial, and space sensors (for example, underwater sensors, navigation and weather systems, and satellites); information transfer systems (for example, communications systems, radios, antennas, and switches); and information management systems (for example, local area networks and routers).
As of October 2000, SPAWAR was managing 21 programs that involved low-rate initial production. These programs had cumulative low-rate initial procurements ranging from 5 to 100 percent of the total inventory objective—7 systems were above 50 percent. The estimated total production costs for the eight systems we analyzed in detail ranged from $31 million to $525 million.
As weapon system programs move through the phases of the acquisition process, they are subject to review at major decision points called milestones. Major defense acquisition programs, known as acquisition category I, and major systems programs, known as acquisition category II,as well as non-major systems programs, known as acquisition category III and IV, follow the same general process. DOD and Navy acquisition policies state that program risks shall be assessed at each milestone decision point before approval is granted for the next phase. The policies add that test and evaluation shall be used to determine system maturity and identify areas of technical risk.
Major milestones in DOD’s systems acquisition process include  Milestone 0, when the determination is made about whether an identified mission need warrants a study of alternative concepts to satisfy the need. If warranted, the program is approved to begin the concept exploration and definition phase.
Milestone I, when the determination is made about whether a new acquisition program is warranted. If so, initial cost, schedule, and performance goals are established for the program, and authorization is given to start the demonstration and validation phase.
Milestone II, when the determination is made about whether continuation of development, testing, and preparation for production is warranted. If so, authorization is given to start the engineering and manufacturing development phase. Approval of this phase will often involve a commitment to low-rate initial production, which is defined as the minimum quantity needed to (1) provide production- representative articles for operational testing and evaluation, (2) establish an initial production base, and (3) permit orderly ramp-up to full-rate production upon completion of operational testing and evaluation. Operational test and evaluation is a key internal control to ensure that decisionmakers have objective information available on a weapon system’s performance to minimize risks of procuring costly and ineffective systems. Operational testing and evaluation uses field tests under realistic conditions to determine the operational effectiveness and suitability of a system for use in combat. DOD acquisition regulations generally provide that programs successfully complete these tests before starting full-rate production.
Milestone III, when operational test and evaluation has been completed, a determination is made about whether to proceed to full- rate production and field the system.
Over the years, we have found instances in which DOD used the low-rate initial production decision phase to purchase significant numbers of major and non-major systems without successfully completing operational testing and evaluation. Often, these systems later experienced significant effectiveness and/or suitability problems. In 1994, we reported that DOD had made large buys of weapon systems during the low-rate initial production phase and prior to completion of operational test and evaluation, which resulted in operational problems. For example, we reported that the Navy procured 100 percent of a system’s inventory objective during low-rate initial production and later found that the system lacked critical hardware and software capabilities. In another case, the Navy procured 100 percent of a system’s inventory objective during low- rate initial production and later terminated the program when it failed operational testing and evaluation. A recent Defense Science Board report found that weapons systems are still being fielded without adequate testing to assure their effectiveness and utility to operating units. We conducted this review because buying systems before completing operational testing has inherent risks, and SPAWAR’s practice of buying high percentages of a system’s total inventory objective during low-rate initial production raised these risk concerns.
section_title: Fleet Demands and Rapidly Advancing Technology Cited for the High Percentage of Low-Rate Initial Production Purchases
paragraphs: SPAWAR officials cited three primary reasons for high-percentage buys during low-rate initial production. First, to meet Navy initiatives, SPAWAR must provide the fleet with large quantities of information technology systems as quickly as possible. Second, many information systems consist of evolving technology that quickly becomes obsolete. Third, additional low-rate initial production buys are approved due to delays in conducting the operational test and evaluation.
The main reason that SPAWAR officials cited for high-percentage buys during low-rate initial production is the need to provide as many information technology systems to the fleet as quickly as possible to meet several Navy initiatives. The Navy’s current vision for the 21st century, Forward From The Sea, involves innovations in technology to rapidly transform the Navy into a 21st century force. SPAWAR provides or contributes to many of the operational capabilities that support the vision. Officials in the Chief of Naval Operations’ Fleet and Allied Requirements Division stated that the fleets put pressure on SPAWAR to provide information systems faster. These officials, as well as SPAWAR officials, contend that if SPAWAR does not provide systems to the fleet quickly, then the fleet will bypass the Chief of Naval Operations and SPAWAR and procure some systems with fleet funding. If the fleet buys the systems, SPAWAR cannot control the configuration of these systems, which can eventually result in interoperability problems with systems that SPAWAR procures. An official in the Office of the Deputy Assistant Secretary of the Navy responsible for communications and space systems also agreed that there is pressure on SPAWAR to meet fleet demands. He said that the Under Secretary of Defense for Acquisition, Technology, and Logistics stated about 3 years ago that the pace of developing systems was too slow and called for shortening the development cycle by incorporating evolutionary development and acquisition. Through evolutionary development and acquisition, systems are continuously improved, as new technology becomes available.
According to the SPAWAR commander, another reason for quickly providing systems to the fleet is that information systems consist of rapidly advancing technology, which can become obsolete within 18 months. He said that procuring and fielding a large percentage of a system’s inventory objective while still in low-rate initial production quickly provide the fleet with better information systems and provide important operational data so that any system performance problems can be quickly fixed. Officials in the Navy Fleet and Allied Requirements Division said that the 18-month obsolescence cycle is the main reason that the fleet would rather have a system now with 75 to 80 percent of its full capability as opposed to waiting until the system has all of its capability. However, officials in the Chief of Naval Operations’ Office of Test, Evaluation, and Technology Requirements disputed that rapidly advancing technology is a legitimate reason for making high-percentage buys during low-rate initial production and before completing operational testing. These officials concluded that making high-percentage buys during low- rate initial production circumvents the operational testing and evaluation process and increases the risk that systems will not work as intended when fielded.
High-percentage buys during low-rate initial production also were the result of delays in conducting the operational test and evaluation. According to the SPAWAR commander, the pass or fail nature of operational testing contributed to delays. Rather than fail a test and risk program reduction or termination, operational tests were delayed until there was a good probability that the system would pass the tests. As tests were delayed, additional low-rate initial production buys were approved. Six of the eight systems we analyzed had additions to the original low-rate initial production quantities.
The SPAWAR commander said that making high-percentage buys of a system while still in low-rate initial production is low risk when proven commercial technology items are being procured and they are relatively low-cost items—when compared to the cost of ships and aircraft. Further, if problems arise after the low-rate initial production systems are fielded, the cost to fix them is not significant. However, SPAWAR officials agreed that none of the eight systems we analyzed are entirely commercial and that all of them have military requirements that must be tested in a realistic operational environment.
The DOD Director of Operational Test and Evaluation agreed that low-rate initial production items can be used in an operational environment to learn about problems and fix them, but he said that procuring large quantities of a system before operational test and evaluation is a risky strategy.
section_title: Problems Experienced When Procuring High Percentages of Units In Low-Rate Initial Production
paragraphs: We analyzed eight SPAWAR systems and found seven of them had a combination of problems that adversely affected fleet operations—all had performance problems, all had interoperability problems, and six had suitability problems. A performance problem is the inability of a system to effectively and efficiently perform its assigned mission. An interoperability problem is the inability of systems to work together effectively to provide services to and accept services from other systems. A suitability problem involves a system not satisfactorily meeting one or more requirements, including reliability, maintainability, logistics support, or training. These problems may delay progress in achieving the Navy’s vision for using information technology to attain and maintain network-centric warfighting knowledge and decision-making superiority. The last of the eight systems had not been installed when we analyzed the eight systems. Table 1 illustrates the types of problems identified for seven of the eight systems.
Two of the systems—the Digital Wideband Transmission System and the Command and Control Processor—illustrate how interoperability, performance, and suitability problems impact fleet operations. The Digital Wideband Transmission System is a radio transmission system supporting voice, video, and data communications. It is required on all aircraft carriers and amphibious ships and at training facilities. SPAWAR approved 100 percent of the total inventory objective to be bought under low-rate initial production; however, the system does not work due to a number of problems with the antenna, power amplifier, and radio frequency control. The system also created, and was affected by, electromagnetic interference, which caused severe interoperability problems. For example, the interference caused a complete loss of Global Positioning System navigation capability. Furthermore, in order for the digital system to work, an air-search radar had to be shut down. Consequently, the Navy turned off the system the first time it was used during an amphibious ready group deployment in the Pacific, replaced it with a legacy system, and placed the digital system in an inoperative status for 9 months. By November 2000, SPAWAR had installed 78 percent of the systems in the fleet, but it will cost $4.3 million to fix the problems for this $40-million program— $1.2 million for engineering work and $3.1 million for retrofit costs. SPAWAR is currently developing and testing improvements to system performance.
The Command and Control Processor acquires information from other communications systems, stores the information, and reformats it for use by the Aegis combat system on aircraft carriers and naval combat surface ships. Although SPAWAR originally approved a small purchase of the processor during the low-rate initial production, three subsequent low-rate initial production increases bought the total to 41 percent of the inventory objective. From 1995 to 2000, there were 263 problems noted with the system, mostly involving software, during battle group system integration tests. The processor has severe suitability problems because it breaks down unpredictably up to 12 hours at a time (due to software problems) and freezes up, which eliminates the system’s capability to provide current situational awareness. Also, on an aircraft carrier that we visited, operators said that the processor would not integrate with other systems, even though it is designed to do so. To prevent the breakdowns, SPAWAR developed a workaround procedure, which involves resetting the system every 2 hours instead of every 24 hours. In addition, the breakdowns and workarounds put more pressure on operators and maintainers during combat or hostile situations.
Some of SPAWAR’s 13 other systems with a high percentage of low-rate initial production buys also had operational effectiveness and suitability problems. The Pacific Fleet experienced 46 problems with the Navy Extremely High Frequency Satellite Communications System from 1995 to 2000. The problems involved hardware and software, interoperability, and training. For example, in February 2000, system performance was degraded due to a part failure. In April 2000, SPAWAR reported that the problem had been solved. However, about a month later, the system had problems again, resulting in no response from the satellite and time- tracking errors being returned. The system was again placed in a degraded status and, as of August 2000, the problem was still unresolved. The Pacific Fleet also experienced problems with the High Frequency Radio Group, mainly due to system performance problems and training shortfalls. For example, on a ship visit in October 2000, ship communications personnel said that the system had broken down several times for a duration of 1 week to 1 month at a time. They said that, when it breaks down, the operators must tune in the radio frequency manually, but ship operators have not been trained to do this because they were used to relying on the system to tune into a particular frequency automatically. The Pacific Fleet also identified several other problems with the radio group, including a ship that experienced 15 failures of a system switch within 11 months.
The SPAWAR commander said that the Command did not have adequate controls and oversight, at the time of most of these low-rate initial production decisions, to either mitigate or manage risks associated with procuring and fielding large percentages of systems during low-rate initial production. He said the need for more discipline in the acquisition process contributed to the interoperability, performance, and suitability deficiencies we identified. He further noted, however, that some problems are part of the cost of doing business with new systems and are worth the risk to provide systems to the fleet quickly. According to the SPAWAR commander, the most meaningful measure of success is whether the systems are meeting their operational requirements, and he said that SPAWAR’s systems are meeting theirs based on a performance parameter called operational availability. However, according to the DOD Director of Operational Test and Evaluation and the Chief of Naval Operations’ Office of Test, Evaluation, and Technology Requirements, operational availability is only one of a number of key performance measures, and an overall assessment of system performance should not be based solely on that parameter.
section_title: Navy and SPAWAR Take Steps to Reduce Low-Rate Initial Production Risks
paragraphs: The Navy and SPAWAR have taken or plan to take a number steps to mitigate the risks of large low-rate initial production procurements. To add more discipline and rigor to the low-rate initial production decision process, the Command now requires program managers to use a standardized checklist and report template as part of reviewing and approving low-rate initial production purchase requests. SPAWAR has also established an Acquisition Reform Office to serve as a focal point and command-wide disseminator of lessons learned and process improvements. The reform office is currently developing a “Rules of the Road” Acquisition Guidebook for SPAWAR program managers. Further, in discussions of our findings and observations during this review, the commander called for the development of risk management guidance for information systems and agreed to suggested improvements in documenting and justifying low-rate initial production decisions.
The SPAWAR commander said better risk management guidance would improve low-rate initial production decisions on information systems, especially when milestone decision authorities and program managers rotate in and out over time. The commander stated that he and the program managers primarily use their acquisition knowledge, wisdom, and experience when making risk management decisions.
In discussions with Navy and SPAWAR officials, we noted that the Acquisition Decision Memorandums, used to document and support milestone decisions, did not always include the low-rate initial production quantity being approved, the cumulative number of low-rate initial production items that had been approved, or the cumulative low-rate initial production percentage. We also noted that, at SPAWAR, the justification for approving low-rate initial production purchases was not always documented. The Navy and SPAWAR officials agreed to document in the Acquisition Decision Memorandum the justification for all low-rate initial production approvals, as well as the current inventory objective and the cumulative number of units bought under low-rate initial production. SPAWAR also agreed to include in its quarterly program status report when cumulative low-rate initial production approvals reach 50 percent or more of the current inventory objective. Recognizing that other Navy activities can benefit from the low-rate initial production decision checklist and report template, we recommended that this guidance be distributed throughout the Navy. The Navy subsequently distributed the guidance DOD-wide. Finally, the Navy and SPAWAR agreed to supplement acquisition training for program managers and staff by incorporating risk management tools into existing courses.
section_title: Conclusions
paragraphs: In seeking to provide new information systems to the fleet as quickly as possible, SPAWAR officials procured and fielded relatively large quantities of systems during low-rate initial production and before completing operational testing. Our subsequent review of seven of these systems found that six had experienced operational problems that negatively impacted the fleet. The SPAWAR commander noted that controls and oversight, at the time of most of these decisions, were not adequate to either mitigate or manage risks associated with procuring and fielding large percentages of systems during low-rate initial production. He said the need for more discipline in the acquisition process contributed to deficiencies we identified. Since that time, Navy and SPAWAR officials have taken or have plans to take a number steps to mitigate the risks of large low-rate initial production procurements. In addition, they have agreed to implement, and in one case have already implemented, process improvements we suggested during the course of this review. Given these actions, we are not making any recommendations in this report.
section_title: Agency Comments
paragraphs: In commenting on our draft report, the Navy agreed and stated that the actions taken and planned by it and the SPAWAR Command are expected to improve the Navy’s low-rate initial production decision process. The Navy’s comments appear in appendix I.
section_title: Scope and Methodology
paragraphs: To acquire information about the number and status of SPAWAR low-rate initial production programs, we interviewed officials and obtained documentation from the SPAWAR Acquisition Reform Office; selected SPAWAR program offices; the Office of the Assistant Secretary of the Navy (Research, Development, and Acquisition); the Office of the Deputy Assistant Secretary of the Navy (Command, Control, Communications, Computers, and Information/Electronic Warfare/Space); the Office of the Deputy Director Defense Procurement Strategies; the Office of the Under Secretary of Defense (Acquisition and Technology); and the Office of the Chief of Naval Operations.
To obtain detailed information about the impact of the high percentage of SPAWAR low-rate initial production procurements, we selected and reviewed 14 programs, representing 70 percent of all SPAWAR low-rate initial production programs. Of these 14 programs, we examined 8 programs in detail looking at operational, logistic, interoperability, and training issues to determine how well these programs were performing.
To obtain information about the operational testing, evaluation, interoperability, and fielding of low-rate initial production systems, we interviewed officials and obtained documentation from the Office of the Director of Navy Test, Evaluation, and Technology Requirements; the Office of the Navy Commander Operational Testing and Evaluation Force; the Office of the Program Manager for Battle Group Systems Integration Testing; and the Office of the Director of Operational Test and Evaluation, Office of the Assistant Secretary of Defense.
To obtain information about the operation, performance, interoperability, maintenance, repair, retrofit, suitability, and training regarding low-rate initial production systems in the fleet, we visited the Naval Surface Force Command, Pacific; the Naval Air Command, Pacific; and the Naval Submarine Command, Pacific, Squadron Eleven. We also visited specific ships in each command, including the U.S.S. Benfold (DDG-65), the U.S.S. Pearl Harbor (LSD-52), the U.S.S. John C. Stennis (CVN-74), and the U.S.S. Salt Lake City (SSN-716). In addition, we held discussions with SPAWAR program officials and officials from SPAWAR’s In-Service Engineering Activity, Fleet Support Engineering Team, and Installations and Logistics Directorate.
To obtain information about the laws, regulations, procedures, and guidance governing the procurement of information technology systems in low-rate initial production, we interviewed officials and obtained documentation from the Office of the Commander, Space and Naval Warfare Systems Command; the Office of the Chairman Deskbook Working Group (DOD 5000 Rewrite); the Assistant Secretary of the Navy (Research, Development and Acquisition); and the Office of the Assistant Deputy Under Secretary of Defense (Systems Acquisition). We also reviewed selected laws and regulations governing low-rate initial production, including title 10 of the U.S. Code, the DOD 5000 series acquisition regulations (the1996 and revised 2000 version), and Secretary of the Navy Instruction 5000-2B governing acquisition and procurement of low-rate initial production and commercial-off-the-shelf technology.
We conducted our review from June 2000 through May 2001 in accordance with generally accepted government auditing standards.
We are sending copies of this report to interested congressional committees; the Honorable Donald H. Rumsfeld, Secretary of Defense; and the Honorable Mitchell E. Daniels Jr., Director of the Office of Management and Budget. Copies will be made available to others upon request.
Please contact me at (202) 512-4821 if you have any questions regarding this report. Key contributors to this report were Cristina Chaplain, Joe Dewechter, Dorian Dunbar, Stephanie May, Gary Middleton, Sarah Prehoda, Richard Price, and William Woods.
section_title: Appendix I: Comments From the Department of the Navy