Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development
Released: 2004-07-19T00:00:00

Letter

Scope and Methodology

To assess DOD’s progress in attaining the knowledge it needs before the start of product development, we examined the resources (technology, communications infrastructure, and funding) committed and planned for the program as well as the users’ needs for an SBR system. We considered DOD’s plans for maturing the critical technologies when we obtained technology-readiness information for each critical technology (as well as its mature backup technology) against best practice standards to determine if they will be sufficiently mature when DOD plans to start product development. We also reviewed the SBR risk management plans and concept development contract information. We discussed these documents and issues with representatives from Air Force Space Command, Peterson Air Force Base, Colorado; and the SBR Joint Program Office, Space and Missile Systems Center, Los Angeles Air Force Base, California.

To determine SBR’s role in a larger DOD architecture, we met with officials from the Joint Chiefs of Staff, Washington, D.C.; and the Air Force Directorate of Space Acquisitions, Arlington, Virginia. We also consulted past GAO reports to determine the relationship between SBR and the Transformational Communications Architecture.

To determine the scope and completeness of the analysis of alternatives and its follow-on study to identify the optimal ways to gather information on ground moving targets from radars based in space versus air, we met with officials from Air Force Space Command; DOD Office of the Director, Program Analysis and Evaluation, Washington, D.C.; Air Force Directorate of Requirements for Space, Crystal City, Virginia; and the Air Force Studies and Analyses Agency, Arlington, Virginia. We also talked with an official from the Air Force Office of Aerospace Studies, Kirtland Air Force Base, New Mexico.

We discussed overarching programmatic issues—including the level of coordination between DOD and the intelligence community—with representatives from the Air Force Directorate of Space Acquisitions. We were not able to obtain meetings with members of the Mission Requirements Board (a board within the intelligence community responsible for approving program requirements) or the intelligence agencies to discuss their stake in the SBR program.

We performed our work from November 2003 through June 2004 in accordance with generally accepted government auditing standards.

Background

SBR represents the first time that DOD has taken the lead on developing a major national security space capability with the intelligence community as a partner. Because of this partnership, SBR’s acquisition process is more complex than that used for typical DOD programs. While DOD and the intelligence community will likely use all the data that SBR produces, their priorities differ. DOD’s warfighting community is particularly interested in tracking targets moving over land or sea as well as other objects of interest. The intelligence community is more focused on obtaining detailed global imagery and combining it with other data for advanced processing. SBR is expected to meet both needs and be fully integrated with other space and non-space systems, including TCA, which is to transmit SBR’s data to receivers in the air, at sea, or on the ground.

A key advantage of radar in space is having the ability to “see” through clouds and sand storms and any type of weather, day or night. Radar-equipped aircrafts, on the other hand, require U.S. air dominance to collect radar information and must steer clear of hostile areas—the result being limited radar coverage. The SBR concept offers other added features, including electronic steering of the radar signal toward a particular area and capturing high volumes of very fine resolution radar images of targets and terrain. With the ability to perform these functions almost simultaneously, SBR is expected to help analysts gain a better understanding of what is occurring in specific locations.

To help meet some of its goals, DOD plans to leverage key technologies that were developed in the late 1990s to demonstrate a space-based radar capability. According to DOD officials, contractors developed some satellite hardware and prototype components under the Discoverer II program, which began in 1998 and was to identify and validate by 2008 the capability of tracking mobile ground targets from space. Discoverer II, comprising two radar demonstration satellites, was a joint initiative by the Air Force, DOD’s Defense Advanced Research Projects Agency, and the intelligence community’s National Reconnaissance Office. DOD officials told us that the Discoverer II program had reached the preliminary design review phase when it was cancelled in 2000 because of cost and schedule uncertainties, poorly explained requirements, and the lack of a coherent vision to transition the system to operational use.

The Secretary of Defense concluded that space-based radar could provide a military advantage and in 2001 approved SBR as a new major defense acquisition program, delegating it to the Air Force. In July 2003, an independent cost assessment team consisting of representatives from DOD and the intelligence community estimated that $28.6 billion would be needed to pay for SBR’s life-cycle costs—development, production, launch, and operation. The program entered the study phase in August 2003. The Air Force has requested $328 million for SBR in fiscal year 2005 and has programmed about $4 billion for the program from fiscal years 2005 to 2009. Given concerns about affordability and readiness, the Fiscal Year 2005 Defense Appropriations Conference Report reduced funding for SBR to $75 million, with the direction to return this effort back to the technology development phase. In 2003, Congress reduced the Air Force’s $274 million budget request for SBR by $100 million due to concerns about technology maturity and schedule. DOD has scheduled the start of product development for mid-fiscal year 2006, with production starting at the end of fiscal year 2008 and the first satellite to be launched at the end of fiscal year 2012. Figure 1 shows SBR’s acquisition schedule in fiscal years.

Gaining Knowledge about Requirements and Resources before Product Development Is Important for Space Acquisition Success

In the past several decades, DOD’s space acquisitions have experienced problems that have driven up costs by hundreds of millions, even billions, of dollars; have stretched schedules by years; and have increased performance risks. In some cases, capabilities have not been delivered to the warfighter after decades of development. Our reports have shown that these problems, common among many weapon acquisitions, are largely rooted in a failure to match the customer’s requirements (desired capabilities) with the developer’s resources (technical knowledge, timing, and funding) when starting an acquisition program.

In particular, our past work has shown that for space systems, product development was often started based on a rigid set of requirements that proved to be unachievable within a reasonable development time frame. Other cases involved unstable requirements. In some cases where requirements had been identified and approved, even more requirements were added after the program began. When technology did not perform as planned, adding resources in terms of time and money became the primary option for solving problems because the customer’s expectations about the product’s performance capabilities already had been set.

The path traditionally taken by space programs—and other DOD weapon system programs—stands in sharp contrast to that taken by leading commercial firms. Our extensive body of work shows that leading companies use a product development model that helps reduce risks and increase knowledge when developing new products. This best practices model enables decision makers to be reasonably certain about their products at critical junctures during development and helps them make informed investment decisions. This knowledge-based process can be broken down into three cumulative knowledge points.

Knowledge point 1: A match must be made between the customer’s requirements and the developer’s available resources before product development starts. As noted earlier, DOD plans to start SBR product development in 2006.

Knowledge point 2: The product’s design must be stable and must meet performance requirements before initial manufacturing begins.

Knowledge point 3: The product must be producible within cost, schedule, and quality targets and demonstrated to be reliable before production begins.

Systems engineering is a technical management tool that provides the knowledge necessary at knowledge point 1 to translate requirements into specific, achievable capabilities. With systems engineering knowledge in hand, acquisition decision makers and developers can work together to close gaps between requirements and available resources—well before product development starts. Some gaps can be resolved by the developer’s investments, while others can be closed by finding technical or design alternatives. Remaining gaps—capabilities the developer does not have or cannot get without increasing the price and timing of the product beyond what decision makers will accept—must be resolved through trade-offs and negotiations. Effective use of this tool enables decision makers to move on to knowledge point 2 and to produce a stable product design.

DOD has recently issued a new acquisition policy for space systems, partly intended to address past acquisition problems and provide capability to users quicker. However, we recently reported that the policy is not likely to achieve these goals because it allows programs to continue to develop technologies after product development starts. Our past work has shown that this approach makes it more difficult to estimate cost and schedule at the onset of product development and increases the likelihood that programs will encounter technical problems that could disrupt design and production and require more time and money to address than anticipated. Over the long run, the extra investment required to address these problems could reduce funding for developing other technological advances, slow the overall modernization effort, delay capabilities for the warfighter, and force unplanned—and possibly unnecessary—trade-offs between space and other weapon system programs. By contrast, DOD’s revised policy for other weapon acquisitions encourages programs to mature technologies to the point of being tested in an operational environment before beginning product development. We recommended that DOD modify its policy to separate technology development from product development so that needs can be matched with available technology, time, and money at the start of a new program.

We also reported that DOD’s space acquisition policy does not require DOD to commit to setting aside funding for space acquisitions. Hence, there is no guarantee that the resources needed to meet requirements will be there on any individual program when needed. This makes it difficult for DOD as a whole to make corporate-level and trade-off decisions— which will likely be needed when DOD begins the SBR acquisition because (1) costs are significantly increasing for other critical space systems such as the Space-Based Infrared System High, the Transformational Satellite, and the Evolved Expendable Launch Vehicle and (2) DOD is planning to undertake additional new programs, such as the Space-Based Space Surveillance system and a new version of the Global Positioning System.

DOD is revising its new space acquisition policy partly to address these issues; however, the revision was not available for review at the time of this review.

DOD Moving Forward on Acquiring Critical Knowledge but Gaps Remain in Approval for SBR Requirements

DOD has bolstered the SBR acquisition program by increasing senior leader and stakeholder involvement in setting requirements. However, DOD is not fully documenting commitments made during the requirements approval process before progressing to the next acquisition phase, nor has it established a process to resolve potential disagreements that may occur after approval. Clouding the approval of requirements is that DOD’s current space acquisition policy does not provide specific guidance for acquisitions that involve partnerships between DOD and the intelligence community.

SBR Managed by New Executive Oversight Structure

Providing senior-level oversight are three new groups created expressly for the SBR program: the Executive Steering Group, which advises the Requirements/Capabilities Group and the Joint Senior Acquisition Group. Members of these groups come from DOD, National Reconnaissance Office, and National Geospatial-Intelligence Agency. All key stakeholders are expected to have open and honest discussions about what can and cannot be done within desired time frames, budgetary constraints, and achievable technologies. Figure 2 shows how these groups work with SBR’s joint program office and requirements review boards for DOD and the intelligence community.

A primary benefit of having an oversight structure for the SBR program, which involves many decision makers from across multiple organizations, is that the right people are involved in the decision-making process and can work together to lock in their requirements. The intent is to avoid problems of the past in which a program incurs cost, schedule, and performance risks because decision makers continue to negotiate and make trade-offs even after designers and engineers have started technology development and design work. Figure 3 shows the likely outcomes if requirements are poorly defined and are not approved or, in the case of SBR, if requirements are adequately defined and approved early in the study phase.

SBR’s Requirements- Setting Process Lacks Formal Approval and Documentation

DOD officials reported to us that the oversight groups have achieved informal consensus on requirements for SBR. However, this approval has not been formalized and it is unclear as to whether and how it might be formalized. Moreover, it is unclear how disagreements that may occur after initial approval will be resolved.

Regardless of how many stakeholders have been invited to join in decision making or how much expertise is included in SBR’s oversight function, overall success of the SBR program hinges in part on whether the requirements are clear, stable, and achievable and whether DOD and the intelligence community demonstrate commitment and accountability by formally approving the requirements. In an acquisition decision memorandum, the Under Secretary of the Air Force requested that DOD and the intelligence community approve the initial capabilities document and concept of operations before the request for proposals was released in January 2004 for concept development contracts. DOD officials told us that the Joint Requirements Oversight Council and the intelligence community’s Mission Requirements Board approved the initial capabilities document, and there are memoranda documenting these decisions. The Joint Requirements Oversight Council reviewed the concept of operations, provided comments, but did not approve it. According to DOD officials, during a meeting of the SBR Executive Steering Group, high-level officials from the intelligence community verbally approved the concept of operations, but there is no documentation recording this approval.

Agreement is critical because DOD and the intelligence community are placing different emphasis on desired capabilities for SBR. An independent assessment of the SBR program determined that requirements were adequate to enter the study phase, which started in August 2003, but cautioned that the requirements needed to be converged among all stakeholders and users. Table 1 shows the type of knowledge that decision makers expect to gain from the initial capabilities document and the concept of operations.

A defined requirements approval process helps decision makers resolve disagreements that may occur and ensure they will remain committed to their decisions after formal approval. Based on our past reports on uncovering problems and our best practice work, we believe that the steps in a formal approval process include: explaining how decision makers’ requirements and comments are obtained and addressed; identifying the officials and/or the organizations responsible for taking specific approval action; establishing a mechanism and time frame for providing approval or establishing a system for addressing unresolved issues as they relate to key program documentation; and assessing changes to approved requirements based on their effect on the program’s cost and schedule.

While DOD has taken steps to increase senior leader and stakeholder involvement in setting requirements and addressing acquisition issues, DOD is not fully documenting commitments made during the requirements approval process, nor has it established a process to resolve potential disagreements that may occur after approval.

DOD Taking Proactive Steps to Gain Knowledge about Resources, but Critical Gaps May Remain at Product Development

DOD is also taking positive steps to attain the knowledge needed to understand what resources will be needed to develop SBR’s capabilities and to mitigate risks. These include: relying on systems engineering to translate requirements into specific, achievable capabilities and to close gaps between requirements and resources; adopting a more comprehensive cost estimating technique to identify SBR’s life-cycle costs; exploring alternatives for SBR if TCA—the infrastructure that DOD is depending on to transmit SBR’s data—incurs schedule slips; and asking two concept development contractors to each propose at least two different operations concepts for SBR with and without TCA.

However, the path that SBR is on has potential for knowledge gaps when making investment decisions, the types of gaps that have hampered other space programs in the past. Specifically, it is expected that some critical SBR technologies will not be mature when product development starts, that is, not tested in a relevant or operational environment. Typical outcomes of this lack of knowledge are significant cost and schedule increases because of the need to fix problems later in development. Furthermore, TCA, a new, more robust communications infrastructure that could transmit SBR’s imagery data much more quickly than the current infrastructure, is facing uncertainties. Specifically, one of TCA’s primary components, the Transformational Satellite, may not be ready in time to support SBR. Without mature technologies and faced with a possible slip in the Transformational Satellite’s schedule, DOD will be less able to accurately estimate total system costs before the start of product development. In addition, DOD and the Air Force may not have knowledge needed to make corporate level trade-offs between SBR and other air-based radar systems at the time it plans to make a commitment to invest in the SBR acquisition program. DOD has undertaken an analysis to weigh the merits of space-based radar. At this time, it is not known whether this analysis will be a detailed examination of the capabilities and costs of each individual radar option and combined with other radar platforms or whether the analysis will be a less rigorous examination of the mix of radar options.

DOD Taking Positive Steps to Build Foundation of Knowledge about SBR Resources

DOD is planning to aggressively address technology, affordability, and integration issues by, in part, instituting robust systems engineering processes and procedures. Systems engineering is a technical management tool for gaining information on a broad array of activities related to the development of a system. For SBR, DOD plans to perform systems engineering work on requirements and their allocation, interface definitions, trade studies, risk management, performance analysis and modeling, environmental and safety planning, test planning, program protection planning, information assurance, and configuration control. Applying systems engineering to these activities would give DOD the insight and knowledge it needs to better manage the program, including ways to reduce risk and ensure the viability of concepts and requirements.

DOD has also decided to take a more comprehensive approach to estimating SBR’s life-cycle costs. According to the SBR program director, this marks the first time DOD has willingly presented all related costs to develop, acquire, produce, maintain, operate, and sustain the system. DOD officials stated that they wanted to identify not just direct costs, but also costs for associated infrastructure such as the costs related to modifying the ground system that will be used to support SBR as well as other systems. According to DOD, about $8 billion of the $28.6 billion life-cycle cost estimate represents costs that in the past, would not have been included in space program total cost estimates. Taking steps to more comprehensively identify SBR and SBR related costs is a positive step and will help DOD manage its portfolio of space programs.

Although DOD hopes to rely on TCA to support SBR data transmissions, it is taking a proactive approach to identify and assess the viability of TCA alternatives. First, in April 2004, DOD awarded two 2-year contracts for concept development efforts that call for the identification of alternatives to TCA. For each alternative identified, the contractor is to conduct an assessment of the cost, risk, and effect on SBR’s performance. DOD officials told us that when SBR initiates product development in 2006, it would know whether TCA will be available to support SBR or whether to pursue a TCA alternative. In addition, DOD also awarded two contracts totaling $510,000 for a yearlong study to propose several alternatives to TCA capable of supporting SBR’s communications requirements and to analyze the viability of such alternatives. These actions have put DOD in a better position to ensure the program is successful.

The two 2-year contracts that DOD awarded in April 2004 also require that at least two different viable SBR operations concepts be proposed. DOD is expecting each contractor to fully develop the alternative operations concepts. These alternative concepts could involve using unique radar processing techniques. According to DOD, it will work with each of the contractors to pare down the alternatives to a single best concept for each contractor. For the remainder of the contract performance period, the contractors would focus their attention on fleshing out the details associated with these concepts. This approach will put DOD in a better position when the time comes to select a single contractor to design the SBR system.

Technologies Will Not Be Mature at Product Development Start

DOD officials have said that SBR will likely be the most technically challenging, software-intensive, and complex space system ever built by DOD. The two key pieces of hardware needed to give SBR a radar capability from space—the electronically scanned array (which steers the radar signal to an area of interest) and the on-board processor (the radar-processing unit aboard SBR)—face the highest amount of risk. The electronically scanned array can scan multiple areas of interest virtually simultaneously, allowing for simplified satellite design over conventional technology offering mechanical slew radar. The on-board processor is expected to allow the processing radar data to assure the timely and thorough delivery of imagery data that will be downlinked for transmission to the warfighter.

To minimize the potential for technology development problems after the start of product development, DOD uses an analytical tool to assess technology maturity for many weapon system acquisition programs. Called Technology Readiness Levels (TRL), this tool associates a TRL with different levels of demonstrated performance, ranging from paper studies to actual application of the technology in its final form. The value of using a tool based on demonstrated performance is that it can presage the likely consequences of incorporating a technology at a given level of maturity into a product’s development, enabling decision makers to make informed choices. Our previous reviews have found the use of TRLs, which range from 1 to 9, to be a best practice. (See app. I for a description of the TRL levels.)

The critical technologies that will support the SBR program currently range from TRL 3 to 5. A TRL 3 means that most of the work performed so far has been based on analytical and laboratory studies. At a TRL 5, the basic technology components are integrated and tested in a simulated or laboratory environment. Table 2 shows the current TRL for each of SBR’s critical technologies and the expected TRL at product development start in 2006. In general, the program office’s key risk reduction efforts are scheduled to mature these technologies to TRL 5 by the middle of fiscal year 2006. These efforts include the awarding of research and development contracts to three payload contractors for efforts to continue to develop and mature these components (the electronically scanned array and on-board processor). The period of performance of each contract is about 2.5 years.

To mature the electronically scanned array and on-board processor technologies from a TRL 3/4 to 5, the contractors plan to conduct various developmental and integrative tasks in about 3 years. For example, one contractor plans to conduct 18 tasks to develop the electronically scanned array and 8 tasks to integrate the on-board processor with other system components. In addition, the development of the integrated circuits and programmable microcircuits that support the on-board processor requires extensive tests and evaluations and the radiation-hardening requirement further complicates the development. Given the challenges of the state-of-the-art technologies being developed and the algorithms involved, the testing programs must be rigorous and transparent and the results fully documented. We have determined that the time allotted to mature the SBR technologies to TRL 5 is ambitious given the tasks that need to be accomplished. Furthermore, the development of the signal processing algorithms and communications downlink involves significant software development. Based on our past experience of software assessments in other programs, the establishment of a structured testing regime for software development has always been underestimated.

By planning to start product development in fiscal year 2006 with technologies at TRL 5, DOD is very likely to continue designing the system and to conduct other program activities at the same time it builds representative models of key technologies and tests them in an environment that simulates space conditions (such as a vacuum chamber). This approach is common with DOD space acquisitions but has a problematic history. Our past work has shown that it can lead to significant cost and schedule increases because of the need to fix problems later in development. A continuing problem is that software needs are poorly understood at the beginning of a program. We have previously recommended that DOD not allow technologies to enter into a weapon system’s product development until they are assessed at a TRL 7, meaning a prototype has been demonstrated in an operational environment. DOD has accepted lower TRL thresholds for space programs because testing in an operational environment—in space, for example, or even in a relevant environment—is difficult and costly. However, DOD’s new space acquisition policy does not identify what the minimum TRL level should be before starting product development for space programs, how risks should be mitigated if technologies are included in programs without full testing, or how lower TRL levels affect the confidence of cost and schedule estimates. Moreover, the policy does not address the option of maturing technologies outside a program and pulling them in once they prove to be viable.

One way to mitigate technology risk is to rely on backup technologies, should newer technologies prove to be problematic during product development. According to DOD officials, there are backup technologies that are more mature for each of SBR’s critical technologies. The backups are the same technologies but rely on a previous and more mature version. Using previous versions of these technologies would result in a lower level of desired performance—such as a reduced area collection rate, a reduction in the total number of targets collected per satellite per day, increased product delivery time frames to the user, an increased weight of the spacecraft, and higher cost. For example, more mature versions of the electronically scanned array exist and if used, would result in a reduction in its performance level. In addition, some previous versions of SBR technologies have not been demonstrated or tested in space. But according to DOD officials, even with backup technologies, the total performance of the SBR system can be maintained through systems engineering trades. DOD says it has been able to leverage some of the key technologies (such as the electronically scanned array) that were under development during the previous effort, Discoverer II, to demonstrate a space-based radar capability.

Communications Infrastructure May Not Be Ready in Time to Support SBR

Current plans call for TCA to transmit SBR’s large volume of data to ground-, air-, ship-, and space-based systems. However, one of TCA’s primary components, the Transformational Satellite—which will use technologies that DOD has never before tried in space—is facing uncertainties in its scheduled 2011 launch. DOD started product development for the Transformational Satellite in December 2003 even though technologies were immature. If the Transformational Satellite falters but SBR launches as expected in 2012, then DOD will have a fully operational, new-generation satellite that is missing its primary means of data transmission. Recognizing the challenges, DOD is to decide by November 2004 whether to move forward or delay the Transformational Satellite’s acquisition program and instead procure another Advanced Extremely High Frequency satellite, which already are under development and are based on mature technologies.

Our analysis shows that alternatives to TCA may involve a greater reliance on processing aboard the SBR satellites, thereby increasing software development efforts. This approach would reduce the volume of data requiring transmission, allowing conventional satellite systems, such as the Advanced Extremely High Frequency satellites, to handle the transmission. Another likely alternative is to have SBR satellites transmit only selected portions of data, again, so that the Advanced Extremely High Frequency satellite could handle the lower volume of information. Finally, a dedicated system of satellites could be fielded for the sole purpose of transmitting SBR data, significantly increasing program cost and raising affordability issues. Currently, DOD is working closely with officials from the Transformational Satellite program office to evaluate the relative merits of various alternatives and to document the interfaces needed between SBR and the Transformational Satellite for each alternative. During the course of our audit work, SBR program officials met weekly with the Transformational Satellite program’s integrated product teams and were coordinating efforts on a memorandum of agreement on requirements development, joint engineering practices, and studies of air- and space-based options.

SBR’s Cost Estimate Unlikely to Be Realistic Because of Multiple Uncertainties

Based on a notional constellation of nine (plus one spare) satellites operating in low-earth orbit, an independent cost assessment in 2003 put SBR’s cost at the $28.6 billion mark, making SBR the most expensive DOD space system ever built. When this initial cost estimate is revised in 2006, before SBR’s product development starts, DOD is to have decided a number of issues, such as how many satellites are to be acquired, what their capabilities will be, and at what altitude(s) the satellites are to operate. This system refinement allows DOD to develop a more realistic total system cost estimate—a critical knowledge point if a successful match between requirements and resources is to be made. However, if DOD begins product development with less than mature technologies and without knowing the availability of TCA, accurate cost estimates for SBR will be much more difficult to prepare. We have previously reported that improving the reliability of cost estimates is critical and affords DOD decision makers with the appropriate information to decide whether a weapon system is worth the overall investment and whether the time is right to proceed with such an investment. Once a total cost is known, DOD needs to secure the funding so it can design, produce, operate, and sustain the system.

DOD may also lack knowledge needed to make a corporate-level decision as to how much it should invest in SBR versus air platforms with similar capabilities at the time it begins the SBR acquisition program. In November 2003, the Air Force completed an analysis of alternatives (AOA) for SBR, which was supposed to evaluate whether space- or air-based radar platforms (such as manned and unmanned aircraft with radar capabilities) or a combination of both are better suited for tracking moving targets on land or at sea and analyze the capabilities and costs of each suitable option. However, DOD officials raised a concern that the AOA only weighed the merits of various space-based solutions. The Air Force decided to undertake a follow-on study to explore the optimal ways to gather information on ground moving targets from radars based in space versus air. The plan is to also use this follow-on study as part of DOD’s preparations for submitting a fiscal year 2006 budget to Congress to secure funding for SBR and other radar systems on air platforms. A more thorough AOA, completed before the start of the study phase, might conceivably have determined that air-based radar could provide many or most of the capabilities promised by space-based radar but at a fraction of the cost. Moreover, this type of analysis could help DOD officials better decide whether SBR should be initiated at a later date, when critical technologies will have been matured, or when the communications infrastructure to support SBR will be available.

DOD officials have mentioned other ongoing studies that are examining the optimal mix between SBR and other platforms for specific capabilities, such as ground-moving target indication. However, it is unclear as to the extent these studies will be factored into the SBR product development start decision.

Conclusions

DOD has recently embarked on a discovery and exploration phase for its SBR program. During this period, it is critical for programs to work toward closing knowledge gaps about requirements, technologies, funding, and other resources so they can be positioned to succeed when DOD decides to commit to making significant investments. For SBR, this would mean testing technologies to the point of knowing they can work as intended before starting program development, securing agreement on requirements with the intelligence community, and fully assessing the cost and benefits and risks of relying on TCA and alternatives, including different mixes of air and space-based platforms. DOD is taking positive steps toward this end, but without maturing critical technologies or securing formal commitment on requirements, it will not be able to assure decision makers that the program can be completed within cost and schedule estimates. Should DOD decide to proceed on a path that leaves open important questions, including those about technologies, then it should do so with (1) assessments of technical risks and what additional resources (in terms of time and money) would be needed to address problems that may occur during development as well as what trade-offs would need to be made with other space programs should DOD need to invest additional resources in SBR, and (2) a formal commitment for providing additional resources if problems do occur.

Recommendations for Executive Action

To better ensure that DOD and its intelligence community partners obtain the additional knowledge they need to determine whether and when to begin the SBR acquisition program, we recommend that the Secretary of Defense direct the Under Secretary of the Air Force to: Direct the SBR Executive Steering Group to ensure that outcomes from the requirements management process are formally approved and documented as the program proceeds through product development before an investment is made beyond technology and concept development for the SBR program. This group should identify how key document review comments are to be obtained and addressed and identify all the officials and/or organizations responsible for taking specific approval action. In addition, the group should establish a mechanism and time frame for providing approval/disapproval. Finally, the group should establish a formal mechanism for addressing unresolved issues as they relate to key program documentation, as well as how changes to approved requirements will be assessed.

Modify DOD’s space acquisition policy to reflect protocols for setting requirements when DOD undertakes programs in partnership with the intelligence community.

Delay approval to commit funding to product development (key decision point B) for SBR until technologies have been demonstrated in a relevant or operational environment so DOD can more reliably estimate the resources needed to complete the program. If the Under Secretary determines that the program should go forward with less mature technologies, then we recommend that the Under Secretary (1) undertake an assessment of the backup technologies that may lessen capability and add cost to the program and the additional time and money that may be required to meet SBR’s performance objectives to address those risks, (2) undertake an assessment of trade-offs that may need to be made with other space programs to assure SBR’s successful outcome, and (3) secure formal commitments from DOD to provide funding for total estimated costs as well as costs estimated to address potential technical risks. Strengthen the ongoing study of options for tracking ground-moving targets by ensuring this work includes: (1) a full range of air and space options; (2) measures of effectiveness that would help justify choosing SBR over air options; and (3) the possibility of having to rely on TCA alternatives for space options. This work should also consider the results of analyses being conducted by other DOD entities on tracking ground- moving targets.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Deputy Under Secretary of Defense (Programs, Requirements, and Resources) within the Office of the Under Secretary of Defense for Intelligence. DOD generally agreed with our findings and our recommendation to strengthen its study of SBR alternatives. DOD partially agreed with our recommendations to strengthen its requirement setting process for SBR and to demonstrate SBR technologies in a relevant or operational environment before committing to product development. DOD did not agree with our recommendation to modify its acquisition policy to strengthen requirements setting.

In commenting on our recommendations, DOD agreed in principle with the need to extensively define, analyze, and validate requirements for SBR, but it did not believe this necessitated a different requirements setting process than the one that is in place for SBR or changes to its space acquisition policy or that additional controls were needed within the program’s study phase. To clarify, our recommendation was not intended to construct a new requirements setting process or supplant activities undertaken by the Joint Requirements Oversight Council or the Mission Requirements Board, as DOD asserts. Rather, we recommend that DOD build on the positive requirements setting procedures it has already put in place by instituting controls and mechanisms that ensure transparency, discipline, and accountability with requirements setting. As noted in our report, while DOD has taken steps to increase senior leader and stakeholder involvement in requirements setting, it is not fully documenting commitments made during the requirements approval process, nor has it established a process to resolve potential disagreements that may occur after approval. It is important that this discipline be instilled in the study phase and throughout the SBR effort. As noted in previous reports, many space programs have not been executed within cost and schedule estimates because of an inability to establish firm requirements and to make and enforce trade-off decisions. For SBR, the potential for difficulty in requirements setting is higher because of the distinct needs of the intelligence community and DOD’s desire to integrate SBR with other radar platforms. Moreover, revising the acquisition policy to clearly communicate protocols that should be followed when DOD undertakes space programs in the future involving diverse users—such as the intelligence community, military services, industry, and/or other agencies—would further help DOD to rationalize requirements setting and to solidify relationships with users, which DOD reported was a top SBR management issue.

In regard to our recommendation to delay product development until SBR technologies are sufficiently matured, DOD stated that it has planned for critical and most other enabling technologies to be demonstrated at least at the component level in a relevant environment on the ground. DOD also stated that where technically feasible and fiscally feasible, it planned to pursue on-orbit demonstrations. It also stated it has taken some actions relating to our recommendation such as accounting for technical risks in the costing and budgeting process. DOD asserted, however, that our recommendation encourages pursuit of older, more proven, technologies. We recommended that DOD pursue relevant or operational environment demonstrations of all critical technologies and even an integrated system before committing to a formal acquisition program because this practice enables a program to align customer expectations with resources, and therefore minimize problems that could hurt a program in its design and production phase and drive up costs and schedule. Further, we agree that continuing to develop leading edge technology is important for space system capabilities. However, history has shown and we have repeatedly reported that conducting technology development within a product environment consistently delays the delivery of capability to the user, robs other programs of necessary funds through unanticipated cost overruns, and, consequently, can result in money wasted and fewer units produced than originally stated as necessary. A technology development environment is more forgiving and less costly than a delivery-oriented acquisition program environment. Events such as test “failures,” new discoveries, and time spent in attaining knowledge are considered normal in this environment. Further, judgments of technology maturity have proven to be insufficient as the basis for accurate estimates of program risks as it relates to cost, schedule, and capability. Lastly, our report noted that DOD was taking positive actions to gain knowledge about technology readiness, including strengthening systems engineering, undertaking risk assessments, and assessing various technical concepts. Given the potential cost of the program, our recommendation focuses on taking these steps further by assessing what trade-offs may need to be made with other space programs should the program encounter technical problems that require more time and money than anticipated and securing commitments to provide resources needed to address such problems.

DOD’s detailed comments are provided in appendix II.

We plan to provide copies of this report to the Secretary of Defense, the Secretary of the Air Force, and interested congressional committees.

We will make copies available to others upon request. In addition, the report will be available on the GAO Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please contact me at (202) 512-4841 or Arthur Gallegos at (303) 572-7368. Other key contributors to the report include Tony Beckham, Cristina Chaplain, Lily Chin, Maria Durant, Nancy Rothlisberger, and Hai V. Tran.

Appendix I: TRL Scale for Assessing Critical Technologies

Appendix I: TRL Scale for Assessing Critical Technologies Lowest level of technology readiness. Scientific research begins to be translated into applied research and development (R&D). Examples might include paper studies.

Invention begins. Once basic principles are observed, practical applications can be invented. Examples are still limited to paper studies.

Active R&D is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

Basic technological components are integrated to establish that they will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory.

Fidelity of breadboard technology increases significantly. Basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in simulated environment. Examples include “high fidelity” laboratory integration of components.

Representative model or prototype system, which is well beyond the breadboard tested for level 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated operational environment.

Prototype near, or at, planned operational system. Represents a major step up from TRL 6, requiring demonstration of an actual system prototype in an operational environment such as an aircraft, vehicle, or space.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include Developmental Test and Evaluation of the system in its intended weapon system to determine if it meets design specifications.

Actual application of the technology in its final form and under mission conditions, such as those encountered in Operational Test and Evaluation. Examples include using the system under operational mission conditions.

Appendix II: Comments from the Department of Defense

GAO’s Comments

The following are GAO’s comments on the Department of Defense’s letter dated June 29, 2004. 1. DOD stated that DOD does not require formal approval for the concepts of operations from the Joint Requirements Oversight Council or the Mission Requirements Board, but noted that the Joint Requirements Oversight Council communicated agreement in a memo. As we reported, the Under Secretary of the Air Force requested that both DOD and the intelligence community approve the initial capabilities document and concept of operations in light of the complexity of SBR’s acquisition process, the partnership with the intelligence community, and the proposed integration with other radar platforms. 2. DOD stated that it is not engaged in a partnership with the intelligence community on SBR, as our report states. Specifically, DOD stated that SBR is wholly funded in the defense budget and that a programmatic commitment with the intelligence community does not exist. DOD’s SBR System Acquisition Strategy was signed by senior-level officials from DOD, National Reconnaissance Office, and the National Geospatial-Intelligence Agency and approved on January 14, 2004. This strategy states that the Air Force, in close partnership with the National Reconnaissance Office and National Geospatial-Intelligence Agency, is responsible for leading development of an SBR capability. This strategy further identifies the responsibilities related to SBR that each mission partner (National Reconnaissance Office and National Geospatial-Intelligence Agency) is supposed to carry out. We disagree with DOD’s assertion that these organizations must provide funding to SBR in order to consummate a partnership. Because SBR is being justified on the basis of the system’s ability to provide intelligence, surveillance, and reconnaissance products to both DOD and the intelligence community, the part of the budget used is not relevant to our finding. 3. To clarify, we did not recommend that DOD pursue lower risk technologies that would result in lower levels of desired performance. Instead, we reported that DOD might have to resort to using backup technologies if the current ones prove to be problematic during product development. We recommended that DOD should assess the cost to the program of having to use the backup technologies DOD has already identified in terms of time and money.

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs