Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development
Published: 2004-07-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Missing among the Department of Defense's (DOD) portfolio of systems is a capability to track stationary and moving enemy vehicles on land or at sea in any type of weather, day or night, from space. To meet this need, DOD and the intelligence community are collaborating on the ambitious Space-Based Radar (SBR) program. By leveraging the newest generation of radar technologies, the SBR concept promises to deliver high-quality data to a wide array of users. DOD intends to start product development in 2006 and to field SBR satellites as quickly as possible so that warfighters, the intelligence community, and national decision makers can gain a better understanding of what adversaries are doing in specific locations around the world. GAO reviewed the SBR program to assess DOD's progress in attaining the knowledge it needs by 2006 in terms of customer needs (or requirements) and resources.
section_title: What GAO Found
paragraphs: Although SBR is 2 years away from product development, the program already faces major challenges. DOD officials say SBR will likely be the most expensive and technically challenging space system ever built by DOD. The acquisition time frame is much shorter than what has been achieved in the past for other complex satellite systems. Finally, DOD is setting precedence by taking the lead on developing SBR with the intelligence community as a partner. Most DOD space programs that GAO has reviewed in the past several decades were hampered by schedule and cost growth and performance shortfalls. Problems were largely rooted in a failure to match requirements with resources when starting product development. Commitments were made without knowing whether technologies being pursued would work as intended. To avoid these problems, leading commercial firms have adopted a knowledge-based model that enables decision makers to be reasonably certain about their products at critical junctures and helps them make informed investment decisions. Although DOD has taken positive steps to strengthen the involvement of senior leaders within DOD and the intelligence community in setting requirements, SBR's concept of operations has not been approved and signed by requirements boards for either of the two partners. Without documentation and formal approval, it is unclear who will be held accountable for setting requirements or how disagreements among SBR's partners will be resolved when DOD moves SBR into ensuing phases of acquisition. DOD has adopted noteworthy practices to gain knowledge about SBR's resources. These include maximizing the use of systems engineering to close gaps between requirements and resources; estimating all of SBR's costs; exploring alternatives for SBR if the Transformational Communications Architecture (TCA)--the communications infrastructure that is expected to relay SBR data across a network of users--incurs schedule and performance shortfalls; and asking contractors to propose multiple operations concepts for SBR with or without TCA. Despite these accomplishments, DOD is at risk of knowledge gaps. SBR's critical technologies will not be mature when product development starts, as called for by best practices. One of TCA's primary components may not be ready in time to support SBR data. These knowledge gaps make it harder for DOD to reliably estimate how much time and money are needed to complete SBR's development. If TCA is delayed, DOD's alternatives may involve reducing SBR's capabilities or significantly increasing program cost. Without sufficient knowledge, DOD may not be able to determine by the time SBR's product development starts in 2006 whether space-based radar is best suited to tracking moving targets on land or at sea or whether air-based radar would provide enough capabilities at far less cost. More specific analyses would help DOD weigh the merits of various alternatives and assess how much to invest in the SBR acquisition program versus air platforms with similar capabilities.

section_title: Letter
paragraphs: The Department of Defense (DOD) currently has ground-based and  air-based radars but no capability to track moving targets from space.  To meet a need for persistent global observation, DOD and the intelligence  community are working together to develop the Space-Based Radar (SBR)  system to find, identify, track, and monitor ground and sea targets— mobile or immobile—under all-weather conditions and on a near-continual  basis across large swaths of the earth’s surface. SBR is to enhance  information gathering by providing intelligence, surveillance, and  reconnaissance data in a meaningful and timely manner.
This is an ambitious program for DOD. DOD’s initial total cost estimate for  SBR is about $28.6 billion from fiscal year 2003 to 2024. And although the  technologies for tracking moving targets from space are still in  development, DOD is scheduling almost 7 years between the start of  product development and launch of the first SBR satellite—a time frame  that is considerably shorter than what has been achieved in the past for  other complex satellite systems. DOD would like to field SBR satellites as  quickly as possible because it believes that SBR represents a major leap  forward in providing the warfighters, intelligence community, and national  decision makers with significant tactical, operational, and strategic  advantages over potential adversaries. For example, DOD envisions that  SBR will be able to see deep inside enemy territory without risk to  personnel or resources and that it will operate over areas where  conventional airborne surveillance systems are at risk from the enemy’s  surface-to-air missiles, which can travel increasingly longer ranges. DOD  also envisions that SBR will generate high-quality radar imagery of targets  and terrain and interface with ground, air, and other space systems so that  users can gain a better understanding of what is occurring in specific  locations.
DOD is in the early exploratory phase of the SBR program—a period when  it gathers knowledge about its needs, or requirements, and explores the  feasibility of meeting those requirements. DOD expects to begin product  development in fiscal year 2006; at which point it will make a commitment  to invest in a formal acquisition program for SBR. Our past work has  shown that successful weapon system programs are able to match their  needs to their resources—that is, money, technology, and time—before  product development. With achievable requirements and commitment of  sufficient investment to complete development, programs are better able  to deliver products at cost and on schedule. Most space programs over the  past several decades have not been able to achieve a match between needs  and resources before product development and have incurred significant  cost and schedule increases due in part to the need to rework technologies  in the later stages of their acquisition.
We conducted our review of the SBR program on the initiative of the  Comptroller General. Given SBR’s overall importance to DOD and the  intelligence community, we reviewed the SBR program to assess DOD’s  progress in attaining the knowledge it needs by 2006 in terms of  (1) requirements and (2) resources—technology, communications  infrastructure, and funding.
We are addressing this report to you because of your jurisdiction over  weapon systems acquisition.
subsections: section_title: Scope and Methodology
paragraphs: To assess DOD’s progress in attaining the knowledge it needs before the  start of product development, we examined the resources (technology,  communications infrastructure, and funding) committed and planned for  the program as well as the users’ needs for an SBR system. We considered  DOD’s plans for maturing the critical technologies when we obtained  technology-readiness information for each critical technology (as well as  its mature backup technology) against best practice standards to  determine if they will be sufficiently mature when DOD plans to start  product development. We also reviewed the SBR risk management plans  and concept development contract information. We discussed these  documents and issues with representatives from Air Force Space  Command, Peterson Air Force Base, Colorado; and the SBR Joint Program  Office, Space and Missile Systems Center, Los Angeles Air Force Base,  California.
To determine SBR’s role in a larger DOD architecture, we met with  officials from the Joint Chiefs of Staff, Washington, D.C.; and the Air Force  Directorate of Space Acquisitions, Arlington, Virginia. We also consulted  past GAO reports to determine the relationship between SBR and the  Transformational Communications Architecture.
To determine the scope and completeness of the analysis of alternatives  and its follow-on study to identify the optimal ways to gather information  on ground moving targets from radars based in space versus air, we met  with officials from Air Force Space Command; DOD Office of the Director,  Program Analysis and Evaluation, Washington, D.C.; Air Force Directorate  of Requirements for Space, Crystal City, Virginia; and the Air Force  Studies and Analyses Agency, Arlington, Virginia. We also talked with an  official from the Air Force Office of Aerospace Studies, Kirtland Air Force  Base, New Mexico.
We discussed overarching programmatic issues—including the level of  coordination between DOD and the intelligence community—with  representatives from the Air Force Directorate of Space Acquisitions.  We were not able to obtain meetings with members of the Mission  Requirements Board (a board within the intelligence community  responsible for approving program requirements) or the intelligence  agencies to discuss their stake in the SBR program.
We performed our work from November 2003 through June 2004 in  accordance with generally accepted government auditing standards.
section_title: Background
paragraphs: SBR represents the first time that DOD has taken the lead on developing a  major national security space capability with the intelligence community  as a partner. Because of this partnership, SBR’s acquisition process is  more complex than that used for typical DOD programs. While DOD and  the intelligence community will likely use all the data that SBR produces,  their priorities differ. DOD’s warfighting community is particularly  interested in tracking targets moving over land or sea as well as other  objects of interest. The intelligence community is more focused on  obtaining detailed global imagery and combining it with other data for  advanced processing. SBR is expected to meet both needs and be fully  integrated with other space and non-space systems, including TCA, which  is to transmit SBR’s data to receivers in the air, at sea, or on the ground.
A key advantage of radar in space is having the ability to “see”  through clouds and sand storms and any type of weather, day or night.  Radar-equipped aircrafts, on the other hand, require U.S. air dominance to  collect radar information and must steer clear of hostile areas—the result  being limited radar coverage. The SBR concept offers other added  features, including electronic steering of the radar signal toward a  particular area and capturing high volumes of very fine resolution radar  images of targets and terrain. With the ability to perform these functions  almost simultaneously, SBR is expected to help analysts gain a better  understanding of what is occurring in specific locations.
To help meet some of its goals, DOD plans to leverage key technologies  that were developed in the late 1990s to demonstrate a space-based radar  capability. According to DOD officials, contractors developed some  satellite hardware and prototype components under the Discoverer II  program, which began in 1998 and was to identify and validate by 2008 the  capability of tracking mobile ground targets from space. Discoverer II,  comprising two radar demonstration satellites, was a joint initiative by the  Air Force, DOD’s Defense Advanced Research Projects Agency, and the  intelligence community’s National Reconnaissance Office. DOD officials  told us that the Discoverer II program had reached the preliminary design  review phase when it was cancelled in 2000 because of cost and schedule  uncertainties, poorly explained requirements, and the lack of a coherent  vision to transition the system to operational use.
The Secretary of Defense concluded that space-based radar could provide  a military advantage and in 2001 approved SBR as a new major defense  acquisition program, delegating it to the Air Force. In July 2003, an  independent cost assessment team consisting of representatives from  DOD and the intelligence community estimated that $28.6 billion would be  needed to pay for SBR’s life-cycle costs—development, production,  launch, and operation. The program entered the study phase in August  2003. The Air Force has requested $328 million for SBR in fiscal year 2005  and has programmed about $4 billion for the program from fiscal years  2005 to 2009. Given concerns about affordability and readiness, the Fiscal  Year 2005 Defense Appropriations Conference Report reduced funding for  SBR to $75 million, with the direction to return this effort back to the  technology development phase. In 2003, Congress reduced the Air Force’s  $274 million budget request for SBR by $100 million due to concerns about  technology maturity and schedule. DOD has scheduled the start of product  development for mid-fiscal year 2006, with production starting at the end  of fiscal year 2008 and the first satellite to be launched at the end of fiscal  year 2012. Figure 1 shows SBR’s acquisition schedule in fiscal years.
section_title: Gaining Knowledge about Requirements and Resources before Product Development Is Important for Space Acquisition Success
paragraphs: In the past several decades, DOD’s space acquisitions have experienced  problems that have driven up costs by hundreds of millions, even billions,  of dollars; have stretched schedules by years; and have increased  performance risks. In some cases, capabilities have not been delivered to  the warfighter after decades of development. Our reports have shown that  these problems, common among many weapon acquisitions, are largely  rooted in a failure to match the customer’s requirements (desired  capabilities) with the developer’s resources (technical knowledge, timing,  and funding) when starting an acquisition program.
In particular, our past work has shown that for space systems, product  development was often started based on a rigid set of requirements that  proved to be unachievable within a reasonable development time frame.  Other cases involved unstable requirements. In some cases where  requirements had been identified and approved, even more requirements  were added after the program began. When technology did not perform as  planned, adding resources in terms of time and money became the primary  option for solving problems because the customer’s expectations about  the product’s performance capabilities already had been set.
The path traditionally taken by space programs—and other DOD weapon  system programs—stands in sharp contrast to that taken by leading  commercial firms. Our extensive body of work shows that leading  companies use a product development model that helps reduce risks and  increase knowledge when developing new products. This best practices  model enables decision makers to be reasonably certain about their  products at critical junctures during development and helps them make  informed investment decisions. This knowledge-based process can be  broken down into three cumulative knowledge points.
Knowledge point 1: A match must be made between the customer’s  requirements and the developer’s available resources before product  development starts. As noted earlier, DOD plans to start SBR product  development in 2006.
Knowledge point 2: The product’s design must be stable and must meet  performance requirements before initial manufacturing begins.
Knowledge point 3: The product must be producible within cost, schedule,  and quality targets and demonstrated to be reliable before production  begins.
Systems engineering is a technical management tool that provides the  knowledge necessary at knowledge point 1 to translate requirements into  specific, achievable capabilities. With systems engineering knowledge in  hand, acquisition decision makers and developers can work together to  close gaps between requirements and available resources—well before  product development starts. Some gaps can be resolved by the developer’s  investments, while others can be closed by finding technical or design  alternatives. Remaining gaps—capabilities the developer does not have or  cannot get without increasing the price and timing of the product beyond  what decision makers will accept—must be resolved through trade-offs  and negotiations. Effective use of this tool enables decision makers to  move on to knowledge point 2 and to produce a stable product design.
DOD has recently issued a new acquisition policy for space systems, partly  intended to address past acquisition problems and provide capability to  users quicker. However, we recently reported that the policy is not likely  to achieve these goals because it allows programs to continue to develop  technologies after product development starts. Our past work has shown  that this approach makes it more difficult to estimate cost and schedule at  the onset of product development and increases the likelihood that  programs will encounter technical problems that could disrupt design and  production and require more time and money to address than anticipated.  Over the long run, the extra investment required to address these  problems could reduce funding for developing other technological  advances, slow the overall modernization effort, delay capabilities for the  warfighter, and force unplanned—and possibly unnecessary—trade-offs  between space and other weapon system programs. By contrast, DOD’s  revised policy for other weapon acquisitions encourages programs to  mature technologies to the point of being tested in an operational  environment before beginning product development. We recommended  that DOD modify its policy to separate technology development from  product development so that needs can be matched with available  technology, time, and money at the start of a new program.
We also reported that DOD’s space acquisition policy does not require  DOD to commit to setting aside funding for space acquisitions. Hence,  there is no guarantee that the resources needed to meet requirements will  be there on any individual program when needed. This makes it difficult  for DOD as a whole to make corporate-level and trade-off decisions— which will likely be needed when DOD begins the SBR acquisition because  (1) costs are significantly increasing for other critical space systems such  as the Space-Based Infrared System High, the Transformational Satellite,  and the Evolved Expendable Launch Vehicle and (2) DOD is planning to  undertake additional new programs, such as the Space-Based Space  Surveillance system and a new version of the Global Positioning System.
DOD is revising its new space acquisition policy partly to address these  issues; however, the revision was not available for review at the time of  this review.
section_title: DOD Moving Forward on Acquiring Critical Knowledge but Gaps Remain in Approval for SBR Requirements
paragraphs: DOD has bolstered the SBR acquisition program by increasing senior  leader and stakeholder involvement in setting requirements. However,  DOD is not fully documenting commitments made during the requirements  approval process before progressing to the next acquisition phase, nor has  it established a process to resolve potential disagreements that may occur  after approval. Clouding the approval of requirements is that DOD’s  current space acquisition policy does not provide specific guidance for  acquisitions that involve partnerships between DOD and the intelligence  community.
subsections: section_title: SBR Managed by New Executive Oversight Structure
paragraphs: Providing senior-level oversight are three new groups created expressly  for the SBR program: the Executive Steering Group, which advises the  Requirements/Capabilities Group and the Joint Senior Acquisition Group.  Members of these groups come from DOD, National Reconnaissance  Office, and National Geospatial-Intelligence Agency. All key stakeholders  are expected to have open and honest discussions about what can and  cannot be done within desired time frames, budgetary constraints, and  achievable technologies. Figure 2 shows how these groups work with  SBR’s joint program office and requirements review boards for DOD and  the intelligence community.
A primary benefit of having an oversight structure for the SBR program,  which involves many decision makers from across multiple organizations,  is that the right people are involved in the decision-making process and  can work together to lock in their requirements. The intent is to avoid  problems of the past in which a program incurs cost, schedule, and  performance risks because decision makers continue to negotiate and  make trade-offs even after designers and engineers have started  technology development and design work. Figure 3 shows the likely  outcomes if requirements are poorly defined and are not approved or, in  the case of SBR, if requirements are adequately defined and approved  early in the study phase.
section_title: SBR’s Requirements- Setting Process Lacks Formal Approval and Documentation
paragraphs: DOD officials reported to us that the oversight groups have achieved  informal consensus on requirements for SBR. However, this approval has  not been formalized and it is unclear as to whether and how it might be  formalized. Moreover, it is unclear how disagreements that may occur  after initial approval will be resolved.
Regardless of how many stakeholders have been invited to join in decision  making or how much expertise is included in SBR’s oversight function,  overall success of the SBR program hinges in part on whether the  requirements are clear, stable, and achievable and whether DOD and the  intelligence community demonstrate commitment and accountability by  formally approving the requirements. In an acquisition decision  memorandum, the Under Secretary of the Air Force requested that DOD  and the intelligence community approve the initial capabilities document  and concept of operations before the request for proposals was released in  January 2004 for concept development contracts. DOD officials told us  that the Joint Requirements Oversight Council and the intelligence  community’s Mission Requirements Board approved the initial capabilities  document, and there are memoranda documenting these decisions. The  Joint Requirements Oversight Council reviewed the concept of operations,  provided comments, but did not approve it. According to DOD officials,  during a meeting of the SBR Executive Steering Group, high-level officials  from the intelligence community verbally approved the concept of  operations, but there is no documentation recording this approval.
Agreement is critical because DOD and the intelligence community are  placing different emphasis on desired capabilities for SBR. An  independent assessment of the SBR program determined that  requirements were adequate to enter the study phase, which started in  August 2003, but cautioned that the requirements needed to be converged  among all stakeholders and users. Table 1 shows the type of knowledge  that decision makers expect to gain from the initial capabilities document  and the concept of operations.
A defined requirements approval process helps decision makers resolve  disagreements that may occur and ensure they will remain committed to  their decisions after formal approval. Based on our past reports on  uncovering problems and our best practice work, we believe that the steps  in a formal approval process include:    explaining how decision makers’ requirements and comments are  obtained and addressed;  identifying the officials and/or the organizations responsible for taking  specific approval action;    establishing a mechanism and time frame for providing approval or    establishing a system for addressing unresolved issues as they relate to  key program documentation; and    assessing changes to approved requirements based on their effect on the  program’s cost and schedule.
While DOD has taken steps to increase senior leader and stakeholder  involvement in setting requirements and addressing acquisition issues,  DOD is not fully documenting commitments made during the requirements  approval process, nor has it established a process to resolve potential  disagreements that may occur after approval.
section_title: DOD Taking Proactive Steps to Gain Knowledge about Resources, but Critical Gaps May Remain at Product Development
paragraphs: DOD is also taking positive steps to attain the knowledge needed to  understand what resources will be needed to develop SBR’s capabilities  and to mitigate risks. These include:  relying on systems engineering to translate requirements into specific,  achievable capabilities and to close gaps between requirements and  resources;    adopting a more comprehensive cost estimating technique to identify  SBR’s life-cycle costs;    exploring alternatives for SBR if TCA—the infrastructure that DOD is  depending on to transmit SBR’s data—incurs schedule slips; and    asking two concept development contractors to each propose at least two  different operations concepts for SBR with and without TCA.
However, the path that SBR is on has potential for knowledge gaps when  making investment decisions, the types of gaps that have hampered other  space programs in the past. Specifically, it is expected that some critical  SBR technologies will not be mature when product development starts,  that is, not tested in a relevant or operational environment. Typical  outcomes of this lack of knowledge are significant cost and schedule  increases because of the need to fix problems later in development.  Furthermore, TCA, a new, more robust communications infrastructure  that could transmit SBR’s imagery data much more quickly than the  current infrastructure, is facing uncertainties. Specifically, one of TCA’s  primary components, the Transformational Satellite, may not be ready in  time to support SBR. Without mature technologies and faced with a  possible slip in the Transformational Satellite’s schedule, DOD will be less  able to accurately estimate total system costs before the start of product  development. In addition, DOD and the Air Force may not have knowledge  needed to make corporate level trade-offs between SBR and other  air-based radar systems at the time it plans to make a commitment to  invest in the SBR acquisition program. DOD has undertaken an analysis to  weigh the merits of space-based radar. At this time, it is not known  whether this analysis will be a detailed examination of the capabilities and  costs of each individual radar option and combined with other radar  platforms or whether the analysis will be a less rigorous examination of  the mix of radar options.
subsections: section_title: DOD Taking Positive Steps to Build Foundation of Knowledge about SBR Resources
paragraphs: DOD is planning to aggressively address technology, affordability, and  integration issues by, in part, instituting robust systems engineering  processes and procedures. Systems engineering is a technical management  tool for gaining information on a broad array of activities related to the  development of a system. For SBR, DOD plans to perform systems  engineering work on requirements and their allocation, interface  definitions, trade studies, risk management, performance analysis and  modeling, environmental and safety planning, test planning, program  protection planning, information assurance, and configuration control.  Applying systems engineering to these activities would give DOD the  insight and knowledge it needs to better manage the program, including  ways to reduce risk and ensure the viability of concepts and requirements.
DOD has also decided to take a more comprehensive approach to  estimating SBR’s life-cycle costs. According to the SBR program director,  this marks the first time DOD has willingly presented all related costs to  develop, acquire, produce, maintain, operate, and sustain the system. DOD  officials stated that they wanted to identify not just direct costs, but also  costs for associated infrastructure such as the costs related to modifying  the ground system that will be used to support SBR as well as other  systems. According to DOD, about $8 billion of the $28.6 billion life-cycle  cost estimate represents costs that in the past, would not have been  included in space program total cost estimates. Taking steps to more  comprehensively identify SBR and SBR related costs is a positive step and  will help DOD manage its portfolio of space programs.
Although DOD hopes to rely on TCA to support SBR data transmissions, it  is taking a proactive approach to identify and assess the viability of TCA  alternatives. First, in April 2004, DOD awarded two 2-year contracts for  concept development efforts that call for the identification of alternatives  to TCA. For each alternative identified, the contractor is to conduct an  assessment of the cost, risk, and effect on SBR’s performance. DOD  officials told us that when SBR initiates product development in 2006, it  would know whether TCA will be available to support SBR or whether to  pursue a TCA alternative. In addition, DOD also awarded two contracts  totaling $510,000 for a yearlong study to propose several alternatives to  TCA capable of supporting SBR’s communications requirements and to  analyze the viability of such alternatives. These actions have put DOD in a  better position to ensure the program is successful.
The two 2-year contracts that DOD awarded in April 2004 also require that  at least two different viable SBR operations concepts be proposed. DOD is  expecting each contractor to fully develop the alternative operations  concepts. These alternative concepts could involve using unique radar  processing techniques. According to DOD, it will work with each of the  contractors to pare down the alternatives to a single best concept for each  contractor. For the remainder of the contract performance period, the  contractors would focus their attention on fleshing out the details  associated with these concepts. This approach will put DOD in a better  position when the time comes to select a single contractor to design the  SBR system.
section_title: Technologies Will Not Be Mature at Product Development Start
paragraphs: DOD officials have said that SBR will likely be the most technically  challenging, software-intensive, and complex space system ever built by  DOD. The two key pieces of hardware needed to give SBR a radar  capability from space—the electronically scanned array (which steers the  radar signal to an area of interest) and the on-board processor (the  radar-processing unit aboard SBR)—face the highest amount of risk. The  electronically scanned array can scan multiple areas of interest virtually  simultaneously, allowing for simplified satellite design over conventional  technology offering mechanical slew radar. The on-board processor is  expected to allow the processing radar data to assure the timely and  thorough delivery of imagery data that will be downlinked for  transmission to the warfighter.
To minimize the potential for technology development problems after the  start of product development, DOD uses an analytical tool to assess  technology maturity for many weapon system acquisition programs.  Called Technology Readiness Levels (TRL), this tool associates a TRL with  different levels of demonstrated performance, ranging from paper studies  to actual application of the technology in its final form. The value of using  a tool based on demonstrated performance is that it can presage the likely  consequences of incorporating a technology at a given level of maturity  into a product’s development, enabling decision makers to make informed  choices. Our previous reviews have found the use of TRLs, which range  from 1 to 9, to be a best practice. (See app. I for a description of the  TRL levels.)
The critical technologies that will support the SBR program currently  range from TRL 3 to 5. A TRL 3 means that most of the work performed so  far has been based on analytical and laboratory studies. At a TRL 5, the  basic technology components are integrated and tested in a simulated or  laboratory environment. Table 2 shows the current TRL for each of SBR’s  critical technologies and the expected TRL at product development start  in 2006. In general, the program office’s key risk reduction efforts are  scheduled to mature these technologies to TRL 5 by the middle of  fiscal year 2006. These efforts include the awarding of research and  development contracts to three payload contractors for efforts to continue  to develop and mature these components (the electronically scanned array  and on-board processor). The period of performance of each contract is  about 2.5 years.
To mature the electronically scanned array and on-board processor  technologies from a TRL 3/4 to 5, the contractors plan to conduct various  developmental and integrative tasks in about 3 years. For example, one  contractor plans to conduct 18 tasks to develop the electronically scanned  array and 8 tasks to integrate the on-board processor with other system  components. In addition, the development of the integrated circuits and  programmable microcircuits that support the on-board processor  requires extensive tests and evaluations and the radiation-hardening  requirement further complicates the development. Given the challenges  of the state-of-the-art technologies being developed and the algorithms  involved, the testing programs must be rigorous and transparent and the  results fully documented. We have determined that the time allotted to  mature the SBR technologies to TRL 5 is ambitious given the tasks that  need to be accomplished. Furthermore, the development of the signal  processing algorithms and communications downlink involves significant  software development. Based on our past experience of software  assessments in other programs, the establishment of a structured testing  regime for software development has always been underestimated.
By planning to start product development in fiscal year 2006 with  technologies at TRL 5, DOD is very likely to continue designing the  system and to conduct other program activities at the same time it  builds representative models of key technologies and tests them in an  environment that simulates space conditions (such as a vacuum chamber).  This approach is common with DOD space acquisitions but has a  problematic history. Our past work has shown that it can lead to  significant cost and schedule increases because of the need to fix  problems later in development. A continuing problem is that software  needs are poorly understood at the beginning of a program. We have  previously recommended that DOD not allow technologies to enter into a  weapon system’s product development until they are assessed at a TRL 7,  meaning a prototype has been demonstrated in an operational  environment. DOD has accepted lower TRL thresholds for space  programs because testing in an operational environment—in space, for  example, or even in a relevant environment—is difficult and costly.  However, DOD’s new space acquisition policy does not identify what the  minimum TRL level should be before starting product development for  space programs, how risks should be mitigated if technologies are  included in programs without full testing, or how lower TRL levels affect  the confidence of cost and schedule estimates. Moreover, the policy does  not address the option of maturing technologies outside a program and  pulling them in once they prove to be viable.
One way to mitigate technology risk is to rely on backup technologies,  should newer technologies prove to be problematic during product  development. According to DOD officials, there are backup technologies  that are more mature for each of SBR’s critical technologies. The backups  are the same technologies but rely on a previous and more mature version.  Using previous versions of these technologies would result in a lower level  of desired performance—such as a reduced area collection rate, a  reduction in the total number of targets collected per satellite per day,  increased product delivery time frames to the user, an increased weight of  the spacecraft, and higher cost. For example, more mature versions of the  electronically scanned array exist and if used, would result in a reduction  in its performance level. In addition, some previous versions of SBR  technologies have not been demonstrated or tested in space. But  according to DOD officials, even with backup technologies, the total  performance of the SBR system can be maintained through systems  engineering trades. DOD says it has been able to leverage some of the key  technologies (such as the electronically scanned array) that were under  development during the previous effort, Discoverer II, to demonstrate a  space-based radar capability.
section_title: Communications Infrastructure May Not Be Ready in Time to Support SBR
paragraphs: Current plans call for TCA to transmit SBR’s large volume of data to  ground-, air-, ship-, and space-based systems. However, one of TCA’s  primary components, the Transformational Satellite—which will use  technologies that DOD has never before tried in space—is facing  uncertainties in its scheduled 2011 launch. DOD started product  development for the Transformational Satellite in December 2003 even  though technologies were immature. If the Transformational Satellite  falters but SBR launches as expected in 2012, then DOD will have a fully  operational, new-generation satellite that is missing its primary means of  data transmission. Recognizing the challenges, DOD is to decide by  November 2004 whether to move forward or delay the Transformational  Satellite’s acquisition program and instead procure another Advanced  Extremely High Frequency satellite, which already are under development  and are based on mature technologies.
Our analysis shows that alternatives to TCA may involve a greater reliance  on processing aboard the SBR satellites, thereby increasing software  development efforts. This approach would reduce the volume of data  requiring transmission, allowing conventional satellite systems, such as  the Advanced Extremely High Frequency satellites, to handle the  transmission. Another likely alternative is to have SBR satellites transmit  only selected portions of data, again, so that the Advanced Extremely High  Frequency satellite could handle the lower volume of information. Finally,  a dedicated system of satellites could be fielded for the sole purpose of  transmitting SBR data, significantly increasing program cost and raising  affordability issues. Currently, DOD is working closely with officials from  the Transformational Satellite program office to evaluate the relative  merits of various alternatives and to document the interfaces needed  between SBR and the Transformational Satellite for each alternative.  During the course of our audit work, SBR program officials met weekly  with the Transformational Satellite program’s integrated product teams  and were coordinating efforts on a memorandum of agreement on  requirements development, joint engineering practices, and studies of   air- and space-based options.
section_title: SBR’s Cost Estimate Unlikely to Be Realistic Because of Multiple Uncertainties
paragraphs: Based on a notional constellation of nine (plus one spare) satellites  operating in low-earth orbit, an independent cost assessment in 2003 put  SBR’s cost at the $28.6 billion mark, making SBR the most expensive DOD  space system ever built. When this initial cost estimate is revised in 2006,  before SBR’s product development starts, DOD is to have decided a  number of issues, such as how many satellites are to be acquired, what  their capabilities will be, and at what altitude(s) the satellites are to  operate. This system refinement allows DOD to develop a more realistic  total system cost estimate—a critical knowledge point if a successful  match between requirements and resources is to be made. However, if  DOD begins product development with less than mature technologies and  without knowing the availability of TCA, accurate cost estimates for SBR  will be much more difficult to prepare. We have previously reported that  improving the reliability of cost estimates is critical and affords DOD  decision makers with the appropriate information to decide whether a  weapon system is worth the overall investment and whether the time is  right to proceed with such an investment. Once a total cost is known, DOD  needs to secure the funding so it can design, produce, operate, and sustain  the system.
DOD may also lack knowledge needed to make a corporate-level decision  as to how much it should invest in SBR versus air platforms with similar  capabilities at the time it begins the SBR acquisition program. In  November 2003, the Air Force completed an analysis of alternatives (AOA)  for SBR, which was supposed to evaluate whether space- or air-based  radar platforms (such as manned and unmanned aircraft with radar  capabilities) or a combination of both are better suited for tracking  moving targets on land or at sea and analyze the capabilities and costs of  each suitable option. However, DOD officials raised a concern that the  AOA only weighed the merits of various space-based solutions. The Air  Force decided to undertake a follow-on study to explore the optimal ways  to gather information on ground moving targets from radars based in  space versus air. The plan is to also use this follow-on study as part of  DOD’s preparations for submitting a fiscal year 2006 budget to Congress to  secure funding for SBR and other radar systems on air platforms. A more  thorough AOA, completed before the start of the study phase, might  conceivably have determined that air-based radar could provide many or  most of the capabilities promised by space-based radar but at a fraction of  the cost. Moreover, this type of analysis could help DOD officials better  decide whether SBR should be initiated at a later date, when critical  technologies will have been matured, or when the communications  infrastructure to support SBR will be available.
DOD officials have mentioned other ongoing studies that are examining  the optimal mix between SBR and other platforms for specific capabilities,  such as ground-moving target indication. However, it is unclear as to the  extent these studies will be factored into the SBR product development  start decision.
section_title: Conclusions
paragraphs: DOD has recently embarked on a discovery and exploration phase for its  SBR program. During this period, it is critical for programs to work toward  closing knowledge gaps about requirements, technologies, funding, and  other resources so they can be positioned to succeed when DOD decides  to commit to making significant investments. For SBR, this would mean  testing technologies to the point of knowing they can work as intended  before starting program development, securing agreement on  requirements with the intelligence community, and fully assessing the  cost and benefits and risks of relying on TCA and alternatives, including  different mixes of air and space-based platforms. DOD is taking positive  steps toward this end, but without maturing critical technologies or  securing formal commitment on requirements, it will not be able to assure  decision makers that the program can be completed within cost and  schedule estimates. Should DOD decide to proceed on a path that leaves  open important questions, including those about technologies, then it  should do so with (1) assessments of technical risks and what additional  resources (in terms of time and money) would be needed to address  problems that may occur during development as well as what trade-offs  would need to be made with other space programs should DOD need to  invest additional resources in SBR, and (2) a formal commitment for  providing additional resources if problems do occur.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that DOD and its intelligence community partners obtain  the additional knowledge they need to determine whether and when to  begin the SBR acquisition program, we recommend that the Secretary of  Defense direct the Under Secretary of the Air Force to:    Direct the SBR Executive Steering Group to ensure that outcomes from  the requirements management process are formally approved and  documented as the program proceeds through product development  before an investment is made beyond technology and concept  development for the SBR program. This group should identify how key  document review comments are to be obtained and addressed and identify  all the officials and/or organizations responsible for taking specific  approval action. In addition, the group should establish a mechanism and  time frame for providing approval/disapproval. Finally, the group should  establish a formal mechanism for addressing unresolved issues as they  relate to key program documentation, as well as how changes to approved  requirements will be assessed.
Modify DOD’s space acquisition policy to reflect protocols for setting  requirements when DOD undertakes programs in partnership with the  intelligence community.
Delay approval to commit funding to product development (key decision  point B) for SBR until technologies have been demonstrated in a relevant  or operational environment so DOD can more reliably estimate the  resources needed to complete the program. If the Under Secretary  determines that the program should go forward with less mature  technologies, then we recommend that the Under Secretary (1) undertake  an assessment of the backup technologies that may lessen capability and  add cost to the program and the additional time and money that may be  required to meet SBR’s performance objectives to address those risks,  (2) undertake an assessment of trade-offs that may need to be made with  other space programs to assure SBR’s successful outcome, and (3) secure  formal commitments from DOD to provide funding for total estimated  costs as well as costs estimated to address potential technical risks.    Strengthen the ongoing study of options for tracking ground-moving  targets by ensuring this work includes: (1) a full range of air and space  options; (2) measures of effectiveness that would help justify choosing  SBR over air options; and (3) the possibility of having to rely on TCA  alternatives for space options. This work should also consider the results  of analyses being conducted by other DOD entities on tracking ground- moving targets.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Deputy  Under Secretary of Defense (Programs, Requirements, and Resources)  within the Office of the Under Secretary of Defense for Intelligence.  DOD generally agreed with our findings and our recommendation to  strengthen its study of SBR alternatives. DOD partially agreed with our  recommendations to strengthen its requirement setting process for SBR  and to demonstrate SBR technologies in a relevant or operational  environment before committing to product development. DOD did not  agree with our recommendation to modify its acquisition policy to  strengthen requirements setting.
In commenting on our recommendations, DOD agreed in principle with  the need to extensively define, analyze, and validate requirements for SBR,  but it did not believe this necessitated a different requirements setting  process than the one that is in place for SBR or changes to its space  acquisition policy or that additional controls were needed within the  program’s study phase. To clarify, our recommendation was not intended  to construct a new requirements setting process or supplant activities  undertaken by the Joint Requirements Oversight Council or the Mission  Requirements Board, as DOD asserts. Rather, we recommend that DOD  build on the positive requirements setting procedures it has already put in  place by instituting controls and mechanisms that ensure transparency,  discipline, and accountability with requirements setting. As noted in our  report, while DOD has taken steps to increase senior leader and  stakeholder involvement in requirements setting, it is not fully  documenting commitments made during the requirements approval  process, nor has it established a process to resolve potential  disagreements that may occur after approval. It is important that this  discipline be instilled in the study phase and throughout the SBR effort. As  noted in previous reports, many space programs have not been executed  within cost and schedule estimates because of an inability to establish firm  requirements and to make and enforce trade-off decisions. For SBR, the  potential for difficulty in requirements setting is higher because of the  distinct needs of the intelligence community and DOD’s desire to integrate  SBR with other radar platforms. Moreover, revising the acquisition policy  to clearly communicate protocols that should be followed when DOD  undertakes space programs in the future involving diverse users—such as  the intelligence community, military services, industry, and/or other  agencies—would further help DOD to rationalize requirements setting and  to solidify relationships with users, which DOD reported was a top SBR  management issue.
In regard to our recommendation to delay product development until SBR  technologies are sufficiently matured, DOD stated that it has planned for  critical and most other enabling technologies to be demonstrated at least  at the component level in a relevant environment on the ground. DOD also  stated that where technically feasible and fiscally feasible, it planned to  pursue on-orbit demonstrations. It also stated it has taken some actions  relating to our recommendation such as accounting for technical risks in  the costing and budgeting process. DOD asserted, however, that our  recommendation encourages pursuit of older, more proven, technologies.  We recommended that DOD pursue relevant or operational environment  demonstrations of all critical technologies and even an integrated system  before committing to a formal acquisition program because this practice  enables a program to align customer expectations with resources, and  therefore minimize problems that could hurt a program in its design and  production phase and drive up costs and schedule. Further, we agree that  continuing to develop leading edge technology is important for space  system capabilities. However, history has shown and we have repeatedly  reported that conducting technology development within a product  environment consistently delays the delivery of capability to the user, robs  other programs of necessary funds through unanticipated cost overruns,  and, consequently, can result in money wasted and fewer units produced  than originally stated as necessary. A technology development  environment is more forgiving and less costly than a delivery-oriented  acquisition program environment. Events such as test “failures,” new  discoveries, and time spent in attaining knowledge are considered normal  in this environment. Further, judgments of technology maturity have  proven to be insufficient as the basis for accurate estimates of program  risks as it relates to cost, schedule, and capability. Lastly, our report noted  that DOD was taking positive actions to gain knowledge about technology  readiness, including strengthening systems engineering, undertaking risk  assessments, and assessing various technical concepts. Given the potential  cost of the program, our recommendation focuses on taking these steps  further by assessing what trade-offs may need to be made with other space  programs should the program encounter technical problems that require  more time and money than anticipated and securing commitments to  provide resources needed to address such problems.
DOD’s detailed comments are provided in appendix II.
We plan to provide copies of this report to the Secretary of Defense, the  Secretary of the Air Force, and interested congressional committees.
We will make copies available to others upon request. In addition, the  report will be available on the GAO Web site at http://www.gao.gov.
If you or your staff has any questions concerning this report, please  contact me at (202) 512-4841 or Arthur Gallegos at (303) 572-7368. Other  key contributors to the report include Tony Beckham, Cristina Chaplain,  Lily Chin, Maria Durant, Nancy Rothlisberger, and Hai V. Tran.
section_title: Appendix I: TRL Scale for Assessing Critical Technologies
paragraphs: Appendix I: TRL Scale for Assessing Critical  Technologies  Lowest level of technology readiness. Scientific research begins to be translated  into applied research and development (R&D). Examples might include paper  studies.
Invention begins. Once basic principles are observed, practical applications can be  invented. Examples are still limited to paper studies.
Active R&D is initiated. This includes analytical studies and laboratory studies to  physically validate analytical predictions of separate elements of the technology.  Examples include components that are not yet integrated or representative.
Basic technological components are integrated to establish that they will work  together. This is relatively “low fidelity” compared to the eventual system. Examples  include integration of “ad hoc” hardware in a laboratory.
Fidelity of breadboard technology increases significantly. Basic technological  components are integrated with reasonably realistic supporting elements so that the  technology can be tested in simulated environment. Examples include “high fidelity”  laboratory integration of components.
Representative model or prototype system, which is well beyond the breadboard  tested for level 5, is tested in a relevant environment. Represents a major step up in  a technology’s demonstrated readiness. Examples include testing a prototype in a  high-fidelity laboratory environment or in a simulated operational environment.
Prototype near, or at, planned operational system. Represents a major step up from  TRL 6, requiring demonstration of an actual system prototype in an operational  environment such as an aircraft, vehicle, or space.
Technology has been proven to work in its final form and under expected  conditions. In almost all cases, this TRL represents the end of true system  development. Examples include Developmental Test and Evaluation of the system  in its intended weapon system to determine if it meets design specifications.
Actual application of the technology in its final form and under mission conditions,  such as those encountered in Operational Test and Evaluation. Examples include  using the system under operational mission conditions.
section_title: Appendix II: Comments from the Department of Defense
subsections: section_title: GAO’s Comments
paragraphs: The following are GAO’s comments on the Department of Defense’s letter  dated June 29, 2004.  1.  DOD stated that DOD does not require formal approval for the  concepts of operations from the Joint Requirements Oversight Council  or the Mission Requirements Board, but noted that the Joint  Requirements Oversight Council communicated agreement in a memo.  As we reported, the Under Secretary of the Air Force requested that  both DOD and the intelligence community approve the initial  capabilities document and concept of operations in light of the  complexity of SBR’s acquisition process, the partnership with the  intelligence community, and the proposed integration with other radar  platforms.  2.  DOD stated that it is not engaged in a partnership with the intelligence  community on SBR, as our report states. Specifically, DOD stated that  SBR is wholly funded in the defense budget and that a programmatic  commitment with the intelligence community does not exist. DOD’s  SBR System Acquisition Strategy was signed by senior-level officials  from DOD, National Reconnaissance Office, and the National  Geospatial-Intelligence Agency and approved on January 14, 2004. This  strategy states that the Air Force, in close partnership with the  National Reconnaissance Office and National Geospatial-Intelligence  Agency, is responsible for leading development of an SBR capability.  This strategy further identifies the responsibilities related to SBR that  each mission partner (National Reconnaissance Office and National  Geospatial-Intelligence Agency) is supposed to carry out. We disagree  with DOD’s assertion that these organizations must provide funding to  SBR in order to consummate a partnership. Because SBR is being  justified on the basis of the system’s ability to provide intelligence,  surveillance, and reconnaissance products to both DOD and the  intelligence community, the part of the budget used is not relevant to  our finding.  3.  To clarify, we did not recommend that DOD pursue lower risk  technologies that would result in lower levels of desired performance.  Instead, we reported that DOD might have to resort to using backup  technologies if the current ones prove to be problematic during  product development. We recommended that DOD should assess the  cost to the program of having to use the backup technologies DOD has  already identified in terms of time and money.
section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs