GPS: Actions Needed to Address Ground System Development Problems and User Equipment Production Readiness
Published: 2015-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The satellite-based GPS provides positioning, navigation, and timing data to users worldwide. The Air Force is modernizing the satellite, ground control, and user equipment segments to enhance GPS performance.
The Senate and House Armed Services Committee reports accompanying bills for the National Defense Authorization Act for Fiscal Year 2015 included provisions for GAO to review the status of OCX development and DOD's efforts to field M-code signal capability. This report addresses (1) the extent to which DOD is meeting cost, schedule, and performance requirements for OCX; (2) the progress DOD is making in delivering M-code capable MGUE by the end of fiscal year 2017; and (3) the challenges DOD faces in synchronizing the development of GPS III, OCX, and MGUE to deploy M-code.
To conduct this work, GAO analyzed program documents such as acquisition strategies; reviewed oversight reporting; assessed constellation reliability metrics; and interviewed officials from DOD programs and contractors.
section_title: What GAO Found
paragraphs: The Air Force has experienced significant difficulties developing the Global Positioning System (GPS) next generation operational control system (OCX) and consistently overstated progress to the Office of the Secretary of Defense (OSD) compared to advisory independent assessments it received. It needs $1.1 billion and 4 years more than planned to deliver OCX due to poor acquisition decisions and a slow recognition of development problems. The Air Force began OCX development in 2010 prior to completing preliminary development reviews in contrast with best acquisition practices. It accelerated OCX development in 2012 to meet optimistic GPS III satellite launch timeframes even as OCX development problems and costs grew, and then paused development in 2013 to address problems and resolve what it believed were root causes. However, as the figure below shows, OCX cost and schedule growth have persisted due in part to a high defect rate, which may result from systemic issues. Further, unrealistic cost and schedule estimates limit OSD visibility into and oversight over OCX progress.
The Air Force is implementing the military GPS user equipment (MGUE) program to develop for the military services GPS receiver cards capable of receiving the military-code (M-code) signal—which can help users operate in jamming environments. The Air Force has revised MGUE's acquisition strategy several times in its quest to develop the cards. Even so, the military services are unlikely to have sufficient knowledge to make informed procurement decisions starting in fiscal year 2018 because operational testing that provides valuable information about MGUE performance will not be complete until fiscal year 2019.
The current GPS constellation has proven to be much more reliable than the Air Force predicted when GAO last reported on it in 2010. Given delays to OCX, the Air Force has prepared contingency plans for sustaining the GPS constellation, but these plans may not deliver the full range of GPS capability. Initial M-code capability will not be available until OCX delivery in mid-2019 at the earliest and full M-code capability is likely at least a decade away—once the services are able to deploy MGUE receivers in sufficient numbers. Until the OCX program trajectory is corrected, additional delays to it may likely pose significant risks to sustaining the GPS constellation and delivering GPS capability.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD obtain a more robust independent assessment of OCX to identify and resolve root causes, and ensure MGUE design is stable to inform testing and procurement decisions. DOD concurred on OCX but stated that actions taken to date are sufficient. DOD partially concurred on MGUE. GAO believes all recommended actions are necessary to address systemic problems.

section_title: Letter
paragraphs: The satellite-based Global Positioning System (GPS) provides  positioning, navigation, and timing (PNT) data to users worldwide.  Besides being a household term and synonymous with navigation, GPS is  an essential U.S. national security asset and a key component of  economic growth, national infrastructure, and transportation safety. The  Department of Defense (DOD)—specifically, the Air Force—develops and  operates the GPS system, which consists of three segments: space,  which comprises a constellation of PNT satellites orbiting the earth— currently, approximately 40; ground control, which primarily consists of  software and operates the satellites as well as monitors and corrects  signal data; and receivers, which help civil and military users employ GPS  signals to determine their location. For about the past 8 years, the Air  Force has been in the process of modernizing all three segments to  enhance GPS performance and security. This modernization effort is  divided into three major programs: GPS III, which is developing a new  generation of satellites; next generation operational control system  (OCX), which will replace the existing ground system to operate most of  the current and future satellites; and, for DOD users, military GPS user  equipment (MGUE), which will provide the military services with new  receivers that can receive GPS signals—particularly, the advanced  military code (M-code) signal—in hostile jamming or challenging  environments.
The Senate and House Armed Services committee reports accompanying  bills for the National Defense Authorization Act for Fiscal Year 2015  included provisions for GAO to review the status of OCX development  and DOD’s efforts to field M-code capability, respectively. This report  assesses the (1) extent to which DOD is meeting cost, schedule, and  performance requirements for OCX, (2) progress DOD has made in  delivering M-code capable MGUE by the end of fiscal year 2017, and (3)  challenges DOD faces in synchronizing the development of GPS III  satellites, OCX, and MGUE to deploy M-code. To conduct our work, we  analyzed program cost and schedule baselines, strategies, schedules,  software development plans, earned value metrics, and other documents.  We also interviewed officials from the government program offices, prime  contractors, Defense Contract Management Agency (DCMA), the Office  of the Director, Operational Test and Evaluation (DOT&E), each of the  military services, and Air Force Space Command, and analyzed program  and management briefings and GPS constellation reliability parameters.  We examined only OCX development contract costs rather than full  program acquisition costs because the latter includes prior expenditures  on technology development as well as annual management and GPS  enterprise integration support services. We did not include an  assessment of cost, schedule, or performance for GPS III satellites in the  scope of our work. In addition, we did not review MGUE Increment 2  because that development effort has not yet reached its technology  development milestone. Appendix I contains a more detailed description  of our scope and methodology.
We conducted this performance audit from August 2014 to September  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPS is a global positioning, navigation, and timing system consisting of  space, ground control, and user equipment segments that support the  broadcasts of military and civil GPS signals. These signals each include  positioning and timing information, which enables users with  appropriately-equipped GPS receivers to determine their position,  velocity, and time, 24 hours a day, in all weather, worldwide. GPS is used  by all branches of the military to guide troop movements, assist with  logistics support and situational awareness, and synchronize  communications networks. In addition, weapon systems, including  munitions, are guided to their targets by GPS signals, and GPS is used to  locate military personnel in distress. Early in the development of GPS, its  scope was expanded to include complementary civil capabilities such as  civil, maritime, and land navigation.
subsections: section_title: GPS System Description
paragraphs: Space, ground control, and user equipment segments are needed to take  full advantage of GPS capabilities. The GPS space segment consists of a  constellation of satellites that broadcast encrypted military signals and  civil signals. In recent years, because numerous satellites have lasted  longer than anticipated, the constellation has grown well beyond the  minimum requirement of 24 satellites to approximately 40 satellites of  various generations, with 8 in residual status. The satellites are operated  by a master control station that regularly updates navigation signals on  the satellites. Using these navigation signals, GPS military and civilian  user equipment receivers determine a user’s location. Figure 1 below  illustrates how GPS satellites, ground control, and user equipment  function together as an operational system.
The GPS ground control segment primarily consists of software deployed  at a master control station at Schriever Air Force Base, Colorado, and at  an alternate master control station at Vandenberg Air Force Base,  California. The ground control software is supported by 6 Air Force and  11 National Geospatial-Intelligence Agency monitoring stations located  around the globe along with four ground antennas with uplink capabilities.  Information from the monitoring stations is processed at the Master  Control Station to determine satellite clock and orbit status.
The GPS user equipment segment includes military and civilian GPS  receivers. These receivers determine a user’s position and time by  calculating the distance from four or more satellites using the navigation  signals on the satellites to determine its location. Military GPS receivers  are designed to utilize the encrypted military GPS signals that are only  available to authorized users, including military and allied forces and  some authorized civil agencies. Civilian—including commercial— receivers use the civil GPS signal, which is publicly available worldwide.
section_title: Current Modernization Efforts
paragraphs: In 2000, DOD began an effort to modernize the space, ground control,  and user equipment segments of GPS to enhance the system’s  performance, accuracy, and integrity. To that end, the Air Force is now in  the process of developing a new generation of GPS III satellites, OCX,  and M-code capable MGUE receivers.
GPS III satellites are planned to supplement and eventually replace the  constellation of GPS satellites now in orbit; these satellites consist of  multiple versions or generations developed and launched over the years.  The first GPS III satellite was originally expected to be available for  launch in April 2014; however, due to development problems it is now  expected to be ready for launch in May 2017. A complete GPS III satellite  has not yet been tested, and the program is now rebaselining its cost  estimates as a result of the schedule delay and associated increased  costs.
The GPS ground control segment is being modernized under the OCX  program. OCX is required because the existing GPS control system,  Operational Control System (OCS) cannot control—and therefore  enable—the modernized features of the two latest generations of GPS  satellites—IIR-M and IIF—currently in orbit. The Air Force plans to  develop OCX in blocks, with each block delivering upgrades as they  become available. Block 0 is intended to support the launch and initial  testing of GPS III satellites; block 1 is designed to command and control  the GPS II and III satellites and basic modernized signals; and block 2 is  to enable the full modernized M-code signal and support, monitor, and  control additional navigation signals. OCX block 1 is needed to  incorporate GPS III satellites into the operational constellation to sustain  required levels of GPS signal coverage, because the legacy OCS system  cannot support the GPS III satellites. OCX is also required to enable  military and civil use of modernized GPS signals. In particular, the military  cannot use M-code signals for enhanced warfighting until OCX block 1 is  delivered.
The Air Force began the OCX program in 2007 with a technology  development phase, referred to as phase A. The Air Force awarded  phase A contracts, for approximately $160 million each, to Northrop  Grumman in Redondo Beach, California, and Raytheon Intelligence and  Information Systems in Aurora, Colorado to produce competitive  prototypes. Following the competitive down-select between these firms’  prototypes, the Air Force awarded the development contract to Raytheon  in February 2010. This $886 million contract covered the development of  OCX blocks 1 and 2 (as discussed in this report, block 0 was added as a  contract modification later), with an option to begin preliminary work on  blocks 3 and 4 which are to provide additional capabilities to support  follow-on, upgraded versions of GPS III satellites.
At the formal start of development (milestone B), the Air Force estimated  the total OCX acquisition costs—including other costs such as  expenditures on technology development prior to 2010 and annual  management support and enterprise integrator services—at $3.5 billion.  According to Air Force documentation, enterprise integrator services are  required to ensure GPS enterprise coordination among the ground,  space, and user equipment segments. These costs do not include funding  contributions from civilian agencies to support OCX; according to Air  Force documentation, the Department of Transportation and other  agencies are to provide resources to DOD to develop and operate GPS  civil capabilities. For the purpose of this report, OCX development costs  refer to the costs of the development contract with Raytheon.
Military GPS receivers are also being modernized under the MGUE  program. The Air Force was directed by the Assistant Secretary of  Defense for Networks and Information Integration in August 2006 to  develop M-code capable GPS receiver cards to meet military services’  needs. In January 2011, the Ike Skelton National Defense Authorization  Act for Fiscal Year 2011 directed that DOD not obligate or expend funds  to procure GPS user equipment after fiscal year 2017 unless that  equipment is capable of receiving M-code. Figure 2 below shows an  illustration of a MGUE receiver card.
The modernized receiver cards are to provide U.S. forces with enhanced  position, navigation, and time capabilities, while improving resistance to  existing and emerging threats, such as jamming. The Air Force plans to  develop MGUE in two increments. The first increment, now under way  and expected to cost about $1.7 billion, utilizes three prime contractors.  All three contractors—L-3 Interstate Electronics Corporation, Raytheon  Space and Airborne Systems, and Rockwell Collins—are developing  receiver cards for ground environments. Raytheon and Rockwell Collins  are also developing combined aviation/maritime receiver cards for use in  air and sea environments. The Air Force plans to build on the work  conducted in Increment 1 to develop a more compact receiver card in  Increment 2 that can be used when size, weight, and power need to be  minimized and that can serve as an “engine” for future military GPS  receivers. At this time, the Air Force is also exploring the possibility of  using Increment 1 technologies to support munitions, handheld devices,  and space applications that it previously anticipated supporting with  Increment 2.
The Air Force initially began development of M-code in fiscal year 2003. It  then transitioned that work to the modernized user equipment (MUE)  program in 2006. According to GPS program updates, MUE was troubled  by issues such as underestimation of software complexity, longer than  anticipated software development time, and more difficult than expected  software and hardware integration. These issues, among others, resulted  in significant cost and schedule growth. The MUE program yielded  aviation and ground prototypes of M-code-capable components, cost  about $498 million, and spanned about 7 years, ending in 2013.
Concurrent with and based on MUE development work, the Air Force  initiated the MGUE program in 2011. While DOD policy on GPS user  equipment and procurement in 2006 indicated the Air Force was to  develop MGUE to production-ready status, the Air Force currently plans  to develop MGUE Increment 1 ground and aviation/maritime receiver  cards to the point of production representative test articles. It then  intends for the MGUE program office to provide funding to the military  services so that they can acquire, integrate, and operationally test the  receiver cards on service-specific “lead platforms.” These platforms are  expected to serve as pathfinders for the military services’ ground,  aviation, and maritime environments and are currently designated by the  services as follows: Army–Defense Advanced GPS Receiver (DAGR)  Distributed Device (D3) onboard a Stryker ground combat vehicle; Air  Force–B-2 Spirit aircraft; Navy–DDG-51 Arleigh Burke Destroyer hosted  by GPS-based Positioning, Navigation, and Timing Service (GPNTS); and  Marine Corps–Joint Light Tactical Vehicle (JLTV). Once operational  testing is complete, the military services will then be responsible for  procuring MGUE Increment 1 for their weapons systems.
DOD indicates that the military services anticipate procurement of about  207,000 MGUE Increment 1 receiver cards.  Historically, full fielding of  user equipment has lagged behind the anticipated availability of GPS  satellite and ground systems. For example, in 2009 we reported that the  Air Force expected GPS satellite and ground systems to be available in  2013 to transmit and process the M-code signal, but that fielding of M- code equipment to all designated military users would not be complete  until 2025.
section_title: Poor Acquisition Decisions and a Slow Recognition of Development Problems Led to Significant OCX Cost Increases and Schedule Delays
paragraphs: OCX development contract costs have more than doubled since the  contract was awarded in February 2010, increasing by approximately  $1.1 billion to $1.98 billion, and the program’s schedule has roughly  doubled over estimates at contract award. The Air Force awarded the  contract to begin OCX development but did not follow key acquisition  practices such as completing a preliminary design review before  development start as called for by best practices. In addition, key  requirements, particularly for cybersecurity, were not well understood by  the Air Force and contractor at the time of contract award. The contractor,  Raytheon, experienced significant software development challenges from  the onset, but the Air Force consistently presented optimistic  assessments of OCX progress to acquisition overseers. Figure 3 below  shows select key events related to the OCX program since the  development contract was awarded in 2010.
Further, the Air Force complicated matters by accelerating OCX  development to better synchronize it with the projected completion time  lines of the GPS III satellite program, but this resulted in disruptions to the  OCX development effort. As Raytheon continued to struggle developing  OCX, the program office paused development in late 2013 to fix what it  believed were the root causes of the development issues, and  significantly increased the program’s cost and schedule estimates.  However, progress reports to DOD acquisition leadership continued to be  overly optimistic relative to the reality of OCX problems. OCX issues  appear to be persistent and systemic, raising doubts whether all root  causes have been adequately identified, let alone addressed, and  whether realistic cost and schedule estimates have been developed.  Figure 4 below shows how OCX costs have grown and the schedule  delayed since contract award in February 2010.
subsections: section_title: The Air Force Did Not Follow Key Acquisition Practices When It Initiated OCX Development
paragraphs: The Air Force awarded the OCX development contract to Raytheon in  February 2010 for $886 million; blocks 1 and 2 were forecast for  completion in August 2015 and March 2016, respectively. The  development contract was awarded before completing a milestone B  decision formally authorizing the start of development. In addition, the  program did not complete a preliminary design review (PDR) as called for  by best practices; the Air Force subsequently acknowledged in the OCX  acquisition strategy approved in September 2012 that the contract was  awarded earlier than normal. Figure 5 below depicts key events in DOD’s  typical acquisition process and the corresponding knowledge called for in  GAO’s prior work on best product development practices. As shown in the  figure, our prior work has identified several proven management practices  that, if fully implemented, can help DOD minimize cost overruns by  ensuring programs are established after matching requirements and  resources.
The Air Force did not conduct a milestone B review prior to awarding the  development contract, missing an opportunity to ensure the program  began on a sound foundation. A milestone B review is an important point  in a program where requirements and resources—that is, technology,  design, time, and funding—should be properly matched to make sure the  program can be executed as planned. The Under Secretary of Defense  for Acquisition, Technology, and Logistics (USD AT&L) approved the Air  Force’s request in January 2010 to award the OCX development contract  prior to milestone B. The Air Force and USD AT&L decided to award the  contract early for several reasons. First, the Air Force believed that the 3- year competitive technology development effort from 2007-2010, where  two contractors (Northrop Grumman and Raytheon) worked on proof-of- concept prototypes for OCX, successfully demonstrated initial  functionality and reduced risk. Second, the Air Force wanted to reduce  the costs of carrying two contractors through PDR. Finally, the Air Force  and USD AT&L believed that splitting efforts between two contractors was  slowing progress on the program, and decided that down-selecting to one  contractor prior to milestone B would help accelerate OCX completion.  According to Raytheon officials, a contributing factor was the need to  align OCX development with that of GPS III development, the  development contract for which was awarded 21 months earlier, in May  2008.
In addition, prior to development start, the program did not complete a  PDR—which assesses the maturity of the preliminary design and  confirms that the system is ready to proceed into detailed design with  acceptable risk. As GAO’s best practices work has shown, PDR is a  critical step at which customer needs are balanced with available  resources. Programs that are launched prior to completing a preliminary  design tend to experience more problems compared to programs where  launch occurs after completing a preliminary design. Figure 6 below  depicts how programs generally have better outcomes when they are  launched after conducting relatively more detailed systems engineering:  particularly, completing a preliminary design before formally launching.  Programs launched after only completing a notional design—a general  concept, unconstrained by resources, of what the product will look like  and what it might be capable of—often experienced problems and  needed more time or money than had been estimated at program launch.
OCX program office and Raytheon officials stated that certain OCX  requirements were not well understood at the time of contract award. In  particular, Raytheon officials stated that the company did not understand  the extent to which it would be required to implement Information  Assurance (IA) requirements until as late as 2013, 3 years after beginning  OCX development. The purpose of IA, also referred to as cybersecurity,  is to ensure that DOD systems can resist and continue to operate during  cyber-attacks by managing risks and implementing safeguards. OCX was  required to be compliant with the DOD directive that prescribes IA  policies, responsibilities, and procedures, among other things, for DOD  information technology and defense components. In light of increasing  cyber threats and given that satellite ground systems are the most  vulnerable components to potential attacks, the Air Force plans for OCX  to have an improved IA capability over the current GPS control system.  According to program and Raytheon officials, OCX is going to be one of  the first large-scale programs within DOD to fully implement IA  requirements. In addition, Raytheon officials described cybersecurity  threats as continuously evolving, and that both Raytheon and the Air  Force have had to adapt their interpretation over time of how to meet IA  requirements on OCX development to address changing threats.
Program office and Raytheon officials noted that past DOD acquisition  programs routinely waived some of the IA requirements in prior programs,  and that Raytheon entered OCX development with the expectation that  some IA requirements would be waived as in the past. According to  Raytheon officials, at the time OCX development began, neither the Air  Force nor Raytheon had the experience—and therefore, the knowledge— of developing such a complex IA-intensive program. Given the  importance of GPS to the military and civil communities and with the  increase in cybersecurity threats, the Air Force did not waive any IA  requirements for OCX. Consequently, Raytheon found that it had greatly  underestimated the cost and time to meet these requirements. According  to program officials, most of the requirements issues were resolved in  early 2015.
subsections: section_title: Preliminary Design Review Held More Than a Year after Contract Award
paragraphs: The OCX program held PDR in August 2011—more than a year after  contract award and after the Air Force had spent over $1 billion on the  overall OCX program since 2007 including phase A technology  development—with the Air Force affirming that the OCX architecture and  design was solid and that the program was ready to begin formal system  development. However, at PDR, Raytheon did not prepare—and the Air  Force did not assess—a preliminary design for the entire OCX system  because Raytheon followed an iterative software development process  unlike the traditional, “waterfall” software development process. An  iterative approach is a common industry practice that consists of  developing software in a series of iterations and blocks, where developers  go through multiple cycles of breaking down requirements, and designing,  coding, and integrating software. By contrast, in the traditional waterfall  approach the system is fully designed before coding and testing follow in  a linear sequence. In theory, an iterative approach allows for a balanced  and efficient use of resources when developing complex systems.  Smaller iterative development cycles—as opposed to a lengthy, linear,  waterfall development process—allow for capabilities to more easily be  added incrementally, lessons learned to be incorporated, and early  integration testing to be conducted to minimize cost and schedule risk. To  achieve this, effective systems engineering—the process of deriving and  allocating requirements for individual software iterations—is key. The Air  Force, acknowledging Raytheon’s use of an iterative approach in late  2010, tailored its preliminary design review criteria—which originally were  based on a traditional, waterfall approach—to only review the two  iterations completed by Raytheon at that point. For the remaining six  iterations, Raytheon had only completed the initial allocation of  requirements at a high level. However, as Raytheon encountered  problems in software development, it began deferring difficult work to later  iterations.
Raytheon was experiencing difficulties developing OCX, but the Air Force  presented optimistic assessments of progress to USD AT&L. By October  2011, an independent OSD-chartered review team—one of a series of  independent reviews—warned of severe software development problems,  particularly with mounting deferred work, ineffective integration testing,  and overestimated software productivity rates, and predicted a 15 to 19  month delay to block 1 delivery. According to an official with USD AT&L,  these independent assessments are designed to identify program cost,  schedule, and performance risks and provide feedback to the program  manager and the milestone decision authority. The official further noted  that the assessments were considered helpful, but essentially nonbinding  advice from department experts to the program manager. The  independent team followed up in February 2012 for a more detailed  assessment of OCX and predicted a higher, 24-month slip to block 1.  However, at the annual GPS enterprise review (AGER) with USD AT&L in  April 2012, the Air Force acknowledged the software issues but projected  a shorter, 11-month delay citing various corrective actions already taken,  including greater oversight. This projection proved to be overly optimistic.  In light of the challenges experienced to that point, at the April 2012  AGER, USD AT&L postponed the scheduled milestone B decision for  OCX and directed the program to return for a formal Defense Acquisition  Board (DAB) review within 120 days and to report on progress.
section_title: The Air Force Accelerated OCX Development to Meet Optimistic GPS III Satellite Launch Time Frames Even as OCX Was Experiencing Problems
paragraphs: In the midst of mounting OCX problems, the Air Force disrupted the  ongoing software development effort. As it became apparent that OCX  block 1 would not be ready in time to support the then-projected launch  time frames of the first GPS III satellite (August 2015 and May 2014  respectively), USD AT&L directed the Air Force in January 2011 to  separate out development of the satellite launch and initial testing portion  of the block 1 software. Dubbed block 0, this subset of software is to  contain the capabilities needed to launch and test the initial GPS III  satellites. The Air Force modified the OCX contract in January 2012 to  implement this change, which included bringing forward IA capabilities  originally scheduled for later iterations. Block 0 is a temporary measure  because the Air Force anticipated that some GPS III satellites would need  to be launched before the entirety of block 1 was to be developed. Block  0 does not allow the GPS III satellites to be incorporated into the overall  GPS constellation and used by the military or civil community—which  requires block 1 implementation. Further, this decision to accelerate the  GPS III launch and test capability was based on Air Force assessments  of GPS III progress that appeared overly optimistic, as GAO found at the  time. Nevertheless, according to Raytheon officials, the creation of block  0 and acceleration of some software capabilities caused it to have to  revamp the OCX software development plan 2 years into development— not in accordance with best practices in software development, which call  for stabilizing requirements and design prior to coding. Our prior work in  this area shows that too many changes to requirements can result in  additional, sometimes unmanageable risk.
Meanwhile, Raytheon continued to struggle with developing OCX,  specifically with implementing IA requirements. The contractor was overly  optimistic in its initial estimates of the work associated with incorporating  open source and reused software, and, according to the Air Force, did not  appear to follow IA screening or software assurance processes as  required, for example, incorporating open source software without  ensuring that it was IA-compliant. The Air Force stated that it was not  aware that Raytheon’s software assurance processes were not in  compliance with the OCX software development plan until it performed an  audit of Raytheon’s secure coding process in August 2012. According to  Raytheon officials, however, the development contract did not specifically  address the extent that open source software had to be scanned for IA  compliance. Nevertheless, this led to significant rework and added cost to  remediate the security vulnerabilities and meet IA standards. In addition,  Raytheon’s systems engineering was incomplete, resulting in an inability  to build code as planned and work being consistently deferred to later  iterations. The Air Force noted that it became aware of these systems  engineering issues early in development and took some corrective  actions such as defining the completion of certain components; however,  these actions were focused only on high-priority OCX components.
Nevertheless, in July 2012, the Air Force presented a positive progress  report at the mandated DAB review, stating essentially that all technical  issues had been identified and mitigated, the design fully validated, and  an executable and realistic schedule put in place after a thorough internal  examination of the program. While the program stated that sufficient  margin had been incorporated into the cost and schedule estimates, the  program’s estimates for block 1 delivery—between February and October  2016—were still optimistic compared to the time frames (November 2016  to March 2017) forecast by the independent team in October 2011. The  Air Force formally completed milestone B for OCX in November 2012— more than 2 years after contract award. At this point, the Air Force  developed its first detailed cost estimate for the program as required by  DOD policy, forecasting that OCX development would cost approximately  $1.6 billion, which, according to the Air Force, was its first, formal  realization of the magnitude to which the contract was initially underbid.
section_title: The Air Force Paused OCX Development amid Ongoing Problems and to Identify and Resolve Problems, but Continued to Present Optimistic Assessments to Acquisition Overseers
paragraphs: According to the OCX program office and independent OSD-chartered  reviews, Raytheon’s incomplete systems engineering led to continuous  rework and deferred requirements to later iterations. In addition, the Air  Force made significant changes to certain requirements, particularly with  updating the specifications for OCX’s connections to other government  systems, and M-code signal requirements. As a result, Raytheon officials  estimated that nearly two-thirds of the requirements baseline as of PDR  had changed by mid-2012. In March 2013, an independent OSD-led team  praised the program for the corrective steps taken, but pointed to a  rapidly deteriorating delivery schedule as a result of software  development taking much longer than planned and a high defect rate,  among other factors. The independent team projected block 1  completion between February and July 2018, a schedule slip of 16 to 21  months over the estimate at milestone B. DOD’s contract performance  reviewer, Defense Contract Management Agency (DCMA), highlighted  major technical difficulties in OCX development in its monthly analyses  from the June to October 2013 time frame, including missed software  iteration milestones, concurrent systems engineering and software  development, and a high defect rate, among other issues. In addition,  DCMA reported as early as August 2013 that the Air Force and Raytheon  were going to soon begin an over target baseline (OTB) process where  DOD determines that contract budgets are unrealistic and formally  increases the program’s budget. An OTB is intended to allow for more  realistic budget and work estimates and, therefore, more meaningful  performance measurement against the updated budget.
However, at the September 2013 AGER, the Air Force again presented  an optimistic assessment of the program to USD AT&L, stating that OCX  was on track to meet milestone B cost and schedule estimates, IA  challenges had been identified, and key metrics had been established to  ensure progress. The program projected just a 2-month delay to block 1  (to December 2016).
Meanwhile, the Air Force noted that an excessive amount of rework was  occurring at the time, and it directed Raytheon in November 2013 to  pause development and complete a greater level of systems engineering  for block 1. In December the Air Force and Raytheon began an OTB  process to identify root causes and corrective actions and establish  revised cost and schedule goals for OCX, and notified USD AT&L that  block 1 delivery was likely to slip by approximately 9 months, to  September 2017.
The OTB concluded in June 2014, and identified root causes, many of  which were similar to those identified at previous AGERs and  independent assessments: incomplete systems engineering, inadequate  process discipline, and IA implementation difficulties due to complexity. At  this point, the Air Force’s estimate of the program’s development cost had  grown to $1.7 billion—about 6 percent more than the milestone B  estimate and nearly twice the initial contract estimate in 2010. The Air  Force also added a total of 2 years to the program over its estimate at the  September 2013 AGER with block 1 delivery moved to November 2018.
Despite the ongoing problems with OCX development, the Air Force,  faced with the statutory deadline to deliver M-code capable receivers— and consequently, OCX support for M-code—by fiscal year 2018, added  some uncertainty to the OCX development plan. The Air Force agreed  with Raytheon during the OTB to concurrently develop a portion of block  2 M-code capability during the block 1 effort; however, the Air Force  noted in technical comments on a draft of this report that Raytheon is  under no contractual obligation to do so. Complicating matters, the Air  Force made significant updates to M-code signal requirements in 2014,  contributing to additional OCX requirements changes. At the time of  contract award in 2010, the Air Force had only developed a preliminary  M-code specification, and it took time for the Air Force to fully mature M- code requirements and specify the functionality required of OCX.  According to Raytheon officials, this had a significant effect on the IA  requirements and design of certain components of OCX.
section_title: The OCX Contractor Has Continued to Struggle with OCX Development and a Realistic, Executable Schedule Is Not Yet Evident
paragraphs: The OCX program’s path forward following the conclusion of the OTB  process in June 2014 depended on resolving certain problems in order to  meet revised cost and schedule targets. In particular, the Air Force  assumed that the contractor would (1) resolve software defects quickly,  (2) bring greater discipline to software development processes, and (3)  achieve higher software productivity rates than previously demonstrated.  However, latest available data as of May 2015 shows that the contractor  has not yet resolved these issues, increasing the risk that additional OCX  cost and schedule growth is likely.
subsections: section_title: Defect Rate Remains High and Root Causes May Not Be Fully Identified
paragraphs: A key expectation underpinning the post-OTB baseline was that  Raytheon would be able to sharply reduce the number of new defects and  resolve them within 6 months. However, the opposite took place.  Raytheon uncovered defects at a faster pace than it could resolve them,  causing further delays. By October 2014, just 4 months after the OTB,  DCMA reported that the defect resolution rate was unsustainable,  needing a continual increase in cost and schedule, with the trend likely to  continue. In addition, DCMA found that the majority of defects were  identified during the later phases of software development at which point  it is harder and much more expensive to resolve than if found earlier.  Latest available DCMA reports, as of May 2015, showed that Raytheon  had yet to bring the defect rate within planned levels. Raytheon officials  noted in early July 2015 that the defect discovery rate and backlog have  been greatly reduced as OCX prepares to begin testing on iteration 1.5.  However, data on the number of defects open and resolved show that the  defect backlog as of July 2015 was still more than three times that  predicted in February 2015. In addition, DCMA has pointed out that the  test activities will likely identify new defects, potentially increasing the  backlog further. DCMA further noted that the current defect backlog  consists of more complex and difficult defects, which will require  considerable effort to close compared to those resolved so far.
The persistently high defect rate for OCX may be a result of as-yet  unidentified systemic issues. An October 2014 independent review noted  that, given the difficulty in resolving defects as planned, systemic issues  may remain but that neither the OCX program office nor Raytheon had  conducted detailed engineering assessments to determine if there were  any systemic issues. As GAO’s prior work has shown, effective defect  management requires a realistic schedule in that it takes time to be able  to fully identify, analyze, prioritize, and track defects. Without investing  the time and resources to conduct detailed engineering assessments, the  program cannot know if any systemic issues are causing the persistently  high software defect rate.
section_title: Processes Remain Undisciplined
paragraphs: The high defect rate is a symptom of continued struggles with the root  cause of undisciplined processes at Raytheon, noted to-date by the OCX  program office, independent OSD-chartered review teams, and DCMA.  For example, Raytheon has had difficulty establishing consistent software  development environments—the computer infrastructure including  hardware, operating systems, and databases—across the OCX program.  Developers built each of the environments with different hardware and  operating system versions and settings. According to Raytheon officials,  this was partly because the Air Force’s requirements called for multiple  tailored environments to save on hardware costs, but that this drove  complexity and posed technical difficulties to Raytheon. This meant that  the contractor could not deploy a given software build onto all the  environments without a lot of rework—and consequently, time and  expense. Following the June 2014 OTB, the Air Force and Raytheon  reported that they had taken steps to correct this issue, such as using  automated tools to ensure consistency. However, latest available data, as  of May 2015, showed that inconsistent environments were still  contributing to a high defect rate and consequent cost growth and delays.  Raytheon noted that it had greatly reduced the amount of time required to  deploy software builds onto a given environment to 3 days as of July  2015; however, this was still well short of the 1-day limit called for by the  October 2014 independent review.
Another example of undisciplined processes at Raytheon is its peer  review process. DCMA reported in a November 2014 audit that peer  reviews—shown by our prior work to be a crucial quality assurance  component of software development—were inconsistent and less  effective than planned. The OCX software development plan identifies  peer reviews as an integral part of the software development process, the  main purpose of which is to identify and report defects in software  artifacts—work products such as code, software class libraries, and  design models—as early as possible in the development life cycle.  Effective peer reviews depend, in part, on clear instructions, compliance,  and standards, as well as consistent processes. DCMA reported  Raytheon having problems with all of these items. In its audit, DCMA  noted that the work instructions were often unclear, contradictory, or  contained loopholes; many peer reviews were non-compliant with the  work instructions; and a high percentage of reviews were held “virtually”  where users review artifacts independently instead of during a formal, in- person meeting as presumed in the OCX software development plan.  DCMA concluded that the quality of the peer review process was  questionable because peer-reviewed software artifacts resulted in an  abnormally high number of defects being discovered later. According to  Raytheon officials, Raytheon has incorporated changes to its peer review  instructions in its software development plan, which is awaiting Air Force  approval as of July 2015 and will govern future software development.
subsections: section_title: Software Productivity Assumptions Are Optimistic
paragraphs: The Air Force and Raytheon also assumed overly optimistic software  productivity rates—considering Raytheon’s track record on OCX—when  developing the post-OTB schedule. For example, although the post-OTB  baseline added nearly 2 years to the prior plan, both the Air Force and  Raytheon assumed productivity rates for iterations 1.6 and 1.7 that were  approximately two-thirds higher than the rate achieved for iteration 1.5,  which was the most complex software effort undertaken to that point. As  of the latest available data, from February 2015, Raytheon’s productivity  rate had increased above the rate achieved for 1.5 but was roughly only  half the forecast increase because of continued difficulties, increasing risk  that OCX delivery will be further delayed. According to DCMA, the  program has had a history of being overly optimistic with forecasting  schedules, for example, using data from Raytheon’s prior experience on  other programs that have consistently proven inaccurate when applied to  OCX, and assuming efficiencies based on learning curves which have not  materialized due to staff turnover, process changes, and poor retention of  lessons learned from past development difficulties. This tendency of  overestimating software productivity rates was noted as far back as  October 2011 as part of an independent OSD-led assessment that  highlighted several OCX development problems. Raytheon noted that  corrective actions were taken as part of the OTB, such as revalidating  and completing systems engineering, establishing a common  environment infrastructure, and validating IA implementation during block  0 tests will result in software productivity improvement for iterations 1.6  and beyond beginning in 2016.
section_title: USD AT&L May Not Have Adequate Insight into OCX Progress
paragraphs: USD AT&L may not have adequate insight into the full extent of OCX  development problems given the Air Force’s consistently optimistic  assessments of the program’s progress. The October 2014 advisory  independent review estimated that block 1 was likely to take  approximately 2 years longer than the June 2014 OTB estimate, putting  probable block 1 delivery around November 2020. DOT&E also  expressed concerns about OCX delays in a memorandum sent to USD  AT&L in early November 2014, citing the negative effect of those delays  on the Air Force’s ability to deliver overall GPS capability. In addition,  DOT&E called the Air Force’s schedules for the overall GPS enterprise,  including that of OCX, “inaccurate, implausible, and incoherent” given  OCX development difficulties to date.
Nevertheless, at the ensuing fiscal year 2015 AGER that was held in  November 2014, the Air Force acknowledged additional development  difficulties but noted that OCX systems engineering was improving and  would lead to better software development in the future. At that time, the  OCX program office forecast block 1 completion in August 2019—a delay  of 9 months over the OTB estimate, but optimistic compared to the  independent review’s assessment. One month later, in December 2014,  the Air Force presented its official update for the program and forecast a  block 1 completion date of July 2019. It also estimated that contract costs  had grown to nearly $2 billion—representing cost growth of 16 percent in  just 5 months over the OTB estimate, and more than 120 percent over  initial contract estimates in 2010.
Following the November 2014 AGER, in December 2014, USD AT&L  expressed concern about the continued deterioration in the cost  performance of OCX and stated that the program’s trajectory must be  corrected. USD AT&L directed the Air Force to provide by January 2015 a  “deep dive” program review of OCX focused on the status of and ways to  improve program execution. At this review, which occurred in February  2015, the Air Force acknowledged the program’s volatile cost and  schedule history and that the contractor’s schedules were aggressive  relative to the risk and amount of work remaining. However, the Air Force  also highlighted signs that program execution was stabilizing and noted  that it had reduced program risk to medium-low by including an additional  $331 million and 3 months above contractor estimates. To address USD  AT&L’s directive to detail OCX’s path forward and key decision points  should OCX continue to sustain cost and schedule growth, the Air Force  established a process whereby it would closely monitor key software  development events roughly every 6 months beginning in July 2015 and  report any deviations from cost performance to the Air Force’s senior  acquisition executive as well as to USD AT&L as needed.
While the closer monitoring of key software development events provides  USD AT&L with opportunities to spot cost growth on a more timely basis  than at the annual AGERs, there is little reason to believe that OCX  systemic problems have been adequately addressed. Notwithstanding the  Air Force’s optimistic report to USD AT&L in February 2015, additional  OCX cost growth is quite likely. First, our analysis of detailed earned  value data from November 2013 through November 2014 showed that the  program office significantly underestimated the anticipated cost of  resolving the risks that could affect OCX development and may have also  significantly underestimated the extent of risks. Second, DCMA’s  reporting of earned value performance for the program shows a sustained  deterioration in program cost and schedule performance immediately  following the OTB—where cost and schedule variances were reset to  zero in order to begin measuring performance against the new baseline.  Cost and schedule variances measure the differences in expectations  between the value of work accomplished in a given period with the value  of the work expected in that period. Negative variances indicate that the  program is either overrunning cost or performing less work than planned;  conversely, positive variances indicate the program is either underrunning  cost or performing more work than planned. Figure 7 below shows the  cumulative cost and schedule variances from October 2013 through last  available data as of May 2015, and how the OCX program has been both  overrunning cost and performing less work than planned immediately  after the OTB concluded. While the OTB process completed in June  2014, the program formally reset the cost and schedule baselines at the  end of July.
Further, DCMA’s June 2015 analysis forecast that OCX costs were likely  to increase to $2.15 billion based upon, among other things, the higher  than expected defect rate and poor comprehension of requirements by  Raytheon. From November 2014 through last available data as of May  2015, Raytheon depended on shifting between approximately 100 and  180 additional staff than planned to resolve defects, slowing work on later  iterations. According to DCMA, Raytheon will likely need even more staff  in the future to maintain schedule, and, consequently, incur additional  cost growth. The Air Force’s current schedule estimates, forecasting  block 1 completion in July 2019, are still optimistic by at least a year  compared to the October 2014 independent team’s assessment that  block 1 will most likely be delivered in November 2020. Figure 8 below  summarizes the differences in Air Force estimates for the total number of  months to complete block 1 and those predicted by independent reviews.
In addition, our prior work on space acquisitions has shown that  unrealistic estimates of the achievability of planned schedules, among  other things, directly contributed to unrealistic cost estimates and,  consequently, distorted management decisions, increased risks to  mission success, and virtually guaranteed program delays. By contrast,  senior leaders in successful organizations in the commercial sector  actively encouraged program managers to share bad news about their  programs. These organizations took pains to ensure program estimates  are complete and accurate. Based on the persistently high software  defect rate, continued undisciplined processes, lower than expected  software productivity rates, the downward trend in cost and schedule  performance following the OTB, and lack of realism in Air Force  depictions of OCX progress, root causes do not appear to have been  adequately addressed or perhaps even fully identified. Until all root  causes of OCX problems are fixed, the program is likely to continue to  struggle to achieve desired outcomes.
section_title: Lack of Knowledge about MGUE Likely to Hinder Military Services’ Ability to Make Procurement Decisions
paragraphs: The Air Force has revised the MGUE acquisition strategy several times in  attempts to develop M-code capability. Even so, the military services are  unlikely to have sufficient knowledge about MGUE design and  performance to make informed procurement decisions starting in fiscal  year 2018 because it is uncertain whether an important design review will  be conducted prior to that time and because operational testing will still  be under way.
subsections: section_title: MGUE’s Acquisition Strategy May Limit Insight into MGUE Design
paragraphs: The Air Force has revised the MGUE acquisition strategy several times  as it pursued the program’s development. The latest strategy of record  eliminated a key design assessment, the critical design review (CDR),  and it is uncertain whether a recent revision to that strategy will include a  CDR. As a result, the military services may face a knowledge gap about  MGUE’s design stability and maturity. As we have noted in our prior work,  positive acquisition outcomes typically require the use of a knowledge- based approach to product development that demonstrates high levels of  knowledge before significant commitments are made. We determined  that, in keeping with that approach, knowledge gained through a CDR  helps supplant risk over time by ensuring that a product’s design will meet  customer requirements as well as cost, schedule, and reliability targets.  As part of that work, we also found that a knowledge deficit early in a  program can cascade through design and production, leaving decision  makers with less knowledge to support decisions about when and how  best to move into subsequent acquisition phases that commit more  budgetary resources. Additionally, we found that demonstrating a stable  and mature design, typically via a CDR, is generally considered a  prerequisite by leading commercial firms and successful DOD programs  for moving forward with a program because it assesses final product  design and provides assurance that product specifications have been  captured in detailed design documentation.
The 2011 preliminary strategy for MGUE called for development of  separate ground, air, and sea receiver cards within a traditional DOD  three-phase acquisition process that encompassed separate technology  development, system development, and production phases. The  program’s 2012 baseline strategy was similar and rolled development of  air and sea cards together based on the Navy’s assessment that aviation  card development could support the maritime environment. In 2014, the  Air Force received approval from USD AT&L to revise the MGUE  acquisition strategy again by bypassing a formal system development  phase and combining the development and production decision points.  USD AT&L approved the associated Air Force acquisition strategy  document in April 2015, but just two months later an official from AT&L  indicated that—based on a June 2015 memorandum from the Assistant  Secretary of Defense for Acquisition—those milestones would be split  apart again. According to the AT&L official, it was determined to be  impractical to combine the milestones because there were many events  that needed to be conducted between them. The official indicated that  USD AT&L plans to oversee the program until the development decision  point, at which time it would then delegate program oversight to the Air  Force.
The last MGUE acquisition strategy of record eliminated a key  assessment of MGUE design maturity, the CDR, which we have found in  our prior best practices work is typically held mid-way through a  program’s development phase and is essentially an assessment of  whether a product’s design meets the customer’s requirements. The Air  Force stated that a CDR was unnecessary because, among other things,  detailed design work normally approved at CDR was completed for  MGUE’s PDR in September 2014 and hardware and software designs  and cost, schedule, and performance risks typically reviewed at CDR  were assessed as part of that PDR. It also anticipated other events would  cover and go beyond the purpose of a CDR, including a compatibility  certification process to ensure MGUE receiver cards were compatible  with the GPS satellite signal; security validation to ensure contractors’  designs met security certification criteria; initial and final verification  reviews to assess product maturity and deficiencies; and utilizing multiple  contractors with multiple designs from which the military services can  choose to meet their requirements.
However, the MGUE program carries design-related risks that would  typically be revisited within the CDR context. An April 2015 assessment of  the MGUE PDR by the Office of the Undersecretary of Defense for  System Engineering noted, for example, that some security and  information assurance design details were not addressed as part of PDR  and had been deferred to the security verification review scheduled to  finish in late summer/early fall 2015. The Undersecretary’s office also  pointed out that the refinement of security countermeasures may result in  later design changes. In addition, the office stated that MGUE PDR  interface designs with the military services’ lead platforms may not be  rigorous enough to account for implementing those designs across  various operating environments.
Moreover, it is unclear whether MGUE designs meet the military services’  requirements. According to Army program officials, the Army has  identified a set of performance gaps for ground and aviation receiver  cards between the MGUE program’s technical requirements document  and Army operational requirements, including power and thermal  incompatibility issues between the ground receiver card and the platform  the cards are to support. The MGUE program said that it is unaware of  any data showing the ground cards do not meet power and thermal  requirements; analysis shows the cards meet service requirements for  power as captured in MGUE’s capability development document; the  power and thermal issues may be caused by the host platforms rather  than the ground cards; and as such, the lead platform program offices  would need to make modification to their power supplies or platforms to  resolve the issues. Army officials, in contrast, said that the MGUE  capability development document does not contain specific power  consumption limits and that the Air Force’s GPS Directorate had recently  changed power limitations by unilaterally editing the MGUE technical  requirements document, against significant objection by the Army. A  DOT&E official explained to us that the Air Force believes it has  addressed this issue by clarifying the specification to be an average  maximum power requirement for MGUE ground cards but also  emphasized that this clarification relieves the MGUE contractors of  meeting more stringent instantaneous power limits. As a result, the official  said, there are potential adverse effects to host platform interface designs  which may then require additional development and integration by the  military services in order to adapt their platforms to MGUE.
section_title: Military Services Are Unlikely to Have Complete Knowledge of MGUE Performance by Fiscal Year 2018
paragraphs: The military services are unlikely to have the knowledge to begin  informed MGUE procurement at the start of fiscal year 2018 because  operational testing that will provide knowledge about MGUE’s operational  effectiveness and suitability for ground, air, and sea environments will not  yet have been completed. Prior to operational testing, the Air Force is  conducting activities and demonstrations that it believes reduce program  risk by providing knowledge about MGUE performance. Those efforts,  however, have limitations which call into question their value in assessing  MGUE’s ability to meet the military services’ needs.
Based on the MGUE program’s April 2015 acquisition strategy document,  as shown in Figure 9 below, formal integration and operational testing for  the MGUE aviation receiver card on the Air Force’s B-2 aircraft abuts  fiscal year 2018; that same effort for the Army’s DAGR D3/Stryker  extends into fiscal year 2018; and it carries into fiscal year 2019 on the  GPNTS/Navy’s DDG-51 Arleigh Burke-class destroyer and Marine Corps’  JLTV. In accordance with DOD guidance, these operational tests mark an  important point in the assessment of MGUE design and performance— the first time production representative test articles are planned to be  incorporated on the lead platforms and tested in a realistic environment.  Among other things, per DOD guidance on test and evaluation, the  operational tests are expected to serve as field tests that assess the  receiver cards’ ability to satisfy the military services’ requirements; current  capabilities, including operational benefits or burdens; and the need for  further development of the cards to correct performance deficiencies.
The Air Force believes that completion of operational testing is not  required for the military services to begin informed procurement planning  because data from prior test events can be leveraged. It stated that the  MGUE program has participated in various testing and integration risk  reduction activities prior to the time lines shown in figure 9, and that those  efforts can count towards MGUE integration. Those activities include,  according to the Air Force, successful demonstration of technologies on  the Army’s Raven unmanned aerial vehicle; demonstration of MGUE  receiver cards in operationally relevant environments on a surrogate C-12  aircraft flying similarly enough to the B-2 lead platform to serve as a  surrogate; and lead platform-based risk reduction events via prototype  GPS units that will be incorporated into the B-2. The Air Force said that  testing on surrogate platforms is intended as early risk reduction activity  rather than emulation of lead platform environments. Additionally, it stated  that it plans to conduct subsystem-level integration on lead platforms prior  to system-level integration, and that these efforts also would provide the  military services with further information to asses MGUE performance.
Some of the military services, however, believe that the integration  activities to date have been more limited than indicated by the Air Force  and that they are now responsible for more of the MGUE development  effort than indicated earlier in DOD policy. Army officials stated, for  example, that the MGUE program began initial contact with the DAGR D3  program office in fiscal year 2014, but that integration with the DAGR  D3/Stryker per se has not yet started due to lack of MGUE devices for  platform integration. Instead, they said, “fit check” tests were conducted in  early 2015 and showed the DAGR D3 is unable to provide sufficient  power to two of the three MGUE contractors’ ground receiver cards. The  officials also said that the Army planned to perform thermal testing with  the DAGR D3/Stryker, but such testing has not yet been executed due to  lack of MGUE devices. They emphasized that they have begun planning  for MGUE procurement but cannot initiate actual procurement at the start  of fiscal year 2018 for several reasons, including the need to modify as- built MGUE devices to close performance gaps, resolve power and  thermal issues by either modifying one MGUE card design or the  integration on some 100 platforms; and developing and implementing a  means for MGUE and the currently-fielded Selective Availability Anti- Spoofing Module GPS receiver to coexist and interact in the field. In  addition, both Army and Navy officials stated that, based on their  understanding of DOD’s 2006 policy on GPS user equipment and  procurement, the Air Force was to develop MGUE to production-ready  status. They noted that, in order to bring MGUE to true production-ready  status, their respective military services will need to do further  development after the Air Force completes its MGUE development effort.
Moreover, the demonstrations and risk reduction activities that the Air  Force points to may not establish MGUE performance to the extent that  the Air Force believes. For example, DOT&E—the DOD independent test  authority charged with ensuring a program’s operational testing confirms  operational effectiveness and suitability in combat use—expressed  serious concerns about the Air Force’s characterization of the results from  one of the past demonstrations. In a November 2014 memorandum to  USD AT&L, DOT&E emphasized that the Air Force had overstated MGUE  development maturity and that the demonstrations to-date had achieved  more mixed results than the Air Force indicated. DOT&E noted, for  example, that the testing with the Army’s Raven did not use final  development models; had employed a limited subset of realistic threats;  successfully flew its intended flight pattern in just 7 out of 51 attempts;  and that none of those flights was conducted in the presence of electronic  jamming. It went on to say that developmental activity with the Raven  should in no way be construed as having the level of rigor of operational  testing or even formal developmental testing. Furthermore, Army program  officials stated that the Army’s lead platform had been changed from the  Raven to the DAGR D3/Stryker not only because the Army does not  intend to procure additional Ravens but also because the Raven does not  adequately stress the capabilities of the ground receiver card.
In addition, DOT&E concluded in its 2014 memorandum that MGUE  cannot be considered effective until it is successfully integrated on host  platforms. According to a DOT&E official, subsystem integration does not  guarantee proper integration into a larger system and may not necessarily  demonstrate integration and performance in the intended operational  environment—particularly in light of the fact that the MGUE test strategy  is inherently risky in that only 4 lead platforms are designated to represent  the operating conditions for over 100 platforms service-wide. He also  emphasized—as did DOT&E’s 2014 memorandum —that DOD has found  integration historically challenging.
Our prior work on integration and testing has also shown that integration  is a common risk in system development, and that commercial firms  reduce risk by capturing design and manufacturing knowledge early.  Our previous work pointed out, for example, that systems integration  problems can occur even though the various components performed  successfully on previous systems. We have found, in illustration, that the  Air Force’s C-17 aircraft program intended to use current, available, and  proven technology to minimize development costs and structure a low  technical risk effort but that the integration of technologies was a major  engineering and management task that eventually contributed to  significant cost increases. In addition, we have also determined that it is  during integration and testing that problems are likely to be found.  Furthermore, as we pointed out in a 2000 study of best acquisition  practices for testing among commercial firms, test weaknesses invariably  cause negative program outcomes, such as cost increases, schedule  delays, or performance shortfalls. Our work on best commercial  acquisition practices also shows that such negative outcomes can be  avoided by accumulating knowledge prior to beginning production. In a  2002 study of best commercial practices, for example, we found that  leading commercial firms reduce program risk by demonstrating fully- integrated prototypes prior to making production decisions. Such  demonstrations, we determined, help the firms decide when to make the  transition from product development to production and to ensure that  transition is smooth.
The military services can postpone procuring MGUE if they lack sufficient  knowledge about MGUE design and performance after fiscal year 2017.  They can, as provided by the National Defense Authorization Act for  Fiscal Year 2011, request a waiver of such procurement. Specifically, the  Secretary of Defense may waive this limitation upon a determination that  suitable M-code capable user equipment is either not available or DOD  does not require that user equipment be capable of receiving M-code  from GPS. An official from USD AT&L stated that the waiver process  has not yet been established.
section_title: GPS Constellation Has Been More Reliable Than Expected, but M-code Deployment Is Likely More Than a Decade Away
paragraphs: In 2010, we examined GPS satellite reliability data and reported that a  delay of 2 or more years to the launches of GPS III satellites—originally  projected to begin launching in April 2014—would likely reduce the  constellation size below the minimum requirement of 24 operational GPS  satellites by fiscal year 2018. Since that report, while the GPS III and  OCX programs have been delayed, the GPS constellation has proven to  be more reliable than previously expected because the Air Force  successfully extended the life of existing satellites, primarily by modifying  the satellites’ battery charging procedures. However, the Air Force now  needs OCX block 1 to be operational by late 2019 to launch and  incorporate GPS IIIs into the constellation. Given the ongoing  development problems with OCX, delivery by that date—and  consequently, overall constellation reliability—is at some risk. The Air  Force is exploring contingency plans, including modifying the current  ground system to launch and operate GPS IIIs. However, the modified  ground system would operate GPS III satellites as current legacy GPS  satellites, thereby not utilizing all the capabilities on the new satellites.  Moreover, this approach would increase the risk that several IIIs may be  launched before they can be fully tested with OCX. Delays to OCX  delivery also mean that M-code functionality—which requires OCX block  1 to be fully enabled—is not scheduled to be deployed until 2019 at the  earliest, and likely for another decade or more until the military services  can widely deploy modernized MGUE receivers.
subsections: section_title: The Air Force Has Found Ways to Extend the Life of the GPS Constellation
paragraphs: The performance standards for (1) the standard positioning service  provided to civil and commercial GPS users, and (2) the precise  positioning service provided to military GPS users, commit the U.S.  government to at least a 95 percent probability of maintaining a  constellation of 24 operational GPS satellites. In September 2010, we  reported that delays to the launch of the GPS III satellites—originally  projected to begin in April 2014—could affect the long-term probability of  maintaining the minimum constellation size. Based on satellite reliability  data provided by the Air Force, we had predicted that, for example, a 2- year delay in the production and launch of the first and all subsequent  GPS III satellites would reduce the guaranteed size of the constellation  (at the 95 percent confidence level) to about 18 satellites by around fiscal  year 2018.
Since our 2010 report, the launch availability date for the first GPS III  satellite has, in fact, slipped by about 3 years, to May 2017, mainly due to  development difficulties with its navigation payload. Moreover, as  discussed earlier, the delivery of OCX block 1—required to operate the  IIIs—has been delayed by about 4 years, to mid-2019. However, the 3- year delay in GPS III’s delivery has not resulted in a predicted significant  reduction in the size of the GPS constellation largely because the  expected life of the existing generations of GPS satellites has risen  dramatically. Figure 10 below displays the probability of maintaining a 24- satellite GPS constellation as a function of time based on March 2014  satellite reliability data and launch schedules—the latest date for which a  complete set of approved parameters is available. The launch schedule  has been adjusted to reflect the successful launches of four GPS IIF  satellites through June 2015.
The expected lifetimes of the GPS IIR and IIR-M satellites—the types of  satellites that currently comprise the majority of the current  constellation—have risen dramatically since our 2010 report, from 14.8  and 10.5 years, respectively, to 20.4 and 17.5 years, respectively. This  increase in expected lifetime is due mainly to the Air Force’s  implementation of a modified charging procedure that will prolong the life  of the satellites’ batteries. Spacecraft batteries—like all other  rechargeable batteries—are subject to cycle life, that is, they can only be  charged and discharged so many times before they will not charge  efficiently anymore. Because satellite batteries were predicted to be one  of the primary life-limiting components of these satellites, the Air Force  developed and tested a procedure in 2012 to reduce the battery charge  rates during certain times of the year. The Air Force estimated that this  new procedure would increase the expected lifetimes of the IIR and IIR-M  satellites by 1 to 2 years. Based on recent telemetry data, the new  procedure has proved successful. In 2014, the Air Force announced  that this modified battery charging procedure has added, in aggregate, 27  additional years of operational life to the GPS IIR and IIR-M satellites.  Moreover, according to Air Force representatives, additional information  suggests that this estimate is probably conservative, and that the next  formal review of satellite reliability data will likely reveal an additional  increase in the expected life of the IIR satellites. Because these satellites  form the mainstay of the current GPS constellation, and because most of  them are still expected to be operational in 2017, when the first GPS III  satellite is planned to be available for launch, an increase in the expected  life of these older satellites significantly improves the Air Force’s ability to  meet the 24-satellite performance standard despite the 3-year slip in GPS  III and 4-year slip in OCX.
The expected lifetimes of the GPS IIA satellites—the oldest version of  GPS satellites on orbit—have also risen since our 2010 report, from 16.5  years to 19.7 years, due to positive trends in on orbit performance.  However, the effect of this increase on the Air Force’s ability to meet the  24-satellite performance standards is less significant than the effect of the  IIR/IIR-M life expectancy increase. The current constellation availability  analysis assumes each of these satellites is only available to replace  other satellites in the unlikely event of a large number of satellite failures  until OCX is operational. Because OCX is not expected to be capable of  operating the IIA satellites, the analysis assumes that any remaining IIA  satellites would be turned off by July 2019, the date that OCX is  scheduled to be required to be ready to transition to operations.
section_title: Additional OCX Delays Could Affect Constellation Availability
paragraphs: Despite the delays to GPS III and OCX, the GPS constellation is  expected to meet the constellation size requirement if OCX block 1 is  available by July 2019 as needed. However, further OCX delays could  affect overall constellation availability, bringing it below the 95-percent  performance standard for maintaining a 24-satellite constellation for a  year or more. For example, if OCX block 1 is not ready to transition to  operations until November 2020—as the most recent (October 2014)  independent review of OCX estimated—the Air Force could experience a  15-month period during which it would not meet its commitment to  maintaining a 24-satellite constellation with a 95 percent confidence level.  While GPS III satellites could still be launched beginning in 2017, they  would not be added to the constellation until OCX comes on line or until a  contingency operational control capability able to command GPS III  satellites is available. Figure 11 depicts the predicted constellation size  based on OCX block 1 being potentially delayed until November 2020.
section_title: The Air Force Is Taking Actions to Sustain the GPS Constellation Given Delays to OCX, but These Efforts Will Not Fully Exploit GPS Capabilities
paragraphs: GPS capabilities are unbalanced, with satellite capabilities outstripping  those supported by the ground and user equipment segments. In  particular, the Air Force has launched two generations of satellites—IIR-M  and IIF—capabilities of which are still not fully utilized because they have  been transmitting three signals, including M-code, that are not supported  by the current GPS ground control system, OCS.
GPS IIFs are nearing the end of their launch, with the last three  scheduled to be launched between July 2015 and February 2016. GPS  IIIs are needed to continue to sustain the constellation. Since OCX block  1 is needed to operate the GPS III satellites as part of the constellation,  the Air Force is preparing contingency plans in the event that OCX block  1 is delayed beyond August 2019, the need date for the first GPS III  satellite. The Air Force has pinpointed one option as the most viable— modifying the current OCS system to operate the GPS III satellites but at  the older, IIF satellite level of functionality; this option would deliver M- code support without all modernized GPS III functionality, specifically  support for the L1C signal. Under this plan, the Air Force would use OCX  block 0 to launch and initially test the GPS III satellites, and use the  modified OCS to control the satellites’ navigation signals as part of the  overall constellation. As figure 12 shows, successfully implementing this  contingency plan should enable the Air Force to meet its commitment to  maintaining a 24-satellite constellation with a 95 percent confidence level,  even if the OCX block 1 delivery date were to slip to November 2020 (or  later), but the plan would require funding and other resources.
The Air Force estimated the cost of this option at $105 million as of  February 2015; however, pursuing this option means that the Air Force  may launch up to 5 GPS III satellites without fully testing them with OCX  block 1, increasing the risk that issues may be found during testing  without the ability to fix the satellites already launched.
section_title: Initial M-code Capability Will Not Be Available until OCX Delivery in Mid-2019, and Full M-code Deployment Is Likely at Least Another Decade Away
paragraphs: M-code initial operational capability is defined as having 18 M-code  capable satellites on orbit, the control segment able to command/upload  M-code capabilities to the space segment, and MGUE receivers fielded  across the military services to utilize M-code capabilities operationally.  Full operational capability occurs when 24 M-code satellites are on orbit,  and a larger portion of MGUE is fielded.
The GPS constellation will likely include 18 M-code capable satellites by  September 2015, but M-code capabilities will not be available to users— assuming MGUE receivers can be fielded—until OCX delivery in mid- 2019. The launching of M-code capable satellites has been stretched  over a much longer period than originally envisioned because of delays to  the IIF, III, OCX, and MGUE programs. However, because the oldest of  the M-code capable satellites are lasting longer than originally predicted,  the effect of this stretched deployment has been mitigated. In particular  the oldest M-code capable satellite—the first IIR-M satellite, launched in  September 2005—has a better than a 75 probability of still being  operational in October 2020, and better than a 50 percent probability of  still being operational in October 2023. Nevertheless, DOD cannot take  full advantage of M-code capability until MGUE receivers are deployed in  sufficient numbers across the services. As noted earlier, the earliest  MGUE operational test will not be completed until the end of fiscal year  2017 and the latest at the end of fiscal year 2019. Accordingly, some of  the military services will not be able to make informed fielding decisions  until fiscal year 2020 at the earliest, assuming operational testing goes as  planned.
Furthermore, as discussed earlier, the Air Force plans to deliver  production representative test articles, which the military services will then  acquire, integrate, and operationally test only on selected military  platforms. As GAO reported in 2009, each of the military services would  still need to add the new user equipment to other platforms, which could  take an additional 10 or more years based on the need to perform such  activities as coordinating installation with the platforms’ maintenance and  upgrade schedules.
section_title: Conclusions
paragraphs: GPS is a global utility that is integral to U.S. national security and civilian  use. As a result, any decision about GPS has far-reaching consequences.  A combination of many factors, including technical challenges, poor  contractor execution and program management, and ineffective  acquisition oversight have all put GPS modernization at significant risk.
OCX is the key to enabling the full GPS capability that both the military  and civilian communities depend on, from command and control of the  satellite constellation including the new GPS III satellites, to allowing  military receivers to take advantage of M-code signals for more robust  warfighting capability, to enabling advanced civil GPS signals. However,  by any measure, OCX development has been mired in development  difficulties resulting in steady cost growth and schedule delays. Moreover,  despite a 7 month pause ending in mid-2014, OCX has yet to turn the  corner on resolving the problems that have affected the program since  development began in 2010. Independent observers such as OSD-led  teams, DCMA, and DOT&E have raised red flags about the effectiveness  of the Air Force’s oversight and Raytheon’s ability to deliver promised  outcomes. The Air Force has compounded matters by consistently  presenting overly optimistic assessments about OCX progress— demonstrating a pattern of marginalizing warnings about OCX delays  presented by independent assessors, likely because their assessments  are considered advisory in nature. Driving towards unrealistic timeframes  is compounding the program’s inability to meet stated cost and schedule  goals. Five years into what was originally estimated to be a 5-year effort,  OCX is still roughly 5 years away from completion. Without  comprehensively identifying systemic causes for OCX problems, DOD  cannot have high-confidence cost and schedule estimates for OCX. The  Air Force and Raytheon have noted that OTB corrective actions will begin  paying off in early 2016 as block 1 development resumes in earnest. But  given the lack of success in prior attempts to turn the program around, the  Air Force could benefit from external expertise and guidance on what is  necessary to address systemic issues.
Since the enactment of the statute directing DOD to generally procure  only M-code capable user equipment after fiscal year 2017, the Air Force  has struggled to deliver a MGUE acquisition strategy that would allow the  military services to comply with that direction. As it stands now, the Air  Force essentially truncates its MGUE development work and hands the  result of its efforts off to the military services to continue development on  their respective platforms. At the point the Air Force provides funds to the  military services for them to acquire and test MGUE, it also transfers the  onus of the development work to the military services’ shoulders because  MGUE development cannot be considered complete until the cards work  in the lead platforms. Unfortunately, the military services will absorb an  added development burden because the Air Force plans to transfer its  work without conducting design and performance assessments that could  help the services decide the extent to which the cards are ready to  integrate and test and how much additional work they will need to do  before beginning to procure the cards.
Fortunately, many GPS satellites now on orbit have served the nation  particularly well by working considerably longer than expected. Even so, it  cannot be presumed that they will continue to do so. Therefore, it is  critical that the modernized GPS III, OCX, and MGUE development  efforts succeed sooner rather than later. Of those three programs, OCX is  now the pacing item for modernization due to its many past delays and  probable future delays. Until the OCX program trajectory is corrected,  those delays are likely to pose significant risks to sustaining the GPS  constellation, and consequently, delivering GPS capability to the military  community.
section_title: Recommendations for Executive Action
paragraphs: To better position DOD as it continues pursuing GPS modernization, to  have the information necessary to make decisions on how best to  improve that modernization, and to mitigate risks to sustaining the GPS  constellation, we recommend that the Secretary of Defense take the  following five actions:
Convene an independent task force comprising experts from other  military services and defense agencies with substantial knowledge  and expertise to provide an assessment to USD AT&L of the OCX  program and concrete guidance for addressing the OCX program’s  underlying problems, particularly including:
A detailed engineering assessment of OCX defects to determine  the systemic root causes of the defects;
Whether the contractor’s software development procedures and  practices match the levels described in the OCX systems  engineering and software development plans; and
Whether the contractor is capable of executing the program as  currently resourced and structured.
Develop high confidence OCX cost and schedule estimates based on  actual track record for productivity and learning curves.
Direct the Air Force to retain experts from the independent task force  as a management advisory team to assist the OCX program office in  conducting regular systemic analysis of defects and to help ensure  OCX corrective measures are implemented successfully and  sustained.
Put in place a mechanism for ensuring that the knowledge gained  from the OCX assessment is used to determine whether further  programmatic changes are needed to strengthen oversight.
To allow the military services to fully assess the maturity of the MGUE  design before committing test and procurement resources,  incorporate a CDR in the Air Force’s MGUE development effort.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Defense for review  and comment. In its written comments, reproduced in appendix II, DOD  concurred with the four OCX-related recommendations calling for a more  robust independent review of the program, stating that the intent of those  recommendations has been met by OSD-chartered independent reviews  conducted to-date, which included both Air Force and OSD staff, and  other DOD activities governing OCX oversight, such as the use of  independent cost estimates. DOD partially concurred with the  recommendation calling for incorporating a CDR in the MGUE program’s  development effort, noting that adding a CDR would delay the program  amidst its efforts to compress the acquisition process to deliver MGUE  capability by fiscal year 2018. DOD also provided technical comments,  which we have incorporated as appropriate.
While DOD concurred with our recommendations to put OCX on a better  path forward, it responded that it had essentially been following the intent  of our recommendations all along, requiring no further action from the  department. These comments provide little confidence that the  department intends to fully implement our recommendations to fix the  development problems that have beset the program since its inception.  As we noted in our report, the independent reviews conducted to-date  have been nonbinding and advisory in nature. Even as the OSD-led  reviews have warned of significant delays and inadequate insight into  systemic root causes of development problems, the Air Force has  repeatedly developed unrealistic timeframes, ostensibly with OSD’s  knowledge. If business continues as usual without swift and thoughtful  action, OCX will likely continue on its path of demonstrating poor cost and  schedule outcomes. We continue to stand by our recommendations  calling for a fresh review—this time an in-depth and comprehensive  critical review of the program—to identify the true root causes of OCX  development difficulties and to ensure the Air Force implements the  corrective actions.
Regarding DOD’s response to our recommendation that DOD hold a CDR  for MGUE, we note that DOD’s rationale for skipping this best practice  step is based on a desire to accelerate fielding of the units. However, our  past work has consistently shown that taking shortcuts and skipping  important knowledge points in the acquisition process generally results in  an inability to deliver promised cost and schedule outcomes. To minimize  any potential disruption to development efforts, a CDR could be held after  production-representative test articles are delivered. The results of a CDR  will show whether the MGUE design is stable. Rushing into lead platform  testing without an approved and stable design means that DOD will have  to concurrently test and correct the receivers’ design at a point where  resolving issues are typically more expensive than earlier in  development—not in accordance with best practices that are important to  achieving cost and schedule outcomes with a high degree of confidence.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of the Defense, the Secretary of the Air Force,  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In May 2014 House Report No. 113-446, accompanying H.R. 4435, the  Howard P. “Buck” McKeon National Defense Authorization Act for Fiscal  Year 2015, noted the requirements stated in section 913 of the Ike  Skelton National Defense Authorization Act for Fiscal Year 2011 (Public  Law No. 111–383) requiring the Department of Defense (DOD) to  purchase M-code capable user equipment during the fiscal years after  fiscal year 2017 and included a provision for us to report on DOD’s  progress in deploying M-code capability. The House report also provided  that our report assess current and planned investments; whether key  milestones are being met; the projected ability to the meet the  requirements in section 913 of Public Law No. 111–383; and an  identification of the challenges that Global Positioning System (GPS)  faces and possible recommendations on how to make the program more  successful in delivering M-code capabilities. Additionally, in June 2014,  Senate Report No. 113-176, accompanying the Carl Levin National  Defense Authorization Act for Fiscal Year 2015, included a provision for  us to review the cost, scope, and schedule of the GPS III operational  control system, including synchronization with the launch of the GPS III  constellation with recommendations for improvement.
In response, for this report our objectives were to assess (1) the extent to  which DOD is meeting cost, schedule, and performance requirements for  next generation operational control system (OCX), (2) the progress DOD  has made in delivering M-code capable military GPS user equipment  (MGUE) by the end of fiscal year 2017, and (3) the challenges DOD faces  in synchronizing the development of GPS III satellites, OCX, and MGUE  to deploy M-code.
To assess the extent to which DOD is meeting cost, schedule, and  performance requirements for OCX, we reviewed program and contractor  cost and schedule documentation, including program acquisition  baselines, earned value metrics, and test plans. Throughout this report,  we focused on the costs of the OCX development contract instead of the  full program acquisition costs because the latter includes prior,  technology development expenditures as well as costs of management  support and enterprise integration support services. We analyzed the  progress made against planned program milestones and reviewed  technical documentation such as software development plans to gain  insights into OCX progress. In addition, we reviewed briefings and  schedule documentation provided by program and contractor officials to  determine changes in OCX cost, schedule, and performance over time.  These documents included annual GPS enterprise reviews, OCX  program assessments, and program status briefings. We also interviewed  officials from the Air Force Space and Missile Systems Center GPS and  OCX program offices, OCX prime contractor Raytheon, and Defense  Contract Management Agency (DCMA) officials charged with oversight of  the OCX contractor efforts to identify and assess cost and schedule  issues facing the program’s development efforts, major program risks,  and technical challenges. Finally, we interviewed officials from DOD’s  Office of Cost Assessment and Program Evaluation and Director,  Operational Test and Evaluation office to discuss cost, schedule, and  performance challenges for OCX. When program documents identified  program events by fiscal quarter rather than by month, we used the last  month of the given quarter as the date of the event.
To determine the progress DOD has made in delivering M-code capable  MGUE by the end of fiscal year 2017, we reviewed and analyzed program  plans and documentation related to cost, schedule, acquisition strategy,  technology development, and major challenges to delivering MGUE  Increment 1. We then compared the information we obtained to GAO’s  criteria for best practices in system development. To assess the  program’s progress and challenges, we held discussions with and  received information from officials at the Air Force Space and Missile  Systems Center GPS and MGUE program offices; Office of the  Undersecretary of Defense for Acquisition, Technology, and Logistics;  DOD’s Office of Cost Assessment and Program Evaluation; and DOD’s  Director, Operational Test and Evaluation. Additionally, to understand  their respective development efforts and challenges, we interviewed  contractor officials from the three MGUE prime contractors, L-3 Interstate  Electronics Corporation, Raytheon Space and Airborne Systems, and  Rockwell Collins Inc. as well as Defense Contract Management Agency  officials overseeing those contractors. To identify the military services’  concerns about MGUE development, schedule, and integration and  testing with their respective platforms, we interviewed officials from the  lead platform program offices for the Army’s Defense Advanced GPS  Receiver Distributed Device/Stryker, Air Force’s B-2 aircraft, Navy’s  Arleigh Burke Class DDG-51 ship, and Marine Corps Joint Light Tactical  Vehicle. As with OCX, when program documents identified program  events by fiscal quarter rather than by month, we used the last month of  the given quarter as the date of the event. We did not assess MGUE  Increment 2 because it has not yet reached its technology development  milestone.
To identify the challenges faced in synchronizing GPS III satellite, OCX,  and MGUE to deploy M-code capability, we built on our work assessing  the plans, schedules, and challenges of the OCX and MGUE programs by  reviewing integrated master schedules and analyzing approved GPS  constellation reliability parameters. We also interviewed cognizant  officials within Air Force Space Command (AFSPC) and the Aerospace  Corporation, which supports the Air Force’s Space and Missile Systems  Center (SMC), to evaluate predicted reliability of the GPS satellite  constellation. We did not assess cost, schedule, or performance for GPS  III satellites.
To assess the status of the GPS constellation, we interviewed officials  from the Air Force SMC GPS program office and AFSPC. To assess the  risks that a delay in the acquisition and fielding of GPS III satellites could  result in the GPS constellation falling below the 24 satellites required by  the standard positioning service and precise positioning service  performance standards, we employed a methodology very similar to the  one we had used to assess constellation performance in 2009 and 2010.  We obtained information from the Air Force predicting the reliability for 57  GPS satellites—each of the 39 current (on-orbit as of March 2015) and 18  future GPS satellites—as a function of time. Each satellite’s total reliability  curve defines the probability that the satellite will still be operational at a  given time in the future. It is generated from the product of two reliability  curves—a wear-out reliability curve defined by the cumulative normal  distribution, and a random reliability curve defined by the cumulative  Weibull distribution. For each of the 57 satellites, we obtained the two  parameters defining the cumulative normal distribution, and the two  parameters defining the cumulative Weibull distribution. For each of the  18 unlaunched satellites we included in our model, we also obtained a  parameter defining its probability of successful launch, and its current  scheduled launch date. The 18 unlaunched satellites include 3 IIF  satellites and 15 III satellites; launch of the final III satellite we included in  our model is scheduled for March 2025. Using this information, we  generated overall reliability curves for each of the 57 GPS satellites. We  discussed with Air Force and Aerospace Corporation representatives, in  general terms, how each satellite’s normal and Weibull parameters were  calculated. However, we did not analyze any of the data used to calculate  these parameters.
Using the reliability curves for each of the 57 GPS satellites, we  developed a Monte Carlo simulation to predict the probability that at least  a given number of satellites would be operational as a function of time,  based on the GPS launch schedule as of December 2014. We conducted  several runs of our simulation—each run consisting of 10,000 trials—and  generated “sawtoothed” curves depicting the probability that at least 24  satellites would still be operational as a function of time. We then used  our Monte Carlo simulation model to examine the effect of a delay to the  delivery of OCX block 1 until November 2020, which would delay the  introduction of GPS III satellites into the operational constellation. We  then reran the model based on this assumption, and calculated new  probabilities that at least 24 satellites would still be operational as a  function of time. Finally, we simulated the effect of the Air Force’s  proposed contingency plan, which would enable GPS III satellites to be  added to the operational constellation, with limited functionality, prior to  the delivery of OCX block 1.
We conducted this performance audit from August 2014 to September  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Dr. Nabajyoti Barkakati, Chief  Technologist; Art Gallegos, Assistant Director; Jason Lee, Assistant  Director; Karen Richey, Assistant Director; Jay Tallon, Assistant Director;  Marie P. Ahearn; Pete Anderson; Brandon Booth; Brian Bothwell; Raj  Chitikila; Tana Davis; Roxanna Sun; and Hai V. Tran made key  contributions to this report.