GPS: Actions Needed to Address Ground System Development Problems and User Equipment Production Readiness
Released: 2015-09-09T00:00:00

Letter

Background

GPS is a global positioning, navigation, and timing system consisting of space, ground control, and user equipment segments that support the broadcasts of military and civil GPS signals. These signals each include positioning and timing information, which enables users with appropriately-equipped GPS receivers to determine their position, velocity, and time, 24 hours a day, in all weather, worldwide. GPS is used by all branches of the military to guide troop movements, assist with logistics support and situational awareness, and synchronize communications networks. In addition, weapon systems, including munitions, are guided to their targets by GPS signals, and GPS is used to locate military personnel in distress. Early in the development of GPS, its scope was expanded to include complementary civil capabilities such as civil, maritime, and land navigation.

GPS System Description

Space, ground control, and user equipment segments are needed to take full advantage of GPS capabilities. The GPS space segment consists of a constellation of satellites that broadcast encrypted military signals and civil signals. In recent years, because numerous satellites have lasted longer than anticipated, the constellation has grown well beyond the minimum requirement of 24 satellites to approximately 40 satellites of various generations, with 8 in residual status. The satellites are operated by a master control station that regularly updates navigation signals on the satellites. Using these navigation signals, GPS military and civilian user equipment receivers determine a user’s location. Figure 1 below illustrates how GPS satellites, ground control, and user equipment function together as an operational system.

The GPS ground control segment primarily consists of software deployed at a master control station at Schriever Air Force Base, Colorado, and at an alternate master control station at Vandenberg Air Force Base, California. The ground control software is supported by 6 Air Force and 11 National Geospatial-Intelligence Agency monitoring stations located around the globe along with four ground antennas with uplink capabilities. Information from the monitoring stations is processed at the Master Control Station to determine satellite clock and orbit status.

The GPS user equipment segment includes military and civilian GPS receivers. These receivers determine a user’s position and time by calculating the distance from four or more satellites using the navigation signals on the satellites to determine its location. Military GPS receivers are designed to utilize the encrypted military GPS signals that are only available to authorized users, including military and allied forces and some authorized civil agencies. Civilian—including commercial— receivers use the civil GPS signal, which is publicly available worldwide.

Current Modernization Efforts

In 2000, DOD began an effort to modernize the space, ground control, and user equipment segments of GPS to enhance the system’s performance, accuracy, and integrity. To that end, the Air Force is now in the process of developing a new generation of GPS III satellites, OCX, and M-code capable MGUE receivers.

GPS III satellites are planned to supplement and eventually replace the constellation of GPS satellites now in orbit; these satellites consist of multiple versions or generations developed and launched over the years. The first GPS III satellite was originally expected to be available for launch in April 2014; however, due to development problems it is now expected to be ready for launch in May 2017. A complete GPS III satellite has not yet been tested, and the program is now rebaselining its cost estimates as a result of the schedule delay and associated increased costs.

The GPS ground control segment is being modernized under the OCX program. OCX is required because the existing GPS control system, Operational Control System (OCS) cannot control—and therefore enable—the modernized features of the two latest generations of GPS satellites—IIR-M and IIF—currently in orbit. The Air Force plans to develop OCX in blocks, with each block delivering upgrades as they become available. Block 0 is intended to support the launch and initial testing of GPS III satellites; block 1 is designed to command and control the GPS II and III satellites and basic modernized signals; and block 2 is to enable the full modernized M-code signal and support, monitor, and control additional navigation signals. OCX block 1 is needed to incorporate GPS III satellites into the operational constellation to sustain required levels of GPS signal coverage, because the legacy OCS system cannot support the GPS III satellites. OCX is also required to enable military and civil use of modernized GPS signals. In particular, the military cannot use M-code signals for enhanced warfighting until OCX block 1 is delivered.

The Air Force began the OCX program in 2007 with a technology development phase, referred to as phase A. The Air Force awarded phase A contracts, for approximately $160 million each, to Northrop Grumman in Redondo Beach, California, and Raytheon Intelligence and Information Systems in Aurora, Colorado to produce competitive prototypes. Following the competitive down-select between these firms’ prototypes, the Air Force awarded the development contract to Raytheon in February 2010. This $886 million contract covered the development of OCX blocks 1 and 2 (as discussed in this report, block 0 was added as a contract modification later), with an option to begin preliminary work on blocks 3 and 4 which are to provide additional capabilities to support follow-on, upgraded versions of GPS III satellites.

At the formal start of development (milestone B), the Air Force estimated the total OCX acquisition costs—including other costs such as expenditures on technology development prior to 2010 and annual management support and enterprise integrator services—at $3.5 billion. According to Air Force documentation, enterprise integrator services are required to ensure GPS enterprise coordination among the ground, space, and user equipment segments. These costs do not include funding contributions from civilian agencies to support OCX; according to Air Force documentation, the Department of Transportation and other agencies are to provide resources to DOD to develop and operate GPS civil capabilities. For the purpose of this report, OCX development costs refer to the costs of the development contract with Raytheon.

Military GPS receivers are also being modernized under the MGUE program. The Air Force was directed by the Assistant Secretary of Defense for Networks and Information Integration in August 2006 to develop M-code capable GPS receiver cards to meet military services’ needs. In January 2011, the Ike Skelton National Defense Authorization Act for Fiscal Year 2011 directed that DOD not obligate or expend funds to procure GPS user equipment after fiscal year 2017 unless that equipment is capable of receiving M-code. Figure 2 below shows an illustration of a MGUE receiver card.

The modernized receiver cards are to provide U.S. forces with enhanced position, navigation, and time capabilities, while improving resistance to existing and emerging threats, such as jamming. The Air Force plans to develop MGUE in two increments. The first increment, now under way and expected to cost about $1.7 billion, utilizes three prime contractors. All three contractors—L-3 Interstate Electronics Corporation, Raytheon Space and Airborne Systems, and Rockwell Collins—are developing receiver cards for ground environments. Raytheon and Rockwell Collins are also developing combined aviation/maritime receiver cards for use in air and sea environments. The Air Force plans to build on the work conducted in Increment 1 to develop a more compact receiver card in Increment 2 that can be used when size, weight, and power need to be minimized and that can serve as an “engine” for future military GPS receivers. At this time, the Air Force is also exploring the possibility of using Increment 1 technologies to support munitions, handheld devices, and space applications that it previously anticipated supporting with Increment 2.

The Air Force initially began development of M-code in fiscal year 2003. It then transitioned that work to the modernized user equipment (MUE) program in 2006. According to GPS program updates, MUE was troubled by issues such as underestimation of software complexity, longer than anticipated software development time, and more difficult than expected software and hardware integration. These issues, among others, resulted in significant cost and schedule growth. The MUE program yielded aviation and ground prototypes of M-code-capable components, cost about $498 million, and spanned about 7 years, ending in 2013.

Concurrent with and based on MUE development work, the Air Force initiated the MGUE program in 2011. While DOD policy on GPS user equipment and procurement in 2006 indicated the Air Force was to develop MGUE to production-ready status, the Air Force currently plans to develop MGUE Increment 1 ground and aviation/maritime receiver cards to the point of production representative test articles. It then intends for the MGUE program office to provide funding to the military services so that they can acquire, integrate, and operationally test the receiver cards on service-specific “lead platforms.” These platforms are expected to serve as pathfinders for the military services’ ground, aviation, and maritime environments and are currently designated by the services as follows: Army–Defense Advanced GPS Receiver (DAGR) Distributed Device (D3) onboard a Stryker ground combat vehicle; Air Force–B-2 Spirit aircraft; Navy–DDG-51 Arleigh Burke Destroyer hosted by GPS-based Positioning, Navigation, and Timing Service (GPNTS); and Marine Corps–Joint Light Tactical Vehicle (JLTV). Once operational testing is complete, the military services will then be responsible for procuring MGUE Increment 1 for their weapons systems.

DOD indicates that the military services anticipate procurement of about 207,000 MGUE Increment 1 receiver cards. Historically, full fielding of user equipment has lagged behind the anticipated availability of GPS satellite and ground systems. For example, in 2009 we reported that the Air Force expected GPS satellite and ground systems to be available in 2013 to transmit and process the M-code signal, but that fielding of M- code equipment to all designated military users would not be complete until 2025.

Poor Acquisition Decisions and a Slow Recognition of Development Problems Led to Significant OCX Cost Increases and Schedule Delays

OCX development contract costs have more than doubled since the contract was awarded in February 2010, increasing by approximately $1.1 billion to $1.98 billion, and the program’s schedule has roughly doubled over estimates at contract award. The Air Force awarded the contract to begin OCX development but did not follow key acquisition practices such as completing a preliminary design review before development start as called for by best practices. In addition, key requirements, particularly for cybersecurity, were not well understood by the Air Force and contractor at the time of contract award. The contractor, Raytheon, experienced significant software development challenges from the onset, but the Air Force consistently presented optimistic assessments of OCX progress to acquisition overseers. Figure 3 below shows select key events related to the OCX program since the development contract was awarded in 2010.

Further, the Air Force complicated matters by accelerating OCX development to better synchronize it with the projected completion time lines of the GPS III satellite program, but this resulted in disruptions to the OCX development effort. As Raytheon continued to struggle developing OCX, the program office paused development in late 2013 to fix what it believed were the root causes of the development issues, and significantly increased the program’s cost and schedule estimates. However, progress reports to DOD acquisition leadership continued to be overly optimistic relative to the reality of OCX problems. OCX issues appear to be persistent and systemic, raising doubts whether all root causes have been adequately identified, let alone addressed, and whether realistic cost and schedule estimates have been developed. Figure 4 below shows how OCX costs have grown and the schedule delayed since contract award in February 2010.

The Air Force Did Not Follow Key Acquisition Practices When It Initiated OCX Development

The Air Force awarded the OCX development contract to Raytheon in February 2010 for $886 million; blocks 1 and 2 were forecast for completion in August 2015 and March 2016, respectively. The development contract was awarded before completing a milestone B decision formally authorizing the start of development. In addition, the program did not complete a preliminary design review (PDR) as called for by best practices; the Air Force subsequently acknowledged in the OCX acquisition strategy approved in September 2012 that the contract was awarded earlier than normal. Figure 5 below depicts key events in DOD’s typical acquisition process and the corresponding knowledge called for in GAO’s prior work on best product development practices. As shown in the figure, our prior work has identified several proven management practices that, if fully implemented, can help DOD minimize cost overruns by ensuring programs are established after matching requirements and resources.

The Air Force did not conduct a milestone B review prior to awarding the development contract, missing an opportunity to ensure the program began on a sound foundation. A milestone B review is an important point in a program where requirements and resources—that is, technology, design, time, and funding—should be properly matched to make sure the program can be executed as planned. The Under Secretary of Defense for Acquisition, Technology, and Logistics (USD AT&L) approved the Air Force’s request in January 2010 to award the OCX development contract prior to milestone B. The Air Force and USD AT&L decided to award the contract early for several reasons. First, the Air Force believed that the 3- year competitive technology development effort from 2007-2010, where two contractors (Northrop Grumman and Raytheon) worked on proof-of- concept prototypes for OCX, successfully demonstrated initial functionality and reduced risk. Second, the Air Force wanted to reduce the costs of carrying two contractors through PDR. Finally, the Air Force and USD AT&L believed that splitting efforts between two contractors was slowing progress on the program, and decided that down-selecting to one contractor prior to milestone B would help accelerate OCX completion. According to Raytheon officials, a contributing factor was the need to align OCX development with that of GPS III development, the development contract for which was awarded 21 months earlier, in May 2008.

In addition, prior to development start, the program did not complete a PDR—which assesses the maturity of the preliminary design and confirms that the system is ready to proceed into detailed design with acceptable risk. As GAO’s best practices work has shown, PDR is a critical step at which customer needs are balanced with available resources. Programs that are launched prior to completing a preliminary design tend to experience more problems compared to programs where launch occurs after completing a preliminary design. Figure 6 below depicts how programs generally have better outcomes when they are launched after conducting relatively more detailed systems engineering: particularly, completing a preliminary design before formally launching. Programs launched after only completing a notional design—a general concept, unconstrained by resources, of what the product will look like and what it might be capable of—often experienced problems and needed more time or money than had been estimated at program launch.

OCX program office and Raytheon officials stated that certain OCX requirements were not well understood at the time of contract award. In particular, Raytheon officials stated that the company did not understand the extent to which it would be required to implement Information Assurance (IA) requirements until as late as 2013, 3 years after beginning OCX development. The purpose of IA, also referred to as cybersecurity, is to ensure that DOD systems can resist and continue to operate during cyber-attacks by managing risks and implementing safeguards. OCX was required to be compliant with the DOD directive that prescribes IA policies, responsibilities, and procedures, among other things, for DOD information technology and defense components. In light of increasing cyber threats and given that satellite ground systems are the most vulnerable components to potential attacks, the Air Force plans for OCX to have an improved IA capability over the current GPS control system. According to program and Raytheon officials, OCX is going to be one of the first large-scale programs within DOD to fully implement IA requirements. In addition, Raytheon officials described cybersecurity threats as continuously evolving, and that both Raytheon and the Air Force have had to adapt their interpretation over time of how to meet IA requirements on OCX development to address changing threats.

Program office and Raytheon officials noted that past DOD acquisition programs routinely waived some of the IA requirements in prior programs, and that Raytheon entered OCX development with the expectation that some IA requirements would be waived as in the past. According to Raytheon officials, at the time OCX development began, neither the Air Force nor Raytheon had the experience—and therefore, the knowledge— of developing such a complex IA-intensive program. Given the importance of GPS to the military and civil communities and with the increase in cybersecurity threats, the Air Force did not waive any IA requirements for OCX. Consequently, Raytheon found that it had greatly underestimated the cost and time to meet these requirements. According to program officials, most of the requirements issues were resolved in early 2015.

Preliminary Design Review Held More Than a Year after Contract Award

The OCX program held PDR in August 2011—more than a year after contract award and after the Air Force had spent over $1 billion on the overall OCX program since 2007 including phase A technology development—with the Air Force affirming that the OCX architecture and design was solid and that the program was ready to begin formal system development. However, at PDR, Raytheon did not prepare—and the Air Force did not assess—a preliminary design for the entire OCX system because Raytheon followed an iterative software development process unlike the traditional, “waterfall” software development process. An iterative approach is a common industry practice that consists of developing software in a series of iterations and blocks, where developers go through multiple cycles of breaking down requirements, and designing, coding, and integrating software. By contrast, in the traditional waterfall approach the system is fully designed before coding and testing follow in a linear sequence. In theory, an iterative approach allows for a balanced and efficient use of resources when developing complex systems. Smaller iterative development cycles—as opposed to a lengthy, linear, waterfall development process—allow for capabilities to more easily be added incrementally, lessons learned to be incorporated, and early integration testing to be conducted to minimize cost and schedule risk. To achieve this, effective systems engineering—the process of deriving and allocating requirements for individual software iterations—is key. The Air Force, acknowledging Raytheon’s use of an iterative approach in late 2010, tailored its preliminary design review criteria—which originally were based on a traditional, waterfall approach—to only review the two iterations completed by Raytheon at that point. For the remaining six iterations, Raytheon had only completed the initial allocation of requirements at a high level. However, as Raytheon encountered problems in software development, it began deferring difficult work to later iterations.

Raytheon was experiencing difficulties developing OCX, but the Air Force presented optimistic assessments of progress to USD AT&L. By October 2011, an independent OSD-chartered review team—one of a series of independent reviews—warned of severe software development problems, particularly with mounting deferred work, ineffective integration testing, and overestimated software productivity rates, and predicted a 15 to 19 month delay to block 1 delivery. According to an official with USD AT&L, these independent assessments are designed to identify program cost, schedule, and performance risks and provide feedback to the program manager and the milestone decision authority. The official further noted that the assessments were considered helpful, but essentially nonbinding advice from department experts to the program manager. The independent team followed up in February 2012 for a more detailed assessment of OCX and predicted a higher, 24-month slip to block 1. However, at the annual GPS enterprise review (AGER) with USD AT&L in April 2012, the Air Force acknowledged the software issues but projected a shorter, 11-month delay citing various corrective actions already taken, including greater oversight. This projection proved to be overly optimistic. In light of the challenges experienced to that point, at the April 2012 AGER, USD AT&L postponed the scheduled milestone B decision for OCX and directed the program to return for a formal Defense Acquisition Board (DAB) review within 120 days and to report on progress.

The Air Force Accelerated OCX Development to Meet Optimistic GPS III Satellite Launch Time Frames Even as OCX Was Experiencing Problems

In the midst of mounting OCX problems, the Air Force disrupted the ongoing software development effort. As it became apparent that OCX block 1 would not be ready in time to support the then-projected launch time frames of the first GPS III satellite (August 2015 and May 2014 respectively), USD AT&L directed the Air Force in January 2011 to separate out development of the satellite launch and initial testing portion of the block 1 software. Dubbed block 0, this subset of software is to contain the capabilities needed to launch and test the initial GPS III satellites. The Air Force modified the OCX contract in January 2012 to implement this change, which included bringing forward IA capabilities originally scheduled for later iterations. Block 0 is a temporary measure because the Air Force anticipated that some GPS III satellites would need to be launched before the entirety of block 1 was to be developed. Block 0 does not allow the GPS III satellites to be incorporated into the overall GPS constellation and used by the military or civil community—which requires block 1 implementation. Further, this decision to accelerate the GPS III launch and test capability was based on Air Force assessments of GPS III progress that appeared overly optimistic, as GAO found at the time. Nevertheless, according to Raytheon officials, the creation of block 0 and acceleration of some software capabilities caused it to have to revamp the OCX software development plan 2 years into development— not in accordance with best practices in software development, which call for stabilizing requirements and design prior to coding. Our prior work in this area shows that too many changes to requirements can result in additional, sometimes unmanageable risk.

Meanwhile, Raytheon continued to struggle with developing OCX, specifically with implementing IA requirements. The contractor was overly optimistic in its initial estimates of the work associated with incorporating open source and reused software, and, according to the Air Force, did not appear to follow IA screening or software assurance processes as required, for example, incorporating open source software without ensuring that it was IA-compliant. The Air Force stated that it was not aware that Raytheon’s software assurance processes were not in compliance with the OCX software development plan until it performed an audit of Raytheon’s secure coding process in August 2012. According to Raytheon officials, however, the development contract did not specifically address the extent that open source software had to be scanned for IA compliance. Nevertheless, this led to significant rework and added cost to remediate the security vulnerabilities and meet IA standards. In addition, Raytheon’s systems engineering was incomplete, resulting in an inability to build code as planned and work being consistently deferred to later iterations. The Air Force noted that it became aware of these systems engineering issues early in development and took some corrective actions such as defining the completion of certain components; however, these actions were focused only on high-priority OCX components.

Nevertheless, in July 2012, the Air Force presented a positive progress report at the mandated DAB review, stating essentially that all technical issues had been identified and mitigated, the design fully validated, and an executable and realistic schedule put in place after a thorough internal examination of the program. While the program stated that sufficient margin had been incorporated into the cost and schedule estimates, the program’s estimates for block 1 delivery—between February and October 2016—were still optimistic compared to the time frames (November 2016 to March 2017) forecast by the independent team in October 2011. The Air Force formally completed milestone B for OCX in November 2012— more than 2 years after contract award. At this point, the Air Force developed its first detailed cost estimate for the program as required by DOD policy, forecasting that OCX development would cost approximately $1.6 billion, which, according to the Air Force, was its first, formal realization of the magnitude to which the contract was initially underbid.

The Air Force Paused OCX Development amid Ongoing Problems and to Identify and Resolve Problems, but Continued to Present Optimistic Assessments to Acquisition Overseers

According to the OCX program office and independent OSD-chartered reviews, Raytheon’s incomplete systems engineering led to continuous rework and deferred requirements to later iterations. In addition, the Air Force made significant changes to certain requirements, particularly with updating the specifications for OCX’s connections to other government systems, and M-code signal requirements. As a result, Raytheon officials estimated that nearly two-thirds of the requirements baseline as of PDR had changed by mid-2012. In March 2013, an independent OSD-led team praised the program for the corrective steps taken, but pointed to a rapidly deteriorating delivery schedule as a result of software development taking much longer than planned and a high defect rate, among other factors. The independent team projected block 1 completion between February and July 2018, a schedule slip of 16 to 21 months over the estimate at milestone B. DOD’s contract performance reviewer, Defense Contract Management Agency (DCMA), highlighted major technical difficulties in OCX development in its monthly analyses from the June to October 2013 time frame, including missed software iteration milestones, concurrent systems engineering and software development, and a high defect rate, among other issues. In addition, DCMA reported as early as August 2013 that the Air Force and Raytheon were going to soon begin an over target baseline (OTB) process where DOD determines that contract budgets are unrealistic and formally increases the program’s budget. An OTB is intended to allow for more realistic budget and work estimates and, therefore, more meaningful performance measurement against the updated budget.

However, at the September 2013 AGER, the Air Force again presented an optimistic assessment of the program to USD AT&L, stating that OCX was on track to meet milestone B cost and schedule estimates, IA challenges had been identified, and key metrics had been established to ensure progress. The program projected just a 2-month delay to block 1 (to December 2016).

Meanwhile, the Air Force noted that an excessive amount of rework was occurring at the time, and it directed Raytheon in November 2013 to pause development and complete a greater level of systems engineering for block 1. In December the Air Force and Raytheon began an OTB process to identify root causes and corrective actions and establish revised cost and schedule goals for OCX, and notified USD AT&L that block 1 delivery was likely to slip by approximately 9 months, to September 2017.

The OTB concluded in June 2014, and identified root causes, many of which were similar to those identified at previous AGERs and independent assessments: incomplete systems engineering, inadequate process discipline, and IA implementation difficulties due to complexity. At this point, the Air Force’s estimate of the program’s development cost had grown to $1.7 billion—about 6 percent more than the milestone B estimate and nearly twice the initial contract estimate in 2010. The Air Force also added a total of 2 years to the program over its estimate at the September 2013 AGER with block 1 delivery moved to November 2018.

Despite the ongoing problems with OCX development, the Air Force, faced with the statutory deadline to deliver M-code capable receivers— and consequently, OCX support for M-code—by fiscal year 2018, added some uncertainty to the OCX development plan. The Air Force agreed with Raytheon during the OTB to concurrently develop a portion of block 2 M-code capability during the block 1 effort; however, the Air Force noted in technical comments on a draft of this report that Raytheon is under no contractual obligation to do so. Complicating matters, the Air Force made significant updates to M-code signal requirements in 2014, contributing to additional OCX requirements changes. At the time of contract award in 2010, the Air Force had only developed a preliminary M-code specification, and it took time for the Air Force to fully mature M- code requirements and specify the functionality required of OCX. According to Raytheon officials, this had a significant effect on the IA requirements and design of certain components of OCX.

The OCX Contractor Has Continued to Struggle with OCX Development and a Realistic, Executable Schedule Is Not Yet Evident

The OCX program’s path forward following the conclusion of the OTB process in June 2014 depended on resolving certain problems in order to meet revised cost and schedule targets. In particular, the Air Force assumed that the contractor would (1) resolve software defects quickly, (2) bring greater discipline to software development processes, and (3) achieve higher software productivity rates than previously demonstrated. However, latest available data as of May 2015 shows that the contractor has not yet resolved these issues, increasing the risk that additional OCX cost and schedule growth is likely.

Defect Rate Remains High and Root Causes May Not Be Fully Identified

A key expectation underpinning the post-OTB baseline was that Raytheon would be able to sharply reduce the number of new defects and resolve them within 6 months. However, the opposite took place. Raytheon uncovered defects at a faster pace than it could resolve them, causing further delays. By October 2014, just 4 months after the OTB, DCMA reported that the defect resolution rate was unsustainable, needing a continual increase in cost and schedule, with the trend likely to continue. In addition, DCMA found that the majority of defects were identified during the later phases of software development at which point it is harder and much more expensive to resolve than if found earlier. Latest available DCMA reports, as of May 2015, showed that Raytheon had yet to bring the defect rate within planned levels. Raytheon officials noted in early July 2015 that the defect discovery rate and backlog have been greatly reduced as OCX prepares to begin testing on iteration 1.5. However, data on the number of defects open and resolved show that the defect backlog as of July 2015 was still more than three times that predicted in February 2015. In addition, DCMA has pointed out that the test activities will likely identify new defects, potentially increasing the backlog further. DCMA further noted that the current defect backlog consists of more complex and difficult defects, which will require considerable effort to close compared to those resolved so far.

The persistently high defect rate for OCX may be a result of as-yet unidentified systemic issues. An October 2014 independent review noted that, given the difficulty in resolving defects as planned, systemic issues may remain but that neither the OCX program office nor Raytheon had conducted detailed engineering assessments to determine if there were any systemic issues. As GAO’s prior work has shown, effective defect management requires a realistic schedule in that it takes time to be able to fully identify, analyze, prioritize, and track defects. Without investing the time and resources to conduct detailed engineering assessments, the program cannot know if any systemic issues are causing the persistently high software defect rate.

Processes Remain Undisciplined

The high defect rate is a symptom of continued struggles with the root cause of undisciplined processes at Raytheon, noted to-date by the OCX program office, independent OSD-chartered review teams, and DCMA. For example, Raytheon has had difficulty establishing consistent software development environments—the computer infrastructure including hardware, operating systems, and databases—across the OCX program. Developers built each of the environments with different hardware and operating system versions and settings. According to Raytheon officials, this was partly because the Air Force’s requirements called for multiple tailored environments to save on hardware costs, but that this drove complexity and posed technical difficulties to Raytheon. This meant that the contractor could not deploy a given software build onto all the environments without a lot of rework—and consequently, time and expense. Following the June 2014 OTB, the Air Force and Raytheon reported that they had taken steps to correct this issue, such as using automated tools to ensure consistency. However, latest available data, as of May 2015, showed that inconsistent environments were still contributing to a high defect rate and consequent cost growth and delays. Raytheon noted that it had greatly reduced the amount of time required to deploy software builds onto a given environment to 3 days as of July 2015; however, this was still well short of the 1-day limit called for by the October 2014 independent review.

Another example of undisciplined processes at Raytheon is its peer review process. DCMA reported in a November 2014 audit that peer reviews—shown by our prior work to be a crucial quality assurance component of software development—were inconsistent and less effective than planned. The OCX software development plan identifies peer reviews as an integral part of the software development process, the main purpose of which is to identify and report defects in software artifacts—work products such as code, software class libraries, and design models—as early as possible in the development life cycle. Effective peer reviews depend, in part, on clear instructions, compliance, and standards, as well as consistent processes. DCMA reported Raytheon having problems with all of these items. In its audit, DCMA noted that the work instructions were often unclear, contradictory, or contained loopholes; many peer reviews were non-compliant with the work instructions; and a high percentage of reviews were held “virtually” where users review artifacts independently instead of during a formal, in- person meeting as presumed in the OCX software development plan. DCMA concluded that the quality of the peer review process was questionable because peer-reviewed software artifacts resulted in an abnormally high number of defects being discovered later. According to Raytheon officials, Raytheon has incorporated changes to its peer review instructions in its software development plan, which is awaiting Air Force approval as of July 2015 and will govern future software development.

Software Productivity Assumptions Are Optimistic

The Air Force and Raytheon also assumed overly optimistic software productivity rates—considering Raytheon’s track record on OCX—when developing the post-OTB schedule. For example, although the post-OTB baseline added nearly 2 years to the prior plan, both the Air Force and Raytheon assumed productivity rates for iterations 1.6 and 1.7 that were approximately two-thirds higher than the rate achieved for iteration 1.5, which was the most complex software effort undertaken to that point. As of the latest available data, from February 2015, Raytheon’s productivity rate had increased above the rate achieved for 1.5 but was roughly only half the forecast increase because of continued difficulties, increasing risk that OCX delivery will be further delayed. According to DCMA, the program has had a history of being overly optimistic with forecasting schedules, for example, using data from Raytheon’s prior experience on other programs that have consistently proven inaccurate when applied to OCX, and assuming efficiencies based on learning curves which have not materialized due to staff turnover, process changes, and poor retention of lessons learned from past development difficulties. This tendency of overestimating software productivity rates was noted as far back as October 2011 as part of an independent OSD-led assessment that highlighted several OCX development problems. Raytheon noted that corrective actions were taken as part of the OTB, such as revalidating and completing systems engineering, establishing a common environment infrastructure, and validating IA implementation during block 0 tests will result in software productivity improvement for iterations 1.6 and beyond beginning in 2016.

USD AT&L May Not Have Adequate Insight into OCX Progress

USD AT&L may not have adequate insight into the full extent of OCX development problems given the Air Force’s consistently optimistic assessments of the program’s progress. The October 2014 advisory independent review estimated that block 1 was likely to take approximately 2 years longer than the June 2014 OTB estimate, putting probable block 1 delivery around November 2020. DOT&E also expressed concerns about OCX delays in a memorandum sent to USD AT&L in early November 2014, citing the negative effect of those delays on the Air Force’s ability to deliver overall GPS capability. In addition, DOT&E called the Air Force’s schedules for the overall GPS enterprise, including that of OCX, “inaccurate, implausible, and incoherent” given OCX development difficulties to date.

Nevertheless, at the ensuing fiscal year 2015 AGER that was held in November 2014, the Air Force acknowledged additional development difficulties but noted that OCX systems engineering was improving and would lead to better software development in the future. At that time, the OCX program office forecast block 1 completion in August 2019—a delay of 9 months over the OTB estimate, but optimistic compared to the independent review’s assessment. One month later, in December 2014, the Air Force presented its official update for the program and forecast a block 1 completion date of July 2019. It also estimated that contract costs had grown to nearly $2 billion—representing cost growth of 16 percent in just 5 months over the OTB estimate, and more than 120 percent over initial contract estimates in 2010.

Following the November 2014 AGER, in December 2014, USD AT&L expressed concern about the continued deterioration in the cost performance of OCX and stated that the program’s trajectory must be corrected. USD AT&L directed the Air Force to provide by January 2015 a “deep dive” program review of OCX focused on the status of and ways to improve program execution. At this review, which occurred in February 2015, the Air Force acknowledged the program’s volatile cost and schedule history and that the contractor’s schedules were aggressive relative to the risk and amount of work remaining. However, the Air Force also highlighted signs that program execution was stabilizing and noted that it had reduced program risk to medium-low by including an additional $331 million and 3 months above contractor estimates. To address USD AT&L’s directive to detail OCX’s path forward and key decision points should OCX continue to sustain cost and schedule growth, the Air Force established a process whereby it would closely monitor key software development events roughly every 6 months beginning in July 2015 and report any deviations from cost performance to the Air Force’s senior acquisition executive as well as to USD AT&L as needed.

While the closer monitoring of key software development events provides USD AT&L with opportunities to spot cost growth on a more timely basis than at the annual AGERs, there is little reason to believe that OCX systemic problems have been adequately addressed. Notwithstanding the Air Force’s optimistic report to USD AT&L in February 2015, additional OCX cost growth is quite likely. First, our analysis of detailed earned value data from November 2013 through November 2014 showed that the program office significantly underestimated the anticipated cost of resolving the risks that could affect OCX development and may have also significantly underestimated the extent of risks. Second, DCMA’s reporting of earned value performance for the program shows a sustained deterioration in program cost and schedule performance immediately following the OTB—where cost and schedule variances were reset to zero in order to begin measuring performance against the new baseline. Cost and schedule variances measure the differences in expectations between the value of work accomplished in a given period with the value of the work expected in that period. Negative variances indicate that the program is either overrunning cost or performing less work than planned; conversely, positive variances indicate the program is either underrunning cost or performing more work than planned. Figure 7 below shows the cumulative cost and schedule variances from October 2013 through last available data as of May 2015, and how the OCX program has been both overrunning cost and performing less work than planned immediately after the OTB concluded. While the OTB process completed in June 2014, the program formally reset the cost and schedule baselines at the end of July.

Further, DCMA’s June 2015 analysis forecast that OCX costs were likely to increase to $2.15 billion based upon, among other things, the higher than expected defect rate and poor comprehension of requirements by Raytheon. From November 2014 through last available data as of May 2015, Raytheon depended on shifting between approximately 100 and 180 additional staff than planned to resolve defects, slowing work on later iterations. According to DCMA, Raytheon will likely need even more staff in the future to maintain schedule, and, consequently, incur additional cost growth. The Air Force’s current schedule estimates, forecasting block 1 completion in July 2019, are still optimistic by at least a year compared to the October 2014 independent team’s assessment that block 1 will most likely be delivered in November 2020. Figure 8 below summarizes the differences in Air Force estimates for the total number of months to complete block 1 and those predicted by independent reviews.

In addition, our prior work on space acquisitions has shown that unrealistic estimates of the achievability of planned schedules, among other things, directly contributed to unrealistic cost estimates and, consequently, distorted management decisions, increased risks to mission success, and virtually guaranteed program delays. By contrast, senior leaders in successful organizations in the commercial sector actively encouraged program managers to share bad news about their programs. These organizations took pains to ensure program estimates are complete and accurate. Based on the persistently high software defect rate, continued undisciplined processes, lower than expected software productivity rates, the downward trend in cost and schedule performance following the OTB, and lack of realism in Air Force depictions of OCX progress, root causes do not appear to have been adequately addressed or perhaps even fully identified. Until all root causes of OCX problems are fixed, the program is likely to continue to struggle to achieve desired outcomes.

Lack of Knowledge about MGUE Likely to Hinder Military Services’ Ability to Make Procurement Decisions

The Air Force has revised the MGUE acquisition strategy several times in attempts to develop M-code capability. Even so, the military services are unlikely to have sufficient knowledge about MGUE design and performance to make informed procurement decisions starting in fiscal year 2018 because it is uncertain whether an important design review will be conducted prior to that time and because operational testing will still be under way.

MGUE’s Acquisition Strategy May Limit Insight into MGUE Design

The Air Force has revised the MGUE acquisition strategy several times as it pursued the program’s development. The latest strategy of record eliminated a key design assessment, the critical design review (CDR), and it is uncertain whether a recent revision to that strategy will include a CDR. As a result, the military services may face a knowledge gap about MGUE’s design stability and maturity. As we have noted in our prior work, positive acquisition outcomes typically require the use of a knowledge- based approach to product development that demonstrates high levels of knowledge before significant commitments are made. We determined that, in keeping with that approach, knowledge gained through a CDR helps supplant risk over time by ensuring that a product’s design will meet customer requirements as well as cost, schedule, and reliability targets. As part of that work, we also found that a knowledge deficit early in a program can cascade through design and production, leaving decision makers with less knowledge to support decisions about when and how best to move into subsequent acquisition phases that commit more budgetary resources. Additionally, we found that demonstrating a stable and mature design, typically via a CDR, is generally considered a prerequisite by leading commercial firms and successful DOD programs for moving forward with a program because it assesses final product design and provides assurance that product specifications have been captured in detailed design documentation.

The 2011 preliminary strategy for MGUE called for development of separate ground, air, and sea receiver cards within a traditional DOD three-phase acquisition process that encompassed separate technology development, system development, and production phases. The program’s 2012 baseline strategy was similar and rolled development of air and sea cards together based on the Navy’s assessment that aviation card development could support the maritime environment. In 2014, the Air Force received approval from USD AT&L to revise the MGUE acquisition strategy again by bypassing a formal system development phase and combining the development and production decision points. USD AT&L approved the associated Air Force acquisition strategy document in April 2015, but just two months later an official from AT&L indicated that—based on a June 2015 memorandum from the Assistant Secretary of Defense for Acquisition—those milestones would be split apart again. According to the AT&L official, it was determined to be impractical to combine the milestones because there were many events that needed to be conducted between them. The official indicated that USD AT&L plans to oversee the program until the development decision point, at which time it would then delegate program oversight to the Air Force.

The last MGUE acquisition strategy of record eliminated a key assessment of MGUE design maturity, the CDR, which we have found in our prior best practices work is typically held mid-way through a program’s development phase and is essentially an assessment of whether a product’s design meets the customer’s requirements. The Air Force stated that a CDR was unnecessary because, among other things, detailed design work normally approved at CDR was completed for MGUE’s PDR in September 2014 and hardware and software designs and cost, schedule, and performance risks typically reviewed at CDR were assessed as part of that PDR. It also anticipated other events would cover and go beyond the purpose of a CDR, including a compatibility certification process to ensure MGUE receiver cards were compatible with the GPS satellite signal; security validation to ensure contractors’ designs met security certification criteria; initial and final verification reviews to assess product maturity and deficiencies; and utilizing multiple contractors with multiple designs from which the military services can choose to meet their requirements.

However, the MGUE program carries design-related risks that would typically be revisited within the CDR context. An April 2015 assessment of the MGUE PDR by the Office of the Undersecretary of Defense for System Engineering noted, for example, that some security and information assurance design details were not addressed as part of PDR and had been deferred to the security verification review scheduled to finish in late summer/early fall 2015. The Undersecretary’s office also pointed out that the refinement of security countermeasures may result in later design changes. In addition, the office stated that MGUE PDR interface designs with the military services’ lead platforms may not be rigorous enough to account for implementing those designs across various operating environments.

Moreover, it is unclear whether MGUE designs meet the military services’ requirements. According to Army program officials, the Army has identified a set of performance gaps for ground and aviation receiver cards between the MGUE program’s technical requirements document and Army operational requirements, including power and thermal incompatibility issues between the ground receiver card and the platform the cards are to support. The MGUE program said that it is unaware of any data showing the ground cards do not meet power and thermal requirements; analysis shows the cards meet service requirements for power as captured in MGUE’s capability development document; the power and thermal issues may be caused by the host platforms rather than the ground cards; and as such, the lead platform program offices would need to make modification to their power supplies or platforms to resolve the issues. Army officials, in contrast, said that the MGUE capability development document does not contain specific power consumption limits and that the Air Force’s GPS Directorate had recently changed power limitations by unilaterally editing the MGUE technical requirements document, against significant objection by the Army. A DOT&E official explained to us that the Air Force believes it has addressed this issue by clarifying the specification to be an average maximum power requirement for MGUE ground cards but also emphasized that this clarification relieves the MGUE contractors of meeting more stringent instantaneous power limits. As a result, the official said, there are potential adverse effects to host platform interface designs which may then require additional development and integration by the military services in order to adapt their platforms to MGUE.

Military Services Are Unlikely to Have Complete Knowledge of MGUE Performance by Fiscal Year 2018

The military services are unlikely to have the knowledge to begin informed MGUE procurement at the start of fiscal year 2018 because operational testing that will provide knowledge about MGUE’s operational effectiveness and suitability for ground, air, and sea environments will not yet have been completed. Prior to operational testing, the Air Force is conducting activities and demonstrations that it believes reduce program risk by providing knowledge about MGUE performance. Those efforts, however, have limitations which call into question their value in assessing MGUE’s ability to meet the military services’ needs.

Based on the MGUE program’s April 2015 acquisition strategy document, as shown in Figure 9 below, formal integration and operational testing for the MGUE aviation receiver card on the Air Force’s B-2 aircraft abuts fiscal year 2018; that same effort for the Army’s DAGR D3/Stryker extends into fiscal year 2018; and it carries into fiscal year 2019 on the GPNTS/Navy’s DDG-51 Arleigh Burke-class destroyer and Marine Corps’ JLTV. In accordance with DOD guidance, these operational tests mark an important point in the assessment of MGUE design and performance— the first time production representative test articles are planned to be incorporated on the lead platforms and tested in a realistic environment. Among other things, per DOD guidance on test and evaluation, the operational tests are expected to serve as field tests that assess the receiver cards’ ability to satisfy the military services’ requirements; current capabilities, including operational benefits or burdens; and the need for further development of the cards to correct performance deficiencies.

The Air Force believes that completion of operational testing is not required for the military services to begin informed procurement planning because data from prior test events can be leveraged. It stated that the MGUE program has participated in various testing and integration risk reduction activities prior to the time lines shown in figure 9, and that those efforts can count towards MGUE integration. Those activities include, according to the Air Force, successful demonstration of technologies on the Army’s Raven unmanned aerial vehicle; demonstration of MGUE receiver cards in operationally relevant environments on a surrogate C-12 aircraft flying similarly enough to the B-2 lead platform to serve as a surrogate; and lead platform-based risk reduction events via prototype GPS units that will be incorporated into the B-2. The Air Force said that testing on surrogate platforms is intended as early risk reduction activity rather than emulation of lead platform environments. Additionally, it stated that it plans to conduct subsystem-level integration on lead platforms prior to system-level integration, and that these efforts also would provide the military services with further information to asses MGUE performance.

Some of the military services, however, believe that the integration activities to date have been more limited than indicated by the Air Force and that they are now responsible for more of the MGUE development effort than indicated earlier in DOD policy. Army officials stated, for example, that the MGUE program began initial contact with the DAGR D3 program office in fiscal year 2014, but that integration with the DAGR D3/Stryker per se has not yet started due to lack of MGUE devices for platform integration. Instead, they said, “fit check” tests were conducted in early 2015 and showed the DAGR D3 is unable to provide sufficient power to two of the three MGUE contractors’ ground receiver cards. The officials also said that the Army planned to perform thermal testing with the DAGR D3/Stryker, but such testing has not yet been executed due to lack of MGUE devices. They emphasized that they have begun planning for MGUE procurement but cannot initiate actual procurement at the start of fiscal year 2018 for several reasons, including the need to modify as- built MGUE devices to close performance gaps, resolve power and thermal issues by either modifying one MGUE card design or the integration on some 100 platforms; and developing and implementing a means for MGUE and the currently-fielded Selective Availability Anti- Spoofing Module GPS receiver to coexist and interact in the field. In addition, both Army and Navy officials stated that, based on their understanding of DOD’s 2006 policy on GPS user equipment and procurement, the Air Force was to develop MGUE to production-ready status. They noted that, in order to bring MGUE to true production-ready status, their respective military services will need to do further development after the Air Force completes its MGUE development effort.

Moreover, the demonstrations and risk reduction activities that the Air Force points to may not establish MGUE performance to the extent that the Air Force believes. For example, DOT&E—the DOD independent test authority charged with ensuring a program’s operational testing confirms operational effectiveness and suitability in combat use—expressed serious concerns about the Air Force’s characterization of the results from one of the past demonstrations. In a November 2014 memorandum to USD AT&L, DOT&E emphasized that the Air Force had overstated MGUE development maturity and that the demonstrations to-date had achieved more mixed results than the Air Force indicated. DOT&E noted, for example, that the testing with the Army’s Raven did not use final development models; had employed a limited subset of realistic threats; successfully flew its intended flight pattern in just 7 out of 51 attempts; and that none of those flights was conducted in the presence of electronic jamming. It went on to say that developmental activity with the Raven should in no way be construed as having the level of rigor of operational testing or even formal developmental testing. Furthermore, Army program officials stated that the Army’s lead platform had been changed from the Raven to the DAGR D3/Stryker not only because the Army does not intend to procure additional Ravens but also because the Raven does not adequately stress the capabilities of the ground receiver card.

In addition, DOT&E concluded in its 2014 memorandum that MGUE cannot be considered effective until it is successfully integrated on host platforms. According to a DOT&E official, subsystem integration does not guarantee proper integration into a larger system and may not necessarily demonstrate integration and performance in the intended operational environment—particularly in light of the fact that the MGUE test strategy is inherently risky in that only 4 lead platforms are designated to represent the operating conditions for over 100 platforms service-wide. He also emphasized—as did DOT&E’s 2014 memorandum —that DOD has found integration historically challenging.

Our prior work on integration and testing has also shown that integration is a common risk in system development, and that commercial firms reduce risk by capturing design and manufacturing knowledge early. Our previous work pointed out, for example, that systems integration problems can occur even though the various components performed successfully on previous systems. We have found, in illustration, that the Air Force’s C-17 aircraft program intended to use current, available, and proven technology to minimize development costs and structure a low technical risk effort but that the integration of technologies was a major engineering and management task that eventually contributed to significant cost increases. In addition, we have also determined that it is during integration and testing that problems are likely to be found. Furthermore, as we pointed out in a 2000 study of best acquisition practices for testing among commercial firms, test weaknesses invariably cause negative program outcomes, such as cost increases, schedule delays, or performance shortfalls. Our work on best commercial acquisition practices also shows that such negative outcomes can be avoided by accumulating knowledge prior to beginning production. In a 2002 study of best commercial practices, for example, we found that leading commercial firms reduce program risk by demonstrating fully- integrated prototypes prior to making production decisions. Such demonstrations, we determined, help the firms decide when to make the transition from product development to production and to ensure that transition is smooth.

The military services can postpone procuring MGUE if they lack sufficient knowledge about MGUE design and performance after fiscal year 2017. They can, as provided by the National Defense Authorization Act for Fiscal Year 2011, request a waiver of such procurement. Specifically, the Secretary of Defense may waive this limitation upon a determination that suitable M-code capable user equipment is either not available or DOD does not require that user equipment be capable of receiving M-code from GPS. An official from USD AT&L stated that the waiver process has not yet been established.

GPS Constellation Has Been More Reliable Than Expected, but M-code Deployment Is Likely More Than a Decade Away

In 2010, we examined GPS satellite reliability data and reported that a delay of 2 or more years to the launches of GPS III satellites—originally projected to begin launching in April 2014—would likely reduce the constellation size below the minimum requirement of 24 operational GPS satellites by fiscal year 2018. Since that report, while the GPS III and OCX programs have been delayed, the GPS constellation has proven to be more reliable than previously expected because the Air Force successfully extended the life of existing satellites, primarily by modifying the satellites’ battery charging procedures. However, the Air Force now needs OCX block 1 to be operational by late 2019 to launch and incorporate GPS IIIs into the constellation. Given the ongoing development problems with OCX, delivery by that date—and consequently, overall constellation reliability—is at some risk. The Air Force is exploring contingency plans, including modifying the current ground system to launch and operate GPS IIIs. However, the modified ground system would operate GPS III satellites as current legacy GPS satellites, thereby not utilizing all the capabilities on the new satellites. Moreover, this approach would increase the risk that several IIIs may be launched before they can be fully tested with OCX. Delays to OCX delivery also mean that M-code functionality—which requires OCX block 1 to be fully enabled—is not scheduled to be deployed until 2019 at the earliest, and likely for another decade or more until the military services can widely deploy modernized MGUE receivers.

The Air Force Has Found Ways to Extend the Life of the GPS Constellation

The performance standards for (1) the standard positioning service provided to civil and commercial GPS users, and (2) the precise positioning service provided to military GPS users, commit the U.S. government to at least a 95 percent probability of maintaining a constellation of 24 operational GPS satellites. In September 2010, we reported that delays to the launch of the GPS III satellites—originally projected to begin in April 2014—could affect the long-term probability of maintaining the minimum constellation size. Based on satellite reliability data provided by the Air Force, we had predicted that, for example, a 2- year delay in the production and launch of the first and all subsequent GPS III satellites would reduce the guaranteed size of the constellation (at the 95 percent confidence level) to about 18 satellites by around fiscal year 2018.

Since our 2010 report, the launch availability date for the first GPS III satellite has, in fact, slipped by about 3 years, to May 2017, mainly due to development difficulties with its navigation payload. Moreover, as discussed earlier, the delivery of OCX block 1—required to operate the IIIs—has been delayed by about 4 years, to mid-2019. However, the 3- year delay in GPS III’s delivery has not resulted in a predicted significant reduction in the size of the GPS constellation largely because the expected life of the existing generations of GPS satellites has risen dramatically. Figure 10 below displays the probability of maintaining a 24- satellite GPS constellation as a function of time based on March 2014 satellite reliability data and launch schedules—the latest date for which a complete set of approved parameters is available. The launch schedule has been adjusted to reflect the successful launches of four GPS IIF satellites through June 2015.

The expected lifetimes of the GPS IIR and IIR-M satellites—the types of satellites that currently comprise the majority of the current constellation—have risen dramatically since our 2010 report, from 14.8 and 10.5 years, respectively, to 20.4 and 17.5 years, respectively. This increase in expected lifetime is due mainly to the Air Force’s implementation of a modified charging procedure that will prolong the life of the satellites’ batteries. Spacecraft batteries—like all other rechargeable batteries—are subject to cycle life, that is, they can only be charged and discharged so many times before they will not charge efficiently anymore. Because satellite batteries were predicted to be one of the primary life-limiting components of these satellites, the Air Force developed and tested a procedure in 2012 to reduce the battery charge rates during certain times of the year. The Air Force estimated that this new procedure would increase the expected lifetimes of the IIR and IIR-M satellites by 1 to 2 years. Based on recent telemetry data, the new procedure has proved successful. In 2014, the Air Force announced that this modified battery charging procedure has added, in aggregate, 27 additional years of operational life to the GPS IIR and IIR-M satellites. Moreover, according to Air Force representatives, additional information suggests that this estimate is probably conservative, and that the next formal review of satellite reliability data will likely reveal an additional increase in the expected life of the IIR satellites. Because these satellites form the mainstay of the current GPS constellation, and because most of them are still expected to be operational in 2017, when the first GPS III satellite is planned to be available for launch, an increase in the expected life of these older satellites significantly improves the Air Force’s ability to meet the 24-satellite performance standard despite the 3-year slip in GPS III and 4-year slip in OCX.

The expected lifetimes of the GPS IIA satellites—the oldest version of GPS satellites on orbit—have also risen since our 2010 report, from 16.5 years to 19.7 years, due to positive trends in on orbit performance. However, the effect of this increase on the Air Force’s ability to meet the 24-satellite performance standards is less significant than the effect of the IIR/IIR-M life expectancy increase. The current constellation availability analysis assumes each of these satellites is only available to replace other satellites in the unlikely event of a large number of satellite failures until OCX is operational. Because OCX is not expected to be capable of operating the IIA satellites, the analysis assumes that any remaining IIA satellites would be turned off by July 2019, the date that OCX is scheduled to be required to be ready to transition to operations.

Additional OCX Delays Could Affect Constellation Availability

Despite the delays to GPS III and OCX, the GPS constellation is expected to meet the constellation size requirement if OCX block 1 is available by July 2019 as needed. However, further OCX delays could affect overall constellation availability, bringing it below the 95-percent performance standard for maintaining a 24-satellite constellation for a year or more. For example, if OCX block 1 is not ready to transition to operations until November 2020—as the most recent (October 2014) independent review of OCX estimated—the Air Force could experience a 15-month period during which it would not meet its commitment to maintaining a 24-satellite constellation with a 95 percent confidence level. While GPS III satellites could still be launched beginning in 2017, they would not be added to the constellation until OCX comes on line or until a contingency operational control capability able to command GPS III satellites is available. Figure 11 depicts the predicted constellation size based on OCX block 1 being potentially delayed until November 2020.

The Air Force Is Taking Actions to Sustain the GPS Constellation Given Delays to OCX, but These Efforts Will Not Fully Exploit GPS Capabilities

GPS capabilities are unbalanced, with satellite capabilities outstripping those supported by the ground and user equipment segments. In particular, the Air Force has launched two generations of satellites—IIR-M and IIF—capabilities of which are still not fully utilized because they have been transmitting three signals, including M-code, that are not supported by the current GPS ground control system, OCS.

GPS IIFs are nearing the end of their launch, with the last three scheduled to be launched between July 2015 and February 2016. GPS IIIs are needed to continue to sustain the constellation. Since OCX block 1 is needed to operate the GPS III satellites as part of the constellation, the Air Force is preparing contingency plans in the event that OCX block 1 is delayed beyond August 2019, the need date for the first GPS III satellite. The Air Force has pinpointed one option as the most viable— modifying the current OCS system to operate the GPS III satellites but at the older, IIF satellite level of functionality; this option would deliver M- code support without all modernized GPS III functionality, specifically support for the L1C signal. Under this plan, the Air Force would use OCX block 0 to launch and initially test the GPS III satellites, and use the modified OCS to control the satellites’ navigation signals as part of the overall constellation. As figure 12 shows, successfully implementing this contingency plan should enable the Air Force to meet its commitment to maintaining a 24-satellite constellation with a 95 percent confidence level, even if the OCX block 1 delivery date were to slip to November 2020 (or later), but the plan would require funding and other resources.

The Air Force estimated the cost of this option at $105 million as of February 2015; however, pursuing this option means that the Air Force may launch up to 5 GPS III satellites without fully testing them with OCX block 1, increasing the risk that issues may be found during testing without the ability to fix the satellites already launched.

Initial M-code Capability Will Not Be Available until OCX Delivery in Mid-2019, and Full M-code Deployment Is Likely at Least Another Decade Away

M-code initial operational capability is defined as having 18 M-code capable satellites on orbit, the control segment able to command/upload M-code capabilities to the space segment, and MGUE receivers fielded across the military services to utilize M-code capabilities operationally. Full operational capability occurs when 24 M-code satellites are on orbit, and a larger portion of MGUE is fielded.

The GPS constellation will likely include 18 M-code capable satellites by September 2015, but M-code capabilities will not be available to users— assuming MGUE receivers can be fielded—until OCX delivery in mid- 2019. The launching of M-code capable satellites has been stretched over a much longer period than originally envisioned because of delays to the IIF, III, OCX, and MGUE programs. However, because the oldest of the M-code capable satellites are lasting longer than originally predicted, the effect of this stretched deployment has been mitigated. In particular the oldest M-code capable satellite—the first IIR-M satellite, launched in September 2005—has a better than a 75 probability of still being operational in October 2020, and better than a 50 percent probability of still being operational in October 2023. Nevertheless, DOD cannot take full advantage of M-code capability until MGUE receivers are deployed in sufficient numbers across the services. As noted earlier, the earliest MGUE operational test will not be completed until the end of fiscal year 2017 and the latest at the end of fiscal year 2019. Accordingly, some of the military services will not be able to make informed fielding decisions until fiscal year 2020 at the earliest, assuming operational testing goes as planned.

Furthermore, as discussed earlier, the Air Force plans to deliver production representative test articles, which the military services will then acquire, integrate, and operationally test only on selected military platforms. As GAO reported in 2009, each of the military services would still need to add the new user equipment to other platforms, which could take an additional 10 or more years based on the need to perform such activities as coordinating installation with the platforms’ maintenance and upgrade schedules.

Conclusions

GPS is a global utility that is integral to U.S. national security and civilian use. As a result, any decision about GPS has far-reaching consequences. A combination of many factors, including technical challenges, poor contractor execution and program management, and ineffective acquisition oversight have all put GPS modernization at significant risk.

OCX is the key to enabling the full GPS capability that both the military and civilian communities depend on, from command and control of the satellite constellation including the new GPS III satellites, to allowing military receivers to take advantage of M-code signals for more robust warfighting capability, to enabling advanced civil GPS signals. However, by any measure, OCX development has been mired in development difficulties resulting in steady cost growth and schedule delays. Moreover, despite a 7 month pause ending in mid-2014, OCX has yet to turn the corner on resolving the problems that have affected the program since development began in 2010. Independent observers such as OSD-led teams, DCMA, and DOT&E have raised red flags about the effectiveness of the Air Force’s oversight and Raytheon’s ability to deliver promised outcomes. The Air Force has compounded matters by consistently presenting overly optimistic assessments about OCX progress— demonstrating a pattern of marginalizing warnings about OCX delays presented by independent assessors, likely because their assessments are considered advisory in nature. Driving towards unrealistic timeframes is compounding the program’s inability to meet stated cost and schedule goals. Five years into what was originally estimated to be a 5-year effort, OCX is still roughly 5 years away from completion. Without comprehensively identifying systemic causes for OCX problems, DOD cannot have high-confidence cost and schedule estimates for OCX. The Air Force and Raytheon have noted that OTB corrective actions will begin paying off in early 2016 as block 1 development resumes in earnest. But given the lack of success in prior attempts to turn the program around, the Air Force could benefit from external expertise and guidance on what is necessary to address systemic issues.

Since the enactment of the statute directing DOD to generally procure only M-code capable user equipment after fiscal year 2017, the Air Force has struggled to deliver a MGUE acquisition strategy that would allow the military services to comply with that direction. As it stands now, the Air Force essentially truncates its MGUE development work and hands the result of its efforts off to the military services to continue development on their respective platforms. At the point the Air Force provides funds to the military services for them to acquire and test MGUE, it also transfers the onus of the development work to the military services’ shoulders because MGUE development cannot be considered complete until the cards work in the lead platforms. Unfortunately, the military services will absorb an added development burden because the Air Force plans to transfer its work without conducting design and performance assessments that could help the services decide the extent to which the cards are ready to integrate and test and how much additional work they will need to do before beginning to procure the cards.

Fortunately, many GPS satellites now on orbit have served the nation particularly well by working considerably longer than expected. Even so, it cannot be presumed that they will continue to do so. Therefore, it is critical that the modernized GPS III, OCX, and MGUE development efforts succeed sooner rather than later. Of those three programs, OCX is now the pacing item for modernization due to its many past delays and probable future delays. Until the OCX program trajectory is corrected, those delays are likely to pose significant risks to sustaining the GPS constellation, and consequently, delivering GPS capability to the military community.

Recommendations for Executive Action

To better position DOD as it continues pursuing GPS modernization, to have the information necessary to make decisions on how best to improve that modernization, and to mitigate risks to sustaining the GPS constellation, we recommend that the Secretary of Defense take the following five actions:

Convene an independent task force comprising experts from other military services and defense agencies with substantial knowledge and expertise to provide an assessment to USD AT&L of the OCX program and concrete guidance for addressing the OCX program’s underlying problems, particularly including:

A detailed engineering assessment of OCX defects to determine the systemic root causes of the defects;

Whether the contractor’s software development procedures and practices match the levels described in the OCX systems engineering and software development plans; and

Whether the contractor is capable of executing the program as currently resourced and structured.

Develop high confidence OCX cost and schedule estimates based on actual track record for productivity and learning curves.

Direct the Air Force to retain experts from the independent task force as a management advisory team to assist the OCX program office in conducting regular systemic analysis of defects and to help ensure OCX corrective measures are implemented successfully and sustained.

Put in place a mechanism for ensuring that the knowledge gained from the OCX assessment is used to determine whether further programmatic changes are needed to strengthen oversight.

To allow the military services to fully assess the maturity of the MGUE design before committing test and procurement resources, incorporate a CDR in the Air Force’s MGUE development effort.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Defense for review and comment. In its written comments, reproduced in appendix II, DOD concurred with the four OCX-related recommendations calling for a more robust independent review of the program, stating that the intent of those recommendations has been met by OSD-chartered independent reviews conducted to-date, which included both Air Force and OSD staff, and other DOD activities governing OCX oversight, such as the use of independent cost estimates. DOD partially concurred with the recommendation calling for incorporating a CDR in the MGUE program’s development effort, noting that adding a CDR would delay the program amidst its efforts to compress the acquisition process to deliver MGUE capability by fiscal year 2018. DOD also provided technical comments, which we have incorporated as appropriate.

While DOD concurred with our recommendations to put OCX on a better path forward, it responded that it had essentially been following the intent of our recommendations all along, requiring no further action from the department. These comments provide little confidence that the department intends to fully implement our recommendations to fix the development problems that have beset the program since its inception. As we noted in our report, the independent reviews conducted to-date have been nonbinding and advisory in nature. Even as the OSD-led reviews have warned of significant delays and inadequate insight into systemic root causes of development problems, the Air Force has repeatedly developed unrealistic timeframes, ostensibly with OSD’s knowledge. If business continues as usual without swift and thoughtful action, OCX will likely continue on its path of demonstrating poor cost and schedule outcomes. We continue to stand by our recommendations calling for a fresh review—this time an in-depth and comprehensive critical review of the program—to identify the true root causes of OCX development difficulties and to ensure the Air Force implements the corrective actions.

Regarding DOD’s response to our recommendation that DOD hold a CDR for MGUE, we note that DOD’s rationale for skipping this best practice step is based on a desire to accelerate fielding of the units. However, our past work has consistently shown that taking shortcuts and skipping important knowledge points in the acquisition process generally results in an inability to deliver promised cost and schedule outcomes. To minimize any potential disruption to development efforts, a CDR could be held after production-representative test articles are delivered. The results of a CDR will show whether the MGUE design is stable. Rushing into lead platform testing without an approved and stable design means that DOD will have to concurrently test and correct the receivers’ design at a point where resolving issues are typically more expensive than earlier in development—not in accordance with best practices that are important to achieving cost and schedule outcomes with a high degree of confidence.

We are sending copies of this report to the appropriate congressional committees, the Secretary of the Defense, the Secretary of the Air Force, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

In May 2014 House Report No. 113-446, accompanying H.R. 4435, the Howard P. “Buck” McKeon National Defense Authorization Act for Fiscal Year 2015, noted the requirements stated in section 913 of the Ike Skelton National Defense Authorization Act for Fiscal Year 2011 (Public Law No. 111–383) requiring the Department of Defense (DOD) to purchase M-code capable user equipment during the fiscal years after fiscal year 2017 and included a provision for us to report on DOD’s progress in deploying M-code capability. The House report also provided that our report assess current and planned investments; whether key milestones are being met; the projected ability to the meet the requirements in section 913 of Public Law No. 111–383; and an identification of the challenges that Global Positioning System (GPS) faces and possible recommendations on how to make the program more successful in delivering M-code capabilities. Additionally, in June 2014, Senate Report No. 113-176, accompanying the Carl Levin National Defense Authorization Act for Fiscal Year 2015, included a provision for us to review the cost, scope, and schedule of the GPS III operational control system, including synchronization with the launch of the GPS III constellation with recommendations for improvement.

In response, for this report our objectives were to assess (1) the extent to which DOD is meeting cost, schedule, and performance requirements for next generation operational control system (OCX), (2) the progress DOD has made in delivering M-code capable military GPS user equipment (MGUE) by the end of fiscal year 2017, and (3) the challenges DOD faces in synchronizing the development of GPS III satellites, OCX, and MGUE to deploy M-code.

To assess the extent to which DOD is meeting cost, schedule, and performance requirements for OCX, we reviewed program and contractor cost and schedule documentation, including program acquisition baselines, earned value metrics, and test plans. Throughout this report, we focused on the costs of the OCX development contract instead of the full program acquisition costs because the latter includes prior, technology development expenditures as well as costs of management support and enterprise integration support services. We analyzed the progress made against planned program milestones and reviewed technical documentation such as software development plans to gain insights into OCX progress. In addition, we reviewed briefings and schedule documentation provided by program and contractor officials to determine changes in OCX cost, schedule, and performance over time. These documents included annual GPS enterprise reviews, OCX program assessments, and program status briefings. We also interviewed officials from the Air Force Space and Missile Systems Center GPS and OCX program offices, OCX prime contractor Raytheon, and Defense Contract Management Agency (DCMA) officials charged with oversight of the OCX contractor efforts to identify and assess cost and schedule issues facing the program’s development efforts, major program risks, and technical challenges. Finally, we interviewed officials from DOD’s Office of Cost Assessment and Program Evaluation and Director, Operational Test and Evaluation office to discuss cost, schedule, and performance challenges for OCX. When program documents identified program events by fiscal quarter rather than by month, we used the last month of the given quarter as the date of the event.

To determine the progress DOD has made in delivering M-code capable MGUE by the end of fiscal year 2017, we reviewed and analyzed program plans and documentation related to cost, schedule, acquisition strategy, technology development, and major challenges to delivering MGUE Increment 1. We then compared the information we obtained to GAO’s criteria for best practices in system development. To assess the program’s progress and challenges, we held discussions with and received information from officials at the Air Force Space and Missile Systems Center GPS and MGUE program offices; Office of the Undersecretary of Defense for Acquisition, Technology, and Logistics; DOD’s Office of Cost Assessment and Program Evaluation; and DOD’s Director, Operational Test and Evaluation. Additionally, to understand their respective development efforts and challenges, we interviewed contractor officials from the three MGUE prime contractors, L-3 Interstate Electronics Corporation, Raytheon Space and Airborne Systems, and Rockwell Collins Inc. as well as Defense Contract Management Agency officials overseeing those contractors. To identify the military services’ concerns about MGUE development, schedule, and integration and testing with their respective platforms, we interviewed officials from the lead platform program offices for the Army’s Defense Advanced GPS Receiver Distributed Device/Stryker, Air Force’s B-2 aircraft, Navy’s Arleigh Burke Class DDG-51 ship, and Marine Corps Joint Light Tactical Vehicle. As with OCX, when program documents identified program events by fiscal quarter rather than by month, we used the last month of the given quarter as the date of the event. We did not assess MGUE Increment 2 because it has not yet reached its technology development milestone.

To identify the challenges faced in synchronizing GPS III satellite, OCX, and MGUE to deploy M-code capability, we built on our work assessing the plans, schedules, and challenges of the OCX and MGUE programs by reviewing integrated master schedules and analyzing approved GPS constellation reliability parameters. We also interviewed cognizant officials within Air Force Space Command (AFSPC) and the Aerospace Corporation, which supports the Air Force’s Space and Missile Systems Center (SMC), to evaluate predicted reliability of the GPS satellite constellation. We did not assess cost, schedule, or performance for GPS III satellites.

To assess the status of the GPS constellation, we interviewed officials from the Air Force SMC GPS program office and AFSPC. To assess the risks that a delay in the acquisition and fielding of GPS III satellites could result in the GPS constellation falling below the 24 satellites required by the standard positioning service and precise positioning service performance standards, we employed a methodology very similar to the one we had used to assess constellation performance in 2009 and 2010. We obtained information from the Air Force predicting the reliability for 57 GPS satellites—each of the 39 current (on-orbit as of March 2015) and 18 future GPS satellites—as a function of time. Each satellite’s total reliability curve defines the probability that the satellite will still be operational at a given time in the future. It is generated from the product of two reliability curves—a wear-out reliability curve defined by the cumulative normal distribution, and a random reliability curve defined by the cumulative Weibull distribution. For each of the 57 satellites, we obtained the two parameters defining the cumulative normal distribution, and the two parameters defining the cumulative Weibull distribution. For each of the 18 unlaunched satellites we included in our model, we also obtained a parameter defining its probability of successful launch, and its current scheduled launch date. The 18 unlaunched satellites include 3 IIF satellites and 15 III satellites; launch of the final III satellite we included in our model is scheduled for March 2025. Using this information, we generated overall reliability curves for each of the 57 GPS satellites. We discussed with Air Force and Aerospace Corporation representatives, in general terms, how each satellite’s normal and Weibull parameters were calculated. However, we did not analyze any of the data used to calculate these parameters.

Using the reliability curves for each of the 57 GPS satellites, we developed a Monte Carlo simulation to predict the probability that at least a given number of satellites would be operational as a function of time, based on the GPS launch schedule as of December 2014. We conducted several runs of our simulation—each run consisting of 10,000 trials—and generated “sawtoothed” curves depicting the probability that at least 24 satellites would still be operational as a function of time. We then used our Monte Carlo simulation model to examine the effect of a delay to the delivery of OCX block 1 until November 2020, which would delay the introduction of GPS III satellites into the operational constellation. We then reran the model based on this assumption, and calculated new probabilities that at least 24 satellites would still be operational as a function of time. Finally, we simulated the effect of the Air Force’s proposed contingency plan, which would enable GPS III satellites to be added to the operational constellation, with limited functionality, prior to the delivery of OCX block 1.

We conducted this performance audit from August 2014 to September 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Dr. Nabajyoti Barkakati, Chief Technologist; Art Gallegos, Assistant Director; Jason Lee, Assistant Director; Karen Richey, Assistant Director; Jay Tallon, Assistant Director; Marie P. Ahearn; Pete Anderson; Brandon Booth; Brian Bothwell; Raj Chitikila; Tana Davis; Roxanna Sun; and Hai V. Tran made key contributions to this report.