Global Positioning System: Updated Schedule Assessment Could Help Decision Makers Address Likely Delays Related to New Ground Control System
Released: 2019-05-21T00:00:00

Letter

Background

Collectively, the ongoing GPS acquisition efforts aim to (1) modernize and sustain the existing GPS capability and (2) enhance the current GPS system by adding a more cybersecure ground system that enables M- code. M-code is a stronger, encrypted, military-specific GPS signal designed to meet military positioning, navigation, and timing needs. It will help military users overcome GPS signal jamming by using a more powerful signal and protect against false GPS signals, known as spoofing, by encrypting the signal. Figure 1 below shows how GPS satellites, ground control, and user equipment—in the form of receiver cards embedded in systems—function together as an operational system.

The Air Force’s OCX program is primarily a software development effort to replace the current ground system, the operational control system (OCS), with a modernized and more cybersecure system. OCS lacks modern cybersecurity protections and cannot currently control—or enable—modernized features of the three latest generations of GPS satellites now in orbit, including M-code and three new civilian signals. Because existing military receivers were not designed to work with the new M-code signal, military users will have to make investments in new receiver development and procurement timed to when the new signal will be available before they can use it.

Raytheon is the prime contractor working to deliver OCX in a series of blocks that enable additional capabilities. Block 0, which is a subset of block 1 broken out after development started, was delivered in September 2017. It helped to successfully enable the launch and initial testing of the first GPS III satellite, which was launched in December 2018, and will continue to support subsequent GPS III satellite launches. Blocks 1 and 2, originally planned as separate deliveries, have been combined into a single delivery and are currently in development. This combined delivery enables OCX to command and control each satellite and begin using the full M-code signal, as well as control new civilian signals, among other capabilities.

Because of significant delays to OCX, the Air Force initiated two additional programs to modify OCS to deliver some of the planned capabilities before OCX is operational. The first program is Contingency Operations (COps)—which will enable the control of GPS III satellites to operate with the same capabilities as current GPS satellites without the additional military and civilian signals. The second program is M-code Early Use (MCEU)—which will permit some functions of M-code to be used before OCX is delivered. Neither COps nor MCEU will enable the additional civilian signals or the full M-code functionality that is expected with OCX.

Acquisition Cost and Schedule Baselines

DOD is required by statute to establish and approve cost and schedule baselines for major defense acquisition programs before those programs enter system development, also known as the engineering and manufacturing development phase. As part of program planning, including for major defense acquisition programs, DOD policy requires program managers to establish program goals for cost, schedule, and performance parameters. Approved program baseline parameters are reported in the program’s acquisition program baseline as objective and threshold values. The objective values represent goals in terms of what the user—in the case of GPS, the Air Force—desires and expects. The threshold values represent the limit of what is acceptable—meaning cost or schedule growth above threshold values are outside of the approved cost or schedule limits.

For OCX, the cost and schedule objective and threshold dates in the baseline are tied to an event called “ready to transition to operations,” which will be the completion of the OCX acquisition program schedule. For the OCX program, this is a decision within the Air Force to switch control of the GPS constellation from the current GPS ground system, OCS—at this future point with COps and MCEU modifications already added—to OCX. The delivery date of the system by Raytheon and acceptance date by the Air Force will both come before the ready to transition to operations decision. These two dates are important because their timing may influence when OCX operates.

What is a critical Nunn-McCurdy unit cost breach?

For major defense acquisition programs, a critical Nunn-McCurdy unit cost breach of a unit cost threshold is triggered by cost increases of at least 25 percent or more of a program’s current cost baseline or at least 50 percent or more of a program’s original cost baseline.

As an acquisition program works to achieve its objective and threshold values, the original baseline goals may become unachievable. When this occurs, a revised baseline, or rebaseline, is created so the program’s cost and schedule goals are updated to more realistically reflect the program’s current status. If the increase from the cost baseline meets certain thresholds, DOD is required to notify Congress in writing. This is known as a Nunn-McCurdy breach. This notification assists Congress with monitoring program progress, especially on troubled programs. A critical Nunn-McCurdy unit cost breach is the most serious type of breach and requires a program to be terminated unless the Secretary of Defense submits a written certification to Congress that certain criteria have been met, including that the new estimate of the program’s cost has been determined to be reasonable by the Director of DOD’s Office of Cost Assessment and Program Evaluation, and takes other actions, including restructuring the program.

History of Increasing OCX Cost and Schedule Baselines

As we have previously reported, the Air Force has had significant difficulties developing OCX. The program’s cost and schedule baselines have been unstable and unexecutable since the first baseline was established in 2012. In total, there have been three OCX program baselines: 1. November 2012 original baseline at development start, 2. October 2015 rebaseline due to a schedule breach, and 3. September 2018 rebaseline prompted by a critical Nunn-McCurdy unit cost breach.

Since 2012, reflecting the newest baseline and additional cost and schedule growth since the Nunn-McCurdy breach, the schedule has more than doubled and the costs have grown by approximately 68 percent. Figure 2 shows the three OCX baselines with their schedule and cost growth since the start of development.

The National Defense Authorization Act for Fiscal Year 2017 required an independent assessment of OCX. The Act required an assessment of the Air Force’s ability to complete blocks 0 through 2 on a schedule necessary to transition OCX to full operation and an estimate of the cost, among other issues. The MITRE Corporation conducted the study and DOD provided it to congressional defense committees in December 2017.

As a result of the 2016 Nunn-McCurdy unit cost breach, the program repeated the milestone associated with system development start and established new cost and schedule objectives and thresholds, conducted a baseline review of the schedule to verify the work necessary to complete the program, and received approval of the acquisition program baseline by the milestone decision authority—the Under Secretary of Defense for Acquisition and Sustainment.

To support certification of OCX’s new baseline, in May 2017 the Air Force produced an $8.7 billion OCX service cost position for development, sustainment, and disposal. The Air Force service cost position was subsequently reaffirmed in 2018 by the Air Force and supported by an additional independent cost estimate from DOD’s Office of Cost Assessment and Program Evaluation in June 2018, which was approximately 3 percent higher in cost for the development portion. The Under Secretary of Defense for Acquisition and Sustainment selected the Air Force service cost position for the OCX baseline.

Root Causes of Schedule Delays

In 2014, the Air Force identified root causes for OCX cost and schedule growth and concluded that the problems were driven by (1) incomplete systems engineering, (2) inadequate process discipline, and (3) difficulties implementing cybersecurity due to its complexity.

We reported in 2015 that the program office paused development in late 2013 to fix what it believed were the root causes of development issues, and significantly increased the program’s cost and schedule estimates. Despite the pause to address root causes, problems persisted and in the same report we questioned whether all root causes—such as a persistently high software development defect rate—had been adequately identified, let alone addressed, and whether realistic cost and schedule estimates had been developed. We also found that the program was not following various acquisition best practices, such as the completion of a preliminary design review prior to development start. In 2015, we recommended that DOD assemble a task force to assess the OCX program and provide concrete guidance for addressing program problems, to determine root causes for OCX defects, and to establish a high confidence schedule and cost estimate, among other recommendations. DOD concurred with our four OCX-related recommendations and has taken some steps to implement some of them. However, to date, none have been fully implemented and they remain open.

In 2016, DOD’s Director of Performance Assessments and Root Cause Analyses concluded that the root causes for OCX’s Nunn-McCurdy unit cost breach were (1) an unrealistic schedule driven by the need to sustain the GPS constellation, (2) an underestimation of the cost to fully implement information assurance, or cybersecurity, and (3) poor performance by both the government, caused by a lack of requisite software expertise, and Raytheon, caused by poor systems engineering that led to significant rework. We found and DOD’s 2016 root cause analysis has shown a significant and recurring cause of delays on the OCX program has been a lack of mutual understanding of the work between the Air Force and Raytheon.

In December 2017, we found risks to the latest proposed (but not yet then approved) OCX schedule, noting that the schedule to which the program was working at that time (1) was built on certain unproven assumptions regarding planned coding and testing improvements, (2) had not yet undergone a baseline review to verify that the schedule incorporated all of the work required for program completion, and (3) did not yet include a number of changes that the Air Force needed to incorporate into the contract with Raytheon as modifications, which may lead to additional schedule slips. In 2017, we did not make additional recommendations for OCX because the Air Force had undertaken the COps and MCEU programs to provide interim capabilities to mitigate OCX delays.

Changes in Software Development Methodology During the Program

In 2016, Defense Digital Service—a DOD office established by the Secretary of Defense—engaged with the OCX program to suggest improvements to Raytheon’s software development practices. The office recommended that Raytheon change its software development approach to use an incremental development approach. This approach uses a continuous integration and testing process, where the software code is frequently integrated and tested so that defects are detected and addressed sooner. This is done through automation of the software development process, version control tools, and coordination between different teams building software. Traditional software development methods entail a more linear approach whereby each process is completed before proceeding to the next process in the sequence. By such an approach, the software development processes are completed prior to the testing of a full product before the product’s release to the end user.

In 2016, DOD told the Air Force and Raytheon to utilize the new approach, which Raytheon began implementing in a series of seven phases. The first phase began in late 2016 and the last phase is scheduled to be in place by the end of 2019. According to the Air Force and Raytheon, through this new approach, the program aims to introduce efficiencies building software in several ways: 1. discovering defects in software code earlier; 2. reducing the number of defects; 3. reducing the amount of time it takes to repair defects; and 4. reducing the overall time to code, integrate, and test OCX software through automation for some aspects of the software development.

OCX Schedule at Risk for Additional Delays to Delivery and Operation

OCX delivery, acceptance, and the ready to transition to operations decision will likely be delayed, potentially exceeding the April 2023 threshold date for completing the program. Actual development progress has been mixed, with some improvement to the pace of software development. However, the majority of the schedule reserve has been consumed and defect repairs are taking longer than assumed with significant work remaining. In addition, a number of new cost and schedule risks to OCX delivery have arisen since the program schedule was established. GAO’s schedule and cost estimating best practices recommend that the schedule assessment be periodically updated to reflect actual progress and new risks. To mitigate program optimism, GAO’s cost estimating best practices also state it is important to have an independent view of cost estimates and schedules. While the Air Force and the contractor periodically update their schedule estimates, no plans currently exist for further independent analysis of the full program schedule within DOD, and there is no requirement to do so.

Significant Development and Testing Remains Before OCX Is Operational

The OCX program has significant work remaining before OCX is operational, including years of integration and testing. Achieving the full program schedule requires two interrelated steps. First, in order to meet the program schedule there must be timely delivery by Raytheon and acceptance of the system by the Air Force. Second, there must be timely completion of government-run post-acceptance developmental testing. Once the Air Force determines that the developmental testing is completed, OCX will be ready to transition to operations, which ends the full program schedule. GPS operations will then be transferred from OCS to OCX. Figure 3 shows the major activities until the ready to transition to operations decision.

OCX development is expected to continue for approximately 2 more years, after which Raytheon will submit a Material Inspection and Receiving Report (Form DD 250) at delivery. The Air Force will then evaluate OCX for acceptance. Air Force acceptance will be informed by numerous contractor-run developmental tests conducted to help the Air Force understand the maturity of the system. Air Force officials will use information from these contractor tests to inform their approval and complete acceptance. For example, the Air Force will review data and demonstrated system capabilities from the tests to determine whether OCX is ready for integration into the overall GPS. These tests have formal entrance criteria to demonstrate the system is ready for testing and exit criteria to ensure tests are successful before proceeding to the next activity. At the conclusion of contractor testing and delivery to the Air Force, the Air Force will inspect OCX over approximately 2 months before OCX is officially accepted. The Air Force will indicate acceptance by signing the Form DD 250. Currently, the period of performance under the contract ends June 30, 2021. Consequently, acceptance of the delivered OCX would need to occur prior to that date.

Air Force Developmental Testing and Rehearsals

After acceptance, Air Force program officials said OCX will go through government-run developmental testing—currently scheduled to last 7 months—that includes operator transition exercises and rehearsals of the system. According to OCX program officials, Raytheon will provide interim contractor support to address any defects or incomplete work as well as address any additional issues found during the planned 7 month post- acceptance developmental testing. According to program officials, the ground control operators—who have already been working and providing feedback—and training and readiness oversight personnel will continue to work with the new ground system to assess the system’s readiness through hands-on engagement with the installed system.

Ready to Transition to Operations

At the end of this 7 month period, the Air Force will determine whether the system is ready to transition to operations. To make the ready to transition to operations decision, Air Force officials said the system must receive approval from different groups, including senior leadership within the Air Force. Once the decision has been made, the Air Force will transition ground control of the GPS satellite constellation from OCS to OCX. Additionally, after this transition, which completes the program schedule, OCX will undergo an operational test and evaluation period, which will support the Air Force’s separate operational acceptance decision for OCX.

Delivery: Contractor’s Date Remains Optimistic Compared to Other Estimates

The OCX contractor’s delivery date is optimistic and much earlier than Air Force and independent projections. All government and independent analyses project OCX delivery will exceed June 2021 by at least 6 months, but still deliver in time to support the April 2023 threshold (latest acceptable) date for the full program schedule. However, meeting the ready to transition to operations threshold date depends on acceptance of OCX by September 2022, at the latest. This will allow for a planned 7 months of government-run developmental testing that must occur before April 2023.

Numerous OCX schedule estimates were produced between December 2017 and January 2019. Table 1 indicates the estimator, date of the estimate, and the reason the estimate was completed.

Figure 4 shows the results of the varying estimates for the start of OCX operations in months as measured from the beginning of calendar year 2019.

The most recent independent OCX assessment of the delivery date is from the Defense Contract Management Agency in January 2019. That assessment estimates that Raytheon’s projected delivery and the cost at completion are both unrealistic based on staffing profiles, task movement, completion rates, baseline execution, and schedule performance metrics. The Defense Contract Management Agency projects that there are not enough cost and schedule reserves left to cover its own estimate to complete the work plus all of the identified risks. In fact, the Defense Contract Management Agency estimates Raytheon will need over $400 million more in cost reserves and that OCX will likely be delivered 11 months after June 2021.

Delivery: Actual Development Progress Is Mixed

Actual development progress has been mixed. While the pace of software development has improved, implementing the new software development approach has been slower than expected. As a result, Raytheon has used the majority of its schedule reserve and delayed planned staff reductions, indicating that work is not being completed as quickly as planned. In addition, the schedule assumes improvements to software defect discovery have not all come to fruition and repair rates have not been achieved.

Pace of Software Development Has Improved

Under its new software development approach, Raytheon is building and testing OCX software more quickly than under its previous approach. In 2018, the software development rate to build and test software was reduced in duration from 4 weeks or more to less than 7 hours on average—better than planned. Defense Contract Management Agency officials said that software development has improved compared to block 0 by having a better software development process in place. These officials cited in particular the improvement that has occurred with the introduction of software testing automation in some areas. The pace of software development is one area of many that is necessary to improve overall performance and achieve the delivery schedule.

Implementing the New Software Development Approach Took Longer than Planned

OCX program officials told us that the full implementation of the new software approach is foundational to the success of the program; failure to successfully implement the new approach on time would lead to cost growth and schedule delays. However, implementation of the new software approach has taken longer than planned, using a majority of the available schedule reserve. Defense Contract Management Agency officials found that since the current baseline was established, Raytheon consistently takes 5 months to perform 4 months of planned work. This has not yet delayed the delivery schedule because the program has been able to use cost and schedule reserves to cover the delays. Between April 2017, when the current schedule baseline was established, and January 2019, Raytheon used 4 of the 6 months of total schedule reserve. As of April 2019, Raytheon had approximately 26 months of work remaining until June 2021, but only 2 months of schedule reserve. As a result, there will not be enough time to complete OCX development and have it accepted by June 2021 unless the contractor significantly reduces its use of schedule reserve.

Raytheon started using the new software approach on April 1, 2018 to improve software development, but implementation took longer than planned for six of the seven initial adoption phases, with two completing more than a year late. Some of the subsequent expansion phases are also experiencing delays. For example, phase 3 expansion was completed more than a year behind the planned schedule. Three other expansion phases are still in progress and scheduled to complete in mid- to late-2019.

Raytheon’s divergence from the baseline staffing plan indicates that work is not being completed as quickly as planned, and more staff have been needed to prevent additional delivery delays. Raytheon had planned to reduce the number of staff working on the program from approximately 1,000 to 700 between the autumn of 2017 and the end of 2018. However, to maintain schedule, Raytheon delayed those reduction plans and increased staff by approximately 10 percent from January to August 2018. Figure 5 shows the difference between the staffing baseline and actuals for OCX between January 2018 and December 2018.

Our analysis shows a gap between the January 2018 baseline planned staffing reduction and actual contractor staffing levels in each month from January to December 2018, collectively indicating an increase of approximately 29 percent above the plan. According to DOD’s Office of Cost Assessment and Program Evaluation officials, this increase is likely to continue through mid-2019. These officials estimated in their June 2018 independent cost estimate that contractor staffing levels will be higher than planned through May 2019 so that Raytheon can complete key software coding events. OCX program officials told us that the program has been able to afford the additional contractor staff as there are cost risks to support higher than anticipated staffing levels. They said that continued increases too far into 2019, however, will result in a breach of the cost threshold. Further, they said the increased contractor staffing is consistent with their priority on achieving the delivery schedule.

The new software approach implementation will remain a cost and schedule risk until at least late 2019. At this time, the final expansion phase for the new software development approach is planned to be completed in order to support final testing of the entire system.

Assumed Earlier Defect Discovery Shows Mixed Results and Reduction of Time to Repair Defects Has Not Occurred

Progress finding software defects sooner in development is also mixed. Raytheon officials told us that cost reductions are possible if they are able to find defects earlier, as this approach would lead to earlier defect resolution and reduce any backlog of defects. Further, they said there is efficiency in having the same developers repair software code that they created instead of different developers repairing the code later. In March 2018, Raytheon reported increasing the percentage of defects found in the phase of development where the defect was created from 27 percent in block 0 to 66 percent in blocks 1 and 2. However, an independent DOD assessment contradicted this improvement. DOD’s Office of Cost Assessment and Program Evaluation analyzed Raytheon’s defect discovery progress a few months into 2018 and found that after showing some initial improvement, the defect discovery rate dropped from approximately 53 percent to 24 percent.

In addition, assessing progress discovering defects is now more difficult to compare with earlier development since Raytheon changed how it tracks and counts defects in 2018. According to OCX program officials, Raytheon now only counts a defect if it is repaired in a later phase. Therefore, if a defect is found and repaired in the same phase, it is not counted. As of November 2018, Raytheon officials said the predictive measure they are now using estimates the total number of defects expected while measuring the actual defects discovered. From this data, Raytheon found fewer total defects than it predicted, which Raytheon officials said will result in fewer defects likely to be discovered later in subsequent phases.

Further, the defect repair-rate—or how many hours it takes to find and repair a defect—is currently projected to be higher than planned, placing additional pressure on the delivery schedule. According to Defense Contract Management Agency officials, the delivery schedule included defect repair assumptions that were unrealistic. That schedule assumed 30 hours to repair each defect. However, as of November 2018, Raytheon projects it will need 52 hours to repair each defect on average. For example, in one area of the program, defects required 61 hours to repair on average as of December 2018. Defense Contract Management Agency officials told us that they had concerns that the complexity of the defects was driving the time needed to repair them. They said that the more mature software created under the new software approach may be creating much more complex defects to repair than originally planned. This may lead to additional schedule delays as the time to repair these more complex defects may continue to be significantly higher than the delivery schedule assumed.

More complete data on defects and defect repair rates will likely be available by the end of 2019, when the final expansion phases of the new software approach and more software development are completed.

Delivery: Risks Have Changed Substantially Since Schedule Established

How do programs track risk as progress is made and risks evolve?

A risk is an uncertain event that could

Programs track risks to help manage and mitigate their effect on cost and schedule. completion requires knowing potential risks and identifying ways to respond to them before they happen—using risk management to identify, mitigate, and assign resources to manage risks so that their effects can be minimized.

Raytheon’s estimate that OCX will be accepted by the end of June 2021 is further challenged because of significant identified risks that remain in the schedule and changeover in those risks in 2018. As of January 2019, Raytheon was tracking 48 risks it has identified with cost effects—26 with a moderate likelihood of occurring. For example, a moderate risk includes the possibility of finding more defects than planned, which could have both cost and schedule consequences. Other moderate risks include the possibility of software development taking longer to complete or needing to create more software code than planned. If realized, both of these risks have cost effects to pay for additional work and schedule effects to allow additional time to complete work. As of January 2019, Raytheon has no high risks that it tracks. There was also a significant amount of change in the risks themselves in 2018, as Raytheon added 27 new risks while closing 30. The majority of the risks that are currently tracked will not be realized or retired until late 2019, with at least one key risk of concern to the Air Force not realized or retired until 2020.

As the program progresses, risks can (1)

According to OCX program officials, approval to transition OCX to operations assumes a 7-month developmental test schedule after acceptance. As currently formulated, this period will be used to prepare for the transition from OCS to OCX via (1) transition exercises to train operators, (2) transition rehearsals to practice the actual handover from OCS to OCX, and (3) a 156-day integrated system test to verify OCX’s requirements, operational suitability, and readiness to enter operational testing.

However, that 7-month duration may not be sufficient to conduct all of the activities that are necessary to verify OCX is ready to transition to operations. First, the head of the GPS Directorate’s Lead Development Test Organization, which plans and executes the 7-month developmental testing, said that there is some schedule risk because of concurrent activities that need to be accomplished, including crew rehearsals and other test events. Second, the content of the test period has not yet been finalized. The planned testing events will be reviewed and refined about 6 months before beginning the test as it becomes clearer what can be tested and what data will be available from the system. The test director and the OCX program manager are considering combining some test events and, if possible, starting some testing prior to acceptance. Third, the test director and the OCX program manager described a number of risks that could delay completion of developmental testing including (1) the late identification of issues requiring significant new software coding and retesting and (2) identification of new requirements that are not in the scope of the current effort.

In addition, OCX program officials stated that neither they nor senior Air Force leadership would transition OCX to operations if the operators are not ready or requirements have not been verified. They also stated that there are numerous levels of review within the Air Force, and any of these decision makers can refuse to approve the transition of OCX to operations. As a result, according to OCX program officials it could take 5 to 7 months longer than planned, or potentially 14 months total, to complete developmental testing.

In addition, experience with prior upgrades to the current GPS ground system indicates that the completion of developmental testing may require more time than the 7 months assumed in the schedule. Air Force Cost Analysis Agency officials provided us with data for two upgrades that were made to OCS, the existing operational ground system. Those upgrades took 11 and 8 months, respectively. The 11-month upgrade to OCS from 2006 to 2007 was for an effort that was significantly smaller in software size in comparison to the size of OCX, but similarly brought new capabilities to OCS related to the command and control of satellites. The 8-month upgrade to OCS from 2009 to 2010 also provided command and control of a new type of GPS satellite and enhanced security for the current GPS receiver cards. Figure 6 shows the different forecasts with 7- and 14-month developmental test periods as measured from the beginning of calendar year 2019.

If the time doubles for the completion of post-acceptance government-run developmental testing, most OCX schedule estimates would exceed the program schedule threshold.

Air Force and Contractor Updating Schedule Assessments in Accordance with Best Practices, but No Independent Assessments Are Planned of the Full Program Schedule

GAO’s Cost Estimating and Assessment Guide (Cost Guide) and Schedule Assessment Guide (Schedule Guide) identify best practices for managing a program’s cost and schedule. According to these best practices, a well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed and measuring program performance against an approved plan. Typically, schedule delays are followed by cost growth. When this occurs management tends to respond to schedule delays by adding more resources or authorizing overtime. Therefore, a reliable schedule can contribute to an understanding of the cost effect if the program does not finish on time. Moreover, an integrated and reliable schedule can show when major events are expected, as well as the completion dates for all activities leading up to them, which can help determine whether a program’s parameters are realistic and achievable.

Further, the Cost Guide states that, too often, programs overrun costs and schedule because estimates fail to account for the full technical definition, unexpected changes, and risks. The Cost Guide states that one of many challenges program managers face is too much optimism in the original estimate. The Cost Guide also states that because optimism is often prevalent, organizations will encourage goals that are unattainable by accentuating the positive. Because over-optimism potentially affects both cost estimates and schedules, an independent view and analysis is important to properly overcome this bias. An independent view also allows decision makers to react sooner and take steps to minimize any identified risks, like schedule delays. The following best practices recommend that the schedule estimate should be periodically updated to reflect (1) actual progress and (2) newly identified risks.

Periodic Updates and Actual Progress: GAO’s Schedule Guide states that updating a schedule to reflect actual progress is important when assessing the realism of the initial schedule duration assumptions. Programs should make adjustments, if necessary, to the forecast of the remaining effort.

Periodic Updates and Risk: GAO’s Schedule Guide states that prudent organizations recognize that uncertainties and risks can become better defined as the program advances and should conduct periodic reevaluations of risks. GAO’s Cost Guide states that program managers often do not sufficiently account for risks because they tend to be optimistic and because they believe in the original estimates for the plan without allowing for additional changes in scope, schedule delays, or other elements of risk.

Since the current schedule was approved in September 2018, Raytheon has updated its delivery schedule estimate quarterly or as needed to reflect changes, and modifies the delivery and acceptance dates accordingly. Raytheon does not update the full program schedule because the government-run developmental testing is not included in its schedule estimate. OCX program officials said they are currently updating their program schedule estimate by incorporating Raytheon’s data through the end of 2018.

No plans currently exist to conduct another OCX independent cost estimate—which would include a full, independent program schedule assessment—at the DOD-level, and currently there is no requirement to do so. An independent assessment of the schedule would normally be produced in conjunction with the statutory requirement to conduct another independent cost estimate at the next major program milestone. However, in September 2018 the milestone decision authority waived the requirement to hold the next major program milestone. DOD’s Office of Cost Assessment and Program Evaluation conducts independent cost estimates which account for a full program schedule when statutorily required. In addition, according to an official in that office, they also conduct only schedule assessments, without completing a full independent cost estimate, when requested by a program’s milestone decision authority. In June 2018, the Office of Cost Assessment and Program Evaluation provided the last full, independent cost estimate with a schedule assessment to the Under Secretary of Defense for Acquisition and Sustainment, the milestone decision authority, to support the decision to approve the OCX baseline. Officials from the Office of Cost Assessment and Program Evaluation said that they have not been asked by the OCX milestone decision authority to conduct another independent assessment. Without an independent schedule assessment, decision makers may lack updated information when determining whether to take new steps to avoid or mitigate additional delays.

Conclusions

It is still unknown when OCX will be ready to support the command and control of the next generation of GPS satellites. While Raytheon has improved the pace of building and testing software, the majority of schedule reserve has already been consumed and work is not being completed as quickly and efficiently as the delivery schedule predicted. Once software development is complete, it must go through developmental testing. The schedule for this phase may also be optimistic as risks associated with competing activities have the potential to double the amount of time needed for testing.

DOD will be in a better position to assess OCX’s progress and the potential for additional delays when the majority of its changes to its software development approach are completed at the end of 2019. At this time, however, while the program plans to continue assessing schedule progress, there are no plans in place for an independent schedule assessment. The program’s history has consistently shown program and contractor estimates to be optimistic and that independent assessments have provided useful insights about risks as well as past experience with similar activities. Our best practice guidance also emphasize that independent assessments are a necessary step to counter balance schedule optimism. Decision makers in DOD and Congress could use realistic knowledge about the schedule to either request or provide additional funds to complete the acquisition of OCX or develop contingency plans for delays.

Recommendations for Executive Action

We are making the following recommendation to DOD: The Secretary of Defense should direct the Director, Office of Cost Assessment and Program Evaluation to conduct an independent schedule assessment of the full program schedule for the Global Positioning System’s next generation operational control system based on progress made through the end of calendar year 2019. (Recommendation 1)

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In its written comments (reproduced in appendix II), DOD did not concur with our recommendation to conduct an independent assessment of the full OCX program schedule based on progress made through the end of calendar year 2019. DOD said that the Office of Cost Assessment and Program Evaluation conducted an independent cost and schedule estimate supporting the OCX program’s September 2018 system development milestone and that DOD subsequently funded OCX consistent with that estimate. Further, DOD said that the Office of Cost Assessment and Program Evaluation as well as the Defense Contract Management Agency continually assess the program’s ability to meet cost, schedule, and performance objectives. DOD also said the OCX forecast is currently holding to the government schedule, which is ahead of the Office of Cost Assessment and Program Evaluation’s independent cost estimate. Finally, DOD said senior executive reviews continue on a bi-annual basis to monitor the program’s progress.

We continue to believe the recommendation is necessary. As stated in our report, DOD has not conducted an assessment of the full schedule since June 2018, since which time program risks have evolved. In addition, the other potential sources for schedule oversight suggested by DOD are limited in scope. The Defense Contract Management Agency does not look at the full OCX program schedule, as it examines the schedule only until contractor delivery. Officials from the Office of Cost Assessment and Program Evaluation said they do some programmatic monitoring of OCX, including on selected metrics, to inform DOD’s annual program and budget submission. But those metrics do not examine the full schedule that includes the developmental test period after delivery. The Office of Cost Assessment and Program Evaluation is in a position to independently assess the full OCX program schedule, as it has previously done, but only if DOD requests that it do so.

We maintain that for complex programs, such as OCX, best practices state an independent view is necessary and that a periodic schedule assessment should be performed as progress is made and risks change. Given the mixed progress developing software, the number of new contractor risks discovered in 2018, the limited remaining schedule reserve held by the contractor (with at least two years of significant work remaining), and the potential for doubling the time frame for the planned 7-month post-acceptance government-run developmental testing period, we determined that the recommendation remains a prudent step. Such an assessment would help inform congressional and DOD decision makers as they consider what steps may be taken to address delays to the start of OCX operations and ensure the investments in needed new receivers are properly aligned.

We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Secretary of the Air Force, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerns this report, please contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To determine the extent to which schedule risks may delay the delivery, acceptance, and approval for the operation of the Global Positioning System (GPS) next generation operational control system (OCX), we reviewed information relevant to OCX from Air Force GPS quarterly reports, senior management reviews, the program acquisition baseline, software development plans, monthly program management reviews that included schedule risks and progress updates, Air Force monthly acquisition reports, Air Force service cost position documentation, independent cost estimate documentation and analysis, earned value management data, Defense Contract Management Agency performance assessment reports, and slides and information provided by Raytheon Company (Raytheon), the prime contractor, on topics of our request. We reviewed the Air Force’s 2018 integrated baseline review results of the period until government acceptance and assessed the full program schedule—which includes the contractor’s schedule, government acceptance, and post-acceptance government-run developmental testing—until OCX is ready to transition to operations. We reviewed GAO’s best practice guides for cost estimating and assessment and schedule assessment to identify best practices for assessing a program’s cost and schedule and applied selected best practices. We also reviewed relevant reports and assessments focused on OCX completed by the government or required by Congress. We interviewed officials from the OCX program office and GPS Directorate, Defense Contract Management Agency, DOD’s Office of Cost Assessment and Program Evaluation, Air Force Cost Analysis Agency, the Lead Development Test Organization for the GPS Directorate, Defense Digital Services, Office of the Director, Operational Test and Evaluation, the MITRE Corporation, and Raytheon.

We conducted this performance audit from November 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: David Best (Assistant Director), Patrick Breiding (Analyst-in-Charge), Marie P. Ahearn, Pete Anderson, Brian Bothwell, Jonathan Mulcare, Andrew Redd, Karen Richey, Roxanna Sun, and Robin Wilson.