NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective Program Management
Released: 2004-06-22T00:00:00

Letter

Background

NASA’s programs encompass a broad range of complex and technical activities—from investigating the composition and resources of Mars to providing satellite and aircraft observations of Earth for scientific and weather forecasting. NASA currently funds more than 100 programs and projects in various phases of execution in 7 strategic Enterprises: Space Science, Earth Science, Biological and Physical Research, Aeronautics, Space Flight, Education, and Exploration Systems. Two NASA offices have key responsibilities in ensuring the effective execution of these programs: the Office of the Chief Financial Officer, which is responsible for providing oversight and financial management of agency resources and establishing related policy guidance, and the Office of Chief Engineer, which is responsible for ensuring development efforts and mission operations are planned and conducted using sound engineering practices.

More than two-thirds of NASA’s work force is made up of contractors and grantees, and 90 percent—or roughly $13 billion—of NASA’s annual budget is spent on work performed by its contractors. Since 1990, we have identified NASA’s contract management as a high-risk area. This assessment has been based in part on our repeated finding that NASA does not have good cost-estimating processes or the financial information needed to develop good cost estimates for its programs, making it difficult for NASA to oversee its contracts and control costs. For example, in July 2002, we reported that an independent task force convened to assess the management of the International Space Station concluded that the program’s fiscal year 2002 through fiscal 2006 budget was not credible because of weaknesses in its cost-estimating processes. The task force pointed out that these problems occurred because NASA had not instituted or had ignored many of the program’s control and contract oversight procedures—such as preparing a full life-cycle cost estimate—that should have alerted the agency to the growing cost problem and the need for mitigating actions. According to the cost analysis team that supported the task force, NASA’s focus on staying within annual budgets instead of managing total program costs was perhaps the single greatest factor in the program’s cost growth.

NASA’s unreliable cost estimates have significant implications for potential future endeavors, such as those outlined by the President in January of this year. Specifically, the President called for a shift in NASA’s long-term focus, envisioning that NASA will retire the shuttle program as soon as assembly of the International Space Station is completed, planned for the end of the decade; develop a new crew exploration vehicle as well as launch human missions to the moon between 2015 and 2020, and build a permanent lunar base as a stepping stone for more ambitious missions. To achieve these goals, the President proposed spending $12 billion over the next 5 years— about $1 billion of which would come from an increase in NASA’s budget, currently $15.4 billion—with the remaining $11 billion being reallocated from existing NASA programs.

Developing reliable cost estimates has been difficult for agencies across the federal government. The need for reliable cost estimates is at the heart of two of the five-governmentwide initiatives in the 2002 President’s Management Agenda (PMA); the two are “improved financial performance” and “budget and performance integration.” These initiatives are aimed at ensuring that federal financial systems produce accurate and timely information to support operating, budget, and policy decisions and that budgets are performance-based. As part of these initiatives, the President calls for changes to the budget process to better measure the real cost and performance of programs. According to the PMA, accomplishing all of the crosscutting initiatives will matter little without the integration of agency budgets with performance.

Development Cost Estimates Frequently Changed

As of April 2003, the baseline development cost estimates for the programs we reviewed varied considerably from the programs’ initial baseline estimates. More than half of the programs’ development cost estimates increased, and for some programs, this increase was significant. The baseline development cost estimates for each of the 10 programs we reviewed in detail were rebaselined—that is, recalculated to reflect new costs, time frames, or resources associated with program changes in program objectives, deliverables, or scope and plans. Although NASA provided specific reasons for the increased cost estimates and rebaselinings—such as delays in the development or delivery of key system components and funding shortages—it does not have guidance for determining when rebaselinings are justified. Such criteria are important to instilling discipline in the cost-estimating process.

Most of the 27 programs we reviewed experienced a change in their development costs estimates. While 8 of the 27 programs experienced slight decreases in their development cost estimates, 17 experienced cost growth—as much as almost 94 percent. The remaining two programs had no change. Ten of the 17 programs’ cost growth was greater than 25 percent. Table 1 shows the development cost estimate changes from the initial baseline to the baseline as of April 2003 and the life-cycle cost estimate for each of the 27 programs. The 10 programs that we reviewed in detail are shaded and italicized. (See app. II for assessments of the 10 programs and app. III for descriptions of the remaining 17 programs.)

The development cost estimates for each of the 10 programs that we reviewed in detail have been rebaselined—for some programs, as many as four times—and for 7 of the 10 programs, the cost estimate increased each time it was rebaselined (see fig. 1).

For the 10 programs we reviewed in detail, NASA cited specific reasons for changes in the baseline development cost estimates and the recalculated baselines—many of which were related to technical problems and subsequent delays in the development or delivery of key system components, and insufficient funding and reserves, as illustrated in the following examples: Technical problems in the MERs program required a significant redesign of components and the development of a new landing system. Two of MERs’ three rebaselinings were also the result of inadequate reserves. According to NASA officials, without the rebaselinings, the development cost “to go” would have drained the program’s reserves.

The increase in CLCS’s development cost estimate and rebaselining was the result of poorly defined requirements and design, software integration problems, and fundamental changes in the project’s management structure and contractors’ approach to the work. The project, which experienced an almost 94 percent increase in its baseline development cost estimate, was ultimately terminated.

The GP-B program—which was rebaselined four times—experienced significant schedule slippages due to repeated technical problems, including failures in the probe’s heat exchanger, the need for additional testing, payload electronics delays, and thermal vacuum test failures.

Schedule slippages in the SIRTF program—which contributed to increases in the program’s baseline development cost estimate and four rebaselinings of the estimate—were caused by delays in the delivery of components, flight software, and the mission operation system as well as launch delays that resulted from a handling accident involving a global positioning system payload and concerns of delamination on the launch vehicle’s solid rocket motors.

Changes in development cost estimates for the CAU program were primarily the result of the program’s expanded scope, which occurred in October 2002, to produce modification kits that would allow the CAU upgrade to be installed into the orbiters.

The Hyper-X program experienced three rebaselinings, and according to the project manager, the program will be rebaselined again in the near future. The rebaselinings were due to schedule slippages resulting from the need to fund an investigation of the problems experienced in the first Mach 7 flight vehicle—which was destroyed in flight—and related corrective actions to the second Mach 7 flight.

Revised contract requirements, funding changes, or the realization that program goals are not achievable may require a formal rebaselining. However, NASA has not defined or provided guidance or restrictions on rebaselining to ensure that programs consistently and appropriately apply rebaselinings and do not adjust their baseline cost estimates whenever the estimates become unmanageable. Further, NASA lacks a process for systematically identifying and assessing programs that are not achieving their cost, schedule, and performance goals. Such a process has been employed by the Department of Defense (DOD), which also relies heavily on contractors to deliver complex, cutting-edge technologies to meet its mission. Specifically, DOD must report to the Congress programs that incur a cost growth of 15 percent or more in the program baseline. Moreover, DOD must justify the continuation of acquisition programs that incur a cost growth of 25 percent or more in the program baseline by certifying that specific criteria have been met—including that the new cost estimates are reasonable. Under such a process, 5 of the 10 programs that we reviewed in detail would have been required to report to the Congress, and 4 of the 5 programs would have had to certify that their new cost estimates were reasonable.

Poor Estimating Processes and Methodologies Contributed to Wide Variations in Baseline Cost Estimates

NASA has yet to implement a well-defined process for estimating the cost of its programs—a weakness we and NASA’s Inspector General have repeatedly reported. Recognizing the need for such a process, NASA developed a cost-estimating handbook in 2002—the first such guidance provided to its cost-estimating community and program and project managers. Despite this effort, the programs we reviewed failed to follow key cost-estimating processes, including developing and documenting full life-cycle cost estimates, summarizing estimates according to the current breakdown of work to be performed, conducting an uncertainty analysis, performing an independent review of contractors’ cost estimates, and later using earned value management (EVM) to assess progress.

Reflecting Office of Management and Budget (OMB) guidance and best practices of government and industry leaders, NASA requires that full life-cycle cost estimates be prepared using full cost accounting, that estimates be summarized according to the current breakdown of work to be performed, and that major changes be tracked to the life-cycle cost. In its draft cost-estimating handbook, NASA lists a number steps that are integral to preparing a reliable life-cycle cost estimate, including preparing or obtaining a cost analysis requirements description (CARD), developing ground rules and assumptions, and developing cost range and risk assessments.

Carnegie Mellon University’s Software Engineering Institute (SEI) echoes the need for reliable cost-estimating processes in managing software implementations—identifying tasks to be estimated, mapping the estimates to the breakdown of work to be performed, and identifying and explaining assumptions are among SEI’s requisites for producing reliable cost estimates. To evaluate the cost-estimating processes of the 10 NASA programs that we reviewed in detail, we selected 14 criteria based on SEI checklists (see table 2). Many of these criteria are included in NASA’s cost-estimating guidance.

Despite NASA requirements and OMB and SEI guidance, few of the 10 programs that we reviewed in detail met even a third of these criteria; only one met half. Further, none of the programs fully met certain key criteria. For example, none provided a complete life cycle with definitions or a complete description of the methodology used to generate the complete cost estimate, such as data sources and uncertainties. According to the draft NASA cost-estimating handbook, a reliable life-cycle cost estimate is critical to making realistic decisions about developing or producing a system and to determining the appropriate scope or size of a program. NASA guidance also calls for breaking down the work to be performed into smaller units to facilitate cost estimating and program and contract management and to help ensure relevant costs are not omitted. However, only 3 of the 10 programs provided a complete breakdown of the work to be performed. Table 3 shows for each program the applicable criteria that were met, partially met, or not met. (See app. II for a program by program assessment.)

Failing to meet these criteria puts programs at certain risk. For example, underestimating a program’s full life-cycle costs creates the risk that a program could be underfunded and subject to major cost overruns, which would ultimately result in the program being reduced in scope or additional funding being requested and appropriated to ensure the program meets its objectives. Conversely, overestimating life-cycle costs creates the risk that a program will be deemed unaffordable and would, therefore, go unfunded. Without a complete WBS, NASA programs cannot ensure that the life-cycle cost estimates have captured all relevant costs, which again can result in underfunding and cost overruns. Further, inconsistent WBS estimates across programs can create problems of double counting or, worse, underestimating costs when using historical program costs as a basis for projecting future costs on similar programs.

Despite the uncertainty inherent in estimating the cost of emerging technologies, all of the 10 programs we reviewed also failed to conduct an uncertainty analysis to assess risks associated with the cost estimates. Instead, the programs expressed their cost estimates as point values—which implies certainty—not as ranges or numbers with confidence levels. Performing an uncertainty analysis, such as a Monte Carlo simulation, quantifies the amount of cost risk within a program. Only by quantifying the cost risk can management make informed decisions about risk mitigation strategies. Quantifying cost risks also provides a benchmark against which future progress can be measured. Without this knowledge, NASA may have little specific basis to determine adequate financial reserves, schedule margins, and technical performance margins to provide managers the flexibility needed to address programmatic, technical, cost, and schedule risks, as required by NASA policy.

Seven of the 10 programs also failed to have an independent review of contractors’ cost estimates—as required by NASA. Instead, programs established their budgets based on contractor proposals—particularly problematic since many contractors could bid low in order to win the contract. To ensure contractor costs are realistic, NASA procedures and guidelines specifically require programs to ensure that independent reviews are conducted and that these reviews address project life-cycle costs, risk management plans, as well as technical issues. Without such reviews, NASA decision makers lacked the benchmarks needed to assess the reasonableness of the contractors’ proposed costs, limiting NASA’s ability to make sound investment decisions and accurately assess contractor performance.

Finally, only two programs used EVM—an approach used by DOD and leading companies to provide meaningful assessments of a program’s progress by comparing the value of work performed to its costs, rather than the traditional management approach of comparing budgeted and actual costs, which can provide a distorted view of a program’s progress. (For a detailed discussion of EVM, see app. IV.) By using the value of completed work as a basis for estimating the cost and time needed to complete the program, EVM can alert program managers to potential problems early in the program. NASA requires that EVM be used on all significant contracts—that is, research and development contracts with a total anticipated final value of $70 million or more, and production contracts with a total anticipated final value of $300 million or more— which includes all of the 10 programs we reviewed in detail. Although the program managers for all 10 programs stated that EVM was used in their projects, only two programs provided cost performance reports, indicating a true EVM process was in place. The remaining eight programs relied on NASA Form 533, which captures planned and actual obligations and expenditures—not the value of the work performed. Without a true EVM process, programs cannot readily determine if a program is at risk of cost and schedule overruns until it is too late to make programmatic changes to avoid these risks.

NASA Has Begun to Address Certain Barriers to Effective Cost Estimating

There are several impediments that NASA needs to overcome to implement effective cost-estimating practices. These include the lack of reliable financial data and other performance information; lack of trained EVM staff, data analysis tools, and incentive for supporting and implementing EVM; and ineffective use of cost analysts. NASA has initiated several measures to begin addressing some of these impediments.

Utility of Cost-Estimating Tools Depends on the Reliability of NASA’s Financial and Performance Data

According to NASA officials, state-of-the-art cost-estimating tools have been funded and implemented. For example, NASA officials told us that commercial-off-the-shelf models have been used to estimate hardware and software acquisition costs and quantify the level of uncertainty surrounding cost estimates. However, these cost-estimating tools are only as good as the data they rely on to develop the estimates. For more than a decade, we have reported that NASA has failed to develop a system to capture reliable financial and performance information, posing significant challenges to NASA’s ability to estimate and control program costs. Over the past year alone, we issued numerous reports on NASA’s Integrated Financial Management Program (IFMP)—the agency’s third and most recent effort to implement a modern, integrated financial management system. Specifically, we found that IFMP—which is under the responsibility of the Program Executive Officer for IFMP—will not, as it is being implemented, routinely provide program managers and other key stakeholders and decision makers—including the Congress—with the financial related information needed to measure program performance and ensure accountability. For example, the core financial module (considered the backbone of the system) does not appropriately capture property, plant, and equipment, as well as material in its general ledger at the transaction level—which is needed to provide independent control over these assets. In addition, NASA implemented the system before it had the capability to capture the full costs of its programs and projects. According to headquarters officials, collecting nonfinancial data crucial to cost estimating—such as technology readiness levels, parts counts, and team and management experience and skill ratings—has also been difficult.

Use of EVM Has Been Undermined by a Lack of Trained Staff, Data Analysis Tools, and Incentive

According to headquarters officials, agencywide EVM implementation efforts began in 1996 and are recognized by NASA management as a key tool in monitoring and measuring cost trends in higher risk project elements—a tool that serves as an early warning of the need for cost-risk mitigation actions to maintain control of program costs. These officials stated that EVM has been applied to the International Space Station Program and with varying levels of emphasis to other programs and projects at different NASA centers. While all of the program managers for the 10 programs that we reviewed in detail stated that they used EVM, only 2 of the programs used a true EVM process.

NASA headquarters officials identified several challenges that have affected the agency’s ability to implement EVM effectively, including a lack of staff and data analysis tools. According to officials, resource constraints have prevented the agency from staffing many project offices with appropriate personnel to fulfill all project functions. In addition, there has been little or no priority to include a trained EVM analyst, even if one were available. Headquarters officials also noted that EVM has been hampered by the lack of a practical automated software data analysis tool. Without such a tool, analyzing the contractors’ EVM cost performance reports, which contain significant amounts of data, became a cumbersome undertaking that often resulted in incomplete and untimely analyses, providing little usefulness to inform management decisions. A lack of incentive to support EVM has further undermined its use. Some project managers whom we spoke with are skeptical about the benefits of EVM and argue that it has failed to help them manage or control program costs. According to NASA headquarters officials, during proposal and contract negotiation phases, contractors have also suggested not using EVM as a way to reduce contract costs. While EVM was included in most contracts for the 10 programs we reviewed in detail—as required by NASA policy—it was used only in two programs as a cost-estimating tool. In general, EVM has been viewed by NASA as a financial reporting tool. Consequently, there is little incentive to use EVM because the data needed to report financial activity is captured elsewhere, such as in Form 533.

Ineffective Use and Placement of Cost Analysts across the Agency’s Cost Activities also Hinders NASA’s Efforts to Improve Its Cost-Estimating Practices

NASA’s efforts to improve its cost-estimating processes have also been undermined by ineffective use of its limited number of cost-estimating analysts. For example, headquarters officials state that as projects entered the formulation phase, they have typically relied on program control and budget specialists—not cost analysts—to provide the financial services to manage projects. Yet budget specialists are generally responsible for obligating and expending funding—not for conducting cost analyses that underlie the budget or ensuring budgets are based on reasonable cost estimates—and, therefore, tend to assume that the budget is realistic. While NASA officials state that its cost-estimating staff is too limited to be involved in day-to-day project execution activities, they agreed that the cost analysts could be more effectively used throughout the life cycle— particularly when projects are rebaselined and independent cost estimates of project changes must be performed.

In some cases, cost analysts are not appropriately located in the organization, which may compromise controls NASA has in place to ensure reasonable cost estimates. For example, some cost analysts at NASA’s centers are located with senior systems engineers in systems management organizations, while others are not. According to NASA officials, housing the cost analysts with senior systems engineers has two key benefits. First, the systems engineers generally conduct systems analyses to help ensure that a program’s requirements are properly established and that the design and validity meet the requirements. Such analyses can greatly inform the development of reasonable cost estimates. Second, the systems engineering offices afford some measures of independence for cost estimating, which, according to NASA cost- estimating guidance and procedures, is important to the overall project management process.

However, NASA officials stated that several of its centers’ cost analysts are in the advocacy chain of command—not housed with senior systems engineers. For example, one center’s 15 cost analysts work in the center’s Office of the Chief Financial Officer—which is responsible for directing the development and execution of the center’s budget—not in the systems management organization, which is independent from the rest of the center. As a result, the costs analysts’ estimates may not be adequately informed by the systems engineers and may lack the objectivity required to ensure that the criteria for independence have been met.

Efforts Under Way to Remove Some Barriers and Improve Cost Estimating

NASA has several initiatives under way to improve the agency’s cost-estimating processes. First, NASA has established a Cost Analysis Division in the Office of the Comptroller to strategically manage analyses related to directing and funding research, improving cost-estimating processes and practices, and providing cost-estimating tools and training throughout the agency. The division also provides, along with the Independent Program Assessment Office (IPAO), the last independent cost estimate of projects before the information is released externally. These efforts are being coordinated through a steering committee composed of the managers of the cost analysis organizations from each of the centers and IPAO’s deputy director.

NASA is revising the cost sections in its governing procedures and guidelines and is finalizing its cost-estimating handbook to reflect these changes. These documents will require the routine use of probabilistic cost risk analysis, a CARD document, cost as an independent variable (CAIV), and EVM. The CARD supports the project life-cycle cost estimate and a congressionally required independent cost estimate. Agency officials note that while there has been some use of CARD in the agency, its first concentrated and successful use was in the 2001 to 2002 independent cost estimate for the International Space Station program. According to headquarters officials, NASA’s revised guidance and finalized cost-estimating handbook will provide direction and guidance for fully implementing the use of CARDs for major development projects. Although NASA calls for CAIV to be used routinely and notes that CAIV demonstrates a commitment to evolutionary acquisition, it has yet to provide guidance on its implementation. NASA headquarters officials stated that guidance relating to improvements in the collection of cost data is also being reflected in its revised governing procedures and guidelines.

With respect to EVM, NASA headquarters officials described several efforts under way to ensure agencywide implementation of true EVM. For example, NASA recently revised its EVM policy directives to shift ownership of EVM responsibilities from NASA’s Chief Financial Officer to NASA’s Chief Engineer, to emphasize that EVM is to be considered a project management tool rather than a financial management tool. NASA officials also noted that the agency is working to inform managers of the performance management capabilities available to them through EVM and to emphasize the importance of providing adequate resources and management support to ensure successful EVM implementation. Agencywide goals for EVM implementation include promoting the effective use of EVM and providing needed training and education for program and project staff. These efforts and proposed initiatives should help resolve EVM utilization problems.

Finally, NASA officials told us that the agency is planning to hire additional cost analysts to alleviate understaffing at all of its center cost analysis offices. The agency envisions a total staff of about 100 cost analysts along with additional support contractors. NASA officials also stated that it is necessary to ensure centers address the problem of having cost analysts located in the advocacy chain of command, which could affect five NASA centers.

Because NASA’s initiatives have only recently been implemented or are still in the drafting or planning stage, we cannot determine to what degree these efforts will enable NASA to provide reasonable and defensible cost estimates of its programs and projects.

Conclusions

There are numerous scientific and technical challenges inherent in the successful implementation of many NASA programs. Nevertheless, the need to choose among competing alternatives within limited budget resources makes it essential that the agency and the Congress clearly understand the costs and uncertainties of programs proposed for authorization and funding. Yet, NASA does not have the disciplined cost-estimating process needed to make informed acquisition decisions, nor does the agency have processes and tools for capturing, monitoring, and managing program costs and schedules within an implementation plan on a timely basis. This makes it difficult for senior NASA officials, program and project managers, and other key stakeholders to measure performance and initiate mitigation measures when needed. Taken together, the lack of disciplined and established cost-estimating processes and tools can cause program officials to restructure projects to available resources rather than develop realistic cost estimates and implementation plans for projects. As a result, programs may have to be modified to accommodate emerging technical, cost, and schedule realities. Ultimately, programs cost more, fail to meet their schedules, or deliver less than originally envisioned. To help minimize project costs increases and implementation delays identified in this report, NASA needs to instill disciplined cost-estimating processes into its project development and approval activities and to ensure such processes are integrated with its implementation of an integrated financial management system. Without a process that prevents programs from proceeding before they have sufficiently demonstrated that key cost-estimating criteria have been met, NASA programs will continue to be at risk of cost and schedule overruns.

Recommendations for Executive Action

Improvements to NASA’s cost-estimating processes will partly depend on the agency’s ability to address recommendations that we made in November 2003 to help ensure NASA effectively implements a modern, integrated financial management system. Notwithstanding the need to address those recommendations, to better position NASA to ensure its recent initiatives result in sound cost-estimating practices agencywide, we are making three recommendations with minimum suggested courses of action. First, we are recommending that the NASA Administrator direct the Program Executive Officer for IFMP, the Chief Financial Officer, and the Chief Engineer to develop an integrated plan for improving cost estimating that, at a minimum, includes specific actions for ensuring that guidance is established on rebaselining and that rebaselining is consistently applied to provide accountability among programs, true earned value management is used as an organizational management tool to bring cost to the forefront in NASA’s management decision- making process, acquisition and earned value management policies and procedures are staff and support for cost-estimating and earned value analyses are effectively used.

In addition, we recommend that the NASA Administrator direct the Chief Financial Officer to establish a standard framework for developing life- cycle cost estimates. At a minimum the framework should require each program or project to base its cost estimates on a full life cycle for the program—including all direct and indirect costs for operations and maintenance and disposal as well as planning and procurement—and on a work breakdown structure that encompass both in-house and contractor efforts, prepare a cost analysis requirements description, prepare an independent government estimate at each milestone of the conduct a cost risk assessment that identifies the level of uncertainty inherent in the estimate.

Further, we recommend that the NASA Administrator develop procedures that would prohibit proposed projects from proceeding through the review and approval process when they do not address the elements of the recommended cost-estimating practices.

Agency Comments and Our Evaluation

In written comments on a draft of this report, NASA’s Deputy Administrator stated that the agency concurs with the recommendations, adding that the recommendations validate and reinforce the importance of activities underway at NASA to improve cost estimating and program management.

Notwithstanding agreement with our recommendations, the Deputy Administrator believes NASA has made substantive changes and achieved significant improvements in its cost-estimating processes. For example, NASA’s comments on a draft of this report cite a 1992 GAO report (GAO/NSIAD-93-97) that found a median 77 percent increase in NASA program costs. According to the Deputy Administrator, this contrasts with a 13 percent cost growth in this present study. While there may be improvements in the percent of cost growth of some projects, such declines in cost growth are often achieved by rescoping and rebaselining projects to remain within available resources, as was demonstrated in a number of projects discussed in this report. We do not believe other examples cited by the Deputy Administrator, namely termination of the Checkout and Launch Control System and cost control measures imposed on the International Space Station, demonstrate that NASA has made substantive changes and achieved significant improvements in its cost- estimating processes. Rather, we believe these examples demonstrate what happens when projects are undertaken without a full understanding of the potential costs and management challenges inherent in many of the programs NASA proposes and then implemented without adequate financial management systems in place.

With regard to our recommendation to develop guidelines for rebaselining and ensure effective use of earned value management, the Deputy Administrator cited the development of revised direction on program and project management and a refocus on risk and cost-risk analysis. NASA also now requires the establishment of cost thresholds that, if exceeded, will require a rebaselining review. Further, because much of NASA’s work is performed through grants and contracts, NASA’s revised procedures will emphasize how risk and technical complexity affect contractor performance. New earned value management and acquisition policies and procedures will be implemented through program management councils that will review and approve programs and projects regularly through each step of their development. Also, a new Cost Analysis Division has been established, and cost-estimating staff has been added to it and NASA’s Independent Program Assessment Office. NASA also noted the importance of training needed to match the new requirements.

NASA’s Deputy Administrator also concurred with our recommendation to establish a standard framework for developing life-cycle cost estimates. According to the Deputy Administrator, NASA’s new processes and procedural requirements document will define the full life-cycle cost to include development, operations, maintenance, disposal, and all NASA in- house direct and indirect costs to eliminate ambiguity and ensure consistency. NASA’s revised cost-estimating handbook will provide further guidance for life-cycle cost estimates. Also, project managers will be responsible for developing and maintaining a cost analysis requirements document similar to a tool DOD uses that will include the equivalent of a project and technical description; key performance parameters, including documentation of actual work breakdown structure cost elements; and initial and annual updates of the life-cycle cost estimates. NASA guidance will also require periodic independent cost estimates on major programs and approval by the respective program management council to enter into implementation after an independent estimate has been completed.

Lastly, NASA’s Deputy Administrator concurred with our recommendation to prohibit proposed projects from proceeding through the review and approval process when they do not address the elements of the recommended cost-estimating practices. Accordingly, NASA’s forthcoming procedural requirements will define the authority of the program management councils that will, according to NASA, enforce the requirements, including the required information, documentation, and management methods needed for proceeding through the review and approval process. The Deputy Administrator also noted the availability of recent management information system improvements that enhance visibility over project and program performance. In his general comments, the Deputy Administrator also stated that NASA had recently taken steps to address issues raised in the draft report and suggested a report title that would better reflect that progress.

We agree that NASA has initiated number of reforms to its project development and implementation processes that, if properly implemented, would be positive steps to addressing many of the problems noted in this report. However, we also note that some of these problems have been long-standing in the projects discussed in this report and in a number of other projects we and NASA’s Office of Inspector General have reviewed. Furthermore, planned improvements in the past have fallen short of agencywide implementation. For example, poor or inadequate cost estimates and management oversight have been central to the problems that plagued several programs, including those intended to develop new space transportation and the International Space Station programs. A reliable financial management structure is central to the success of many measures noted by the Deputy Administrator in his reply. We recently reported and testified on the impediments that exist in achieving such a capability. Finally, we note that contract management has been a long- standing problem at NASA. In 1990, we identified NASA’s contract management function as an area at high risk. During that time, there was little emphasis on end results, product performance, and cost control. NASA found itself procuring expensive hardware that did not work properly. This report shows that these types of problems still exist. Regarding the Deputy Administrator’s suggestion that we revise the title of our report to reflect recent progress that NASA has made towards addressing issues that we raise, we believe NASA’s improvements have been properly reflected in our report’s title. We considered the concerns expressed in the Deputy Administrator’s comments, and consistent with our stated position that NASA’s improvements are positive steps but that its problems still persist, we revised the title accordingly.

Finally, until NASA’s integrated financial management system, which is central to providing effective management and oversight, is fully implemented, performance assessments relying on cost data may be incomplete and full costing will be only partially achieved. And until these problems are resolved and the measures the Deputy Administrator noted in commenting on a draft of this report are fully implemented and integrated into the way the agency does business, NASA’s contract management function will continue to be an area of concern.

As agreed with your office, unless you announce its contents earlier, we will not distribute this report further until 30 days from its date. At that time, we will send copies to the NASA Administrator and interested congressional committees. We will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any question concerning this report, please contact me at (202) 512-4841 or lia@gao.gov. Key contributors to this report are acknowledged in appendix VI.

Scope and Methodology

To determine cost estimates in selected NASA programs and any changes in those estimates, we asked NASA to provide a list of programs that were currently in the development phase, and programs that had completed development or were launched in fiscal year 2001or 2002. We also asked NASA to provide the initial baseline development cost estimate and current cost estimate for the development phase and life of the program, and the reasons for changes to initial development cost estimates. NASA identified 68 programs that were currently in development or had completed development in fiscal years 2001 and 2002. These included planetary missions and Earth observatory, aeronautical technology, and space flight systems. From that universe, we selected at least one program (10 in total) from 5 of NASA’s 7 Enterprises. This involved 6 of 9 NASA centers (and the Jet Propulsion Laboratory) with lead responsibility for one or more of these programs. Our selection was generally based on programs with the highest current development cost estimates within an Enterprise. We compared the initial development cost estimates NASA provided to the current development cost estimates for the programs. The initial development estimates generally reflect the projected costs at the time a new program was first approved by the Congress. The current development and life-cycle cost estimates reflect the latest estimates provided by NASA as of April 2003. We also interviewed program officials to obtain additional information related to NASA’s revisions to initially established baseline development cost estimates, including the rationale for changes to the cost estimates.

We also analyzed the initial and current development cost estimates for 17 additional NASA programs, later added to the scope of our review, to ascertain the level of cost growth or decline as those programs progressed through the development phase.

To assess NASA’s cost-estimating processes and methodologies, we used cost-estimating criteria developed by Carnegie Mellon University’s Software Engineering Institute (SEI) designed to assess the reliability of project cost and schedule estimates. SEI is a government-funded research organization that is widely considered an authority on software implementation. SEI developed checklists with these criteria to help evaluate software costs and schedule; however, SEI states that these checklists are equally applicable to hardware and systems engineering projects. We first analyzed NASA’s cost-estimating procedures and guidelines to determine if they incorporated key components of good cost-estimating practices advocated by SEI and other experts.

Based on that analysis, we selected 14 criteria from two SEI reports to use in assessing NASA’s cost-estimating practices for the 10 programs we selected to review in detail. Our selection of the 14 criteria from the SEI reports was based, in part, on their commonality with NASA cost- estimating procedures and guidelines. Finally, using the cost-estimating documentation provided by NASA for the 10 programs, we determined the extent to which the programs met the 14 criteria. If a program provided substantiating evidence for a criterion, we determined that the program “fully met” the criterion. If partial evidence was provided for a criterion, we determined the program “partially met” the criterion. If no evidence was found, then we determined that the criterion was “not met.” Table 2 describes each of the 14 criteria and the significance of each criterion.

To identify any barriers that make it difficult to improve any weaknesses in NASA’s cost-estimating processes, we reviewed our recent work on NASA’s efforts to implement a modern integrated financial management system. We also provided questions to NASA headquarters that asked for information regarding NASA’s ability to use its cost estimates as a management tool for its programs. We also provided questions related to the SEI criteria, and NASA’s responses to these questions provided further insight into the agency’s cost-estimating management process at the organizational level. In addition, we interviewed officials in NASA headquarters’ Office of the Chief Financial Officer and Office of the Chief Engineer, and the center project managers for the 10 programs and other appropriate personnel to obtain further perspective on this issue.

To accomplish our work, we visited NASA headquarters, Washington, D.C., and Goddard Space Flight Center, Maryland. We also contacted officials at Marshall Space Flight Center, Alabama; Jet Propulsion Laboratory, California; Kennedy Space Center, Florida; Glenn Research Center, Ohio; Johnson Space Center, Texas; and Langley Research Center, Virginia.

We conducted our work from February 2003 to March 2004 in accordance with generally accepted government auditing standards.

Assessments of 10 Programs Reviewed in Detail

This appendix provides a program by program assessment of the 10 NASA programs we reviewed in detail. Each assessment provides a brief description of the program’s mission; the status of the program—that is, whether it is in development, operational, or terminated; the year the program was initiated; the fiscal year in which the Congress approved the program—that is, when full-scale design and development funds were appropriated; a comparison of the initial and current (as of April 2003) baseline an assessment of the program’s cost-estimating processes, methodologies, and practices to determine the extent they met the 14 cost-estimating criteria that we used to measure program performance. (Table 4 shows for each criterion the number of programs that met, partially met, or did not meet the criterion.)

Summary Descriptions of the 17 Additional Programs

In addition to the 10 programs that we reviewed in detail, we analyzed the initial and current development cost estimates for 17 other NASA programs.

Space Science Enterprise

Thermosphere, Ionosphere, Mesosphere Energetics and Dynamics (TIMED)

NASA’s TIMED satellite is conducting the first global study of the Earth’s mesosphere, lower thermosphere, and ionosphere—segments of the Earth’s atmosphere located between 40 and 110 miles above the planet. Initially, TIMED’s mission was to last 2 years, beginning with its launch in December 2001, but NASA extended the satellite’s orbital operations through 2006. TIMED’s goal is to improve our understanding of the influences the sun and humans have on this “gateway region” as well as the effects of its atmospheric variability on satellites and spacecraft reentering the Earth’s atmosphere.

International Gamma-Ray Astrophysics Laboratory (INTEGRAL)

INTEGRAL is a European Space Agency mission, with Russian and U.S. involvement. Launched in October 2002, the INTEGRAL satellite is equipped with two telescopes designed to register elusive gamma rays— some of the universe’s most energetic radiation—and give insight into the most violent processes in our universe. Through INTEGRAL, scientists plan to study black holes’ interaction with their surroundings, the explosion of supernovae and their role in forming chemical elements, the nature of powerful gamma-ray bursts, and transient sources that suddenly change brightness. U.S. participation consists of co-investigators providing hardware and software components to the spectrometer and imager instruments, a co-investigator for the data center, a mission scientist, and a provision for ground tracking and data collection.

Rosetta

Rosetta is a European Space Agency mission whose objectives are to study the origin of and the relationship between comets and interstellar material and to improve our knowledge of the origins of the Solar System. The Rosetta satellite was launched in March 2004 and, after a long cruise phase, is planned to rendezvous with comet Churyumov-Gerasimenko in 2014. Plans call for Rosetta to orbit the comet while taking scientific measurements and to position a probe on the comet surface to take in-situ measurements. U.S. involvement includes developing three remote-sensing instruments and a subsystem for a fourth instrument.

Mercury Surface, Space Environment, Geochemistry and Ranging (MESSENGER)

Currently scheduled to launch during a 15-day period that opens July 30, 2004, the MESSENGER spacecraft is intended to collect images of Mercury. Through these images, NASA scientists hope to determine Mercury’s geological history and the nature of its surface composition, core, poles, exosphere and magnetosphere, and magnetic field. This information is expected to provide scientists with a better understanding of how Earth was formed, how it evolved, and how it interacts with the sun.

Solar Terrestrial Relations Observatory (STEREO)

Through STEREO—an international collaboration involving France, Germany, the United Kingdom, and the United States—NASA plans to trace the flow of energy and matter from the sun to Earth by studying the solar origin of coronal mass ejections, their evolution in the heliosphere, and their effects on geospace. Twin STEREO observatories, scheduled to be launched in November 2005, will be used to develop a three- dimensional, time-dependent model of the magnetic topology, temperature, density, and velocity structure of the ambient solar wind. Because coronal mass ejections are the prime drivers of major space weather hazards, STEREO is expected to greatly improve our understanding of the most severe disturbances of the Sun-Earth system. The observatories will also provide a continuous data stream for the purpose of real-time space weather forecasts.

Stratospheric Observatory for Infrared Astronomy (SOFIA)

The SOFIA observatory—a modified Boeing 747 aircraft with a permanently installed telescope, which NASA plans to begin flying in 2005—will be used to study different astronomical objects and phenomena, including star births and deaths; solar system formations; complex molecules in space; planets, comets, and asteroids in our solar system; nebulae and dust in galaxies; and black holes at the centers of galaxies. The telescope, provided through a partnership with the German Aerospace Center, is designed to provide routine access to nearly all of the visual, infrared, far-infrared, and submillimeter parts of the spectrum. As such, SOFIA is expected to extend the range of astrophysical observations significantly beyond that of previous infrared airborne observatories through increases in sensitivity and angular resolution. NASA plans to incorporate new or upgraded technologies over the aircraft’s lifetime to allow additional scientific exploration. Because most of the instruments are to be designed and built by graduate students and post-doctoral scientists in universities throughout the United States, SOFIA will serve as a training ground for the next generation of instrument builders.

Solar-B Observatory

The Solar-B program’s objectives are to investigate the interaction between the Sun’s magnetic field and its corona and to understand the sources of solar variability. Solar-B is a Japanese Institute of Space and Astronautical Science mission, with significant U.S. involvement, and follows the Solar-A collaboration among Japan, the United Kingdom, and the United States. The observatory is designed to consist of a set of optical, extreme ultraviolet, and X-ray instruments, and NASA is expected to provide components for each. The Solar-B observatory is scheduled to be launched on a Japanese M-V rocket out of Kagoshima, Japan, in September 2006.

Herschel Space Observatory The European Space Agency’s Herschel Space Observatory (formerly the Far Infrared and Submillimetre Telescope, or FIRST) houses an infrared telescope that is expected to observe virtually unexplored spectrum wavelengths that cannot be observed from the ground. Scheduled for launch in February 2007, Herschel is expected to enable scientists to better understand galaxy formation, evolution in the early universe, and the nature of active galaxy power sources; star-forming regions and interstellar medium physics in the Milky Way and other galaxies; and the molecular chemistry of cometary, planetary, and satellite atmospheres in our solar system. NASA is providing components for two of the three instruments that will be flown on Herschel: the Heterodyne Instrument for Far Infrared and the Spectral and Photometric Imaging Receiver.

Earth Science Enterprise

Terra

Launched in February 2000, Terra is providing measurements that, according to NASA, are significantly contributing to the understanding of the total Earth system. Specifically, Terra is collecting 200 gigabytes of data each day on the earth’s physical and radiative properties of clouds, air-land and air-sea exchanges of energy, carbon, and water as well as measurements of trace gases and volcanology. One of the first operational uses of Terra was to provide imagery to support the U.S. Forest Service’s efforts to combat forest fires in the western United States. Through Terra, fire fighters were able to identify the locations of active fires, instead of locations of smoke, providing them with the data needed to better control spreading fires. Terra data were also used by the Geography Department of Dartmouth College in New Hampshire to assist in flood hazard reduction programs.

New Millennium Program’s Earth Observing-1 (EO-1)

NASA’s New Millennium Program (NMP) is designed to identify, develop, and flight-validate key instrument and spacecraft technologies that can enable new or more cost-effective approaches to conducting science missions. EO-1—the first NMP mission, launched in November 2000— includes three land imaging instruments that are expected to lead to a new generation of lighter weight, higher performance, and lower cost Landsat-type Earth surface imaging instruments.

Jason-1

The mission of the Jason-1 program, a cooperative effort with the French Space Agency, is to study the global oceans. Launched in December 2001, the Jason-1 satellite was expected to monitor ocean circulation and events such as El Nino and ocean eddies and to improve global climate forecasts and predictions. The Jason-1 satellite was positioned to orbit the earth in tandem with TOPEX/Poseidon, an earlier generation satellite launched in 1992, to provide data to the National Oceanic and Atmospheric Administration.

The SeaWinds satellite, launched in December 2002, is providing high- resolution, ocean surface wind data used for studies of ocean circulation, climate, and air-sea interaction to understand global climate changes and weather patterns better. By using long-term wind data in numerical weather and wave prediction models, SeaWinds is expected to improve weather forecasts near coastlines and storm warning and monitoring.

Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (Calipso)

The Calipso satellite, scheduled for launch in 2005, is being designed to study the effect that aerosols and clouds have on the Earth’s radiation balance, which ultimately controls the temperature of the Earth. Calipso is expected to provide scientists with data to construct three-dimensional structures of the atmosphere, enabling new observationally based assessments of the radiative effects of aerosol and clouds that will greatly improve our ability to predict future climate change. NASA plans to fly Calipso in formation with Aqua and CloudSat, a satellite being designed to measure the vertical structure of clouds from space and contribute to a better understanding of the role of clouds in the Earth’s climate system. The Calipso program is a cooperative effort with France.

Space Flight Enterprise

X-38 Crew Return Vehicle (CRV)

The X-38 Crew Return Vehicle was cancelled in April 2002, due to its single purpose design and the potentially high costs identified by an independent assessment. The purpose of the CRV project was to initiate work toward an independent U.S. crew return capability for the International Space Station. As envisioned, CRV was expected to serve as a back-up to the space shuttle orbiters by providing resupply to the station or change-out crew, or accommodating safe return for up to seven crew members who may be ill or injured or in the event that a catastrophic failure of the station made it unable to support life.

Alternate Turbopump Program (ATP)

ATP’s primary objectives were to significantly improve the safety and operating margins of the high-pressure turbopump in the space shuttle’s main engine and to eliminate the need to remove the turbopump for postflight maintenance. An alternative turbopump was successfully implemented in the shuttle launched in April 2002. According to NASA, ATP’s development contract, signed in December 1986, specifically addressed shortcomings of the previous turbopumps; took advantage of the latest technologies; and applied lessons learned. The contract called for the parallel development of two high-pressure turbopumps—one that operates on oxidization and one on fuel. However, 5 years into the program, technical problems prompted NASA to end parallel development and concentrate first on developing the oxidizer turbopump, which was first flown in July 1995. Although development of the fuel turbopump resumed in 1994, extreme high temperatures, pressures, and rotor speeds resulted in significant design challenges and the design certification review was not completed until March 2001. The full implementation of the fuel turbopump into flight was completed beginning with the April 2002 shuttle flight.

Tracking and Data Relay Satellite (TDRS) Replenishment

In December 2002, the TDRS Replenishment project achieved its goal: launch three geosynchronous satellites to replace the existing aging satellite constellation, and thereby continue to provide space network tracking, data, voice, and video services to NASA scientific satellites, the Space Shuttle program, the International Space Station, and other NASA customers. According to NASA, the functional and technical performance requirements for the replacement satellites—launched in June 2000, March 2002, and December 2002—are virtually identical to those of the previous satellites.

Advanced Health Management System (AHMS) Phase 1

AHMS is expected to provide safe shutdown of the space shuttle main engine during potentially catastrophic high-pressure turbopump failures through improved monitoring of engine vibration and anomaly response capabilities. According to NASA, AHMS modifications include (1) adding a vibration redline monitor for high pressure turbopumps, (2) doubling memory capacity and employing radiation tolerant memory, (3) adding an external communication interface for a potential phase-two health management computer, and (4) eliminating existing memory retention batteries and replacing them with nonvolatile memory. While NASA stated the AHMS will be available for launch in January 2005, the shuttle fleet’s return to flight date is planned for March or April 2005.

Description of Earned Value Management

Earned value management (EVM) goes beyond the two-dimensional approach of comparing budgeted costs to actuals. Instead, it attempts to compare the value of work accomplished during a given period with the work scheduled for that period. By using the value of completed work as a basis for estimating the cost and time needed to complete the program, earned value can alert program managers to potential problems early in the program.

An accurate, valid, and current performance management baseline is needed to perform useful analyses using EVM. In 1996, in response to acquisition reform initiatives, the Department of Defense (DOD) adopted 32 criteria for evaluating the quality of management systems. In general terms, the 32 criteria require contractors to (1) define the contractual scope of work using a work breakdown structure; (2) identify organizational responsibility for the work; (3) integrate internal management subsystems; (4) schedule and budget authorized work; (5) measure the progress of work based on objective indicators; (6) collect the cost of labor and materials associated with the work performed; (7) analyze any variances from planned cost and schedules; (8) forecast costs at contract completion; and (9) control changes. The criteria have become the standard for EVM and have been adopted by major U.S. government agencies, industry, and the governments of Canada and Australia. The full application of EVM system criteria is appropriate for large cost reimbursable contracts where the government bears the cost risk. For such contracts, management discipline prescribed by the criteria is essential. In addition, data from an EVM system have been proved to provide objective reports of contract status, allowing numerous indices and performance measures to be calculated. These can then be used to develop accurate estimates of anticipated costs at completion, providing early warning of impending schedule delays and cost overruns.

Table 5 lists the 32 criteria, organized into five basic categories: organization, planning and budgeting, accounting considerations, analysis and management reports, and revisions and data maintenance.

The standard format for tracking earned value is through a cost performance report (CPR). The CPR is a monthly compilation of cost, schedule, and technical data, which displays the performance measurement baseline, any cost and schedule variances from that baseline, the amount of management reserve used to date, the portion of the contract that is authorized unpriced work, and the contractor’s latest revised estimate to complete the program. As a result, the CPR can be used as an effective management tool because it provides the program manager with early warning of potential cost and schedule overruns.

Using data from the CPR, a program manager can assess trends in cost and schedule performance. This information is useful because trends tend to continue and can be difficult to reverse. Studies have shown that once programs are 15 percent complete, the performance indicators are indicative of the final outcome. For example, a CPR showing a negative trend for schedule status would indicate that the program is behind schedule. By analyzing the CPR, one could determine the cause of the schedule problem such as delayed flight tests, changes in requirements, or test problems because the CPR contains a section that describes the reasons for the negative status. A negative schedule can be a predictor of later cost problems because additional spending is often necessary to resolve problems. CPR data also provide the basis for independent assessments of a program’s cost and schedule status and can be used to project final costs at completion in addition to determining when a program should be completed.

Examining a program’s management reserves is another way that a program can use a CPR to determine potential issues early on. Management reserves, which are funds that may be used as needed, provide flexibility to cope with problems or unexpected events. EVM experts agree that transfers of management reserves should be tracked and reported because they are often problem indicators. An alarming situation arises if the CPR shows that the management reserves are being used at a faster pace than the program is progressing toward completion. For example, a problem would be indicated if a program has used 80 percent of its management reserves, but only completed 40 percent of its work. A program’s management reserves should contain at least 10 percent of the cost to complete a program so that funds will always be available to cover future unexpected problems that are more likely to surface as the program moves into the testing and evaluation phase.

Comments from the National Aeronautics and Space Administration

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

Staff making key contributions to this report were Jerry Herley, Shirley Johnson, Charles Malphurs, Karen Sloan, Madhav Panwar, Karen Richey, Jennifer Echard, and Deborah Lott.

GAO’s Mission

The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail alerts” under the “Order GAO Products” heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs