NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective Program Management
Published: 2004-05-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: For more than a decade, GAO has identified the National Aeronautics and Space Administration's (NASA) contract management as a high-risk area--in part because of NASA's inability to collect, maintain, and report the full cost of its programs and projects. Lacking this information, NASA has been challenged to manage its programs and control program costs. The scientific and technical expectations inherent in NASA's mission create even greater challenges--especially if meeting those expectations requires NASA to reallocate funding from existing programs to support proposed new efforts. Because cost growth has been a persistent problem in a number of NASA programs, GAO was asked to examine NASA's cost estimating for selected programs, assess NASA's cost-estimating processes and methodologies, and describe any barriers to improving NASA's cost-estimating processes. To conduct GAO's work, GAO analyzed a total of 27 NASA programs--10 of which GAO reviewed in detail.
section_title: What GAO Found
paragraphs: Considerable change in NASA's program cost estimates--both increases and decreases--indicates that NASA lacks a clear understanding of how much its programs will cost and how long they will take to achieve their objectives. For example, the development cost estimates for more than half of the 27 programs that GAO reviewed have increased and for some programs this increase was significant--as much as 94 percent. Cost estimates changed for each of 10 programs that GAO reviewed in detail. For 8 of the 10 programs, the estimates increased. Although NASA cited specific reasons for the changes, such as technical problems and funding shortages, the variability in the cost estimates indicates that the programs lacked the sufficient knowledge needed to establish priorities, quantify risks, and make informed investment decisions, and thus predict costs. Most notably, NASA's basic cost-estimating processes--an important tool for managing programs--lack the discipline needed to ensure that program estimates are reasonable. Specifically, GAO found that none of the 10 NASA programs that GAO reviewed in detail met all of GAO's cost-estimating criteria, which are based on criteria developed by Carnegie Mellon University's Software Engineering Institute. Moreover, none of the 10 programs fully met certain key criteria--including clearly defining the program's life cycle to establish program commitment and manage program costs, as required by NASA. In addition, only three programs provided a breakdown of the work to be performed. Without this knowledge, the programs' estimated costs could be understated and thereby subject to underfunding and cost overruns, putting programs at risk of being reduced in scope or requiring additional funding to meet their objectives. Finally, only two programs have a process in place for measuring cost and performance to identify risks. NASA has limited ability to collect the program cost and schedule data needed to meet basic cost-estimating criteria. For example, as GAO has previously reported, NASA does not have a system to capture reliable financial and performance data--key to using effectively the cost-estimating tools that NASA officials state that programs employ. Further, without adequate financial and nonfinancial data, programs cannot easily track an acquisition's progress and assess whether the program can meet its cost and schedule goals before it incurs significant cost and schedule overruns. NASA identified other barriers, including limited cost-estimating staff. According to NASA officials, several initiatives are under way to remove such obstacles and improve the agency's cost-estimating practices.

section_title: Letter
paragraphs: The lack of reliable financial and performance information has posed  significant challenges to the National Aeronautics and Space  Administration’s (NASA) ability to manage its largest and most costly  programs effectively. For nearly 15 years, NASA contract management has  been on GAO’s high-risk list—due in part to NASA’s inability to collect,  maintain, and report the full cost of its programs and projects. Without  such information, NASA has consistently developed unrealistic cost and  schedule estimates, which, at least in part, are reflected in the cost growth  and schedule increases in many of its programs.
The demanding scientific and technical expectations inherent in NASA’s  mission create even greater challenges for the agency to control program  costs—especially if meeting those expectations requires NASA to  reallocate funding from existing programs to support new efforts. Because  cost growth has been a persistent problem on a number of NASA programs,  you asked us to (1) identify initial cost estimates in selected NASA  programs and any changes in those cost estimates, (2) assess NASA’s cost- estimating processes and methodologies, and (3) describe any barriers that  make it difficult for NASA to improve its cost-estimating processes.
Our review focused on 27 of 68 NASA programs in the development phase  as of April 2003 or that completed development in fiscal year 2001 or 2002.  To assess NASA’s cost-estimating processes and methodologies, we  conducted a more in-depth review of 10 of the 27 programs, which  generally had the highest development cost estimate within five of NASA’s seven Enterprises. Our work was conducted between February 2003 and  March 2004 in accordance with generally accepted government auditing  standards. For a complete description of our scope and methodology, see  appendix I.
subsections: section_title: Background
paragraphs: NASA’s programs encompass a broad range of complex and technical  activities—from investigating the composition and resources of Mars to  providing satellite and aircraft observations of Earth for scientific and  weather forecasting. NASA currently funds more than 100 programs and  projects in various phases of execution in 7 strategic Enterprises: Space  Science, Earth Science, Biological and Physical Research, Aeronautics,  Space Flight, Education, and Exploration Systems. Two NASA offices have  key responsibilities in ensuring the effective execution of these programs:  the Office of the Chief Financial Officer, which is responsible for providing  oversight and financial management of agency resources and establishing  related policy guidance, and the Office of Chief Engineer, which is  responsible for ensuring development efforts and mission operations are  planned and conducted using sound engineering practices.
More than two-thirds of NASA’s work force is made up of contractors and  grantees, and 90 percent—or roughly $13 billion—of NASA’s annual budget  is spent on work performed by its contractors. Since 1990, we have  identified NASA’s contract management as a high-risk area. This  assessment has been based in part on our repeated finding that NASA does  not have good cost-estimating processes or the financial information  needed to develop good cost estimates for its programs, making it difficult  for NASA to oversee its contracts and control costs. For example, in July  2002, we reported that an independent task force convened to assess the  management of the International Space Station concluded that the  program’s fiscal year 2002 through fiscal 2006 budget was not credible  because of weaknesses in its cost-estimating processes. The task force  pointed out that these problems occurred because NASA had not instituted  or had ignored many of the program’s control and contract oversight  procedures—such as preparing a full life-cycle cost estimate—that should  have alerted the agency to the growing cost problem and the need for  mitigating actions. According to the cost analysis team that supported the  task force, NASA’s focus on staying within annual budgets instead of  managing total program costs was perhaps the single greatest factor in the  program’s cost growth.
NASA’s unreliable cost estimates have significant implications for potential  future endeavors, such as those outlined by the President in January of this  year. Specifically, the President called for a shift in NASA’s long-term focus,  envisioning that NASA will retire the shuttle program as soon as assembly  of the International Space Station is completed, planned for the end of the  decade; develop a new crew exploration vehicle as well as launch human  missions to the moon between 2015 and 2020, and build a permanent lunar  base as a stepping stone for more ambitious missions. To achieve these  goals, the President proposed spending $12 billion over the next 5 years— about $1 billion of which would come from an increase in NASA’s budget,  currently $15.4 billion—with the remaining $11 billion being reallocated  from existing NASA programs.
Developing reliable cost estimates has been difficult for agencies across  the federal government. The need for reliable cost estimates is at the heart  of two of the five-governmentwide initiatives in the 2002 President’s  Management Agenda (PMA); the two are “improved financial performance”  and “budget and performance integration.” These initiatives are aimed at  ensuring that federal financial systems produce accurate and timely  information to support operating, budget, and policy decisions and that  budgets are performance-based. As part of these initiatives, the President  calls for changes to the budget process to better measure the real cost and  performance of programs. According to the PMA, accomplishing all of the  crosscutting initiatives will matter little without the integration of agency  budgets with performance.
section_title: Development Cost Estimates Frequently Changed
paragraphs: As of April 2003, the baseline development cost estimates for the programs  we reviewed varied considerably from the programs’ initial baseline  estimates. More than half of the programs’ development cost estimates  increased, and for some programs, this increase was significant. The  baseline development cost estimates for each of the 10 programs we  reviewed in detail were rebaselined—that is, recalculated to reflect new  costs, time frames, or resources associated with program changes in  program objectives, deliverables, or scope and plans. Although NASA  provided specific reasons for the increased cost estimates and  rebaselinings—such as delays in the development or delivery of key  system components and funding shortages—it does not have guidance for  determining when rebaselinings are justified. Such criteria are important to  instilling discipline in the cost-estimating process.
Most of the 27 programs we reviewed experienced a change in their  development costs estimates. While 8 of the 27 programs experienced  slight decreases in their development cost estimates, 17 experienced  cost growth—as much as almost 94 percent. The remaining two programs  had no change. Ten of the 17 programs’ cost growth was greater than  25 percent. Table 1 shows the development cost estimate changes from  the initial baseline to the baseline as of April 2003 and the life-cycle cost  estimate for each of the 27 programs. The 10 programs that we reviewed  in detail are shaded and italicized. (See app. II for assessments of the  10 programs and app. III for descriptions of the remaining 17 programs.)
The development cost estimates for each of the 10 programs that we  reviewed in detail have been rebaselined—for some programs, as many as  four times—and for 7 of the 10 programs, the cost estimate increased each  time it was rebaselined (see fig. 1).
For the 10 programs we reviewed in detail, NASA cited specific reasons for  changes in the baseline development cost estimates and the recalculated  baselines—many of which were related to technical problems and  subsequent delays in the development or delivery of key system  components, and insufficient funding and reserves, as illustrated in the  following examples:  Technical problems in the MERs program required a significant redesign  of components and the development of a new landing system. Two of  MERs’ three rebaselinings were also the result of inadequate reserves.  According to NASA officials, without the rebaselinings, the development  cost “to go” would have drained the program’s reserves.
The increase in CLCS’s development cost estimate and rebaselining was  the result of poorly defined requirements and design, software  integration problems, and fundamental changes in the project’s  management structure and contractors’ approach to the work. The  project, which experienced an almost 94 percent increase in its baseline  development cost estimate, was ultimately terminated.
The GP-B program—which was rebaselined four times—experienced  significant schedule slippages due to repeated technical problems,  including failures in the probe’s heat exchanger, the need for additional  testing, payload electronics delays, and thermal vacuum test failures.
Schedule slippages in the SIRTF program—which contributed to  increases in the program’s baseline development cost estimate and four  rebaselinings of the estimate—were caused by delays in the delivery of  components, flight software, and the mission operation system as well  as launch delays that resulted from a handling accident involving a  global positioning system payload and concerns of delamination on the  launch vehicle’s solid rocket motors.
Changes in development cost estimates for the CAU program were  primarily the result of the program’s expanded scope, which occurred in  October 2002, to produce modification kits that would allow the CAU  upgrade to be installed into the orbiters.
The Hyper-X program experienced three rebaselinings, and according to  the project manager, the program will be rebaselined again in the near  future. The rebaselinings were due to schedule slippages resulting from  the need to fund an investigation of the problems experienced in the  first Mach 7 flight vehicle—which was destroyed in flight—and related  corrective actions to the second Mach 7 flight.
Revised contract requirements, funding changes, or the realization that  program goals are not achievable may require a formal rebaselining.  However, NASA has not defined or provided guidance or restrictions on  rebaselining to ensure that programs consistently and appropriately apply  rebaselinings and do not adjust their baseline cost estimates whenever the  estimates become unmanageable. Further, NASA lacks a process for  systematically identifying and assessing programs that are not achieving  their cost, schedule, and performance goals. Such a process has been  employed by the Department of Defense (DOD), which also relies heavily  on contractors to deliver complex, cutting-edge technologies to meet its  mission. Specifically, DOD must report to the Congress programs that incur  a cost growth of 15 percent or more in the program baseline. Moreover,  DOD must justify the continuation of acquisition programs that incur a cost  growth of 25 percent or more in the program baseline by certifying that  specific criteria have been met—including that the new cost estimates are  reasonable. Under such a process, 5 of the 10 programs that we reviewed  in detail would have been required to report to the Congress, and 4 of  the 5 programs would have had to certify that their new cost estimates  were reasonable.
section_title: Poor Estimating Processes and Methodologies Contributed to Wide Variations in Baseline Cost Estimates
paragraphs: NASA has yet to implement a well-defined process for estimating the  cost of its programs—a weakness we and NASA’s Inspector General have  repeatedly reported. Recognizing the need for such a process, NASA  developed a cost-estimating handbook in 2002—the first such guidance  provided to its cost-estimating community and program and project  managers. Despite this effort, the programs we reviewed failed to follow  key cost-estimating processes, including developing and documenting full  life-cycle cost estimates, summarizing estimates according to the current  breakdown of work to be performed, conducting an uncertainty analysis,  performing an independent review of contractors’ cost estimates, and later  using earned value management (EVM) to assess progress.
Reflecting Office of Management and Budget (OMB) guidance and best  practices of government and industry leaders, NASA requires that full  life-cycle cost estimates be prepared using full cost accounting, that  estimates be summarized according to the current breakdown of work to  be performed, and that major changes be tracked to the life-cycle cost. In  its draft cost-estimating handbook, NASA lists a number steps that are  integral to preparing a reliable life-cycle cost estimate, including  preparing or obtaining a cost analysis requirements description (CARD),  developing ground rules and assumptions, and developing cost range and  risk assessments.
Carnegie Mellon University’s Software Engineering Institute (SEI) echoes  the need for reliable cost-estimating processes in managing software  implementations—identifying tasks to be estimated, mapping the estimates  to the breakdown of work to be performed, and identifying and explaining  assumptions are among SEI’s requisites for producing reliable cost  estimates. To evaluate the cost-estimating processes of the 10 NASA  programs that we reviewed in detail, we selected 14 criteria based on SEI  checklists (see table 2). Many of these criteria are included in NASA’s  cost-estimating guidance.
Despite NASA requirements and OMB and SEI guidance, few of the  10 programs that we reviewed in detail met even a third of these criteria;  only one met half. Further, none of the programs fully met certain key  criteria. For example, none provided a complete life cycle with definitions  or a complete description of the methodology used to generate the  complete cost estimate, such as data sources and uncertainties. According  to the draft NASA cost-estimating handbook, a reliable life-cycle cost  estimate is critical to making realistic decisions about developing or  producing a system and to determining the appropriate scope or size of a  program. NASA guidance also calls for breaking down the work to be  performed into smaller units to facilitate cost estimating and program and  contract management and to help ensure relevant costs are not omitted.  However, only 3 of the 10 programs provided a complete breakdown of the  work to be performed. Table 3 shows for each program the applicable  criteria that were met, partially met, or not met. (See app. II for a program  by program assessment.)
Failing to meet these criteria puts programs at certain risk. For example,  underestimating a program’s full life-cycle costs creates the risk that a  program could be underfunded and subject to major cost overruns, which  would ultimately result in the program being reduced in scope or additional  funding being requested and appropriated to ensure the program meets its  objectives. Conversely, overestimating life-cycle costs creates the risk that  a program will be deemed unaffordable and would, therefore, go unfunded.  Without a complete WBS, NASA programs cannot ensure that the life-cycle  cost estimates have captured all relevant costs, which again can result in  underfunding and cost overruns. Further, inconsistent WBS estimates  across programs can create problems of double counting or, worse,  underestimating costs when using historical program costs as a basis for  projecting future costs on similar programs.
Despite the uncertainty inherent in estimating the cost of emerging  technologies, all of the 10 programs we reviewed also failed to conduct  an uncertainty analysis to assess risks associated with the cost  estimates. Instead, the programs expressed their cost estimates as  point values—which implies certainty—not as ranges or numbers with  confidence levels. Performing an uncertainty analysis, such as a Monte Carlo simulation, quantifies the amount of cost risk within a program.  Only by quantifying the cost risk can management make informed  decisions about risk mitigation strategies. Quantifying cost risks also  provides a benchmark against which future progress can be measured.  Without this knowledge, NASA may have little specific basis to  determine adequate financial reserves, schedule margins, and technical  performance margins to provide managers the flexibility needed to  address programmatic, technical, cost, and schedule risks, as required by  NASA policy.
Seven of the 10 programs also failed to have an independent review of  contractors’ cost estimates—as required by NASA. Instead, programs  established their budgets based on contractor proposals—particularly  problematic since many contractors could bid low in order to win the  contract. To ensure contractor costs are realistic, NASA procedures and  guidelines specifically require programs to ensure that independent  reviews are conducted and that these reviews address project life-cycle  costs, risk management plans, as well as technical issues. Without such  reviews, NASA decision makers lacked the benchmarks needed to assess  the reasonableness of the contractors’ proposed costs, limiting NASA’s  ability to make sound investment decisions and accurately assess  contractor performance.
Finally, only two programs used EVM—an approach used by DOD and  leading companies to provide meaningful assessments of a program’s  progress by comparing the value of work performed to its costs, rather  than the traditional management approach of comparing budgeted and  actual costs, which can provide a distorted view of a program’s progress.  (For a detailed discussion of EVM, see app. IV.) By using the value of  completed work as a basis for estimating the cost and time needed to  complete the program, EVM can alert program managers to potential  problems early in the program. NASA requires that EVM be used on all  significant contracts—that is, research and development contracts with  a total anticipated final value of $70 million or more, and production contracts with a total anticipated final value of $300 million or more— which includes all of the 10 programs we reviewed in detail. Although the  program managers for all 10 programs stated that EVM was used in their  projects, only two programs provided cost performance reports, indicating  a true EVM process was in place. The remaining eight programs relied on  NASA Form 533, which captures planned and actual obligations and  expenditures—not the value of the work performed. Without a true EVM  process, programs cannot readily determine if a program is at risk of cost  and schedule overruns until it is too late to make programmatic changes to  avoid these risks.
section_title: NASA Has Begun to Address Certain Barriers to Effective Cost Estimating
paragraphs: There are several impediments that NASA needs to overcome to implement  effective cost-estimating practices. These include the lack of reliable  financial data and other performance information; lack of trained EVM  staff, data analysis tools, and incentive for supporting and implementing  EVM; and ineffective use of cost analysts. NASA has initiated several  measures to begin addressing some of these impediments.
subsections: section_title: Utility of Cost-Estimating Tools Depends on the Reliability of NASA’s Financial and Performance Data
paragraphs: According to NASA officials, state-of-the-art cost-estimating tools have  been funded and implemented. For example, NASA officials told us that  commercial-off-the-shelf models have been used to estimate hardware and  software acquisition costs and quantify the level of uncertainty surrounding  cost estimates. However, these cost-estimating tools are only as good as the  data they rely on to develop the estimates. For more than a decade, we  have reported that NASA has failed to develop a system to capture reliable  financial and performance information, posing significant challenges to  NASA’s ability to estimate and control program costs. Over the past year  alone, we issued numerous reports on NASA’s Integrated Financial  Management Program (IFMP)—the agency’s third and most recent effort to  implement a modern, integrated financial management system. Specifically,  we found that IFMP—which is under the responsibility of the Program  Executive Officer for IFMP—will not, as it is being implemented, routinely  provide program managers and other key stakeholders and decision  makers—including the Congress—with the financial related information  needed to measure program performance and ensure accountability. For  example, the core financial module (considered the backbone of the  system) does not appropriately capture property, plant, and equipment, as  well as material in its general ledger at the transaction level—which is  needed to provide independent control over these assets. In addition,  NASA implemented the system before it had the capability to capture the  full costs of its programs and projects. According to headquarters officials,  collecting nonfinancial data crucial to cost estimating—such as technology  readiness levels, parts counts, and team and management experience and  skill ratings—has also been difficult.
section_title: Use of EVM Has Been Undermined by a Lack of Trained Staff, Data Analysis Tools, and Incentive
paragraphs: According to headquarters officials, agencywide EVM implementation  efforts began in 1996 and are recognized by NASA management as a key  tool in monitoring and measuring cost trends in higher risk project  elements—a tool that serves as an early warning of the need for cost-risk  mitigation actions to maintain control of program costs. These officials  stated that EVM has been applied to the International Space Station  Program and with varying levels of emphasis to other programs and  projects at different NASA centers. While all of the program managers for  the 10 programs that we reviewed in detail stated that they used EVM, only  2 of the programs used a true EVM process.
NASA headquarters officials identified several challenges that have  affected the agency’s ability to implement EVM effectively, including a lack  of staff and data analysis tools. According to officials, resource constraints  have prevented the agency from staffing many project offices with  appropriate personnel to fulfill all project functions. In addition, there has  been little or no priority to include a trained EVM analyst, even if one were  available. Headquarters officials also noted that EVM has been hampered  by the lack of a practical automated software data analysis tool. Without  such a tool, analyzing the contractors’ EVM cost performance reports,  which contain significant amounts of data, became a cumbersome  undertaking that often resulted in incomplete and untimely analyses,  providing little usefulness to inform management decisions. A lack of  incentive to support EVM has further undermined its use. Some project  managers whom we spoke with are skeptical about the benefits of EVM  and argue that it has failed to help them manage or control program costs.  According to NASA headquarters officials, during proposal and contract  negotiation phases, contractors have also suggested not using EVM as a  way to reduce contract costs. While EVM was included in most contracts  for the 10 programs we reviewed in detail—as required by NASA policy—it  was used only in two programs as a cost-estimating tool. In general, EVM  has been viewed by NASA as a financial reporting tool. Consequently, there  is little incentive to use EVM because the data needed to report financial  activity is captured elsewhere, such as in Form 533.
section_title: Ineffective Use and Placement of Cost Analysts across the Agency’s Cost Activities also Hinders NASA’s Efforts to Improve Its Cost-Estimating Practices
paragraphs: NASA’s efforts to improve its cost-estimating processes have also been  undermined by ineffective use of its limited number of cost-estimating  analysts. For example, headquarters officials state that as projects entered  the formulation phase, they have typically relied on program control and  budget specialists—not cost analysts—to provide the financial services to  manage projects. Yet budget specialists are generally responsible for  obligating and expending funding—not for conducting cost analyses that  underlie the budget or ensuring budgets are based on reasonable cost  estimates—and, therefore, tend to assume that the budget is realistic.  While NASA officials state that its cost-estimating staff is too limited to be  involved in day-to-day project execution activities, they agreed that the  cost analysts could be more effectively used throughout the life cycle— particularly when projects are rebaselined and independent cost estimates  of project changes must be performed.
In some cases, cost analysts are not appropriately located in the  organization, which may compromise controls NASA has in place to ensure  reasonable cost estimates. For example, some cost analysts at NASA’s  centers are located with senior systems engineers in systems management  organizations, while others are not. According to NASA officials, housing  the cost analysts with senior systems engineers has two key benefits. First,  the systems engineers generally conduct systems analyses to help ensure  that a program’s requirements are properly established and that the design  and validity meet the requirements. Such analyses can greatly inform the  development of reasonable cost estimates. Second, the systems  engineering offices afford some measures of independence for cost  estimating, which, according to NASA cost- estimating guidance and  procedures, is important to the overall project management process.
However, NASA officials stated that several of its centers’ cost analysts are  in the advocacy chain of command—not housed with senior systems  engineers. For example, one center’s 15 cost analysts work in the center’s  Office of the Chief Financial Officer—which is responsible for directing the  development and execution of the center’s budget—not in the systems  management organization, which is independent from the rest of the  center. As a result, the costs analysts’ estimates may not be adequately  informed by the systems engineers and may lack the objectivity required to  ensure that the criteria for independence have been met.
section_title: Efforts Under Way to Remove Some Barriers and Improve Cost Estimating
paragraphs: NASA has several initiatives under way to improve the agency’s  cost-estimating processes. First, NASA has established a Cost Analysis  Division in the Office of the Comptroller to strategically manage analyses  related to directing and funding research, improving cost-estimating  processes and practices, and providing cost-estimating tools and training  throughout the agency. The division also provides, along with the  Independent Program Assessment Office (IPAO), the last independent cost  estimate of projects before the information is released externally. These  efforts are being coordinated through a steering committee composed of  the managers of the cost analysis organizations from each of the centers  and IPAO’s deputy director.
NASA is revising the cost sections in its governing procedures and  guidelines and is finalizing its cost-estimating handbook to reflect these  changes. These documents will require the routine use of probabilistic  cost risk analysis, a CARD document, cost as an independent variable  (CAIV), and EVM. The CARD supports the project life-cycle cost estimate  and a congressionally required independent cost estimate. Agency officials  note that while there has been some use of CARD in the agency, its first  concentrated and successful use was in the 2001 to 2002 independent  cost estimate for the International Space Station program. According  to headquarters officials, NASA’s revised guidance and finalized  cost-estimating handbook will provide direction and guidance for fully  implementing the use of CARDs for major development projects. Although  NASA calls for CAIV to be used routinely and notes that CAIV demonstrates  a commitment to evolutionary acquisition, it has yet to provide guidance  on its implementation. NASA headquarters officials stated that guidance  relating to improvements in the collection of cost data is also being  reflected in its revised governing procedures and guidelines.
With respect to EVM, NASA headquarters officials described several efforts  under way to ensure agencywide implementation of true EVM. For  example, NASA recently revised its EVM policy directives to shift  ownership of EVM responsibilities from NASA’s Chief Financial Officer to  NASA’s Chief Engineer, to emphasize that EVM is to be considered a project  management tool rather than a financial management tool. NASA officials  also noted that the agency is working to inform managers of the  performance management capabilities available to them through EVM and  to emphasize the importance of providing adequate resources and  management support to ensure successful EVM implementation.  Agencywide goals for EVM implementation include promoting the effective  use of EVM and providing needed training and education for program and  project staff. These efforts and proposed initiatives should help resolve  EVM utilization problems.
Finally, NASA officials told us that the agency is planning to hire additional  cost analysts to alleviate understaffing at all of its center cost analysis  offices. The agency envisions a total staff of about 100 cost analysts  along with additional support contractors. NASA officials also stated that  it is necessary to ensure centers address the problem of having cost  analysts located in the advocacy chain of command, which could affect  five NASA centers.
Because NASA’s initiatives have only recently been implemented or are still  in the drafting or planning stage, we cannot determine to what degree these  efforts will enable NASA to provide reasonable and defensible cost  estimates of its programs and projects.
section_title: Conclusions
paragraphs: There are numerous scientific and technical challenges inherent in the  successful implementation of many NASA programs. Nevertheless, the  need to choose among competing alternatives within limited budget  resources makes it essential that the agency and the Congress clearly  understand the costs and uncertainties of programs proposed for  authorization and funding. Yet, NASA does not have the disciplined  cost-estimating process needed to make informed acquisition decisions,  nor does the agency have processes and tools for capturing, monitoring,  and managing program costs and schedules within an implementation plan  on a timely basis. This makes it difficult for senior NASA officials, program  and project managers, and other key stakeholders to measure performance  and initiate mitigation measures when needed. Taken together, the lack of  disciplined and established cost-estimating processes and tools can cause  program officials to restructure projects to available resources rather than  develop realistic cost estimates and implementation plans for projects. As  a result, programs may have to be modified to accommodate emerging  technical, cost, and schedule realities. Ultimately, programs cost more, fail  to meet their schedules, or deliver less than originally envisioned. To help  minimize project costs increases and implementation delays identified in  this report, NASA needs to instill disciplined cost-estimating processes  into its project development and approval activities and to ensure such  processes are integrated with its implementation of an integrated  financial management system. Without a process that prevents programs  from proceeding before they have sufficiently demonstrated that key  cost-estimating criteria have been met, NASA programs will continue to  be at risk of cost and schedule overruns.
section_title: Recommendations for Executive Action
paragraphs: Improvements to NASA’s cost-estimating processes will partly depend on  the agency’s ability to address recommendations that we made in  November 2003 to help ensure NASA effectively implements a modern,  integrated financial management system. Notwithstanding the need to  address those recommendations, to better position NASA to ensure its  recent initiatives result in sound cost-estimating practices agencywide,  we are making three recommendations with minimum suggested courses  of action. First, we are recommending that the NASA Administrator direct  the Program Executive Officer for IFMP, the Chief Financial Officer, and  the Chief Engineer to develop an integrated plan for improving cost  estimating that, at a minimum, includes specific actions for ensuring that  guidance is established on rebaselining and that rebaselining is  consistently applied to provide accountability among programs,  true earned value management is used as an organizational management  tool to bring cost to the forefront in NASA’s management decision- making process,  acquisition and earned value management policies and procedures are   staff and support for cost-estimating and earned value analyses are  effectively used.
In addition, we recommend that the NASA Administrator direct the Chief  Financial Officer to establish a standard framework for developing life-  cycle cost estimates. At a minimum the framework should require each  program or project to  base its cost estimates on a full life cycle for the program—including all  direct and indirect costs for operations and maintenance and disposal as  well as planning and procurement—and on a work breakdown structure  that encompass both in-house and contractor efforts,  prepare a cost analysis requirements description,  prepare an independent government estimate at each milestone of the   conduct a cost risk assessment that identifies the level of uncertainty  inherent in the estimate.
Further, we recommend that the NASA Administrator develop procedures  that would prohibit proposed projects from proceeding through the review  and approval process when they do not address the elements of the  recommended cost-estimating practices.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, NASA’s Deputy Administrator  stated that the agency concurs with the recommendations, adding that the  recommendations validate and reinforce the importance of activities  underway at NASA to improve cost estimating and program management.
Notwithstanding agreement with our recommendations, the Deputy  Administrator believes NASA has made substantive changes and achieved  significant improvements in its cost-estimating processes. For example,  NASA’s comments on a draft of this report cite a 1992 GAO report  (GAO/NSIAD-93-97) that found a median 77 percent increase in NASA  program costs. According to the Deputy Administrator, this contrasts with  a 13 percent cost growth in this present study. While there may be  improvements in the percent of cost growth of some projects, such  declines in cost growth are often achieved by rescoping and rebaselining  projects to remain within available resources, as was demonstrated in a  number of projects discussed in this report. We do not believe other  examples cited by the Deputy Administrator, namely termination of the  Checkout and Launch Control System and cost control measures imposed  on the International Space Station, demonstrate that NASA has made  substantive changes and achieved significant improvements in its cost- estimating processes. Rather, we believe these examples demonstrate what  happens when projects are undertaken without a full understanding of the  potential costs and management challenges inherent in many of the  programs NASA proposes and then implemented without adequate  financial management systems in place.
With regard to our recommendation to develop guidelines for rebaselining  and ensure effective use of earned value management, the Deputy  Administrator cited the development of revised direction on program and  project management and a refocus on risk and cost-risk analysis. NASA  also now requires the establishment of cost thresholds that, if exceeded,  will require a rebaselining review. Further, because much of NASA’s work is  performed through grants and contracts, NASA’s revised procedures will  emphasize how risk and technical complexity affect contractor  performance. New earned value management and acquisition policies and  procedures will be implemented through program management councils  that will review and approve programs and projects regularly through each  step of their development. Also, a new Cost Analysis Division has been  established, and cost-estimating staff has been added to it and NASA’s  Independent Program Assessment Office. NASA also noted the importance  of training needed to match the new requirements.
NASA’s Deputy Administrator also concurred with our recommendation to  establish a standard framework for developing life-cycle cost estimates.  According to the Deputy Administrator, NASA’s new processes and  procedural requirements document will define the full life-cycle cost to  include development, operations, maintenance, disposal, and all NASA in- house direct and indirect costs to eliminate ambiguity and ensure  consistency. NASA’s revised cost-estimating handbook will provide further  guidance for life-cycle cost estimates. Also, project managers will be  responsible for developing and maintaining a cost analysis requirements  document similar to a tool DOD uses that will include the equivalent of a  project and technical description; key performance parameters, including  documentation of actual work breakdown structure cost elements; and  initial and annual updates of the life-cycle cost estimates. NASA guidance  will also require periodic independent cost estimates on major programs  and approval by the respective program management council to enter into  implementation after an independent estimate has been completed.
Lastly, NASA’s Deputy Administrator concurred with our recommendation  to prohibit proposed projects from proceeding through the review and  approval process when they do not address the elements of the  recommended cost-estimating practices. Accordingly, NASA’s forthcoming  procedural requirements will define the authority of the program  management councils that will, according to NASA, enforce the  requirements, including the required information, documentation, and  management methods needed for proceeding through the review and  approval process. The Deputy Administrator also noted the availability of  recent management information system improvements that enhance  visibility over project and program performance. In his general comments,  the Deputy Administrator also stated that NASA had recently taken steps to  address issues raised in the draft report and suggested a report title that  would better reflect that progress.
We agree that NASA has initiated number of reforms to its project  development and implementation processes that, if properly implemented,  would be positive steps to addressing many of the problems noted in this  report. However, we also note that some of these problems have been  long-standing in the projects discussed in this report and in a number of  other projects we and NASA’s Office of Inspector General have reviewed.  Furthermore, planned improvements in the past have fallen short of  agencywide implementation. For example, poor or inadequate cost  estimates and management oversight have been central to the problems  that plagued several programs, including those intended to develop new  space transportation and the International Space Station programs. A  reliable financial management structure is central to the success of many  measures noted by the Deputy Administrator in his reply. We recently  reported and testified on the impediments that exist in achieving such a  capability. Finally, we note that contract management has been a long- standing problem at NASA. In 1990, we identified NASA’s contract  management function as an area at high risk. During that time, there was  little emphasis on end results, product performance, and cost control.  NASA found itself procuring expensive hardware that did not work  properly. This report shows that these types of problems still exist.  Regarding the Deputy Administrator’s suggestion that we revise the title of  our report to reflect recent progress that NASA has made towards  addressing issues that we raise, we believe NASA’s improvements have  been properly reflected in our report’s title. We considered the concerns  expressed in the Deputy Administrator’s comments, and consistent with  our stated position that NASA’s improvements are positive steps but that its  problems still persist, we revised the title accordingly.
Finally, until NASA’s integrated financial management system, which is  central to providing effective management and oversight, is fully  implemented, performance assessments relying on cost data may be  incomplete and full costing will be only partially achieved. And until these  problems are resolved and the measures the Deputy Administrator noted in  commenting on a draft of this report are fully implemented and integrated  into the way the agency does business, NASA’s contract management  function will continue to be an area of concern.
As agreed with your office, unless you announce its contents earlier, we  will not distribute this report further until 30 days from its date. At that  time, we will send copies to the NASA Administrator and interested  congressional committees. We will make copies available to others upon  request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any question concerning this report, please contact  me at (202) 512-4841 or lia@gao.gov. Key contributors to this report are  acknowledged in appendix VI.
section_title: Scope and Methodology
paragraphs: To determine cost estimates in selected NASA programs and any changes  in those estimates, we asked NASA to provide a list of programs that were  currently in the development phase, and programs that had completed  development or were launched in fiscal year 2001or 2002. We also asked  NASA to provide the initial baseline development cost estimate and current  cost estimate for the development phase and life of the program, and the  reasons for changes to initial development cost estimates. NASA identified  68 programs that were currently in development or had completed  development in fiscal years 2001 and 2002. These included planetary  missions and Earth observatory, aeronautical technology, and space flight  systems. From that universe, we selected at least one program (10 in total)  from 5 of NASA’s 7 Enterprises. This involved 6 of 9 NASA centers (and the  Jet Propulsion Laboratory) with lead responsibility for one or more of  these programs. Our selection was generally based on programs with the  highest current development cost estimates within an Enterprise. We  compared the initial development cost estimates NASA provided to the  current development cost estimates for the programs. The initial  development estimates generally reflect the projected costs at the time a  new program was first approved by the Congress. The current development  and life-cycle cost estimates reflect the latest estimates provided by NASA  as of April 2003. We also interviewed program officials to obtain additional  information related to NASA’s revisions to initially established baseline  development cost estimates, including the rationale for changes to the cost  estimates.
We also analyzed the initial and current development cost estimates for  17 additional NASA programs, later added to the scope of our review, to  ascertain the level of cost growth or decline as those programs progressed  through the development phase.
To assess NASA’s cost-estimating processes and methodologies, we used  cost-estimating criteria developed by Carnegie Mellon University’s  Software Engineering Institute (SEI) designed to assess the reliability of  project cost and schedule estimates. SEI is a government-funded research  organization that is widely considered an authority on software  implementation. SEI developed checklists with these criteria to help  evaluate software costs and schedule; however, SEI states that these  checklists are equally applicable to hardware and systems engineering  projects. We first analyzed NASA’s cost-estimating procedures and  guidelines to determine if they incorporated key components of good  cost-estimating practices advocated by SEI and other experts.
Based on that analysis, we selected 14 criteria from two SEI reports to use  in assessing NASA’s cost-estimating practices for the 10 programs we  selected to review in detail. Our selection of the 14 criteria from the SEI  reports was based, in part, on their commonality with NASA cost- estimating procedures and guidelines. Finally, using the cost-estimating  documentation provided by NASA for the 10 programs, we determined the  extent to which the programs met the 14 criteria. If a program provided  substantiating evidence for a criterion, we determined that the program  “fully met” the criterion. If partial evidence was provided for a criterion, we  determined the program “partially met” the criterion. If no evidence was  found, then we determined that the criterion was “not met.” Table 2  describes each of the 14 criteria and the significance of each criterion.
To identify any barriers that make it difficult to improve any weaknesses in  NASA’s cost-estimating processes, we reviewed our recent work on NASA’s  efforts to implement a modern integrated financial management system.  We also provided questions to NASA headquarters that asked for  information regarding NASA’s ability to use its cost estimates as a  management tool for its programs. We also provided questions related to  the SEI criteria, and NASA’s responses to these questions provided further  insight into the agency’s cost-estimating management process at the  organizational level. In addition, we interviewed officials in NASA  headquarters’ Office of the Chief Financial Officer and Office of the Chief  Engineer, and the center project managers for the 10 programs and other  appropriate personnel to obtain further perspective on this issue.
To accomplish our work, we visited NASA headquarters, Washington, D.C.,  and Goddard Space Flight Center, Maryland. We also contacted officials at  Marshall Space Flight Center, Alabama; Jet Propulsion Laboratory,  California; Kennedy Space Center, Florida; Glenn Research Center, Ohio;  Johnson Space Center, Texas; and Langley Research Center, Virginia.
We conducted our work from February 2003 to March 2004 in accordance  with generally accepted government auditing standards.
section_title: Assessments of 10 Programs Reviewed in Detail
paragraphs: This appendix provides a program by program assessment of the 10 NASA  programs we reviewed in detail. Each assessment provides  a brief description of the program’s mission;  the status of the program—that is, whether it is in development,  operational, or terminated;  the year the program was initiated; the fiscal year in which the Congress approved the program—that is,  when full-scale design and development funds were appropriated;  a comparison of the initial and current (as of April 2003) baseline   an assessment of the program’s cost-estimating processes,  methodologies, and practices to determine the extent they met  the 14 cost-estimating criteria that we used to measure program  performance. (Table 4 shows for each criterion the number of programs  that met, partially met, or did not meet the criterion.)
section_title: Summary Descriptions of the 17 Additional Programs
paragraphs: In addition to the 10 programs that we reviewed in detail, we analyzed  the initial and current development cost estimates for 17 other  NASA programs.
subsections: section_title: Space Science Enterprise
subsections: section_title: Thermosphere, Ionosphere, Mesosphere Energetics and Dynamics (TIMED)
paragraphs: NASA’s TIMED satellite is conducting the first global study of the Earth’s  mesosphere, lower thermosphere, and ionosphere—segments of the  Earth’s atmosphere located between 40 and 110 miles above the planet.  Initially, TIMED’s mission was to last 2 years, beginning with its launch in  December 2001, but NASA extended the satellite’s orbital operations  through 2006. TIMED’s goal is to improve our understanding of the  influences the sun and humans have on this “gateway region” as well as the  effects of its atmospheric variability on satellites and spacecraft reentering  the Earth’s atmosphere.
section_title: International Gamma-Ray Astrophysics Laboratory (INTEGRAL)
paragraphs: INTEGRAL is a European Space Agency mission, with Russian and  U.S. involvement. Launched in October 2002, the INTEGRAL satellite is  equipped with two telescopes designed to register elusive gamma rays— some of the universe’s most energetic radiation—and give insight into the  most violent processes in our universe. Through INTEGRAL, scientists plan  to study black holes’ interaction with their surroundings, the explosion of  supernovae and their role in forming chemical elements, the nature of  powerful gamma-ray bursts, and transient sources that suddenly change  brightness. U.S. participation consists of co-investigators providing  hardware and software components to the spectrometer and imager  instruments, a co-investigator for the data center, a mission scientist, and a  provision for ground tracking and data collection.
section_title: Rosetta
paragraphs: Rosetta is a European Space Agency mission whose objectives are to study  the origin of and the relationship between comets and interstellar material  and to improve our knowledge of the origins of the Solar System. The  Rosetta satellite was launched in March 2004 and, after a long cruise phase,  is planned to rendezvous with comet Churyumov-Gerasimenko in 2014.  Plans call for Rosetta to orbit the comet while taking scientific  measurements and to position a probe on the comet surface to take in-situ  measurements. U.S. involvement includes developing three remote-sensing  instruments and a subsystem for a fourth instrument.
section_title: Mercury Surface, Space Environment, Geochemistry and Ranging (MESSENGER)
paragraphs: Currently scheduled to launch during a 15-day period that opens July 30,  2004, the MESSENGER spacecraft is intended to collect images of Mercury.  Through these images, NASA scientists hope to determine Mercury’s  geological history and the nature of its surface composition, core, poles,  exosphere and magnetosphere, and magnetic field. This information is  expected to provide scientists with a better understanding of how Earth  was formed, how it evolved, and how it interacts with the sun.
section_title: Solar Terrestrial Relations Observatory (STEREO)
paragraphs: Through STEREO—an international collaboration involving France,  Germany, the United Kingdom, and the United States—NASA plans to trace  the flow of energy and matter from the sun to Earth by studying the  solar origin of coronal mass ejections, their evolution in the heliosphere,  and their effects on geospace. Twin STEREO observatories, scheduled to  be launched in November 2005, will be used to develop a three- dimensional, time-dependent model of the magnetic topology, temperature,  density, and velocity structure of the ambient solar wind. Because coronal  mass ejections are the prime drivers of major space weather hazards,  STEREO is expected to greatly improve our understanding of the most  severe disturbances of the Sun-Earth system. The observatories will also  provide a continuous data stream for the purpose of real-time space  weather forecasts.
section_title: Stratospheric Observatory for Infrared Astronomy (SOFIA)
paragraphs: The SOFIA observatory—a modified Boeing 747 aircraft with a  permanently installed telescope, which NASA plans to begin flying in  2005—will be used to study different astronomical objects and phenomena,  including star births and deaths; solar system formations; complex  molecules in space; planets, comets, and asteroids in our solar system;  nebulae and dust in galaxies; and black holes at the centers of galaxies. The  telescope, provided through a partnership with the German Aerospace  Center, is designed to provide routine access to nearly all of the visual,  infrared, far-infrared, and submillimeter parts of the spectrum. As such,  SOFIA is expected to extend the range of astrophysical observations  significantly beyond that of previous infrared airborne observatories  through increases in sensitivity and angular resolution. NASA plans to  incorporate new or upgraded technologies over the aircraft’s lifetime to  allow additional scientific exploration. Because most of the instruments  are to be designed and built by graduate students and post-doctoral  scientists in universities throughout the United States, SOFIA will serve as  a training ground for the next generation of instrument builders.
section_title: Solar-B Observatory
paragraphs: The Solar-B program’s objectives are to investigate the interaction between  the Sun’s magnetic field and its corona and to understand the sources of  solar variability. Solar-B is a Japanese Institute of Space and Astronautical  Science mission, with significant U.S. involvement, and follows the Solar-A  collaboration among Japan, the United Kingdom, and the United States.  The observatory is designed to consist of a set of optical, extreme  ultraviolet, and X-ray instruments, and NASA is expected to provide  components for each. The Solar-B observatory is scheduled to be launched  on a Japanese M-V rocket out of Kagoshima, Japan, in September 2006.
Herschel Space Observatory The European Space Agency’s Herschel Space Observatory (formerly the  Far Infrared and Submillimetre Telescope, or FIRST) houses an infrared  telescope that is expected to observe virtually unexplored spectrum  wavelengths that cannot be observed from the ground. Scheduled for  launch in February 2007, Herschel is expected to enable scientists to better  understand galaxy formation, evolution in the early universe, and the  nature of active galaxy power sources; star-forming regions and interstellar  medium physics in the Milky Way and other galaxies; and the molecular  chemistry of cometary, planetary, and satellite atmospheres in our solar  system. NASA is providing components for two of the three instruments  that will be flown on Herschel: the Heterodyne Instrument for Far Infrared  and the Spectral and Photometric Imaging Receiver.
section_title: Earth Science Enterprise
subsections: section_title: Terra
paragraphs: Launched in February 2000, Terra is providing measurements that,  according to NASA, are significantly contributing to the understanding of  the total Earth system. Specifically, Terra is collecting 200 gigabytes of data  each day on the earth’s physical and radiative properties of clouds, air-land  and air-sea exchanges of energy, carbon, and water as well as  measurements of trace gases and volcanology. One of the first operational  uses of Terra was to provide imagery to support the U.S. Forest Service’s  efforts to combat forest fires in the western United States. Through Terra,  fire fighters were able to identify the locations of active fires, instead of  locations of smoke, providing them with the data needed to better control  spreading fires. Terra data were also used by the Geography Department of  Dartmouth College in New Hampshire to assist in flood hazard reduction  programs.
section_title: New Millennium Program’s Earth Observing-1 (EO-1)
paragraphs: NASA’s New Millennium Program (NMP) is designed to identify, develop,  and flight-validate key instrument and spacecraft technologies that can  enable new or more cost-effective approaches to conducting science  missions. EO-1—the first NMP mission, launched in November 2000— includes three land imaging instruments that are expected to lead to a  new generation of lighter weight, higher performance, and lower cost  Landsat-type Earth surface imaging instruments.
section_title: Jason-1
paragraphs: The mission of the Jason-1 program, a cooperative effort with the French  Space Agency, is to study the global oceans. Launched in December 2001,  the Jason-1 satellite was expected to monitor ocean circulation and events  such as El Nino and ocean eddies and to improve global climate forecasts  and predictions. The Jason-1 satellite was positioned to orbit the earth in  tandem with TOPEX/Poseidon, an earlier generation satellite launched in  1992, to provide data to the National Oceanic and Atmospheric  Administration.
The SeaWinds satellite, launched in December 2002, is providing high- resolution, ocean surface wind data used for studies of ocean circulation,  climate, and air-sea interaction to understand global climate changes and  weather patterns better. By using long-term wind data in numerical  weather and wave prediction models, SeaWinds is expected to improve  weather forecasts near coastlines and storm warning and monitoring.
section_title: Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (Calipso)
paragraphs: The Calipso satellite, scheduled for launch in 2005, is being designed to  study the effect that aerosols and clouds have on the Earth’s radiation  balance, which ultimately controls the temperature of the Earth. Calipso is  expected to provide scientists with data to construct three-dimensional  structures of the atmosphere, enabling new observationally based  assessments of the radiative effects of aerosol and clouds that will greatly  improve our ability to predict future climate change. NASA plans to fly  Calipso in formation with Aqua and CloudSat, a satellite being designed to  measure the vertical structure of clouds from space and contribute to a  better understanding of the role of clouds in the Earth’s climate system.  The Calipso program is a cooperative effort with France.
section_title: Space Flight Enterprise
subsections: section_title: X-38 Crew Return Vehicle (CRV)
paragraphs: The X-38 Crew Return Vehicle was cancelled in April 2002, due to its single  purpose design and the potentially high costs identified by an independent  assessment. The purpose of the CRV project was to initiate work toward an  independent U.S. crew return capability for the International Space Station.  As envisioned, CRV was expected to serve as a back-up to the space shuttle  orbiters by providing resupply to the station or change-out crew, or  accommodating safe return for up to seven crew members who may be ill  or injured or in the event that a catastrophic failure of the station made it  unable to support life.
section_title: Alternate Turbopump Program (ATP)
paragraphs: ATP’s primary objectives were to significantly improve the safety and  operating margins of the high-pressure turbopump in the space shuttle’s  main engine and to eliminate the need to remove the turbopump for  postflight maintenance. An alternative turbopump was successfully  implemented in the shuttle launched in April 2002. According to NASA,  ATP’s development contract, signed in December 1986, specifically  addressed shortcomings of the previous turbopumps; took advantage of  the latest technologies; and applied lessons learned. The contract called for  the parallel development of two high-pressure turbopumps—one that  operates on oxidization and one on fuel. However, 5 years into the  program, technical problems prompted NASA to end parallel development  and concentrate first on developing the oxidizer turbopump, which was  first flown in July 1995. Although development of the fuel turbopump  resumed in 1994, extreme high temperatures, pressures, and rotor speeds  resulted in significant design challenges and the design certification review  was not completed until March 2001. The full implementation of the fuel  turbopump into flight was completed beginning with the April 2002 shuttle  flight.
section_title: Tracking and Data Relay Satellite (TDRS) Replenishment
paragraphs: In December 2002, the TDRS Replenishment project achieved its goal:  launch three geosynchronous satellites to replace the existing aging  satellite constellation, and thereby continue to provide space network  tracking, data, voice, and video services to NASA scientific satellites, the  Space Shuttle program, the International Space Station, and other NASA  customers. According to NASA, the functional and technical performance  requirements for the replacement satellites—launched in June 2000,  March 2002, and December 2002—are virtually identical to those of the  previous satellites.
section_title: Advanced Health Management System (AHMS) Phase 1
paragraphs: AHMS is expected to provide safe shutdown of the space shuttle main  engine during potentially catastrophic high-pressure turbopump failures  through improved monitoring of engine vibration and anomaly response  capabilities. According to NASA, AHMS modifications include (1) adding a  vibration redline monitor for high pressure turbopumps, (2) doubling  memory capacity and employing radiation tolerant memory, (3) adding an  external communication interface for a potential phase-two health  management computer, and (4) eliminating existing memory retention  batteries and replacing them with nonvolatile memory. While NASA stated  the AHMS will be available for launch in January 2005, the shuttle fleet’s  return to flight date is planned for March or April 2005.
section_title: Description of Earned Value Management
paragraphs: Earned value management (EVM) goes beyond the two-dimensional  approach of comparing budgeted costs to actuals. Instead, it attempts to  compare the value of work accomplished during a given period with the  work scheduled for that period. By using the value of completed work as a  basis for estimating the cost and time needed to complete the program,  earned value can alert program managers to potential problems early in  the program.
An accurate, valid, and current performance management baseline is  needed to perform useful analyses using EVM. In 1996, in response to  acquisition reform initiatives, the Department of Defense (DOD) adopted  32 criteria for evaluating the quality of management systems. In general  terms, the 32 criteria require contractors to (1) define the contractual  scope of work using a work breakdown structure; (2) identify  organizational responsibility for the work; (3) integrate internal  management subsystems; (4) schedule and budget authorized work;  (5) measure the progress of work based on objective indicators; (6) collect  the cost of labor and materials associated with the work performed;  (7) analyze any variances from planned cost and schedules; (8) forecast  costs at contract completion; and (9) control changes. The criteria have  become the standard for EVM and have been adopted by major  U.S. government agencies, industry, and the governments of Canada and  Australia. The full application of EVM system criteria is appropriate for  large cost reimbursable contracts where the government bears the cost  risk. For such contracts, management discipline prescribed by the criteria  is essential. In addition, data from an EVM system have been proved to  provide objective reports of contract status, allowing numerous indices  and performance measures to be calculated. These can then be used to  develop accurate estimates of anticipated costs at completion, providing  early warning of impending schedule delays and cost overruns.
Table 5 lists the 32 criteria, organized into five basic categories:  organization, planning and budgeting, accounting considerations, analysis  and management reports, and revisions and data maintenance.
The standard format for tracking earned value is through a cost  performance report (CPR). The CPR is a monthly compilation of  cost, schedule, and technical data, which displays the performance  measurement baseline, any cost and schedule variances from that baseline,  the amount of management reserve used to date, the portion of the  contract that is authorized unpriced work, and the contractor’s latest  revised estimate to complete the program. As a result, the CPR can be used  as an effective management tool because it provides the program manager  with early warning of potential cost and schedule overruns.
Using data from the CPR, a program manager can assess trends in cost and  schedule performance. This information is useful because trends tend to  continue and can be difficult to reverse. Studies have shown that once  programs are 15 percent complete, the performance indicators are  indicative of the final outcome. For example, a CPR showing a negative  trend for schedule status would indicate that the program is behind  schedule. By analyzing the CPR, one could determine the cause of the  schedule problem such as delayed flight tests, changes in requirements, or  test problems because the CPR contains a section that describes the  reasons for the negative status. A negative schedule can be a predictor of  later cost problems because additional spending is often necessary to  resolve problems. CPR data also provide the basis for independent  assessments of a program’s cost and schedule status and can be used to  project final costs at completion in addition to determining when a  program should be completed.
Examining a program’s management reserves is another way that a  program can use a CPR to determine potential issues early on. Management  reserves, which are funds that may be used as needed, provide flexibility to  cope with problems or unexpected events. EVM experts agree that  transfers of management reserves should be tracked and reported because  they are often problem indicators. An alarming situation arises if the CPR  shows that the management reserves are being used at a faster pace than  the program is progressing toward completion. For example, a problem  would be indicated if a program has used 80 percent of its management  reserves, but only completed 40 percent of its work. A program’s  management reserves should contain at least 10 percent of the cost to  complete a program so that funds will always be available to cover future  unexpected problems that are more likely to surface as the program moves  into the testing and evaluation phase.
section_title: Comments from the National Aeronautics and Space Administration
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: Staff making key contributions to this report were Jerry Herley, Shirley  Johnson, Charles Malphurs, Karen Sloan, Madhav Panwar, Karen Richey,  Jennifer Echard, and Deborah Lott.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs